Optimization method of optimizer and related device

Abstract

The embodiment of the invention provides a method for reducing memory occupied by an optimizer and a related device, and the method comprises the steps: selecting one from a high-precision compression algorithm and a high-performance compression algorithm as a target compression algorithm through analyzing the characteristics of parameters of a neural network model, and then compressing the parameters of the optimizer according to the target compression algorithm, and the compressed optimizer parameters are obtained. According to the embodiment of the invention, multiple algorithms for compressing the optimizer can be provided, and the stability of the optimizer in more scenes is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an optimization method and related devices for an optimizer. Background Art

[0002] With the development of computer technology, neural network models have received more and more attention and applications in recommendation scenarios. In recommendation scenarios, the number of samples of neural network models will be in the billions or hundreds of billions. The cross-combination of billions of user dimensions and billions of content dimensions through feature engineering produces explosive feature dimensions. Therefore, neural network models often have high-dimensional sparse features and require a large amount of training memory.

[0003] In the process of training a neural network model, an optimizer is usually used to optimize the neural network model. That is, the optimizer optimizes the parameters of the neural network by converting the parameters of the neural network model into optimizer parameters. When the number of parameters of the neural network model is large, the optimizer parameters are also large, and a large amount of training memory is required. Compression algorithms are usually used to build low-memory optimizers, but different neural network models have different compression requirements in different scenarios, and the configured compression algorithms cannot meet the different compression requirements of different neural network models in different scenarios. Summary of the invention

[0004] The embodiments of the present application provide an optimization method and related devices for an optimizer, which can provide multiple compression optimizer algorithms and improve the stability of the optimizer in more scenarios.

[0005] In a first aspect, an embodiment of the present application provides an optimization method of an optimizer, which may include:

[0006] Obtain the parameters of the neural network model; analyze the characteristics of the parameters of the neural network model, and determine the target compression algorithm from a plurality of compression algorithms according to the characteristics of the parameters of the neural network model; use the target compression algorithm to compress the optimizer parameters in the optimizer of the neural network model, and the optimizer is used to optimize the parameters of the neural network model by converting the parameters of the neural network model into the optimizer parameters. It can be seen that the optimizer is introduced in the training of the neural network model, and the network parameters of the model can be optimized by the optimizer. The model often has high-dimensional sparse features, resulting in a huge amount of optimizer parameters, so the optimizer will occupy a large amount of memory during the model training process. Therefore, the embodiment of the present application provides a plurality of compression optimizer algorithms. In practical applications, the corresponding compression algorithm can be selected according to the characteristics of the parameters of the neural network model to apply complex and changeable scenarios and improve the stability of the optimizer in more scenarios.

[0007] In a possible implementation of the first aspect, when the parameter of the neural network model is analyzed to be greater than a first threshold, the target compression algorithm is a high-performance compression algorithm, and the high-performance compression algorithm is a compression algorithm that improves the training efficiency of the neural network model. It can be seen that the embodiment of the present application can take into account the characteristics of the parameters of the neural network model. When the number of parameters of the neural network model is large, in order to improve the training efficiency of the neural network model, a high-performance compression algorithm can be used to improve the stability of the neural network model in more scenarios.

[0008] In a possible implementation of the first aspect, when the parameters of the neural network model are analyzed to be less than or equal to a first threshold value, and the distribution of the parameters of the neural network model is a smooth distribution, the target compression algorithm is a high-performance compression algorithm, and the high-performance compression algorithm is a compression algorithm that improves the training efficiency of the neural network model. It can be seen that the embodiment of the present application can take into account the characteristics of the parameters of the neural network model. When the number of parameters of the neural network model is not large and the distribution is smooth, it means that the differences between a large number of parameters are not large. Therefore, a high-precision compression algorithm can be used to improve the training efficiency of the neural network model.

[0009] In a possible implementation of the first aspect, the high-performance compression algorithm is a sampling compression algorithm, and the sampling compression algorithm compresses the optimizer parameters by sampling the optimizer parameters. It can be seen that the present application can achieve high-rate compression through a sampling method with low computational complexity without affecting the training performance and convergence accuracy of the neural network model.

[0010] In a possible implementation of the first aspect, compressing the optimizer parameters using the sampling compression algorithm includes:

[0011] The matrix corresponding to the optimizer parameter is sampled in row units, and the sampled optimizer is used as the compressed optimizer parameter. It can be seen that the matrix is ​​sampled in row units, and the sampled optimizer parameter can represent the optimizer parameter of each row to achieve high compression ratio.

[0012] In a possible implementation of the first aspect, compressing the optimizer parameters using the sampling compression algorithm includes:

[0013] The matrix corresponding to the optimizer parameter is merged and reduced in units of rows, and the value obtained after the reduce operation is used as the compressed optimizer parameter. It can be seen that the matrix is ​​merged on the scale of rows, and several optimizer parameters are merged into one optimizer parameter. The merged optimizer parameter can represent the several optimizer parameters to achieve high compression ratio.

[0014] In a possible implementation of the first aspect, when the parameters of the neural network model are analyzed to be less than or equal to a first threshold value, and the distribution of the parameters of the optimizer neural network model is a non-smooth distribution, the target compression algorithm is a high-precision compression algorithm, and the high-precision compression algorithm is a compression algorithm that improves the training accuracy of the neural network model. It can be seen that the embodiment of the present application can take into account the characteristics of the parameters of the neural network model. When the number of parameters of the neural network model is not large and the distribution is not smooth, it means that the difference between the parameters is large. Therefore, a high-performance compression algorithm can be used to improve the training efficiency of the neural network model.

[0015] In a possible implementation of the first aspect, the high-precision compression algorithm is a data quantization compression algorithm, and the data quantization compression algorithm is used to compress the optimizer parameters by converting floating-point numbers into integers. It can be seen that storing the optimizer parameters in integer form can reduce the memory usage of the optimizer than storing the optimizer parameters in floating-point form.

[0016] In a possible implementation of the first aspect, the optimizer parameters are compressed using the data quantization compression algorithm, including: converting the optimizer parameters represented by the floating point number into a sign bit, a first decimal place, and a first numerical value bit represented by the integer according to the data quantization compression algorithm. It can be seen that the number of bytes occupied by the floating point number is more than the number of bytes occupied by the integer, and the sign bit, the first decimal place, and the first numerical value bit obtained after quantization can be used to represent the floating point number. Storing the optimizer parameters with the sign bit, the first decimal place, and the first numerical value bit of the integer can reduce memory usage.

[0017] In a possible implementation of the first aspect, compressing the optimizer parameters using the data quantization compression algorithm includes: converting the optimizer parameters of the floating point number into the second decimal place and the second numerical place of the integer according to the data quantization compression algorithm. It can be seen that the number of bytes occupied by the floating point number is more than the number of bytes occupied by the integer, and the second decimal place and the second numerical place obtained after quantization can be used to represent the floating point number. Storing the optimizer parameters with the second decimal place and the second numerical place of the integer can reduce memory usage.

[0018] In a possible implementation of the first aspect, the neural network model is a recommendation model.

[0019] In a second aspect, an embodiment of the present application provides an optimization device of an optimizer, the device comprising:

[0020] An acquisition unit, used for acquiring parameters of a neural network model;

[0021] A processing unit, configured to analyze characteristics of the parameters of the neural network model, and determine a target compression algorithm from a plurality of compression algorithms according to the characteristics of the parameters of the neural network model;

[0022] A compression unit is used to compress the optimizer parameters in the optimizer of the neural network model using the target compression algorithm, and the optimizer is used to optimize the parameters of the neural network model by converting the parameters of the neural network model into the optimizer parameters. In a possible implementation of the second aspect, when the parameters of the neural network model analyzed are greater than a first threshold, the target compression algorithm is a high-performance compression algorithm, and the high-performance compression algorithm is a compression algorithm that improves the training efficiency of the neural network model.

[0023] In a possible implementation of the second aspect, when the parameters of the neural network model are analyzed to be less than or equal to a first threshold, and the distribution of the parameters of the optimizer neural network model is a smooth distribution, the target compression algorithm is a high-performance compression algorithm, and the high-performance compression algorithm is a compression algorithm that improves the training efficiency of the neural network model.

[0024] In a possible implementation of the second aspect, the high-performance compression algorithm is a sampling compression algorithm, and the sampling compression algorithm compresses the optimizer parameters by sampling the optimizer parameters.

[0025] In a possible implementation manner of the second aspect, the compression unit is specifically used to:

[0026] The matrix corresponding to the optimizer parameter is sampled in units of rows, and the sampled values ​​are used as compressed optimizer parameters.

[0027] In a possible implementation manner of the second aspect, the compression unit is specifically used to:

[0028] The matrix corresponding to the optimizer parameter is merged and reduced in units of rows, and the value obtained after the reduce operation is used as the compressed optimizer parameter.

[0029] In a possible implementation of the second aspect, when the optimizer parameter is analyzed to be less than or equal to a first threshold, and the distribution of the parameters of the optimizer neural network model is a non-smooth distribution, the target compression algorithm is a high-precision compression algorithm, and the high-precision compression algorithm is a compression algorithm that improves the training accuracy of the neural network model.

[0030] In a possible implementation of the second aspect, the high-precision compression algorithm is a data quantization compression algorithm, and the data quantization compression algorithm is used to compress the optimizer parameters by converting floating-point numbers into integers.

[0031] In a possible implementation manner of the second aspect, the compression unit is specifically used to:

[0032] The optimizer parameter of the floating point number is converted into a sign bit, a decimal bit and a value bit of the integer according to the data quantization compression algorithm.

[0033] In a possible implementation manner of the second aspect, the compression unit is specifically used to:

[0034] The optimizer parameters of the floating point number are converted into decimal places and numerical places of the integer according to the data quantization compression algorithm.

[0035] In a possible implementation of the second aspect, the neural network model is a recommendation model.

[0036] In a third aspect, an embodiment of the present application provides a computing device, which includes: a memory and a processor, wherein the processor is used to execute instructions stored in the memory so that the computing device implements any method described in the first aspect.

[0037] Optionally, the computing device further comprises a communication interface, wherein the communication interface is used to receive and / or send data, and / or the communication interface is used to provide input and / or output for the processor.

[0038] It should be noted that the above embodiment is described by taking a processor (or general-purpose processor) that executes the method by calling a computer specification as an example. In the specific implementation process, the processor can also be a dedicated processor, in which case the computer instructions have been pre-loaded in the processor. Optionally, the processor can also include both a dedicated processor and a general-purpose processor.

[0039] Optionally, the processor and the memory may also be integrated into one device, that is, the processor and the memory may also be integrated together.

[0040] In a fourth aspect, an embodiment of the present application further provides a computing device cluster, the computing device cluster comprising at least one computing device, each computing device comprising a processor and a memory;

[0041] The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method described in any one of the first aspects.

[0042] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on at least one processor, the method described in any one of the first aspects is implemented.

[0043] In a sixth aspect, an embodiment of the present application further provides a computer program, which includes computer software instructions. When the computer software instructions are executed by a computer, the computer executes the method described in the first aspect above or any possible implementation method of the first aspect.

[0044] In a seventh aspect, an embodiment of the present application further provides a computer program product, the computer program product comprising computer instructions, which, when executed on at least one processor, implement the method described in any one of the first aspects above.

[0045] Optionally, the computer program product may be a software installation package or an image package. When the aforementioned method is required, the computer program product may be downloaded and executed on a computing device.

[0046] The beneficial effects of the technical solutions provided in the second to seventh aspects of the present application can refer to the beneficial effects of the technical solution of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The following is an introduction to the drawings used in the embodiments of the present application.

[0048] Figure 1 It is a structural diagram of an artificial intelligence main framework provided by an embodiment of the present application;

[0049] Figure 2 is a schematic diagram of a system architecture 100 provided in an embodiment of the present application;

[0050] Figure 3 It is a schematic diagram of a system architecture of a training recommendation model provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of a chip hardware structure provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of a flow chart of an optimization method of an optimizer provided in an embodiment of the present application;

[0053] Figure 6 It is a schematic diagram of a flow chart of determining a target compression algorithm from a plurality of compression algorithms provided in an embodiment of the present application;

[0054] Figure 7 It is a schematic diagram of a compression optimizer for fusion processing of multiple modules provided in an embodiment of the present application;

[0055] Fig. 8A It is a schematic diagram of a dynamic quantification provided by an embodiment of the present application;

[0056] Figure 8B is a schematic diagram of a non-negative dynamic quantization provided by an embodiment of the present application;

[0057] Fig. 9A is a schematic diagram of sampling first-order momentum provided by an embodiment of the present application;

[0058] Fig. 9B is a schematic diagram of sampling second-order momentum provided by an embodiment of the present application;

[0059] Fig.10 is a structural schematic diagram of an optimization device of an optimizer provided in an embodiment of the present application;

[0060] Fig.11 It is a structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application are clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0062] The terms "first" and "second" in the specification and drawings of the present application are used to distinguish different objects, or to distinguish different treatments of the same object, rather than to describe the specific order of objects. In addition, the terms "including" and "having" and any variations thereof mentioned in the description of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes other steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. It should be noted that in the embodiments of the present application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design method described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being superior or more advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way. In the embodiments of the present application, "A and / or B" means A and B, A or B. "A, and / or B, and / or C" means any one of A, B, and C, or any two of A, B, and C, or A, B, and C. The technical solution in the present application will be described below in conjunction with the accompanying drawings.

[0063] The present application involves related applications of neural networks. In order to better understand the solutions of the embodiments of the present application, the relevant terms and probabilities of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0064] (1) Loss Function

[0065] In the process of training a neural network, because we hope that the output of the neural network is as close as possible to the value we really want to predict, we can compare the current network's predicted value with the target value we really want, and then update the weight vector of each layer of the neural network based on the difference between the two. For example, if the network's predicted value is high, adjust the weight vector to make it predict a lower value, and keep adjusting it until the neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.

[0066] (2) Back Propagation

[0067] An algorithm that calculates the gradient of model parameters based on the loss function and updates the model parameters. The neural network can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model is smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0068] (3) Optimizer: During the back-propagation process, it guides the various parameters of the loss function to update in the correct direction to the appropriate size, so that the updated parameters make the loss function value continuously approach the global minimum.

[0069] (4) Gradient: The derivative vector of the loss function with respect to the parameters.

[0070] (5)Embedding: refers to the feature representation or word embedding representation of the sample.

[0071] (6) Recommendation system: The recommendation system uses machine learning algorithms to analyze and learn based on the user's historical click behavior data, then predicts the user's new requests and returns a recommended list of personalized items.

[0072] (7) Quantization: It is a compression method that converts high bits into low bits. For example, the compression technology that converts conventional full-precision 32-bit floating-point operations into low-precision bit integer operations can be called quantization. For example, when the low bit is quantized to 8 bits, it can be called int8 quantization, that is, the original representation of an optimizer parameter requires float32, but after quantization, it only needs to be represented by int8. In theory, 8 bits can reduce storage space by 4 times compared to 32 bits, thus achieving the purpose of compression.

[0073] (8) Tensor: The basic data structure used in neural network models. As a data container, the data it contains is almost all numerical data. Tensor is the generalization of matrix to any dimension.

[0074] A tensor that contains only one number is called a scalar. An array of numbers is called a vector or a one-dimensional tensor. An array of vectors is called a matrix or a two-dimensional tensor. Combining multiple matrices into a new array creates a 3D tensor. Similarly, combining multiple 3D tensors into an array creates a 4D tensor.

[0075] The embodiments of the present application are described below in conjunction with the accompanying drawings. A person skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0076] First, the overall workflow of the AI ​​system is described in Figure 1 , Figure 1 It is a structural diagram of an artificial intelligence main framework provided by an embodiment of the present application. The above artificial intelligence main framework is explained below from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data undergoes a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0077] (1) Infrastructure

[0078] The infrastructure provides computing power support for the artificial intelligence system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, GPU, NPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnection networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0079] (2) Data

[0080] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0081] (3) Data processing

[0082] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0083] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, training, etc. During the training process, the parameters of the neural network model are trained using an optimizer. Therefore, the present application can obtain the parameters of the neural network model, analyze the characteristics of the parameters of the neural network model, determine the target compression algorithm from a variety of algorithms based on the characteristics of the parameters of the neural network model, and use the target compression algorithm to compress the optimizer parameters in the optimizer of the neural network model to reduce the memory space occupied by the optimizer.

[0084] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0085] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0086] (4) General ability

[0087] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, content recommendation, etc.

[0088] (5) Smart products and industry applications

[0089] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, etc.

[0090] The embodiments of the present application can be applied to the compression design of the neural network model, and further, can be applied to the compression design of the model optimizer. The neural network model trained by the optimizer compressed by the present application can be specifically applied to the recommendation field, for example, it can include online services for all recommendation scenarios such as advertising, e-commerce shopping, video, audio, and information information flow recommendations. Specifically, the user's behavior data on the network will be recorded by the server in the form of a log, and these behavior data can be extracted to characterize user features (such as the user's gender, age, city, hobbies, etc.), product features used to characterize product attributes (such as product type, content, display location, recommendation time, etc.), and other related features (such as the terminal used, click / download / browse time, etc.).

[0091] In the embodiment of the present application, the user's behavior data can be divided into a training set and a prediction set. The training set is a collection of a large number of labeled sample data, that is, each sample data includes a label and multiple features, and the training set can be used to train the recommendation model. The prediction set is a collection of one or more unlabeled sample data, that is, each sample data includes multiple features but no label.

[0092] The following introduces the system architecture provided by the embodiments of the present application.

[0093] See also Figure 2 , Figure 2 1 is a schematic diagram of a system architecture 100 provided in an embodiment of the present application. As shown in the system architecture 100, the system architecture 100 includes an execution device 110, a training device 120, a database 130, a client device 140, a data storage system 170 and a data acquisition device 160, and the execution device 110 includes a computing module 111 and an input / output (I / O) interface 112.

[0094] In the training phase, the data acquisition device 160 can be used to obtain the required large-scale data set (i.e., training set) and store the training set in the database. The training set can be image data, video data, audio data, text data, etc., which is related to the target task to be performed and is not limited here. The training device 120 trains the neural network model 121 based on the training set maintained in the database to obtain the trained target model 101.

[0095] Specifically, the training device 120 forward passes the output result according to the current parameters (weights) and input data, calculates the loss function between the output result and the target output, calculates the gradient of the loss function relative to the parameters during back propagation, and uses the gradient to update the parameters to reduce the optimization of the loss of the next iteration. During the back propagation process of the training device 120, the optimizer parameters can be calculated according to the gradient through the optimizer 122, and then the parameters of the neural network model 121 are optimized by the optimizer parameters. In order to reduce the memory occupied by the optimizer 122, the training device 120 can obtain the parameters of the neural network model 121, analyze the characteristics of the parameters of the neural network model 121, determine the target compression algorithm from a plurality of compression algorithms according to the characteristics of the parameters of the neural network model 121, and use the target compression algorithm to compress the optimizer parameters in the optimizer 122 of the neural network model.

[0096] In one possible implementation, when the training device 120 analyzes that the parameters of the neural network model 121 are greater than a first threshold, the target compression algorithm is a high-performance compression algorithm, which is a compression algorithm that improves the training efficiency of the neural network model 121.

[0097] In another possible implementation, when the training device 120 analyzes that the parameters of the neural network model 121 are less than or equal to a first threshold, and the distribution of the optimizer parameters is a smooth distribution, the target compression algorithm is a high-performance compression algorithm, which is a compression algorithm that improves the training efficiency of the neural network model 121.

[0098] In a possible implementation, the high-performance compression algorithm is a sampling compression algorithm, and the sampling compression algorithm compresses the optimizer parameters by sampling the optimizer parameters.

[0099] In another possible implementation, when the training device 120 analyzes that the parameters of the neural network model 121 are less than or equal to a first threshold, and the distribution of the optimizer parameters is a non-smooth distribution, the target compression algorithm is a high-precision compression algorithm, which is a compression algorithm that improves the training accuracy of the neural network model 121.

[0100] In a possible implementation, the high-precision compression algorithm is a data quantization compression algorithm, which is used to compress the optimizer parameters by converting floating-point numbers into integers.

[0101] It should be noted that, in actual applications, the training data maintained in the database 130 may not all come from the data acquisition device 160, but may also be received from other devices. It should also be noted that the training device 120 may not train the neural network model entirely based on the training data maintained by the database 130, but may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0102] The trained target model 101 can be applied to different systems or devices (i.e., execution devices). For example, it can be applied to end-side devices such as mobile phones, tablets, and laptops, as well as wearable devices such as smart watches, smart bracelets, and smart glasses. It can also be applied to smart cars, connected cars, robots, and other intelligent mobile bodies. Furthermore, the target model 101 obtained by the above training is not limited to the above-mentioned devices, but can also be applied to any other device that can deploy a neural network, such as a cloud server and a computing platform, and this application does not limit this.

[0103] In the inference phase, the execution device 110 can call data, codes, etc. in the data storage system 170, or store data, instructions, etc. in the data storage system 170. The data storage system 170 can be placed in the execution device 110, or the data storage system 170 can be an external memory relative to the execution device 110. The computing module 111 processes the input data through the trained model, and the processing process is related to the target task, such as content recommendation.

[0104] exist Figure 3 In the case shown, the user can manually give input data, and the "manually given input data" can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can be used as a data collection terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.

[0105] It is worth noting that Figure 2This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 2 In the embodiment, the data storage system 170 is an external memory relative to the execution device 110. In other cases, the data storage system 170 may also be placed in the execution device 110.

[0106] See also Figure 3 , Figure 3 200 is a schematic diagram of a system architecture for training a recommendation model provided in an embodiment of the present application. The training system 200 includes a first node 201 and a second node 202. The first node 201 and the second node 202 are used to train the recommendation model according to input data. During the training of the recommendation model, an optimizer is used to optimize the parameters of the recommendation model. The first node 201 and the second node 202 can be Figure 3 The training device 120 shown, the recommended model can be Figure 3 The neural network model 121 is shown.

[0107] from Figure 3 It can be seen that the first node 201 includes 4 GPUs, namely GPU0, GPU1, GPU2 and GPU3, and the second node 202 includes 4 GPUs, namely GPU0, GPU1, GPU2 and GPU3. Each GPU can start a training process to perform the training of the recommendation model, that is, each GPU can use the optimizer to optimize the parameters of the recommendation model. Among them, the recommendation model mainly includes a sparse layer 203 and a dense layer (Dense Model) 204. In the training scenario of the recommendation model, large-scale sparse discrete features are used. The sparse discrete features need to be densified through the embedding layer (embedding) to obtain the feature vector corresponding to the sparse features, so the sparse layer 203 can also be called an embedding layer.

[0108] In the sparse layer 203, the first node 201 and / or the second node 202 can use a row vector of length D to represent the number of labels (IDs) of M in the data set, and all row vectors are aggregated into a two-dimensional matrix of dimension [M, D] and stored. In a possible implementation, a hash table data structure can be used for storage. Since the mainstream model is based on the embedding parameter to represent the feature vector, and based on the explicit / implicit combination relationship of the representation learning feature, the number of parameters is huge, usually in the hundreds of millions or even hundreds of billions. Therefore, the embedding parameter will take up a very large amount of memory, so the optimizer parameters determined by the embedding parameter are the optimizer parameters of the sparse feature, which will also take up a very large amount of memory. The video memory of a single GPU is not enough to store all the embedding parameters and optimizer parameters, and multiple nodes are required for distributed storage, which is usually deployed in the video memory of multiple GPU cards in a model parallel manner. During the training process of the recommendation model, the IDs in each batch of data are usually only in the hundreds of thousands, involving only a small part of the embedding parameters and optimizer parameters, showing a storage-intensive and computation-sparse form. Because the IDs have a long-tail distribution feature, the optimizer parameters have semantic sparse features and a large amount of information redundancy.

[0109] In the dense layer 204, the first node 201 and / or the second node 202 trains the recommendation model by aggregating and processing the embedding parameters and optimizer parameters involved in a single batch, and outputs the training target. Because the embedding parameters and optimizer parameters in a single batch are represented as dense features, the number scale is usually only in the millions, and the embedding parameters and optimizer parameters of dense features involved in a single batch can be deployed in a data parallel manner.

[0110] In the process of training the recommendation model, the first node 201 and / or the second node 202 may obtain initialization information of the sparse layer, and obtain the scale of the sparse layer, that is, the characteristics of the embedding parameters in the sparse layer, through the dimension information in the initialization information. The first node 201 and / or the second node 202 analyzes the characteristics of the embedding parameters, determines a target compression algorithm from a plurality of compression algorithms according to the characteristics of the embedding parameters, and compresses the optimizer parameters using the target compression algorithm.

[0111] In a possible implementation, due to the row-wise semantic distribution characteristics of the sparse layer, the target compression algorithm is used to compress the matrix corresponding to the optimizer parameter in row units, which can reduce the compression accuracy loss. Floating-point matrix The following introduces a chip hardware structure provided in an embodiment of the present application.

[0112] Figure 4 A schematic diagram of a chip hardware structure provided in an embodiment of the present application, the chip includes an artificial intelligence processor 40. The chip can be set as follows Figure 3 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 3 The training device 120 shown is used to complete the training work of the training device 120 and output the target model 101.

[0113] The artificial intelligence processor 40 can be a neural network processor (network processing unit, NPU), a tensor processing unit (tensor processing unit, TPU) or a graphics processing unit (graphics processing unit, GPU) and any other processor suitable for large-scale XOR operation processing. Take the GPU as an example: the GPU can be mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks to it. The core part of the GPU is the operation circuit 403, which is controlled by the controller 404 to extract data from the memory (weight memory or input memory) and perform operations.

[0114] In some implementations, the operation circuit 403 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 403 is a two-dimensional systolic array. The operation circuit 403 can also be a one-dimensional systolic array or other electronic circuits capable of performing digital operations such as multiplication and addition. In some implementations, the operation circuit 403 is a general-purpose matrix processor.

[0115] For example, assume that there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit 403 takes the corresponding data of the matrix B from the weight memory 402 and caches it on each PE in the operation circuit 403. The operation circuit 403 takes the input data of the matrix A from the input memory 401, performs matrix operations based on the input data of the matrix A and the weight data of the matrix B, and stores the partial results or the final results of the matrix in the accumulator 408.

[0116] The unified memory 406 is used to store input data and output data. The weight data is directly transferred to the weight memory 402 through the direct memory access controller (DMAC) 405. The input data is also transferred to the unified memory 406 through the DMAC.

[0117] The bus interface unit (BIU) 410 is used for the interaction between the DMCA and the instruction fetch buffer 409; the bus interface unit 410 is also used for the instruction fetch buffer 409 to obtain instructions from the external memory; the bus interface unit 410 is also used for the storage unit access controller 405 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0118] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 406, or to transfer weight data to the weight memory 402, or to transfer input data to the input memory 401.

[0119] The vector calculation unit 407 may include multiple operation processing units, and if necessary, further process the output of the operation circuit 403, such as operations involved in converting floating-point numbers into integers, such as vector multiplication, vector addition, exponential operation, logarithmic operation, and / or operations involved in sampling optimizer parameters, such as logarithmic operation, size comparison, etc. The vector calculation unit 407 is mainly used for calculations of non-convolutional layers or fully connected layers (FC) in neural networks, and can specifically process: pooling, batch normalization, local response normalization, etc. For example, the vector calculation unit 407 can apply a nonlinear function to the output of the operation circuit 403, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 407 generates a normalized value, a merged value, or both.

[0120] In some implementations, the vector calculation unit 407 stores the vector of processed outputs to the unified buffer 406. For example, the vector calculation unit 407 can apply a nonlinear function to the output of the operation circuit 403, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 407 generates a normalized value, a merged value, or both. In some implementations, the vector of processed outputs can be used as an activation input to the operation circuit 403, for example, for use in a subsequent layer in a neural network.

[0121] An instruction fetch buffer 409 connected to the controller 404 is used to store instructions used by the controller 404 .

[0122] The controller 404 is used to call the instructions cached in the instruction fetch memory 409 to control the working process of the computing accelerator.

[0123] Generally, the unified memory 406, the input memory 401, the weight memory 402 and the instruction fetch memory 409 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0124] Introduced above Figure 2 Training equipment in 120, Figure 3 The first node 201 and / or the second node 202 in Figure 4 The chip shown is capable of executing each step of the optimization method of the optimizer of the embodiment of the present application.

[0125] The embodiment of the present application provides a system architecture. The system architecture includes one or more local devices, a training device, and a data storage system. The local device is connected to the execution device via a communication network.

[0126] The training device can be implemented by one or more servers. Optionally, the training device can be used in conjunction with other computing devices, such as data storage devices, routers, load balancers, and other devices. The training device can be arranged at one physical site, or distributed at multiple physical sites. The training device can use data in a data storage system, or call program code in a data storage system to implement the optimization method of the optimizer of the embodiment of the present application.

[0127] The user can operate respective user devices (e.g., one or more local devices) to interact with the training device. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0128] Each user's local device can interact with the training device through a communication network with any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0129] In one implementation, the local device obtains relevant parameters of the target model from the training device, deploys the target model on the local device, and uses the target model to perform content recommendation, etc. In the process of training the target model by the training device, the optimizer in the target model is compressed by the optimization method for reducing the optimizer provided by the embodiment of the present application.

[0130] In another implementation, the target model can be directly deployed on the training device, and the training device obtains the data to be processed from the local device and the local device, and processes the data to be processed according to the target model.

[0131] The above-mentioned training device can also be called a cloud device, and the training device is generally deployed in the cloud.

[0132] The method involved in the embodiments of the present application is described in detail below. Figure 5 A schematic diagram of a flow chart of an optimization method of an optimizer provided in an embodiment of the present application. The method can be Figure 2 The training device 120 shown, Figure 3 The first node 201 and / or the second node 202 shown, and Figure 4 Next, we will introduce the training device as the execution subject. Figure 5 The method shown includes but is not limited to the following steps:

[0133] Step S501, obtaining parameters of the neural network model.

[0134] Specifically, the neural network model includes a sparse layer. In the sparse layer 203, a row vector with a length of D can be used to represent the number of labels (IDs) in the data set, and all row vectors are aggregated into a two-dimensional matrix with a dimension of [M, D] and stored. Therefore, before training, the training device can obtain the initialization information of the sparse layer, and obtain the sparse layer scale through the dimension information in the initialization information. The sparse layer scale can reflect the number and distribution of parameters of the neural network model.

[0135] Step S502, analyzing the characteristics of the parameters of the neural network model, and determining a target compression algorithm from a plurality of compression algorithms according to the characteristics of the parameters of the neural network model.

[0136] Specifically, in recommendation scenarios, the sparse layers of mainstream models are usually based on embedding to represent sparse features, and based on the explicit / implicit combination relationship of the features learned, resulting in a huge number of parameters in the neural network model. Therefore, the training device can analyze the sparse features of the parameters of the neural network model and determine the target compression algorithm from a variety of compression algorithms. That is, for neural network models in different scenarios, different compression algorithms will be used to compress the optimizer parameters in the optimizer of the neural network model to achieve a balance between compression performance and compression accuracy.

[0137] See also Figure 6 , Figure 6 FIG. 1 is a flow chart of determining a target compression algorithm from a plurality of compression algorithms provided by an embodiment of the present application. Figure 6 It can be seen that the training device can obtain the initialization information of the sparse layer, obtain the scale of the sparse layer through the dimension information, and obtain the distribution characteristics of the sparse layer through the smoothness statistics, so as to determine the stability and trend of the parameters of the neural network model. Among them, the smoothness statistics is used to count the smoothness of the parameters in the processing of the parameters of the neural network model of the sparse layer. The smoothness is used to indicate the degree of change between the parameters, that is, the distance or difference between the numerical values ​​corresponding to the parameters. The calculation methods of smoothness usually include mean smoothing, moving smoothing, exponential smoothing, sliding window averaging, etc.

[0138] from Figure 6 It can be seen that in one possible implementation, when the training device analyzes the parameters of the neural network model based on the initialization information of the sparse layer and finds that the parameters are greater than the first threshold, the target compression algorithm is a high-performance compression algorithm, and the high-performance compression algorithm is a compression algorithm that improves the training efficiency of the neural network model. In one possible implementation, the first threshold can be hundreds of billions.

[0139] In another possible implementation, when the training device analyzes the number of parameters of the neural network model based on the initialization information of the sparse layer and finds that the number is less than or equal to a first threshold, and the distribution of the parameters of the neural network model is a smooth distribution, the target compression algorithm is a high-performance compression algorithm.

[0140] In another possible implementation, when the training device analyzes that the number of parameters of the neural network model is less than or equal to a first threshold based on the initialization information of the sparse layer, and the distribution of the parameters of the neural network model is a non-smooth distribution, the target compression algorithm is a high-precision compression algorithm.

[0141] It can be understood that the distribution of the parameters of the neural network model is a smooth distribution, which means that the distance or difference between the values ​​corresponding to the parameters is not large, and the stability of the parameters is good. The distribution of the parameters of the neural network model is a non-smooth distribution, which means that the distance or difference between the values ​​corresponding to the parameters is large, and the stability of the parameters is not very good.

[0142] Step S503: compress the optimizer parameters in the optimizer of the neural network model using a target compression algorithm.

[0143] Among them, the optimizer optimizes the parameters of the neural network model by converting the parameters of the neural network model into optimizer parameters. It can be understood that the neural network model introduces a loss function to calculate the error degree of the target prediction. According to the error result calculated by the loss function, it is necessary to make a small change to the model parameters (i.e., weights and biases) in order to reduce the prediction error. At this time, the optimizer can optimize the loss function, and the work of the optimizer is to change the trainable parameters in a way that minimizes the loss function. For example, the training device uses the sparse features corresponding to the training sample as the input of the neural network, outputs the prediction result, and then calculates the value of the loss function based on the prediction result and the true label of the input sample data. During back propagation, the gradient of the parameters of the neural network is calculated based on the value of the loss function, and the gradient is used to update the parameters to reduce the optimization of the loss of the next iteration. During the back propagation process, the training device calculates the gradient through the optimizer to update the parameters of the neural network. That is, the optimizer parameters can be expressed as gradients. When the data volume of the training sample is large, the parameter volume of the optimizer parameters is also large. In order to reduce the memory occupied by the optimizer, the target compression algorithm can be used to compress the optimizer parameters.

[0144] It should be noted that when training a neural network model, sparse features are generally input into the neural network model in the form of tensors. In order to ensure the training accuracy of the neural network model, the original data information of the tensor needs to be retained as much as possible, and the commonly used solution is the quantization solution. To this end, this application analyzes and obtains the following sparse layer characteristics: (1) The optimizer parameters of the sparse layer have information redundancy and effective data range, and the optimizer parameters can be trimmed to a certain extent; (2) During the training process, the optimizer parameters of the sparse layer only need to obtain the rows involved in the current batch to calculate the optimizer parameters, and there is no need to convert the entire optimizer parameters, and the computational complexity is controllable; (3) The optimizer parameters of the sparse layer are arranged in a row-wise semantic distribution.

[0145] In one possible implementation, the target compression algorithm is a high-precision compression algorithm, the high-precision compression algorithm is a data quantization compression algorithm, and the training device can convert floating-point numbers into integers according to the data quantization compression algorithm to compress the optimizer parameters to obtain compressed optimizer parameters.

[0146] In one possible implementation, the optimizer parameters include positive and negative numbers, and the training device converts the optimizer parameters represented by floating point numbers into a sign bit, a first decimal bit, and a first numerical bit represented by an integer according to a data quantization compression algorithm.

[0147] Specifically, the training device truncates the optimizer parameter according to the first representation range to obtain a first truncated value. The first representation range is determined according to the integer type of the optimizer parameter. The integer type includes low-bit integers, such as int4, int8, int16, and so on. Further, the maximum and minimum values ​​of the negative number range that can be represented, as well as the maximum and minimum values ​​of the positive number range, are determined according to the integer type. Then, the negative number range in the first representation range is determined according to the maximum and minimum values ​​of the negative number range, and the positive number range in the first representation range is determined according to the maximum and minimum values ​​of the positive number range.

[0148] Among them, the first upper boundary threshold and the first lower boundary threshold included in the negative number range are both negative numbers, and the first upper boundary threshold and the first lower boundary threshold included in the positive number range are both positive numbers. Therefore, if the optimizer parameter is a negative number, the optimizer parameter that is less than the first lower boundary threshold of the negative number is truncated to the first lower boundary threshold of the negative number, and the optimizer parameter that is greater than the first upper boundary threshold of the negative number is truncated to the first upper boundary threshold of the negative number. Similarly, if the optimizer parameter is a positive number, the optimizer parameter that is less than the first lower boundary threshold of the positive number is truncated to the first lower boundary threshold of the positive number, and the optimizer parameter that is greater than the first upper boundary threshold of the positive number is truncated to the first upper boundary threshold of the positive number.

[0149] Next, the training device can configure the first decimal place, and perform decimal shift on the first truncated value according to the first decimal place to obtain a first shifted value. When the first shifted value belongs to the first interval, the first decimal place and the first truncated value are determined. Among them, the value corresponding to the first decimal place and the first interval are determined according to the integer type, respectively. For example, if the integer type is int8 type, the optimizer parameter represented as int8 type can be represented as 8 bits. Among them, the first bit is the sign bit, then the numerical range of the first decimal place can be determined according to the second and third bits, and the first representation interval can be determined according to the fourth to eighth bits; or, the numerical range of the first decimal place can be determined according to the second, third and fourth bits, and the first interval can be determined according to the fifth to eighth bits.

[0150] Then, the training device selects a first decimal place from the numerical range of the first decimal place in order from small to large, and performs a decimal shift of X places on the first truncated value according to the numerical value X corresponding to the first decimal place, so that the shifted value (i.e., the first shifted value) falls into the first interval. Then, the first decimal place that makes the shifted value fall into the first interval is the required decimal place. Finally, the sign bit, the first decimal place, and the first shifted value corresponding to the first decimal place are the optimizer parameters of integer type.

[0151] In another possible implementation, the optimizer parameter is non-negative, and the training device converts the optimizer parameter represented by a floating point number into a second decimal place and a second numerical place represented by an integer according to a data quantization compression algorithm.

[0152] In a possible implementation, the training device truncates the optimizer parameter according to the second representation range to obtain a second truncated value. The second representation range is a floating point number matrix determined according to the integer type of the non-negative optimizer parameter. The second representation range includes a second upper threshold and a second lower threshold. Therefore, for non-negative optimizer parameters, the second-order momentum less than the second lower boundary threshold is truncated to the second lower boundary threshold, and the second-order momentum greater than the second upper boundary threshold is truncated to the second upper boundary threshold.

[0153] Next, the training device can configure the second decimal place, and perform a decimal shift on the second truncated value according to the second decimal place to obtain a second shifted value. When the second shifted value belongs to the second interval, the second decimal place and the second truncated value are determined. Among them, the value corresponding to the second decimal place and the second interval are determined according to the integer type respectively. For example, if the integer type is int8 type, the non-negative optimizer parameter represented as int8 type can be represented as 8 bits. Among them, the numerical range of the second decimal place can be determined according to the first to third digits, and the second representation interval can be determined according to the fourth to eighth digits, or the numerical range of the second decimal place can be determined according to the first to fourth digits, and the second interval can be determined according to the fifth to eighth digits. Then, the training device selects a second decimal place from the numerical range of the second decimal place in order from small to large, and performs a decimal shift of Y places on the second truncated value according to the numerical value Y corresponding to the second decimal place, so that the shifted value (i.e., the second shifted value) falls into the second interval. Then, the second decimal place that makes the shifted value fall into the second interval is the required decimal place. Finally, the second decimal place and the second moving value corresponding to the second decimal place are non-negative optimizer parameters of integer type.

[0154] The present application finds that when the amount of data of sparse features is very large, for example, more than 100 billion, for sparse features, the importance of optimizer parameters including positive and negative numbers to the update of model parameters is relatively low, and a greater degree of compression can be allowed. After statistically analyzing the optimizer parameters including positive and negative numbers in multiple trainings in the form of histograms, it is found that on the row-wise scale, there is a high-frequency range with significant statistical differences in optimizer parameters including positive and negative numbers.

[0155] In a possible implementation, the high-performance compression algorithm for floating-point matrices is a sampling compression algorithm, and the training device may sample the optimizer parameters according to the sampling compression algorithm to compress the optimizer parameters.

[0156] In a possible implementation, the optimizer parameters include positive and negative numbers, and the training device can count the values ​​corresponding to the optimizer parameters in the two-dimensional matrix of the floating-point number matrix in the row direction according to the sampling compression algorithm to obtain the value range of the optimizer parameters. Then, the training device uses the values ​​in the value range that appear more than the preset threshold as the features of the rows corresponding to the value range, that is, the values ​​in each row that appear more than the preset threshold are used to characterize the features of each row. Therefore, the training device can use the features of each row obtained to characterize the compressed optimizer parameters.

[0157] The present application found that the positive and negative signs of non-negative optimizer parameters do not need to be considered when compressing. Therefore, after statistically analyzing the values ​​corresponding to non-negative optimizer parameters in multiple trainings, it was found that on a row-wise scale, the values ​​corresponding to adjacent non-negative optimizer parameters have a high probability of smoothness.

[0158] In one possible implementation, the optimizer parameters are non-negative, and the training device performs a merge-reduce operation on the matrices corresponding to the optimizer parameters in the row direction according to the sampling compression algorithm, and uses the values ​​obtained after the reduce operation as the compressed optimizer parameters. In one possible implementation, the optimizer parameters are reduced in units of blocks, that is, the optimizer parameters determined after the reduce operation on the optimizer parameters of a block are used as the optimizer parameters represented by the block. In one possible implementation, the training device can save the compressed optimizer parameters and the compression process in the training graph.

[0159] In one possible implementation, when the target compression algorithm is a high-precision compression algorithm, the training graph stores the process of converting a floating-point number into an integer, as well as the sign bit, the first decimal place and the first numerical place in the form of an integer or the second decimal place and the second numerical place in the form of an integer.

[0160] Therefore, when back propagating through the optimizer to optimize the parameters of the neural network model, the training device can dequantize the floating point optimizer parameters according to the sign bit, the first decimal place and the first numerical place. Alternatively, the training device can dequantize the floating point optimizer parameters according to the second decimal place and the second numerical place.

[0161] In another possible implementation, when the target compression algorithm is a high-performance compression algorithm, the optimizer parameters are sampled on the training graph. Therefore, when back propagating through the optimizer to optimize the network parameters of the neural network model, the training device can optimize the parameters of the neural network model according to the optimizer parameters obtained after sampling.

[0162] See also Figure 7 , Figure 7Schematic diagram of a compression optimizer for fusion processing of multiple modules provided in an embodiment of the present application. Figure 7 It can be seen that the momentum compression scheduling module 701 is used to analyze the characteristics of the parameters of the neural network model, and determine the target compression algorithm from a plurality of compression algorithms according to the characteristics of the parameters of the neural network model. When the parameters of the neural network model analyzed are greater than the first threshold, or when the parameters of the neural network model analyzed are less than or equal to the first threshold, and the distribution of the parameters of the neural network model is a smooth distribution, the optimizer parameters are compressed by the high-performance compression module 703 to obtain the compressed optimizer parameters. When the parameters of the neural network model analyzed are less than or equal to the first threshold, and the distribution of the parameters of the neural network model is a non-smooth distribution, the optimizer is compressed by the high-precision compression module 702 to obtain the compressed optimizer parameters.

[0163] In one possible scenario, if the optimizer is a state optimizer, the state optimizer needs to calculate the statistics of the gradient of each parameter as the optimizer parameter. Therefore, the optimizer parameters include first-order momentum and / or second-order momentum. Among them, the first-order momentum is also called the first-order moment estimate, which is a linear combination of the gradients at each moment in the past and is used to control the direction of the neural network model update. The second-order momentum is also called the second-order moment estimate, which is a linear combination of the squares of the gradients at each moment in the past and is used to control the step size of the neural network model update.

[0164] from Figure 7 It can be seen that the high-precision compression module 702 dynamically quantizes the floating-point matrix 7021 corresponding to the first-order momentum in the row direction, converts the floating-point numbers into integers, and determines a first matrix 7022 containing integer first-order momentums. The floating-point first-order momentum is a first-order momentum of the fp32 type, and the integer first-order momentum is a first-order momentum of the int8 type.

[0165] Specifically, see Fig. 8A , Fig. 8A This is a schematic diagram of a dynamic quantization provided by an embodiment of the present application. Fig. 8A It can be seen that the high-precision compression module 702 dynamically quantizes the first-order momentum of the fp32 type in units of lines, and quantizes the first-order momentum of the fp32 type to the first-order momentum of the int8 type. First, the high-precision compression module 702 determines the first representation range according to the first-order momentum of the int type and the first quantization formula. Among them, the first quantization formula is expressed as: S represents the sign bit, E represents the decimal bit, and M represents the value bit. The int8 type can be represented as an 8-bit binary, with the first bit representing the sign bit S, the second and third bits representing the decimal bit E, and the fourth to eighth bits representing the value bit M. From this, it can be determined that the sign bit S = 0 or 1, the decimal bit E = [0,3], and the value bit M = [0,31]. Ignoring the sign bit, when E is 0 and M is 0, When E is 3 and M is 1, That is, without considering the sign bit, the maximum value that can be determined according to the int8 type and the first quantization formula is 10, and the minimum value is 3.125e -4 Therefore, taking the sign bit into account, the first representation range includes: [-10, -3.125e -4 ] and [3.125e -4 , 10]. The first quantization formula provided in the embodiment of the present application is an example, but is not limited thereto. For example, if M occupies four bits in an 8-bit byte, then the first quantization formula is: That is, the first quantization formula can be expressed as Among them, n is the number of bits occupied by M in the 8-bit byte. If M occupies 4 bits, then n is 4; if M occupies 5 bits, then n is 5.

[0166] Next, the high-precision compression module 702 truncates the high-precision first-order momentum in each row of the floating-point matrix 7021 according to the first representation range to obtain a first truncated value. For example, if the first-order momentum is a negative number, such as -10.325, because -10.325 is smaller than the first representation range [-10, -3.125e -5 ] is the first lower boundary threshold of negative numbers -10, then the first truncation value obtained by truncating -10.325 is -10. If the first-order momentum is a positive number, for example 10.2354, because 10.2354 is greater than [3.125e -4 , 10], the first upper boundary threshold of the positive number is 10, then the first truncation value obtained by truncating 10.2354 is 10.

[0167] Next, the high-precision compression module 702 may select a decimal place from the value range [0,3] corresponding to the decimal place, and perform decimal shift on the first truncated value according to the selected decimal place to obtain a first shift value. When the first shift value belongs to the first interval, the first decimal place and the first shift value corresponding to the first shift value are determined. The first interval is determined according to the int8 type, because M represents the value place, then according to the value of M and the first quantization formula The range of the first moving value can be determined, that is, the first interval is

[0168] For example, a decimal place can be selected from the value range [0,7] corresponding to the decimal place in ascending order. If the first cutoff value is 9.345 and the first decimal place is 0, the first moving value is 9.345, because 9.345 does not fall in the first interval [0.03125,1], indicating that the first decimal place 0 does not meet the condition. When the first decimal place is 1, the decimal point of the first cutoff value 9.345 is moved forward by one place to obtain the first moving value 0.9345. Because 0.9345 falls in the first interval [0.03125,1], it means that the first decimal place 1 meets the condition. Therefore, the first decimal place 1 corresponding to the first moving value 0.9345 can be determined.

[0169] Finally, the high-precision compression module 702 saves the sign bit, the decimal place corresponding to the first moving value, and the first moving value in a matrix form. That is, the sign bit, the decimal place corresponding to the first moving value, and the first moving value are the low-precision first-order momentum corresponding to the full-precision first-order momentum.

[0170] from Figure 7 It can be seen that the high-precision compression module 702 is used to dynamically quantize the floating-point matrix 7023 corresponding to the second-order momentum in the row direction to determine the second matrix 7024 containing the integer second-order momentum. Among them, the high-precision second-order momentum is the second-order momentum of the fp32 type, and the low-precision second-order momentum is the second-order momentum of the int8 type.

[0171] Specifically, see Figure 8B , Figure 8B is a schematic diagram of a non-negative dynamic quantization provided by an embodiment of the present application. Figure 8B It can be seen that, in units of lines, the high-precision compression module 702 performs non-negative dynamic quantization on the second-order momentum of the fp32 type, and quantizes the second-order momentum of the fp32 type to the second-order momentum of the int8 type. First, the high-precision compression module 702 determines the second representation range according to the second-order momentum of the int type and the second quantization formula. Among them, because the second-order momentum is non-negative, the second quantization formula is expressed as: E represents the decimal place, and M represents the value place. The int8 type can be represented as 8-bit binary. The first, second, and third places represent the decimal place E, and the fourth to eighth places represent the value place M. From this, it can be determined that the sign place decimal place E = [0,7], and the value place M = [0,31]. When E is 0 and M is 0, When E is 7 and M is 1, That is, the maximum value that can be determined based on the int8 type and the quantization formula is 100, and the minimum value is 3.125e -7 Therefore, the second representation range includes: [3.125e -7, 100]. The quantization formula provided in the embodiment of the present application is an example, but is not limited thereto. For example, if M occupies four bits in an 8-bit byte, then the quantization formula is: That is, the quantization formula can be expressed as Among them, n is the number of bits occupied by M in the 8-bit byte. If M occupies 4 bits, then n is 4; if M occupies 5 bits, then n is 5.

[0172] Next, the high-precision compression module 702 truncates the high-precision second-order momentum in each row of the floating-point matrix 7023 according to the second representation range to obtain a second truncated value. For example, if the second-order momentum value is 100.3259, because 100.3259 is greater than [3.125e -7 , 100] is 100, then the second truncation value obtained by truncating 100.3259 is 100.

[0173] Next, the high-precision compression module 702 can select a decimal place from the value range [0,7] corresponding to the decimal place, and perform decimal shift on the second truncated value according to the selected decimal place to obtain a second shift value. When the second shift value belongs to the second interval, the second decimal place and the second shift value corresponding to the second shift value are determined. Among them, the second interval is determined according to the int8 type. Since M represents the numerical position, the range of the second shift value can be determined according to the value of M, that is, the second interval is [0.03125, 1]. Finally, the high-precision compression module 702 saves the decimal place and the second shift value corresponding to the second shift value in a matrix. That is, the decimal place and the second shift value corresponding to the second shift value are low-precision second-order momentum corresponding to the full-precision second-order momentum.

[0174] from Figure 7 It can be seen that the high-performance compression module 703 samples the floating-point matrix 7031 corresponding to the first-order momentum in the row direction to determine the third matrix 7032 containing the compressed first-order momentum.

[0175] For details, see Fig. 9A , Fig. 9A It is a schematic diagram of sampling first-order momentum provided in an embodiment of the present application. Fig. 9A The histogram shown is a statistical Figure 7 The value of the first-order momentum of each row in the floating-point matrix 7031 shown in FIG. 7 is obtained. In a possible implementation, the modulus operator (mod) can be used to count Figure 7 The numerical value of the first-order momentum of each row in the floating-point matrix 7031 is shown.

[0176] from Fig. 9AIt can be seen that after most of the first-order momentum is concentrated in a certain value range, the high-performance compression module 703 can use the value in the value range exceeding the preset threshold as the feature of the row corresponding to the value range, and the symbol of the value samples the symbol with the highest frequency in the row. Fig. 9A The values ​​shown are in the range [10 2 ~10 3 ] exceeds the preset threshold, so the value range can be [10 2 ~10 3 ] is used as the feature of this row. That is, the value range is [10 2 ~10 3 ] will represent the characteristics of the row being counted. For example, the value range is [10 2 ~10 3 ] can also represent a value in the range of [0~10 1 ], the value range is [10 4 ~10 5 ] in the range [10 6 ~10 7 ]. In a possible implementation, the bincount operator can be used to determine the values ​​in a row that exceed a preset threshold. The complexity can be reduced by using the mod and bincount operations.

[0177] from Figure 7 It can be seen that the high-performance compression module 703 performs a merge-reduce operation on the floating-point matrix 7031 corresponding to the second-order momentum in the row direction in blocks, and determines a fourth matrix 7033 containing the compressed second-order momentum.

[0178] For details, see Fig. 9B , Fig. 9B is a schematic diagram of sampling the second-order momentum provided by an embodiment of the present application. Fig. 9B It can be seen that after numerical statistics of the second-order momentum, it is found that on the row-wise scale, adjacent data have a high probability of smoothing characteristics, so a block-wise reduce operation can be performed on the row-wise scale, and the calculated value directly represents all the data in the block. That is, the value of each row is divided into one or more blocks, each block includes multiple second-order momentums, and the value of each block is calculated through the reduce operation. Among them, the calculation complexity can be reduced through reduce.

[0179] The device provided in the embodiments of the present application is introduced below.

[0180] See also Fig.10 , Fig.10 1 is a schematic diagram of the structure of an optimization device of an optimizer provided in an embodiment of the present application. The optimization device 100 of the optimizer may include an acquisition unit 1001, a processing unit 1002 and a compression unit 1003. Among them,

[0181] An acquisition unit 1001 is used to acquire parameters of a neural network model;

[0182] The processing unit 1002 is used to analyze the characteristics of the parameters of the neural network model, and determine a target compression algorithm from a plurality of compression algorithms according to the characteristics of the parameters of the neural network model; the compression unit 1003 is used to compress the optimizer parameters in the optimizer of the neural network model using the target compression algorithm, and the optimizer is used to optimize the parameters of the neural network model by converting the parameters of the neural network model into the optimizer parameters.

[0183] In a possible implementation, when the parameters of the neural network model analyzed are greater than a first threshold, the target compression algorithm is a high-performance compression algorithm, and the high-performance compression algorithm is a compression algorithm that improves the training efficiency of the neural network model.

[0184] In one possible implementation, when the parameters of the neural network model are analyzed to be less than or equal to a first threshold, and the distribution of the parameters of the optimizer neural network model is a smooth distribution, the target compression algorithm is a high-performance compression algorithm, and the high-performance compression algorithm is a compression algorithm that improves the training efficiency of the neural network model.

[0185] In a possible implementation, the high-performance compression algorithm is a sampling compression algorithm, and the sampling compression algorithm compresses the optimizer parameters by sampling the optimizer parameters. In a possible implementation, the compression unit 1003 is specifically used for:

[0186] The matrix corresponding to the optimizer parameters is sampled in units of rows, and the sampled optimizer parameters are used as compressed optimizer parameters.

[0187] In a possible implementation manner, the compression unit 1003 is specifically configured to:

[0188] The matrix corresponding to the optimizer parameters is merged and reduced in units of rows, and the optimizer parameters obtained after the reduce operation are used as compressed optimizer parameters.

[0189] In one possible implementation, when the optimizer parameter is analyzed to be less than or equal to a first threshold, and the distribution of the parameters of the optimizer neural network model is a non-smooth distribution, the target compression algorithm is a high-precision compression algorithm, and the high-precision compression algorithm is a compression algorithm that improves the training accuracy of the neural network model.

[0190] In a possible implementation, the high-precision compression algorithm is a data quantization compression algorithm, and the data quantization compression algorithm is used to compress the optimizer parameters by converting floating-point numbers into integers.

[0191] In a possible implementation, the compression unit 1003 is specifically configured to: convert the optimizer parameter represented by the floating point number into a sign bit, a decimal bit, and a value bit represented by the integer according to the data quantization compression algorithm.

[0192] In a possible implementation, the compression unit 1003 is specifically configured to: convert the optimizer parameter represented by the floating point number into decimal places and numerical places represented by the integer according to the data quantization compression algorithm.

[0193] In a possible implementation, the neural network model is a recommendation model.

[0194] like Fig.11 As shown, a schematic diagram of a structure of a computing device provided in an embodiment of the present application, the computing device 1100 may be Figure 2 The training device 120 shown, Figure 3 The first node 201 and / or the second node 202 are shown. The computing device 1100 includes a processor 1101 , a memory 1102 , a communication bus 1103 and a communication interface 1104 . The processor 1101 is connected to the memory 1102 and the communication interface 1104 via the communication bus 1103 .

[0195] Processor 1101 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), a neural network processor (NPU) or one or more integrated circuits to execute relevant programs to execute the optimization method of the optimizer of the method embodiment of the present application.

[0196] The processor 1101 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the method for reducing the memory usage of the optimizer of the present application can be completed by the hardware integrated logic circuit or software instructions in the processor 1101. The above-mentioned processor 1101 may also be a general-purpose processor, a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor to be executed. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1102, and the processor 1101 reads the information in the memory 1102 and executes the data processing method of the embodiment of the method of the present application in combination with its hardware.

[0197] The memory 1102 is used to provide a storage space, and the storage space can optionally store application data, user data, operating system, and computer programs. The memory 1102 may include HBM, and may also include a volatile memory (volatile memory), such as a random access memory (random access memory, RAM). The memory 1102 may also include a non-volatile memory (non-volatile memory), such as a read-only memory (read-only memory, ROM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD).

[0198] For example, a program used to implement the method of reducing the memory occupied by the optimizer in the embodiment of the present application, etc.

[0199] The communication interface 1104 is used to provide information input or output for the at least one processor. And / or, the communication interface 1104 can be used to receive data sent externally and / or send data to the outside. The communication interface 1104 can be a wired link interface including an Ethernet cable, etc., or a wireless link (Wi-Fi, Bluetooth, general wireless transmission and other wireless communication technologies, etc.) interface. Optionally, the communication interface 1104 can also include a transmitter (such as a radio frequency transmitter, an antenna, etc.) coupled to the interface, or a receiver, etc.

[0200] Optionally, the computing device 1100 may also include an artificial intelligence processor 1105, which may be a neural network processor (Network Processing Unit, NPU), a tensor processor (TensorProcessing Unit, TPU), or a graphics processor (Graphics Processing Unit, GPU) and other processors suitable for large-scale XOR operation processing. The artificial intelligence processor 1105 can be mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks to it. The artificial intelligence processor 1105 can implement one or more operations involved in the above-mentioned neural network training method. For example, taking NPU as an example, the core part of the NPU is the operation circuit, which controls the operation circuit through a controller to extract matrix data in the memory 1102 and perform multiplication and addition operations.

[0201] The processor 1101 is used to call the data and program code in the memory to execute the specific operations performed by the computing device 1100 in the above method embodiment, which will not be repeated here.

[0202] It should be understood that the implementation of each device can also refer to the corresponding description in the optimization method embodiment of the above-mentioned optimizer, and the embodiments of the present application will not be repeated.

[0203] The embodiment of the present invention further provides a computer storage medium, in which instructions are stored, and when the instructions are executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above-mentioned embodiment methods. If the components of the above-mentioned device are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer product is stored in a computer-readable storage medium.

[0204] The computer-readable storage medium may be an internal storage unit of the device of the aforementioned embodiment, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0205] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0206] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0207] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0208] It is to be understood that those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the various embodiments of the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0209] Those skilled in the art will appreciate that the functions described in the various illustrative logic blocks, modules, and algorithm steps disclosed in the various embodiments of the present application can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions described in the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media, such as data storage media, or includes any communication media that facilitates the transfer of computer programs from one place to another (e.g., according to a communication protocol). In this way, computer-readable media can generally correspond to (1) non-temporary tangible computer-readable storage media, or (2) communication media, such as signals or carrier waves. Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the technology described in the present application. A computer program product may include computer-readable media.

[0210] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0211] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0212] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0213] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0214] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a base station, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., and other media that can store program codes.

[0215] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An optimization method of an optimizer, characterized in that: The method comprises: Get the parameters of the neural network model; Analyzing characteristics of the parameters of the neural network model, and determining a target compression algorithm from a plurality of compression algorithms according to the characteristics of the parameters of the neural network model; The target compression algorithm is used to compress the optimizer parameters in the optimizer of the neural network model, and the optimizer is used to optimize the parameters of the neural network model by converting the parameters of the neural network model into the optimizer parameters.

2. The method according to claim 1, characterized in that When the parameter of the neural network model is analyzed to be greater than a first threshold, the target compression algorithm is a high-performance compression algorithm, and the high-performance compression algorithm is a compression algorithm that improves the training efficiency of the neural network model.

3. The method according to claim 1, characterized in that When the parameters of the neural network model are analyzed to be less than or equal to a first threshold, and the distribution of the parameters of the neural network model is a smooth distribution, the target compression algorithm is a high-performance compression algorithm, and the high-performance compression algorithm is a compression algorithm that improves the training efficiency of the neural network model.

4. The method according to claim 2 or 3, characterized in that: The high-performance compression algorithm is a sampling compression algorithm, and the sampling compression algorithm compresses the optimizer parameters by sampling the optimizer parameters.

5. The method according to claim 4, characterized in that Compressing the optimizer parameters using the sampling compression algorithm includes: The matrix corresponding to the optimizer parameters is sampled in units of rows, and the sampled optimizer parameters are used as compressed optimizer parameters.

6. The method according to claim 4, characterized in that Compressing the optimizer parameters using the sampling compression algorithm includes: The matrix corresponding to the optimizer parameters is merged and reduced in units of rows, and the optimizer parameters obtained after the reduce operation are used as compressed optimizer parameters.

7. The method according to claim 1, characterized in that When the parameters of the neural network model are analyzed to be less than or equal to a first threshold, and the distribution of the parameters of the optimizer neural network model is a non-smooth distribution, the target compression algorithm is a high-precision compression algorithm, and the high-precision compression algorithm is a compression algorithm that improves the training accuracy of the neural network model.

8. The method according to claim 7, characterized in that The high-precision compression algorithm is a data quantization compression algorithm, which is used to compress the optimizer parameters by converting floating-point numbers into integers.

9. The method according to claim 8, characterized in that Compressing the optimizer parameters using the data quantization compression algorithm includes: The optimizer parameter represented by the floating point number is converted into a sign bit, a first decimal bit and a first value bit represented by the integer according to the data quantization compression algorithm.

10. The method according to claim 8, characterized in that Compressing the optimizer parameters using the data quantization compression algorithm includes: The optimizer parameter of the floating point number is converted into a second decimal place and a second numerical place of the integer according to the data quantization compression algorithm.

11. The method according to any one of claims 1 to 10, characterized in that: The neural network model is a recommendation model.

12. An optimization device of an optimizer, characterized in that: The device comprises: An acquisition unit, used for acquiring parameters of a neural network model; A processing unit, configured to analyze characteristics of the parameters of the neural network model, and determine a target compression algorithm from a plurality of compression algorithms according to the characteristics of the parameters of the neural network model; A compression unit is used to compress the optimizer parameters in the optimizer of the neural network model using the target compression algorithm, and the optimizer is used to optimize the parameters of the neural network model by converting the parameters of the neural network model into the optimizer parameters.

13. The device according to claim 12, characterized in that When the parameter of the neural network model is analyzed to be greater than a first threshold, the target compression algorithm is a high-performance compression algorithm, and the high-performance compression algorithm is a compression algorithm that improves the training efficiency of the neural network model.

14. The device according to claim 12, characterized in that When the parameters of the neural network model are analyzed to be less than or equal to a first threshold, and the distribution of the parameters of the optimizer neural network model is a smooth distribution, the target compression algorithm is a high-performance compression algorithm, and the high-performance compression algorithm is a compression algorithm that improves the training efficiency of the neural network model.

15. The device according to claim 13 or 14, characterized in that The high-performance compression algorithm is a sampling compression algorithm, and the sampling compression algorithm compresses the optimizer parameters by sampling the optimizer parameters.

16. The device according to claim 15, characterized in that The compression unit is specifically used for: The matrix corresponding to the optimizer parameter is sampled in units of rows, and the sampled values ​​are used as compressed optimizer parameters.

17. The device according to claim 15, characterized in that The compression unit is specifically used for: The matrix corresponding to the optimizer parameter is merged and reduced in units of rows, and the value obtained after the reduce operation is used as the compressed optimizer parameter.

18. The device according to claim 12, characterized in that When the optimizer parameter is analyzed to be less than or equal to the first threshold, and the distribution of the parameters of the optimizer neural network model is a non-smooth distribution, the target compression algorithm is a high-precision compression algorithm, and the high-precision compression algorithm is a compression algorithm that improves the training accuracy of the neural network model.

19. The device according to claim 18, characterized in that The high-precision compression algorithm is a data quantization compression algorithm, which is used to compress the optimizer parameters by converting floating-point numbers into integers.

20. The device according to claim 19, characterized in that The compression unit is specifically used for: The optimizer parameter of the floating point number is converted into a sign bit, a decimal bit and a value bit of the integer according to the data quantization compression algorithm.

21. The device according to claim 19, characterized in that The compression unit is specifically used for: The optimizer parameters of the floating point number are converted into decimal places and numerical places of the integer according to the data quantization compression algorithm.

22. The device according to any one of claims 12 to 21, characterized in that The neural network model is a recommendation model.

23. A computing device, characterized in that include: A memory and a processor, wherein the processor is configured to execute instructions stored in the memory so that the processor implements the method according to any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, which, when executed by a computing device, enable the computing device to implement the method according to any one of claims 1 to 11.

25. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device, the computing device is enabled to implement the method according to any one of claims 1 to 11.

Images