Data quantification method and device of optimizer, electronic equipment and medium

Abstract

The invention provides a data quantification method and device of an optimizer, electronic equipment and a medium, and relates to the technical field of data processing, in particular to the technical field of data quantification. According to the implementation scheme, gradient data of an optimizer are obtained, and the gradient data comprise initial first-order momentum and initial second-order momentum; determining an initial update rate according to the initial first-order momentum and the initial second-order momentum, the initial update rate indicating a proportional relationship between the initial first-order momentum and the initial second-order momentum; quantizing the initial first-order momentum and the initial update rate to obtain a quantized first-order momentum and a quantized update rate for storage; performing inverse quantization operation on the quantized first-order momentum and the quantized update rate to obtain a target first-order momentum and a target update rate; and determining a target second-order momentum according to the target first-order momentum and the target update rate, so as to update training parameters of the neural network model based on the target first-order momentum and the target second-order momentum.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, in particular to the field of data quantization technology, and specifically to a data quantization method, device, electronic device, computer-readable storage medium, and computer program product for an optimizer. Background Art

[0002] In the training process of the neural network model based on the gradient algorithm, the weight parameters need to be saved as a checkpoint every certain training step. However, for large models, especially large models using distributed parallel training methods, the data required to save for a single training will take up a lot of storage space.

[0003] In this regard, a method has been proposed to quantize the first-order momentum and second-order momentum in the model training optimizer to save storage space.

[0004] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0005] The present disclosure provides a data quantization method, device, electronic device, computer-readable storage medium, and computer program product for an optimizer.

[0006] According to one aspect of the present disclosure, a data quantization method for an optimizer is provided, comprising: obtaining gradient data of an optimizer, wherein the optimizer is used to optimize a training process of a neural network model based on a gradient algorithm, and the gradient data comprises an initial first-order momentum and an initial second-order momentum; determining an initial update rate according to the initial first-order momentum and the initial second-order momentum, wherein the initial update rate indicates a proportional relationship between the initial first-order momentum and the initial second-order momentum; performing a quantization operation on the initial first-order momentum and the initial update rate to obtain a quantized first-order momentum and a quantized update rate for storage; performing a dequantization operation on the quantized first-order momentum and the quantized update rate to obtain a target first-order momentum and a target update rate; and determining a target second-order momentum according to the target first-order momentum and the target update rate to update the training parameters of the neural network model based on the target first-order momentum and the target second-order momentum.

[0007] According to another aspect of the present disclosure, a data quantization device of an optimizer is provided, comprising: an acquisition module configured to acquire gradient data of an optimizer, wherein the optimizer is used to optimize the training process of a neural network model based on a gradient algorithm, and the gradient data includes an initial first-order momentum and an initial second-order momentum; a first determination module configured to determine an initial update rate based on the initial first-order momentum and the initial second-order momentum, wherein the initial update rate indicates a proportional relationship between the initial first-order momentum and the initial second-order momentum; a quantization module configured to perform a quantization operation on the initial first-order momentum and the initial update rate to obtain a quantized first-order momentum and a quantized update rate for storage; a dequantization module configured to perform a dequantization operation on the quantized first-order momentum and the quantized update rate to obtain a target first-order momentum and a target update rate; and a second determination target configured to determine a target second-order momentum based on the target first-order momentum and the target update rate, so as to update the training parameters of the neural network model based on the target first-order momentum and the target second-order momentum.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above method.

[0010] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0011] According to one or more embodiments of the present disclosure, a data quantization method for an optimizer is provided. When the gradient data of the optimizer is quantized, the update rate is determined based on the proportional relationship between the first-order momentum and the second-order momentum of the optimizer to establish a correlation relationship between the first-order momentum and the second-order momentum. Based on this, there is no need to quantize the second-order momentum separately. As long as the first-order momentum and the update rate of the optimizer are quantized, the second-order momentum can be determined based on the first-order momentum and the update rate after dequantization, which effectively reduces the data loss in the quantization process and achieves the goal of minimizing the storage space while maintaining the training direction of the neural network model to the greatest extent.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figure 1 is a schematic diagram illustrating an example system in which the various methods described herein may be implemented according to an exemplary embodiment;

[0015] Figure 2 A flow chart of a data quantization method of an optimizer according to an embodiment of the present disclosure is shown;

[0016] Figure 3 A partial flow chart of a data quantization method of another optimizer according to an embodiment of the present disclosure is shown;

[0017] Figure 4 A structural block diagram showing a data quantization device of an optimizer according to an embodiment of the present disclosure; and

[0018] Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0019] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0020] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0021] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0022] In the related art, a method is proposed to quantize the first-order momentum and second-order momentum in the model training optimizer to save storage space during the training process of the neural network model based on the gradient algorithm. However, the quantized and dequantized gradient data will cause data loss, which greatly affects the training direction of the model.

[0023] To solve the above problems, the present disclosure provides a data quantization method for an optimizer, which determines the update rate based on the proportional relationship between the first-order momentum and the second-order momentum of the optimizer, so as to establish a correlation relationship between the first-order momentum and the second-order momentum. Therefore, there is no need to quantize the second-order momentum separately. It is only necessary to quantize the first-order momentum and the update rate of the optimizer. The second-order momentum can be determined based on the first-order momentum and the update rate after dequantization, which effectively reduces the data loss in the quantization process and achieves the goal of minimizing the storage space while maintaining the training direction of the neural network model to the greatest extent.

[0024] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0025] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0026] In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable execution of the optimizer's data quantization method.

[0027] In some embodiments, server 120 may also provide other services or software applications that may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0028] exist Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the data quantization method of the optimizer described herein and is not intended to be limiting.

[0029] The user may use client devices 101, 102, 103, 104, 105 and / or 106 to perform the data quantization method of the optimizer. The client device may provide an interface that enables the user of the client device to interact with the client device. The client device may also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0030] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.

[0031] The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0032] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0033] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0034] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0035] In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0036] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store training data and other information for the neural network model. For example, one or more of the databases 130 may be used to store information such as weight parameters for model training. The databases 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the database used by the server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data to and from the database in response to commands.

[0037] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0038] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0039] Figure 2 A flow chart of a data quantization method of an optimizer according to an embodiment of the present disclosure is shown.

[0040] like Figure 2 As shown, the data quantization method 200 of the optimizer includes:

[0041] Step 210: Obtaining gradient data of an optimizer, wherein the optimizer is used to optimize the training process of a neural network model based on a gradient algorithm, and the gradient data includes an initial first-order momentum and an initial second-order momentum;

[0042] Step 220: determining an initial update rate according to the initial first-order momentum and the initial second-order momentum, wherein the initial update rate indicates a proportional relationship between the initial first-order momentum and the initial second-order momentum;

[0043] Step 230: quantize the initial first-order momentum and the initial update rate to obtain a quantized first-order momentum and a quantized update rate for storage;

[0044] Step 240: performing a dequantization operation on the quantized first-order momentum and the quantized update rate to obtain a target first-order momentum and a target update rate; and

[0045] Step 250: Determine the target second-order momentum according to the target first-order momentum and the target update rate, so as to update the training parameters of the neural network model based on the target first-order momentum and the target second-order momentum.

[0046] Therefore, by determining the update rate based on the proportional relationship between the first-order momentum and the second-order momentum of the optimizer, a correlation relationship between the first-order momentum and the second-order momentum is established. There is no need to quantize the second-order momentum separately. As long as the first-order momentum and the update rate of the optimizer are quantized, the second-order momentum can be determined based on the first-order momentum and the update rate after dequantization, which effectively reduces the data loss in the quantization process.

[0047] In addition, each model training requires storing multiple checkpoints to record multiple intermediate parameters of the training process. The quantization operation reduces the storage space occupied by the first-order momentum and second-order momentum in each checkpoint, which greatly reduces the demand for storage space generated to save intermediate parameters during model training, and achieves the goal of minimizing storage space while maintaining the training direction of the neural network model to the greatest extent.

[0048] Furthermore, since the training direction of the neural network model is better maintained, when the checkpoint is returned for inspection during the training process, the model iteration direction indicated by the dequantized first-order momentum and the second-order momentum obtained according to the proportional relationship will be more accurate, thereby avoiding the waste of hard disk storage space due to invalid iterations in the wrong direction and greatly improving the utilization efficiency of storage space.

[0049] In step 210, the optimizer may be, for example, an Adam (Adaptive Moment Estimation) optimizer. It should be understood that the optimizer may also be other optimizers that perform parameter updates based on first-order momentum and second-order momentum.

[0050] In step 210, the neural network model based on the gradient algorithm can be, for example, a PaddlePaddle large language model. This type of model adopts a mixed precision training method during training. That is, compared with the traditional full-precision training method, during the training process of this type of model, each checkpoint saves not only a set of full-precision weight models, but also a set of half-precision models, so as to retain more training process information. Based on this, this type of model has high requirements for storage overhead and data loss when quantizing gradient data. The data quantization method of the optimizer disclosed in the present invention effectively improves the training efficiency of such large models and the performance of the target model obtained through training.

[0051] It should be understood that in the optimizer of the gradient optimization algorithm, the first-order momentum is the sliding average of the gradient, which is used to introduce the momentum term in the iteration process to accelerate the gradient descent, and the second-order momentum is the sliding average of the square of the gradient, which is used to adaptively adjust the learning rate of each parameter.

[0052] According to some embodiments, the initial update rate indicates a negative correlation between the initial first order momentum and the initial second order momentum.

[0053] In step 220, the initial update rate is related to the learning rate of the optimizer. The learning rate in the gradient optimization algorithm can be determined based on the following formula:

[0054]

[0055] Among them, LR represents the learning rate, γ represents a constant, represents the first-order momentum, represents the second-order momentum, ∈ represents the bias constant, which is used to prevent the zero division anomaly.

[0056] It can be seen from the above formula that the learning rate can indicate the proportional relationship between the first-order momentum and the second-order momentum of the optimizer. However, the learning rate may be positive or negative. Therefore, if data quantization is performed directly based on the learning rate, the quantization loss may cause the obtained learning rate to fluctuate greatly during the dequantization process, which seriously affects the training direction of the neural network model.

[0057] Based on this, illustratively, the initial update rate may be determined based on the following formula:

[0058]

[0059] Among them, ratio represents the initial update rate, sqrt() represents the square root operation, and eps represents the bias constant used to prevent zero division exceptions.

[0060] It can be seen from the above formula that based on the numerical change law of the weight parameters before and after quantization and the calculation formula of the learning rate of the optimizer, the numerator of the learning rate can be reduced to 1 while the denominator remains the same to maintain it as the update rate, thereby effectively avoiding large fluctuations in the numerical values ​​during quantization and dequantization, and maintaining the training direction of the neural network model to the greatest extent.

[0061] Figure 3 A partial flow chart of a data quantization method of another optimizer according to an embodiment of the present disclosure is shown.

[0062] According to some embodiments, Figure 3 As shown, step 230 includes:

[0063] Step 231, symmetrically and uniformly quantize the initial first-order momentum to obtain quantized first-order momentum for storage; and

[0064] Step 232: Perform asymmetric uniform quantization on the initial update rate to obtain a quantized update rate for storage.

[0065] For different data of initial first-order momentum and initial update rate, symmetric uniform quantization and asymmetric uniform quantization can be adopted for the two respectively, so as to avoid zero division exception in the quantization process, that is, to prevent invalid values ​​due to data overflow, so as to improve the numerical stability in the quantization process.

[0066] According to some embodiments, the quantization operation is a channel-dimensional quantization operation, thereby providing a relatively coarse-grained quantization operation to adapt to corresponding model training scenarios and improve the efficiency of model training.

[0067] In step 231, exemplarily, the weight parameters (initial first-order momentum) of the neural network model can be a set of two-dimensional matrices, for example, a 100x 100 two-dimensional matrix, with the channel dimension as the granularity reference, and an absolute maximum value is taken from each column of the weight parameter matrix to obtain a 1x 100 long strip matrix as a quantized tensor.

[0068] Based on this, the symmetric uniform quantization process can be implemented based on the following formula:

[0069]

[0070] Among them, quant_weight represents the weight parameter after quantization, clip() represents the clipping operation, round() represents the rounding operation, weight represents the weight parameter before quantization, scales represents the quantized tensor, bcnt represents the constant, which is used to map the numerical value of the ratio of the weight parameter before quantization to the tensor to the corresponding quantization space, -bnt-1 represents the lower bound of the mapped corresponding quantization space, and bnt represents the upper bound of the mapped corresponding quantization space.

[0071] It should be noted that both the symmetric quantization method and the asymmetric quantization method are commonly used data quantization methods in this field. Since the symmetric quantization method is a special case of the asymmetric quantization method, an example is given here for the symmetric quantization method, and the asymmetric method is not repeated here.

[0072] According to some embodiments, the quantization operation is a block-dimensional quantization operation. Compared with the channel dimension, the block dimension can provide a higher granularity, so quantization based on the block dimension can further reduce the data loss caused by quantization by refining the quantization granularity.

[0073] It should be understood that the implementation of the block-dimensional quantization operation is similar to that of the channel-dimensional quantization operation, which will not be described in detail.

[0074] According to some embodiments, the data types of the initial first-order momentum and the initial update rate are floating point types, and the data types of the quantized first-order momentum and the quantized update rate are integer types.

[0075] Exemplarily, the optimizer's initial first-order momentum and initial update rate are typically of float32 (32-bit floating point) type, which can be quantized to int4 (4-bit integer) or int8 (8-bit integer) type for compressed storage, thereby reducing the required storage space to 25% of that before quantization, effectively reducing memory overhead.

[0076] According to some embodiments, the integer type includes an int4 type and an int8 type, and the method 200 further includes:

[0077] Step 260 : In response to the data types of the quantized first-order momentum and the quantized update rate being of the int4 type, the quantized first-order momentum and the quantized update rate are combined into int8 type data for storage through binary shift.

[0078] Currently, most deep learning databases do not support the storage of int4 type integer data. By merging two int4 type data into one int8 type data, it is more convenient to store quantitative data to expand the application scenarios.

[0079] In step 240, illustratively, the quantized first-order momentum and quantized update rate of the int4 type or the int8 type are restored to the target first-order momentum and target update rate of the float32 type through a dequantization operation, so as to be used for updating the training parameters of the neural network model.

[0080] For example, for the quantized first-order momentum and quantized update rate of type in4, if the two are merged into one int8 type data during the quantization process, then during the dequantization process, bit operations are used to restore them to two int4 type data, and then dequantized to float32 type according to their respective tensors.

[0081] In step 250, after obtaining the target first-order momentum and the target update rate, the target second-order momentum can be determined based on the following formula:

[0082]

[0083] It can be seen that this formula is derived from the above formula for determining the initial update rate, which will not be described in detail.

[0084] According to another aspect of the present disclosure, a data quantization device of an optimizer is provided. Figure 4 As shown, the data quantization device 400 of the optimizer includes: an acquisition module 410, configured to acquire gradient data of the optimizer, wherein the optimizer is used to optimize the training process of the neural network model based on the gradient algorithm, and the gradient data includes an initial first-order momentum and an initial second-order momentum; a first determination module 420, configured to determine an initial update rate according to the initial first-order momentum and the initial second-order momentum, wherein the initial update rate indicates the proportional relationship between the initial first-order momentum and the initial second-order momentum; a quantization module 430, configured to perform a quantization operation on the initial first-order momentum and the initial update rate to obtain a quantized first-order momentum and a quantized update rate for storage; a dequantization module 440, configured to perform a dequantization operation on the quantized first-order momentum and the quantized update rate to obtain a target first-order momentum and a target update rate; and a second determination target 450, configured to determine a target second-order momentum according to the target first-order momentum and the target update rate, so as to update the training parameters of the neural network model based on the target first-order momentum and the target second-order momentum.

[0085] According to some embodiments, the initial update rate indicates a negative correlation between the initial first order momentum and the initial second order momentum.

[0086] According to some embodiments, the quantization module 430 includes: a first quantization submodule 431, configured to perform symmetrical uniform quantization on the initial first-order momentum to obtain a quantized first-order momentum for storage; and a second quantization submodule 432, configured to perform asymmetrical uniform quantization on the initial update rate to obtain a quantized update rate for storage.

[0087] According to some embodiments, the data types of the initial first-order momentum and the initial update rate are floating-point types, and the data types of the quantized first-order momentum and the quantized update rate are integer types.

[0088] According to some embodiments, the integer type includes int4 type and int8 type, and the device 400 also includes: a merging module 460, which is configured to merge the quantized first-order momentum and the quantized update rate into int8 type data for storage through binary shift in response to the data type of the quantized first-order momentum and the quantized update rate being int4 type.

[0089] According to some embodiments, the quantization operation is a channel dimension based quantization operation.

[0090] According to some embodiments, the quantization operation is a block dimension based quantization operation.

[0091] According to another aspect of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the aforementioned method.

[0092] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is further provided, wherein the computer instructions are used to enable the computer to execute the aforementioned method.

[0093] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, wherein the computer program implements the aforementioned method when executed by a processor.

[0094] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0095] Multiple components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, an output unit 507, a storage unit 508 and a communication unit 509. The input unit 506 can be any type of device that can input information to the electronic device 500. The input unit 506 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a trackpad, a trackball, a joystick, a microphone and / or a remote control. The output unit 507 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator and / or a printer. The storage unit 508 can include but is not limited to a disk, an optical disk. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as Bluetooth TM devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0096] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as a matrix calculation method based on a GPU. For example, in some embodiments, a matrix calculation method based on a GPU may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via a ROM 502 and / or a communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the matrix calculation method based on the GPU described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform a matrix calculation method based on a GPU in any other appropriate manner (e.g., by means of firmware).

[0097] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0098] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0099] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0100] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0101] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0102] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0103] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0104] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A data quantization method for an optimizer, comprising: Obtaining gradient data of an optimizer, wherein the optimizer is used to optimize a training process of a neural network model based on a gradient algorithm, and the gradient data includes an initial first-order momentum and an initial second-order momentum; determining an initial update rate according to the initial first-order momentum and the initial second-order momentum, wherein the initial update rate indicates a proportional relationship between the initial first-order momentum and the initial second-order momentum; Performing a quantization operation on the initial first-order momentum and the initial update rate to obtain a quantized first-order momentum and a quantized update rate for storage; performing a dequantization operation on the quantized first-order momentum and the quantized update rate to obtain a target first-order momentum and a target update rate; and A target second-order momentum is determined according to the target first-order momentum and the target update rate, so as to update the training parameters of the neural network model based on the target first-order momentum and the target second-order momentum.

2. The method according to claim 1, wherein: The initial update rate indicates a negative correlation between the initial first-order momentum and the initial second-order momentum.

3. The method according to claim 1 or 2, wherein: The step of performing a quantization operation on the initial first-order momentum and the initial update rate to obtain a quantized first-order momentum and a quantized update rate for storage includes: symmetrically and uniformly quantizing the initial first-order momentum to obtain the quantized first-order momentum for storage; and The initial update rate is asymmetrically uniformly quantized to obtain the quantized update rate for storage.

4. The method according to any one of claims 1 to 3, wherein: The data types of the initial first-order momentum and the initial update rate are floating-point types, and the data types of the quantized first-order momentum and the quantized update rate are integer types.

5. The method according to claim 4, wherein: The integer type includes an int4 type and an int8 type, and wherein the method further includes: In response to the data types of the quantized first-order momentum and the quantized update rate being int4 type, the quantized first-order momentum and the quantized update rate are combined into int8 type data for storage through binary shift.

6. The method according to any one of claims 1 to 5, wherein: The quantization operation is a quantization operation based on the channel dimension.

7. The method according to any one of claims 1 to 5, wherein: The quantization operation is a block-dimensional quantization operation.

8. A data quantization device for an optimizer, comprising: An acquisition module is configured to acquire gradient data of an optimizer, wherein the optimizer is used to optimize the training process of a neural network model based on a gradient algorithm, and the gradient data includes an initial first-order momentum and an initial second-order momentum; a first determination module configured to determine an initial update rate according to the initial first-order momentum and the initial second-order momentum, wherein the initial update rate indicates a proportional relationship between the initial first-order momentum and the initial second-order momentum; a quantization module configured to perform a quantization operation on the initial first-order momentum and the initial update rate to obtain a quantized first-order momentum and a quantized update rate for storage; a dequantization module configured to perform a dequantization operation on the quantized first-order momentum and the quantized update rate to obtain a target first-order momentum and a target update rate; and The second determination target is configured to determine a target second-order momentum according to the target first-order momentum and the target update rate, so as to update the training parameters of the neural network model based on the target first-order momentum and the target second-order momentum.

9. The device according to claim 8, wherein: The initial update rate indicates a negative correlation between the initial first-order momentum and the initial second-order momentum.

10. The device according to claim 8 or 9, wherein: The quantization module comprises: A first quantization submodule is configured to perform symmetrical and uniform quantization on the initial first-order momentum to obtain the quantized first-order momentum for storage; and The second quantization submodule is configured to perform asymmetric uniform quantization on the initial update rate to obtain the quantized update rate for storage.

11. The device according to any one of claims 8 to 10, wherein: The data types of the initial first-order momentum and the initial update rate are floating-point types, and the data types of the quantized first-order momentum and the quantized update rate are integer types.

12. The device according to claim 11, wherein The integer type includes an int4 type and an int8 type, and wherein the apparatus further includes: The merging module is configured to merge the quantized first-order momentum and the quantized update rate into int8 type data for storage by binary shift in response to the data types of the quantized first-order momentum and the quantized update rate being int4 type.

13. The device according to any one of claims 8 to 12, wherein: The quantization operation is a quantization operation based on the channel dimension.

14. The device according to any one of claims 8 to 12, wherein: The quantization operation is a block-dimensional quantization operation.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

17. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images