Data quantization method and apparatus of an optimizer, electronic device, and medium

Abstract

The present disclosure provides an optimizer data quantization method, device, electronic equipment and medium, relates to the technical field of data processing, and particularly relates to the technical field of data quantization. The implementation scheme is: gradient data of an optimizer is acquired, wherein the gradient data includes initial first-order momentum and initial second-order momentum; an initial update rate is determined according to the initial first-order momentum and the initial second-order momentum, wherein the initial update rate indicates a proportional relationship between the initial first-order momentum and the initial second-order momentum; a quantization operation is performed on the initial first-order momentum and the initial update rate to obtain a quantized first-order momentum and a quantized update rate for storage; a dequantization operation is performed on the quantized first-order momentum and the quantized update rate to obtain target first-order momentum and target update rate; and target second-order momentum is determined according to the target first-order momentum and the target update rate, so as to update training parameters of a neural network model based on the target first-order momentum and the target second-order momentum.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, in particular to the technical field of data quantization, and more particularly to a data quantization method and device of an optimizer, an electronic device, a computer readable storage medium, and a computer program product. BACKGROUND

[0002] In the training process of a neural network model based on a gradient algorithm, the weight parameters need to be saved as a checkpoint every certain training step. However, for a large model, especially a large model using distributed parallel training, the data to be saved in a single training will occupy a large storage space.

[0003] To this end, a method of quantizing the first-order momentum and the second-order momentum in the model training optimizer to save storage space is currently proposed.

[0004] The methods described in this section can not have been previously conceived or made. Unless otherwise indicated, it should not be assumed that any of the methods described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, issues identified with respect to one or more methods should not be assumed to have been raised with respect to any prior art. SUMMARY

[0005] The present disclosure provides a data quantization method and device of an optimizer, an electronic device, a computer readable storage medium, and a computer program product.

[0006] According to an aspect of the present disclosure, a data quantization method of an optimizer is provided, comprising: obtaining gradient data of an optimizer, wherein the optimizer is used to optimize a training process of a neural network model based on a gradient algorithm, and the gradient data comprises an initial first-order momentum and an initial second-order momentum; determining an initial update rate according to the initial first-order momentum and the initial second-order momentum, wherein the initial update rate indicates a proportional relationship between the initial first-order momentum and the initial second-order momentum; performing a quantization operation on the initial first-order momentum and the initial update rate to obtain a quantized first-order momentum and a quantized update rate for storage; performing a dequantization operation on the quantized first-order momentum and the quantized update rate to obtain a target first-order momentum and a target update rate; and determining a target second-order momentum according to the target first-order momentum and the target update rate, to update a training parameter of the neural network model based on the target first-order momentum and the target second-order momentum.

[0007] According to another aspect of the present disclosure, there is provided a data quantization apparatus of an optimizer, comprising: an obtaining module configured to obtain gradient data of an optimizer, wherein the optimizer is used to optimize a training process of a neural network model based on a gradient algorithm, and the gradient data comprises an initial first-order momentum and an initial second-order momentum; a first determining module configured to determine an initial update rate according to the initial first-order momentum and the initial second-order momentum, wherein the initial update rate indicates a proportional relationship between the initial first-order momentum and the initial second-order momentum; a quantization module configured to perform a quantization operation on the initial first-order momentum and the initial update rate to obtain a quantized first-order momentum and a quantized update rate for storage; a dequantization module configured to perform a dequantization operation on the quantized first-order momentum and the quantized update rate to obtain a target first-order momentum and a target update rate; and a second determining module configured to determine a target second-order momentum according to the target first-order momentum and the target update rate, so as to update a training parameter of the neural network model based on the target first-order momentum and the target second-order momentum.

[0008] According to another aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0009] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the above method.

[0010] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the above method.

[0011] According to one or more embodiments of the present disclosure, there is provided a data quantization method of an optimizer, when quantizing gradient data of an optimizer, an update rate is determined based on a proportional relationship between a first-order momentum and a second-order momentum of the optimizer, so as to establish a correlation between the first-order momentum and the second-order momentum, based on which, without separately quantizing the second-order momentum, only the first-order momentum and the update rate of the optimizer need to be quantized, and the second-order momentum can be determined based on the first-order momentum and the update rate after dequantization, thereby effectively reducing data loss in the quantization process, and realizing the minimization of storage space while maximizing the maintenance of the training direction of the neural network model.

[0012] It is to be understood that the details set forth herein are by way of example and not intended to limit the scope of the subject disclosure, but that the subject disclosure is widely applicable to various embodiments and many options and alternatives are available. Other features, objects, and / or advantages of the disclosed embodiments will be apparent from the description of the disclosed embodiments and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this application, illustrate embodiments of the disclosure and together with the description serve to explain exemplary implementations of the disclosure. The illustrated embodiments are merely examples and other embodiments can be used without departing from the scope of the claims. In all the drawings, like reference numerals refer to like parts throughout the several views.

[0014] Figure 1 is a schematic diagram illustrating an example system in which various methods described herein can be implemented, in accordance with an example embodiment;

[0015] Figure 2 is a flowchart illustrating a data quantization method of an optimizer according to an embodiment of the disclosure;

[0016] Figure 3 is a partial flowchart illustrating a data quantization method of another optimizer according to an embodiment of the disclosure;

[0017] Figure 4 is a block diagram illustrating a structure of a data quantization apparatus of an optimizer according to an embodiment of the disclosure; and

[0018] Figure 5 is a block diagram illustrating a structure of an exemplary electronic device that can be used to implement an embodiment of the disclosure. DETAILED DESCRIPTION

[0019] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, in which various details of the embodiments of the present disclosure are set forth in order to provide an adequate understanding of the present disclosure. It will be readily apparent to those skilled in the art that various modifications and changes can be made without departing from the scope of the present disclosure. Similarly, it should be apparent that the descriptions given with reference to the accompanying drawings are intended to be illustrative only and not restrictive. It is intended that the scope of the present disclosure be limited only by the appended claims.

[0020] In the present disclosure, the terms "first", "second", and the like are used to describe various elements, not to limit the positions, the order, or the importance of the elements. In some examples, the first element and the second element can refer to the same instance of the element, and in some cases, based on the context of the description, they can also refer to different instances.

[0021] The terminology used in the description of the various described examples in the present disclosure is for the purpose of describing particular examples only and is not intended to be limiting. As used in this description, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Additionally, the term "and / or" used in the

[0022] In the related art, a method of quantizing first-order momentum and second-order momentum in a model training optimizer to save storage space in the training process of a neural network model based on a gradient algorithm is proposed, but the gradient data after quantization and dequantization will cause data loss, which greatly affects the training direction of the model.

[0023] To solve the above problems, the present disclosure provides a data quantization method of an optimizer, determines an update rate based on the proportional relationship between the first-order momentum and the second-order momentum of the optimizer, to establish the correlation between the first-order momentum and the second-order momentum, thereby, without separately quantizing the second-order momentum, only the first-order momentum and the update rate of the optimizer need to be quantized, and the second-order momentum can be determined according to the first-order momentum and the update rate after dequantization, effectively reducing the data loss in the quantization process, and realizing the minimization of storage space while maintaining the training direction of the neural network model to the greatest extent.

[0024] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0025] Figure 1 A schematic diagram of an example system 100 in which various methods and apparatus described herein can be implemented in accordance with embodiments of the present disclosure is shown. Referring to Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 can be configured to execute one or more application programs.

[0026] In embodiments of the present disclosure, the server 120 can run one or more services or software applications that enable the execution of a data quantization method of an optimizer.

[0027] In certain embodiments, the server 120 can also provide other services or software applications that can include non-virtual and virtual environments. In certain embodiments, these services can be provided as web-based services or cloud services, for example, to users of the client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0028] In Figure 1 In the illustrated configuration, the server 120 can include one or more components implementing functionality performed by the server 120. These components can include software components, hardware components, or a combination thereof, executable by one or more processors. Users operating the client devices 101, 102, 103, 104, 105, and / or 106 can in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that various different system configurations are possible, which can differ from the system 100. Thus, Figure 1 is one example of a system for implementing the data quantization method of the optimizer described herein, and is not intended to be limiting.

[0029] A user can use the client device 101, 102, 103, 104, 105, and / or 106 to perform the data quantization method of the optimizer. The client device can provide an interface that enables the user of the client device to interact with the client device. The client device can also output information to the user via the interface. Although Figure 1 Only six client devices are depicted, but those skilled in the art will understand that the present disclosure can support any number of client devices.

[0030] The client devices 101, 102, 103, 104, 105, and / or 106 can include various types of computer devices, such as portable handheld devices, general purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service kiosk devices, service robots, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and the like. These computer devices can run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux, or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices can include cellular phones, smartphones, tablet computers, personal digital assistants (PDAs), and the like. Wearable devices can include head-mounted displays (such as smart glasses) and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices, and the like. The client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0031] Network 110 can be any type of network familiar to those skilled in the art, which can use any of a variety of available protocols (including, without limitation, TCP / IP, SNA, IPX, etc.) to support data communications. By way of example, one or more of networks 110 can be a LAN, an Ethernet network, a Token Ring network, a WAN, the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infra-red network, a wireless network (e.g., a Bluetooth network, a WIFI network, and / or the like), and / or any combination of these and / or other networks.

[0032] Server 120 can include one or more general purpose computers, special purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers, large mainframe computers), server clusters, or any other appropriate arrangement and / or combination. Server 120 can include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 can be adapted to run one or more services or software applications provided by the functionality described below.

[0033] Computing units in server 120 can run one or more operating systems, including any of the operating systems described above, as well as any commercially available server operating systems. Server 120 can also run any of a variety of additional server applications and / or mid-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, and the like.

[0034] In some embodiments, server 120 can include one or more applications to analyze and consolidate data feeds and / or event updates from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 can also include one or more applications to display the data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0035] In some embodiments, server 120 can be a server of a distributed system, or a server in combination with a blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The cloud server is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and virtual private server (VPS, Virtual Private Server) service.

[0036] The system 100 can also include one or more databases 130. In certain embodiments, these databases can be used to store training data and other information for neural network models. For example, one or more of the databases 130 can be used to store weight parameter information such as for model training. The databases 130 can reside in various locations. For example, databases used by the servers 120 can be local to the servers 120 or can be remote from the servers 120 and can communicate with the servers 120 via a network-based or dedicated connection. The databases 130 can be of different types. In certain embodiments, databases used by the servers 120 can be, for example, relational databases. One or more of these databases can store, update, and retrieve data to and from the databases in response to commands.

[0037] In certain embodiments, one or more of the databases 130 can also be used by applications to store application data. Databases used by applications can be different types of databases, such as key-value stores, object stores, or regular stores supported by file systems.

[0038] Figure 1 The system 100 can be configured and operated in various ways to enable the application of various methods and apparatuses described in accordance with the present disclosure.

[0039] Figure 2 A flowchart of a data quantization method of an optimizer is shown in accordance with embodiments of the present disclosure.

[0040] As shown in Figure 2 The data quantization method 200 of the optimizer includes:

[0041] Step 210, obtaining gradient data of an optimizer, wherein the optimizer is used to optimize a training process of a neural network model based on a gradient algorithm, and the gradient data includes an initial first-order momentum and an initial second-order momentum;

[0042] Step 220, determining an initial update rate according to the initial first-order momentum and the initial second-order momentum, wherein the initial update rate indicates a proportional relationship between the initial first-order momentum and the initial second-order momentum;

[0043] Step 230, performing a quantization operation on the initial first-order momentum and the initial update rate to obtain a quantized first-order momentum and a quantized update rate for storage;

[0044] Step 240, performing a dequantization operation on the quantized first-order momentum and the quantized update rate to obtain a target first-order momentum and a target update rate; and

[0045] In step 250, the target second-order momentum is determined according to the target first-order momentum and the target update rate, so as to update the training parameters of the neural network model based on the target first-order momentum and the target second-order momentum.

[0046] In this way, by determining the update rate based on the proportional relationship between the first-order momentum and the second-order momentum of the optimizer, a correlation between the first-order momentum and the second-order momentum is established, and the second-order momentum can be determined based on the first-order momentum and the update rate after dequantization without separately quantifying the second-order momentum. Thus, the data loss in the quantization process is effectively reduced.

[0047] In addition, multiple checkpoints need to be stored to record multiple intermediate parameters of the training process each time the model is trained, and the quantization operation reduces the storage space occupied by the first-order momentum and the second-order momentum in each checkpoint, thereby greatly reducing the demand for storage space for saving intermediate parameters in the model training process, and achieving the minimization of storage space while maximizing the maintenance of the training direction of the neural network model.

[0048] Further, since the training direction of the neural network model is well maintained, when the checkpoint is returned for checking during the training process, the model iteration direction indicated by the dequantized first-order momentum and the second-order momentum obtained according to the proportional relationship will be more accurate, thereby avoiding wasting hard disk storage space due to invalid iteration in the wrong direction, and greatly improving the utilization efficiency of the storage space.

[0049] In step 210, the optimizer may be, for example, an Adam (Adaptive Moment Estimation) optimizer. It should be understood that the optimizer may also be other optimizers that update parameters based on first-order momentum and second-order momentum.

[0050] In step 210, the neural network model based on the gradient algorithm may be, for example, a PaddlePaddle large language model. Such a model uses a mixed precision training method during training, that is, compared with the traditional full-precision training method, in the training process of such a model, in addition to saving a set of full-precision weight models, a set of half-precision models is also saved to retain more training process information. Based on this, the storage overhead and data loss of the quantization of gradient data by such a model are required to be higher, and the data quantization method of the optimizer according to the present disclosure effectively improves the training efficiency of such a large model and the performance of the target model obtained by training.

[0051] It should be understood that in the optimizer of the gradient optimization algorithm, the first order momentum is a sliding average of the gradient, used to introduce a momentum term to accelerate the gradient descent in the iteration process, and the second order momentum is a sliding average of the square of the gradient, used to adaptively adjust the learning rate of each parameter.

[0052] According to some embodiments, the initial update rate indicates a negative correlation between the initial first order momentum and the initial second order momentum.

[0053] In step 220, the initial update rate is related to the learning rate of the optimizer, and the learning rate in the gradient optimization algorithm can be determined based on the following formula:

[0054]

[0055] wherein LR represents the learning rate, γ represents a constant, represents the first order momentum, represents the second order momentum, and ∈ represents a bias constant for preventing a division by zero exception.

[0056] As can be seen from the above formula, the learning rate can indicate the proportional relationship between the first order momentum and the second order momentum of the optimizer, however, the learning rate can be positive or negative, and therefore, if the data quantization is directly based on the learning rate, the resulting learning rate can be greatly shocked due to quantization loss in the dequantization process, seriously affecting the training direction of the neural network model.

[0057] Based on this, exemplarily, the initial update rate can be determined based on the following formula:

[0058]

[0059] wherein ratio represents the initial update rate, sqrt() represents a square root operation, and eps represents a bias constant for preventing a division by zero exception.

[0060] As can be seen from the above formula, based on the numerical change rule before and after the quantization of the weight parameter and the calculation formula of the learning rate of the optimizer, the numerator part in the learning rate can be changed to 1 while the denominator part remains the same, so as to keep the update rate, thereby effectively avoiding the large shock of the numerical value in the quantization and dequantization process, and maximizing the training direction of the neural network model.

[0061] Figure 3 A partial flowchart of a data quantization method of another optimizer according to an embodiment of the present disclosure is shown.

[0062] According to some embodiments, as Figure 3 shown, step 230 includes:

[0063] Step 231, symmetrically and uniformly quantize the initial first-order momentum to obtain a quantized first-order momentum for storage; and

[0064] Step 232, asymmetrically and uniformly quantize the initial update rate to obtain a quantized update rate for storage.

[0065] For different data of the initial first-order momentum and the initial update rate, symmetric and asymmetric uniform quantization can be respectively taken for the two, so as to avoid the exception of division by zero in the quantization process, that is, to prevent the invalidity of the numerical value due to data overflow, so as to improve the numerical stability in the quantization process.

[0066] According to some embodiments, the quantization operation is a quantization operation based on the channel dimension. Thus, a relatively coarse-grained quantization operation can be provided to adapt to the corresponding model training scene and improve the efficiency of model training.

[0067] In step 231, exemplarily, the weight parameter (initial first-order momentum) of the neural network model can be a set of two-dimensional matrices, for example, a 100x 100 two-dimensional matrix, taking an absolute maximum value from each column of the weight parameter matrix as a 1x 100 long strip matrix as a quantization tensor based on the channel dimension as the granularity reference.

[0068] Based on this, the symmetric uniform quantization process can be implemented based on the following formula:

[0069]

[0070] Where quant_weight represents the quantized weight parameter, clip() represents the clipping operation, round() represents the rounding operation, weight represents the weight parameter before quantization, scales represents the quantization tensor, bcnt represents a constant for mapping the value of the ratio of the weight parameter before quantization to the corresponding quantization space, -bnt-1 represents the lower bound of the corresponding quantization space of the mapping, and bnt represents the upper bound of the corresponding quantization space of the mapping.

[0071] It should be noted that both the symmetric quantization method and the asymmetric quantization method are common data quantization methods in the art. Since the symmetric quantization method is a special case of the asymmetric quantization method, the symmetric quantization method is exemplified here, and the asymmetric method is not described again.

[0072] According to some embodiments, the quantization operation is a quantization operation based on the block dimension. Compared with the channel dimension, the block dimension can provide higher granularity, and therefore, quantization based on the block dimension can further reduce data loss caused by quantization by refining the quantization granularity.

[0073] It should be understood that the implementation of the quantization operation based on the block dimension is similar to that of the quantization operation based on the channel dimension, and thus will not be described again.

[0074] According to some embodiments, the data types of the initial first-order momentum and the initial update rate are float types, and the data types of the quantized first-order momentum and the quantized update rate are integer types.

[0075] For example, the initial first-order momentum and the initial update rate of the optimizer are usually of the float32 (32-bit floating-point number) type, which can be quantized to the int4 (4-bit integer) or int8 (8-bit integer) type for storage after compression, so that the required storage space can be reduced to 25% of that before quantization, effectively reducing the memory overhead.

[0076] According to some embodiments, the integer type includes the int4 type and the int8 type, and the method 200 further includes:

[0077] In step 260, in response to the data types of the quantized first-order momentum and the quantized update rate being the int4 type, the quantized first-order momentum and the quantized update rate are merged by binary shifting to be stored as data of the int8 type.

[0078] At present, most deep learning databases do not support the storage of int4 type integer data, and by merging two int4 type data into one int8 type data, the quantized data can be more conveniently stored to expand the application scenarios.

[0079] In step 240, for example, the quantized first-order momentum and the quantized update rate of the int4 type or the int8 type are restored to the target first-order momentum and the target update rate of the float32 type by the dequantization operation, so as to be used to update the training parameters of the neural network model.

[0080] For example, for the quantized first-order momentum and the quantized update rate of the int4 type, if they are merged into one int8 type data during the quantization process, they are restored to two int4 type data by using bit operations during the dequantization process, and then dequantized to the float32 type according to respective tensors.

[0081] In step 250, after obtaining the target first-order momentum and the target update rate, the target second-order momentum can be determined based on the following formula:

[0082]

[0083] It can be seen that the formula is derived from the above formula for determining the initial update rate, and thus will not be described again.

[0084] According to another aspect of the present disclosure, a data quantization apparatus of an optimizer is provided. As shown in Figure 4 The data quantization apparatus 400 of the optimizer includes an obtaining module 410 configured to obtain gradient data of an optimizer, wherein the optimizer is used to optimize a training process of a neural network model based on a gradient algorithm, and the gradient data includes an initial first-order momentum and an initial second-order momentum; a first determining module 420 configured to determine an initial update rate according to the initial first-order momentum and the initial second-order momentum, wherein the initial update rate indicates a proportional relationship between the initial first-order momentum and the initial second-order momentum; a quantization module 430 configured to perform a quantization operation on the initial first-order momentum and the initial update rate to obtain a quantized first-order momentum and a quantized update rate for storage; a dequantization module 440 configured to perform a dequantization operation on the quantized first-order momentum and the quantized update rate to obtain a target first-order momentum and a target update rate; and a second determining module 450 configured to determine a target second-order momentum according to the target first-order momentum and the target update rate, so as to update a training parameter of the neural network model based on the target first-order momentum and the target second-order momentum.

[0085] According to some embodiments, the initial update rate indicates a negative correlation between the initial first-order momentum and the initial second-order momentum.

[0086] According to some embodiments, the quantization module 430 includes a first quantization submodule 431 configured to perform a symmetric uniform quantization on the initial first-order momentum to obtain the quantized first-order momentum for storage, and a second quantization submodule 432 configured to perform an asymmetric uniform quantization on the initial update rate to obtain the quantized update rate for storage.

[0087] According to some embodiments, the data types of the initial first-order momentum and the initial update rate are floating-point types, and the data types of the quantized first-order momentum and the quantized update rate are integer types.

[0088] According to some embodiments, the integer types include int4 types and int8 types, and the apparatus 400 further includes a merging module 460 configured to, in response to the data types of the quantized first-order momentum and the quantized update rate being int4 types, merge the quantized first-order momentum and the quantized update rate into int8 type data through binary shifting for storage.

[0089] According to some embodiments, the quantization operation is a quantization operation based on a channel dimension.

[0090] According to some embodiments, the quantization operation is a quantization operation based on a block dimension.

[0091] According to another aspect of this disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the aforementioned method.

[0092] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to cause the computer to perform the aforementioned method.

[0093] According to another aspect of this disclosure, a computer program product is also provided, including a computer program, wherein the computer program implements the aforementioned method when executed by a processor.

[0094] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0095] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, output unit 507, storage unit 508, and communication unit 509. Input unit 506 can be any type of device capable of inputting information to electronic device 500. Input unit 506 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 507 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 508 may include, but is not limited to, disk and optical disk. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth. TM Devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices and / or the like.

[0096] The computing unit 501 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs various methods and processes described above, such as the GPU-based matrix computation method. For example, in some embodiments, the GPU-based matrix computation method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded onto the RAM 503 and executed by the computing unit 501, one or more steps of the GPU-based matrix computation method described above can be performed. Alternatively, in other embodiments, the computing unit 501 can be configured to perform the GPU-based matrix computation method by any other appropriate means, such as by means of firmware.

[0097] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0098] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0099] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0100] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0101] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0102] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0103] It should be understood that the various forms of flow illustrated above can be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technology disclosed in the present disclosure can be achieved, which is not limited herein.

[0104] While embodiments or examples of the present disclosure have been described with reference to the drawings, it should be understood that the above-described methods, systems, and devices are merely exemplary embodiments or examples, and the scope of the present disclosure is not limited by these embodiments or examples, but is only limited by the claims and their equivalents. Various elements in the embodiments or examples can be omitted or replaced by equivalent elements thereof. In addition, each step can be performed in an order different from that described in the present disclosure. Further, various elements in the embodiments or examples can be combined in various ways. It is important that many of the elements described herein can be replaced by equivalent elements that appear after the present disclosure as technology evolves.

Claims

1. A data quantization method for an optimizer, comprising: Acquire gradient data of the optimizer, wherein the optimizer is used to optimize the training process of a gradient-based neural network model, and the gradient data includes initial first-order momentum and initial second-order momentum; An initial renewal rate is determined based on the initial first-order momentum and the initial second-order momentum, wherein the initial renewal rate indicates the proportional relationship between the initial first-order momentum and the initial second-order momentum; The initial first-order momentum and the initial update rate are quantized to obtain quantized first-order momentum and quantized update rate for storage. The data types of the initial first-order momentum and the initial update rate are floating-point numbers, and the data types of the quantized first-order momentum and the quantized update rate are integers, including int4 and int8 types. Since the data types of the quantized first-order momentum and the quantized update rate are int4, the quantized first-order momentum and the quantized update rate are merged into int8 data for storage by binary shifting; Perform dequantization on the quantized first-order momentum and the quantized update rate to obtain the target first-order momentum and the target update rate; and The target second-order momentum is determined based on the target first-order momentum and the target update rate, and the training parameters of the neural network model are updated based on the target first-order momentum and the target second-order momentum.

2. The method according to claim 1, wherein, The initial update rate indicates a negative correlation between the initial first-order momentum and the initial second-order momentum.

3. The method according to claim 1 or 2, wherein, The step of quantizing the initial first-order momentum and the initial update rate to obtain quantized first-order momentum and quantized update rate for storage includes: The initial first-order momentum is symmetrically and uniformly quantized to obtain the quantized first-order momentum for storage; and The initial update rate is subjected to asymmetric uniform quantization to obtain the quantized update rate for storage.

4. The method according to claim 1 or 2, wherein, The quantization operation is a channel-dimensional quantization operation.

5. The method according to claim 1 or 2, wherein, The quantization operation is a block-level quantization operation.

6. A data quantization device for an optimizer, comprising: The acquisition module is configured to acquire gradient data of an optimizer, wherein the optimizer is used to optimize the training process of a gradient-based neural network model, and the gradient data includes initial first-order momentum and initial second-order momentum. The first determining module is configured to determine an initial update rate based on the initial first-order momentum and the initial second-order momentum, wherein the initial update rate indicates the proportional relationship between the initial first-order momentum and the initial second-order momentum. The quantization module is configured to perform quantization operations on the initial first-order momentum and the initial update rate to obtain quantized first-order momentum and quantized update rate for storage. The data types of the initial first-order momentum and the initial update rate are floating-point numbers, and the data types of the quantized first-order momentum and the quantized update rate are integers, including int4 and int8 types. The merging module is configured to, in response to the data types of the quantized first-order momentum and the quantization update rate being int4, merge the quantized first-order momentum and the quantization update rate into int8 type data for storage via binary shift; The dequantization module is configured to perform a dequantization operation on the quantized first-order momentum and the quantization update rate to obtain the target first-order momentum and the target update rate; and The second determining module is configured to determine the target second momentum based on the target first momentum and the target update rate, so as to update the training parameters of the neural network model based on the target first momentum and the target second momentum.

7. The apparatus according to claim 6, wherein, The initial update rate indicates a negative correlation between the initial first-order momentum and the initial second-order momentum.

8. The apparatus according to claim 6 or 7, wherein, The quantization module includes: A first quantization submodule is configured to perform symmetrical uniform quantization on the initial first-order momentum to obtain the quantized first-order momentum for storage; and The second quantization submodule is configured to perform asymmetric uniform quantization on the initial update rate to obtain the quantized update rate for storage.

9. The apparatus according to claim 6 or 7, wherein, The quantization operation is a channel-dimensional quantization operation.

10. The apparatus according to claim 6 or 7, wherein, The quantization operation is a block-level quantization operation.

11. An electronic device, comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.

13. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method of any one of claims 1-5.

Images