A quantification method, recommendation method, and apparatus

Abstract

The application provides a quantization method, a recommendation method and a device, which are used for quantizing each feature in a full-precision embedding representation based on an adaptive step length, so as to improve quantization precision. The method comprises the following steps: firstly, obtaining a full-precision embedding representation, wherein the embedding representation comprises multiple features; determining an adaptive step length corresponding to each feature in the multiple features, wherein the step lengths corresponding to the multiple features may be the same or different; then quantizing the multiple features according to the adaptive step lengths corresponding to the multiple features respectively, so as to obtain a low-precision embedding representation, wherein the precision of the features in the low-precision embedding representation is lower than the precision of the features in the full-precision embedding representation, thereby reducing the storage space required for saving or transmitting the embedding representation.

Description

Technical Field

[0001] This application relates to the field of computers, and more particularly to a quantification method, a recommendation method, and an apparatus. Background Technology

[0002] Machine learning systems, including personalized recommendation systems, train the parameters of machine learning models based on input data and labels using optimization methods such as gradient descent. Once the model parameters converge, the model can be used to predict unknown data.

[0003] For example, taking a click-through rate (CTR) prediction model in a recommender system as an example, this model typically includes an embedding layer and a multilayer perceptron (MLP) layer. The embedding layer is usually used to map high-dimensional sparse data to low-dimensional dense vectors, while the MLP is typically used to fit combinations of features, sequence information, or CTR, etc. However, for some large-scale data scenarios, the input data volume of the recommender model is usually very large, so the size of the embedding layer is very large, resulting in a very large storage space required for storage and training. Summary of the Invention

[0004] This application provides a quantization method, a recommendation method, and an apparatus for quantizing each feature in a full-precision embedded representation based on an adaptive step size, thereby improving quantization accuracy.

[0005] In view of this, in a first aspect, this application provides a quantization method, comprising: first, obtaining a full-precision embedding representation, the embedding representation including multiple features; determining an adaptive step size corresponding to each of the multiple features, the step sizes corresponding to the multiple features may be the same or different; subsequently quantizing the multiple features according to the adaptive step size corresponding to each feature to obtain a low-precision embedding representation, wherein the precision of the features in the low-precision embedding representation is lower than the precision of the features in the full-precision embedding representation, therefore the storage resources or transmission resources required to save or transmit the low-precision embedding representation are lower than the storage resources required to save or transmit the full-precision embedding representation, thereby reducing the storage space required to save or transmit the embedding representation.

[0006] In this embodiment, during the quantization of the full-precision embedded representation, an adaptive step size corresponding to each feature can be calculated, and quantization is performed based on the adaptive step size corresponding to each feature, thereby improving quantization accuracy and avoiding accuracy loss caused by a fixed step size. For example, when the update of a certain feature is relatively small, using a fixed step size may cause the quantization accuracy to decrease due to the small update portion. However, with the quantization method provided in this application, each feature has a corresponding adaptive step size, which matches the length of each feature or the amount of updated data, thereby avoiding data loss and improving quantization accuracy during quantization.

[0007] In one possible implementation, the low-precision embedding representation vocabulary is applied to the neural network. The aforementioned acquisition of the full-precision embedding representation vocabulary may include: acquiring the representation corresponding to the input data of the current iteration from the low-precision embedding representation vocabulary to obtain the low-precision embedding representation of the current iteration; and dequantizing the low-precision embedding representation of the current iteration to obtain the full-precision embedding representation of the current iteration.

[0008] Therefore, the quantization method provided in this application can be applied to quantization in the training process of neural networks. In each iteration, a low-precision embedding representation is transmitted, and a full-precision embedding representation can be obtained by dequantization with the corresponding adaptive step size. This enables the full-precision restoration of the low-precision embedding representation, resulting in a lossless full-precision embedding representation, which can reduce the storage space occupied by the embedding representation during the training process of neural networks.

[0009] In one possible implementation, the aforementioned determination of the adaptive step size corresponding to each of the multiple features may include: using the full-precision embedding representation of the current iteration as input to the neural network to obtain the full-precision gradient corresponding to the prediction result of the current iteration; obtaining an updated full-precision embedding representation based on the full-precision gradient to obtain the updated full-precision embedding representation; and obtaining the adaptive step size corresponding to each feature in the updated full-precision embedding representation based on the full-precision gradient.

[0010] In the embodiments of this application, during the training process of the neural network, the adaptive step size corresponding to each feature can be determined according to the full-precision gradient, so that the step size can be updated adaptively to obtain an adaptive step size that matches each feature. This can avoid the reduction in quantization accuracy due to insufficient update in the embedded representation and can improve quantization accuracy.

[0011] In one possible implementation, the aforementioned quantization of multiple features according to the adaptive step size corresponding to each feature includes: quantizing multiple features in the full-precision low-dimensional representation of the current iteration according to the adaptive step size corresponding to each feature, to obtain a low-precision embedded representation.

[0012] Therefore, in the embodiments of this application, an adaptive step size based on full-precision gradient calculation can be used for quantization, thereby quantizing the embedded representation synchronously during the training process.

[0013] In one possible implementation, the method provided in this application may further include: updating the low-precision embedding representation vocabulary based on the low-precision embedding representation to obtain the updated low-precision embedding representation vocabulary.

[0014] After quantizing to obtain a new low-precision embedding representation, the new low-precision embedding representation can be written back into the low-precision embedding representation vocabulary for subsequent low-precision storage or transmission.

[0015] In one possible implementation, the aforementioned determination of the adaptive step size corresponding to each of the multiple features may include: calculating the adaptive step size corresponding to each feature using a heuristic algorithm.

[0016] In the embodiments of this application, the adaptive step size can be calculated using a heuristic algorithm, which is applicable to scenarios where a low-precision embedded representation vocabulary is stored.

[0017] In one possible implementation, the aforementioned calculation of the adaptive step size for each feature using a heuristic algorithm may include: calculating the adaptive step size for each feature based on the absolute value of its weight. Therefore, the adaptive step size can be calculated based on the weight value of each feature itself, without relying on external data.

[0018] In one possible implementation, the aforementioned quantization of multiple features according to the adaptive step size corresponding to each feature to obtain a low-precision embedding representation vocabulary may further include: obtaining discrete features of each feature according to the adaptive step size corresponding to each feature; and truncating the discrete features of each feature using a random truncation algorithm to obtain a low-precision embedding representation.

[0019] In this embodiment, a random truncation algorithm can be used to truncate each feature, thereby adaptively retaining effective features and improving quantization accuracy.

[0020] In one possible implementation, a low-precision embedded representation vocabulary is applied to a language model or a recommendation model. The language model is used to acquire semantic information from the corpus, and the recommendation model is used to generate recommendation information based on user information. Therefore, the method provided in this application can be applied to natural language processing or recommendation scenarios.

[0021] Secondly, this application provides a recommendation method, comprising: acquiring input data, the input data including data generated by a user for at least one behavior of a terminal; acquiring a low-precision embedding representation corresponding to the input data from a low-precision embedding representation vocabulary, the low-precision embedding representation including multiple features; dequantizing multiple features according to an adaptive step size corresponding to each feature to obtain a full-precision embedding representation, the adaptive step size being the adaptive step size obtained when quantizing the full-precision embedding representation; and outputting recommendation information based on the full-precision embedding representation as input to a neural network, the recommendation information being used to make recommendations for at least one user behavior.

[0022] In this embodiment of the application, during the inference process of the neural network, an adaptive step size can be used to dequantize the low-precision embedding representation to obtain a full-precision embedding representation. Therefore, the low-precision representation can be saved or transmitted during inference, and then losslessly restored using the adaptive step size to obtain the full-precision embedding representation. This reduces the storage space occupied by the embedding representation vocabulary and allows for lossless restoration during use.

[0023] In one possible implementation, the neural network includes a language model or a recommendation model, wherein the language model is used to acquire semantic information from the corpus, and the recommendation model is used to generate recommendation information based on user information.

[0024] Thirdly, this application provides a quantization device, comprising: The acquisition module is used to acquire full-precision embedding representations, which include multiple features. The determination module is used to determine the adaptive step size for each feature among multiple features; The quantization module is used to quantize multiple features according to the adaptive step size corresponding to each feature, so as to obtain a low-precision embedding representation. The precision of the features in the low-precision embedding representation is lower than that of the features in the full-precision embedding representation.

[0025] In one possible implementation, a low-precision embedded representation vocabulary is applied to the neural network. The acquisition module is specifically used to acquire the representation corresponding to the input data of the current iteration from the low-precision embedding representation vocabulary, and obtain the low-precision embedding representation of the current iteration; and to dequantize the low-precision embedding representation of the current iteration to obtain the full-precision embedding representation of the current iteration.

[0026] In one possible implementation, the determining module is specifically used for: taking the full-precision embedding representation of the current iteration as the input of the neural network to obtain the full-precision gradient corresponding to the prediction result of the current iteration; obtaining the updated full-precision embedding representation based on the full-precision gradient to obtain the updated full-precision embedding representation; and obtaining the adaptive step size corresponding to each feature in the updated full-precision embedding representation based on the full-precision gradient.

[0027] In one possible implementation, the quantization module is specifically used to quantize multiple features in the full-precision low-dimensional representation of the current iteration according to the adaptive step size corresponding to each feature, so as to obtain a low-precision embedded representation.

[0028] In one possible implementation, the acquisition module is further configured to update the low-precision embedding representation vocabulary based on the low-precision embedding representation, thereby obtaining the updated low-precision embedding representation vocabulary.

[0029] In one possible implementation, a determination module is specifically used to calculate the adaptive step size corresponding to each feature using a heuristic algorithm.

[0030] In one possible implementation, the determining module is specifically used to calculate the adaptive step size for each feature based on the absolute value of the weights in each feature.

[0031] In one possible implementation, the quantization module is specifically used to: obtain the discrete features of each feature according to the adaptive step size corresponding to each feature; and truncate the discrete features of each feature using a random truncation algorithm to obtain a low-precision embedding representation.

[0032] In one possible implementation, a low-precision embedded representation vocabulary is applied to a language model or a recommendation model. The language model is used to obtain semantic information from the corpus, and the recommendation model is used to generate recommendation information based on the user's information.

[0033] Fourthly, this application provides a recommended device, comprising: The input module is used to acquire input data, which includes data generated by the user's actions on the terminal at least once; The acquisition module is used to acquire low-precision embedding representations corresponding to the input data from the low-precision embedding representation vocabulary. The low-precision embedding representations include a variety of features. The dequantization module is used to dequantize multiple features according to the adaptive step size corresponding to each feature in multiple features, so as to obtain a full-precision embedded representation. The recommendation module is used to output recommendation information based on the full-precision embedded representation as input to the neural network. The recommendation information is used to make recommendations based on at least one user behavior.

[0034] In one possible implementation, the neural network includes a language model or a recommendation model, wherein the language model is used to acquire semantic information from the corpus, and the recommendation model is used to generate recommendation information based on user information.

[0035] Fifthly, this application provides a quantization device, the recommended device comprising: a processor, a memory, an input / output device, and a bus; the memory stores computer instructions; when the processor executes the computer instructions in the memory, the memory stores computer instructions; when the processor executes the computer instructions in the memory, it is used to implement any of the implementation methods of the first aspect.

[0036] In a sixth aspect, this application provides a recommended apparatus comprising: a processor, a memory, an input / output device, and a bus; the memory stores computer instructions; when the processor executes the computer instructions in the memory, the memory stores computer instructions; when the processor executes the computer instructions in the memory, it is used to implement any of the implementation methods of the second aspect.

[0037] In a seventh aspect, embodiments of this application provide a chip system including a processor and input / output ports. The processor is used to implement the processing functions involved in the methods described in the first or second aspect above, and the input / output ports are used to implement the transmission and reception functions involved in the methods described in the first or second aspect above.

[0038] In one possible design, the chip system also includes a memory for storing program instructions and data for implementing the functions involved in the methods described in the first or second aspect above.

[0039] This chip system can consist of chips or include chips and other discrete components.

[0040] Eighthly, embodiments of this application provide a computer-readable storage medium. The computer-readable storage medium stores computer instructions; when the computer instructions are executed on a computer, the computer causes the computer to perform the method as described in any possible implementation of the first or second aspect.

[0041] Ninthly, embodiments of this application provide a computer program product. The computer program product includes a computer program or instructions that, when executed on a computer, cause the computer to perform the method as described in any possible implementation of the first or second aspect. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of an artificial intelligence framework used in this application; Figure 2 A schematic diagram of a system architecture is provided for this application; Figure 3 Another system architecture diagram provided for this application; Figure 4This application provides an illustration of an application scenario. Figure 5A This application provides another application scenario illustration; Figure 5B This application provides another application scenario illustration; Figure 6 The flowchart of a quantification method provided in this application is intended to illustrate the process. Figure 7 The procedural intent of another quantification method provided in this application; Figure 8 The procedural intent of another quantification method provided in this application; Figure 9 This application provides another application scenario illustration; Figure 10 This application provides another application scenario illustration; Figure 11 This application provides another application scenario illustration; Figure 12 A flowchart illustrating a recommended method provided in this application; Figure 13 A schematic diagram of a quantization device provided in this application; Figure 14 A schematic diagram of the structure of a recommended device provided in this application; Figure 15 A schematic diagram of a quantization device provided in this application; Figure 16 A schematic diagram of the structure of a recommended device provided in this application; Figure 17 This is a schematic diagram of the structure of a chip provided in this application. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0044] The recommended methods provided in this application can be applied to artificial intelligence (AI) scenarios. AI is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to have the functions of perception, reasoning, and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0045] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1 The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0046] (1) Infrastructure

[0047] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0048] (2) Data

[0049] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0050] (3) Data processing

[0051] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0052] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0053] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0054] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0055] (4) General ability

[0056] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0057] (5) Smart products and industry applications

[0058] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0059] This application relates to the application of neural networks. In order to better understand the solutions of this application, the relevant terms and concepts of neural networks that may be involved in this application will be introduced below.

[0060] (1) Convolutional Neural Network

[0061] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernel. Shared weights can be understood as the way features are extracted being independent of their location. The convolutional kernel can be formalized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is reducing the connections between layers in the CNN, while also reducing the risk of overfitting.

[0062] (2) Graph Convolutional Network (GCN)

[0063] Graph neural networks are deep learning models for modeling and processing non-Euclidean spatial data (such as graph data). Their principle is based on pairwise message passing, allowing graph nodes to iteratively update their representations by exchanging information with their neighbors.

[0064] Similar to CNNs, GCNs differ in that CNNs typically take two-dimensional data as input, while GCNs typically take graph data as input. GCNs ingeniously design a method to extract features from graph data, which can then be used to perform node classification, graph classification, link prediction, and obtain graph embeddings.

[0065] (3) Loss function

[0066] In training deep neural networks, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value and update the weight vector of each layer based on the difference. (Of course, there's usually a pre-configuration process before the first update, where parameters are pre-configured for each layer.) For example, if the network's prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network can predict the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, so training the deep neural network becomes a process of minimizing this loss. Common loss functions include mean squared error, cross-entropy, logarithmic, and exponential loss functions. For example, mean squared error can be used as the loss function, defined as... The specific loss function can be selected based on the actual application scenario.

[0067] (4) Backpropagation algorithm

[0068] An algorithm for calculating the gradient of model parameters based on a loss function and updating the model parameters. Neural networks can use backpropagation (BP) to correct the initial parameter values ​​during training, thus minimizing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters of the initial neural network model, thereby converging the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0069] In the embodiments of this application, the BP algorithm can be used to train the model in either the training or inference phase to obtain the trained model.

[0070] (5) Gradient: The vector of derivatives of the loss function with respect to the parameters.

[0071] (6) Stochastic gradient: In machine learning, the number of samples is very large, so the loss function is calculated from the randomly sampled data each time, and the corresponding gradient is called stochastic gradient.

[0072] (7) Embedding: refers to the feature representation or word embedding representation of a sample.

[0073] (8) Recommendation system: The recommendation system uses machine learning algorithms to analyze and learn based on the user's historical click behavior data, and then predicts the user's new requests and returns a personalized list of recommended items.

[0074] (9) Model quantization: This is a model compression method that converts high bits to low bits. For example, the model compression technique that converts conventional 32-bit floating-point operations into low-bit integer operations can be called model quantization. When the low bits are quantized to 8 bits, it can be called int8 quantization. That is, originally, it would require float32 to represent a weight, but after quantization, it only needs to be represented by int8. Theoretically, this can achieve a 4x network speedup. At the same time, 8 bits can reduce the storage space by 4 times compared to 32 bits, thus reducing storage space and computation time, thereby achieving the purpose of model compression and acceleration.

[0075] (10) Automated Machine Learning (AutoML): This refers to designing a series of advanced control systems to operate machine learning models, enabling the models to automatically learn appropriate parameters and configurations without human intervention. In deep neural network-based learning models, automated computational learning mainly includes network architecture search and global parameter setting. Among them, network architecture search is used to enable the computer to generate the most suitable neural network architecture for the problem based on the data, and it has the characteristics of high training complexity and significant performance improvement.

[0076] (11) Corpus: also known as free text, which can be words, phrases, sentences, fragments, articles and any combination thereof. For example, “The weather is nice today” is a corpus.

[0077] (12) Neural machine translation: Neural machine translation is a typical task in natural language processing. This task is a technique that outputs the corresponding target language sentence given a sentence in a source language. In commonly used neural machine translation models, the words in both the source and target language sentences are encoded into vector representations. The relationships between words and sentences are calculated in the vector space to perform the translation task.

[0078] (13) Pre-trained language model (PLM): A natural language sequence encoder that encodes each word in a natural language sequence into a vector representation for prediction tasks. PLM training consists of two phases: pre-training and fine-tuning. In the pre-training phase, the model is trained on large-scale unsupervised text to learn word representations. In the fine-tuning phase, the model is initialized using the parameters learned in the pre-training phase and then trained on downstream tasks such as text classification or sequence labeling with fewer steps, successfully transferring the semantic information obtained from pre-training to downstream tasks.

[0079] (14) Click-through rate (CTR): refers to the probability that a user clicks on a displayed item in a specific environment.

[0080] (15) Conversion rate (CVR): refers to the probability that a user will convert a clicked item in a specific environment. For example, if a user clicks on the icon of an APP, conversion means downloading, installing, registering, etc.

[0081] (16) Epoch

[0082] It defines the number of times the learning algorithm works on the entire training set. One epoch can be considered as the number of times the neural network is trained using the entire training set.

[0083] (17) batch; Closely related to the definition of an epoch, an epoch involves training a neural network using the entire dataset, while a batch represents one batch of data within an epoch, specifically represented by the batch_size. batches = epoch, which can be understood as each epoch being divided into one or more batches. Each batch can use a portion of the data in the training set to train the neural network.

[0084] The recommended method provided in this application can be executed on a server or on a terminal device. The terminal device can be a mobile phone with image processing capabilities, a tablet personal computer (TPC), a media player, a smart TV, a laptop computer (LC), a personal digital assistant (PDA), a personal computer (PC), a camera, a camcorder, a smartwatch, a wearable device (WD), or an autonomous vehicle, etc., and this application does not limit the specific device to these.

[0085] The system architecture provided in the embodiments of this application is described below.

[0086] See Figure 2 This application provides a system architecture 200. As shown in the system architecture 200, the data acquisition device 260 can be used to collect training data. After the data acquisition device 260 collects the training data, it stores the training data in the database 230. The training device 220 trains the target model / rule 201 based on the training data maintained in the database 230.

[0087] The following describes how the training device 220 obtains the target model / rule 201 based on the training data. For example, the training device 220 processes multiple frames of sample images, outputs corresponding predicted labels, and calculates the loss between the predicted labels and the original labels of the samples. Based on this loss, the classification network is updated until the predicted labels are close to the original labels of the samples or the difference between the predicted labels and the original labels is less than a threshold, thus completing the training of the target model / rule 201. A detailed description of the training method follows.

[0088] The target model / rule 201 in this embodiment can specifically be a neural network. It should be noted that in practical applications, the training data maintained in the database 230 may not all originate from the data acquisition device 260; it may also be received from other devices. Furthermore, the training device 220 may not necessarily train the target model / rule 201 entirely based on the training data maintained in the database 230; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0089] The target model / rule 201 trained using training device 220 can be applied to different systems or devices, such as... Figure 2The execution device 210 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, in-vehicle terminal, television, etc., or it can be a server or cloud service. Figure 2 In the process, the execution device 210 is equipped with a transceiver 212, which may include an input / output (I / O) interface or other wireless or wired communication interfaces, for data interaction with external devices. Taking the I / O interface as an example, the user can input data to the I / O interface through the client device 240.

[0090] During the preprocessing of input data by the execution device 210, or during the calculation module 111 of the execution device 210 performing calculations and other related processes, the execution device 210 can call data, code, etc. in the data storage system 250 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 250.

[0091] Finally, transceiver 212 returns the processing result to client device 240, thereby providing it to the user.

[0092] It is worth noting that the training device 220 can generate corresponding target models / rules 201 based on different training data for different objectives or tasks. The corresponding target models / rules 201 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0093] In the appendix Figure 2 In the scenario shown, the user can manually provide input data, which can be done through the interface provided by transceiver 212. Alternatively, client device 240 can automatically send input data to transceiver 212. If user authorization is required for client device 240 to automatically send input data, the user can set the corresponding permissions in client device 240. The user can view the output results of execution device 210 on client device 240, which can be presented in various ways such as display, sound, or animation. Client device 240 can also act as a data acquisition terminal, collecting the input data and output results of transceiver 212 as new sample data and storing them in database 230. Alternatively, data can be collected directly from transceiver 212 without going through client device 240; the transceiver 212 can directly store the input data and output results of transceiver 212 as new sample data in database 230.

[0094] It is worth noting that, attached Figure 2This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 2 In this context, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 may also be placed within the execution device 210.

[0095] like Figure 2 As shown, the target model / rule 201 is obtained by training the training device 220. The target model / rule 201 can be the recommended model in this application embodiment.

[0096] For example, the system architecture for applying the neural network training method provided in this application can be as follows: Figure 3 As shown. In this system architecture 300, the server cluster 310 is implemented by one or more servers, optionally in conjunction with other computing devices, such as data storage, routers, load balancers, etc. The server cluster 310 can use data in the data storage system 250, or call the program code in the data storage system 250 to implement the steps of the neural network training method provided in this application.

[0097] Users can interact with the server cluster 310 using their respective user devices (such as local devices 301 and 302). Each local device can represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, smart camera, smart car or other type of cellular phone, media consumption device, wearable device, set-top box, game console, etc.

[0098] Each user's local device can interact with the server cluster 310 through a communication network using any communication mechanism / standard. The communication network can be a wide area network (WAN), a local area network (LAN), a point-to-point connection, or any combination thereof. Specifically, the communication network can include a wireless network, a wired network, or a combination of both. The wireless network includes, but is not limited to, any one or more combinations of: 5th-Generation (5G) systems, Long Term Evolution (LTE) systems, Global System for Mobile Communication (GSM) or Code Division Multiple Access (CDMA) networks, Wideband Code Division Multiple Access (WCDMA) networks, Wireless Fidelity (WiFi), Bluetooth, Zigbee, Radio Frequency Identification (RFID), Long Range (Lora) wireless communication, and Near Field Communication (NFC). The wired network can include fiber optic communication networks or networks composed of coaxial cables.

[0099] In another implementation, one or more aspects of the execution device 210 may be implemented by each local device. For example, local device 301 may provide local data or feedback calculation results to the execution device 210.

[0100] It should be noted that all the functions of execution device 210 can also be implemented by a local device. For example, local device 301 implements the functions of execution device 210 and provides services to its own users, or provides services to users of local device 302.

[0101] Typically, machine learning systems can include personalized recommendation systems. Based on input data and labels, the parameters of the machine learning model are trained using optimization methods such as gradient descent. Once the model parameters converge, the model can be used to predict unknown data. Taking click-through rate prediction in a personalized recommendation system as an example, its input data includes user characteristics, item characteristics, and contextual features. How to predict personalized recommendation lists based on user preferences has a significant impact on improving the user experience and platform revenue of the recommendation system.

[0102] For example, taking a click-through rate prediction model in a recommendation system as an example, such as Figure 4 As shown, it typically includes an embedding and an MLP layer, i.e. Figure 4 The diagram shows a feature interaction layer, a deep neural network layer, and a prediction layer. Embedding is used to map high-dimensional sparse data to low-dimensional dense vectors. The MLP layer is generally used to fit the combination relationships and sequence information between features to approximate the true click-through rate distribution. Mainstream models all represent features based on embedding parameters and learn features based on this representation. Explicit / Implicit The large number of features in recommendation models, coupled with complex relationships, leads to massive embedding sizes, sometimes reaching terabytes (TB) in internet companies. The excessively large embedding table means that the memory of a single GPU or NPU is insufficient to store all parameters, necessitating distributed storage across multiple nodes. However, distributed storage introduces new problems: increased memory overhead; and the need to retrieve embedding parameters over the network during training / inference, leading to increased communication overhead, model computation latency, and ultimately impacting recommendation performance.

[0103] To reduce the memory footprint of the embedding table, it is common practice to quantize the embedding table, thereby compressing it by reducing its precision.

[0104] For example, compression can be achieved by pruning parameters. A parameter threshold is set, and parameters in the embedding table that are below the threshold are pruned. After pruning the embedding parameters, retraining is performed based on the pruned embeddings. However, this only compresses the memory during the inference stage, not the training memory; it requires retraining, increasing the training cost; and the generated embedding table is unstructured data, requiring special storage.

[0105] For example, compression can be achieved using AutoML-based methods, such as reinforcement learning or Differentiable Architecture Learning (DARTS) methods, which allow end-to-end adjustment of the number of features and the size of different features in the embedding table. Once the model converges, it can be retrained. However, this method has a long search time and poor practicality.

[0106] For example, compression based on hashing involves assigning independent embeddings to high-frequency features and mapping low-frequency features using hash functions, thereby compressing the embedding parameters of low-frequency features. However, feature conflicts may occur, leading to a loss of accuracy.

[0107] For example, in some low-precision training methods, all parameters are stored as low-precision parameters during training. Full-precision parameters (fp32) are obtained through dequantization, and then the full-precision gradient is calculated using forward and backward propagation. Finally, the gradient is applied according to the learning rate step size. The fp32 full-precision parameters are updated to obtain the updated parameters. However, when the weights are smaller and their amplitudes are much smaller than the quantization step size, deterministic rounding will erase the parameter updates, causing the network to fail to train and thus affecting the training accuracy.

[0108] Therefore, this application provides a quantization method that preserves more parameter information and improves quantization accuracy by setting an adaptive quantization step size.

[0109] First, to facilitate understanding, the application scenarios of the method provided in this application will be introduced.

[0110] Typically, the quantization method provided in this application can be applied to language models or recommendation models. The language model may include neural machine translation or PLM models, while the recommendation model may include click-through rate prediction models, conversion rate prediction models, etc. For example, an embedding table can be set in the model to extract representations from the input corpus, then the corresponding semantics can be obtained, followed by further translation or semantic recognition. The specific subsequent steps can be performed according to the tasks required by the model.

[0111] For example, taking a recommendation scenario as an example, the recommendation framework applied in this application can be as follows: Figure 5A As shown, the process can be divided into a training part and an online inference part. In the training part, the training set includes input data and corresponding labels. For example, in a user product recommendation scenario, this training set could include products that the user clicked, favorited, or liked, as well as products that were ultimately purchased. The training set is input into the initial model, and the parameters of the machine learning model are trained using optimization methods such as gradient descent to obtain the recommendation model. In the online inference part, the recommendation model can be deployed on a recommendation platform, such as a server or terminal. Taking a server as an example, the server can output a recommendation list for the user. For example, in a product recommendation scenario, information about the recommended products, such as product icons or link titles, can be displayed on the user's homepage, or after the user clicks on a product, the icon or link title of the recommended product can be displayed in the recommendation area.

[0112] In some application scenarios, the recommendation process can be as follows: Figure 5BAs shown, this can include components such as a display list, logs, offline training, and online prediction. Users perform a series of actions on the front-end display list, such as browsing, clicking, commenting, and downloading, generating behavioral data which is stored in the logs. The recommendation system uses this data, including user behavior logs, for offline model training. After training converges, a prediction model is generated. This model is deployed in an online service environment and provides recommendation results based on user requests, product features, and contextual information. Users then provide feedback on these recommendations, forming user data.

[0113] In both offline training and online prediction, a larger embedding table leads to increased training memory usage and computational latency. To reduce the memory footprint of the embedding table during both training and inference phases, this application proposes an end-to-end adaptive low-precision training framework. This framework can compress the memory of the embedding table in recommendation models, including training and inference memory, thereby reducing the storage overhead of saving, using, and training the model.

[0114] The following describes the process of the quantification method provided in this application.

[0115] See Figure 6 The flowchart of a quantification method provided in this application is as follows.

[0116] 601. Obtain full-precision embedding representation.

[0117] This full-precision embedding representation can include multiple features. Each feature can be represented as one or more feature vectors.

[0118] The full-precision embedding representation can include all or part of the features in the embedding table. If a full-precision embedding table is obtained, all or part of the data can be directly read from the full-precision embedding table to obtain the aforementioned full-precision embedding representation. If a low-precision embedding table is obtained, all or part of the features can be read from the low-precision embedding table, and the read features can be dequantized to obtain the full-precision embedding representation.

[0119] Typically, embedding layers in neural networks are used to map high-dimensional sparse data to low-dimensional dense vectors. Specifically, this involves retrieving the low-dimensional representation corresponding to the input data from an embedding table. This can be understood as the embedding table storing low-dimensional representations of various data types. Since the input data is usually high-dimensional and sparse, the embedding table can map it to low-dimensional representations, effectively decomposing the semantics of the multiple dimensions included in the input data.

[0120] Optionally, during the training of the neural network, the representation corresponding to the input data of the current iteration can be obtained from the low-precision embedding representation vocabulary to obtain the low-precision embedding representation of the current iteration; the low-precision embedding representation of the current iteration is dequantized to obtain the full-precision embedding representation of the current iteration.

[0121] Optionally, the neural network may include a language model or a recommendation model. The language model may include models such as neural machine translation or PLM, and the recommendation model may include click-through rate prediction models, conversion rate prediction models, etc. Therefore, the method provided in this application can be applied to language processing or recommendation scenarios.

[0122] 602. Determine the adaptive step size for each feature among multiple features.

[0123] Before quantizing the embedding, an adaptive step size can be determined for each feature.

[0124] Alternatively, a heuristic algorithm can be used to calculate the adaptive step size for each feature, or the adaptive step size can be calculated through learning.

[0125] Specifically, employing heuristic algorithms can include calculating the adaptive step size for each feature based on the absolute value of its weights. For example, the adaptive quantization step size can be calculated based on the maximum absolute value of the weights in each embedding vector. ,in For the embedding parameter vector, To find the maximum value, take the maximum value of the current vector. Divide into equal parts, where m is the number of bits.

[0126] Quantization is achieved by applying adaptive compensation to the training process of neural networks. For example, the adaptive step size in the current iteration can be calculated based on the weights in the neural network after the current iteration and the step size updated in the previous iteration of training the neural network, thereby achieving higher training accuracy.

[0127] Typically, different methods can be used to calculate the adaptive step size in different scenarios. For example, in training neural networks, either heuristic or learning-based methods can be chosen. For instance, if high accuracy is required and training resources are plentiful, a learning-based method can be chosen to calculate the adaptive step size, while if computational efficiency is a priority, a heuristic method can be used for quantization. Another example is when saving an embedding table; a heuristic algorithm can be used to calculate the adaptive step size, thus efficiently obtaining it without relying on the neural network's training parameters.

[0128] Furthermore, after calculating the adaptive step size for each feature, the adaptive step size for each feature can be saved so that when performing dequantization later, the low-precision features can be dequantized without loss based on the adaptive step size to obtain the full-precision features.

[0129] Optionally, during the training of the neural network, the full-precision embedding representation of the current iteration can be used as the input of the neural network to obtain the full-precision gradient corresponding to the prediction result of the current iteration; the updated full-precision embedding representation is obtained based on the full-precision gradient; and the adaptive step size corresponding to each feature in the updated full-precision embedding representation is obtained based on the full-precision gradient. Therefore, during training, the adaptive step size adapted to the updated parameters can be updated in real time according to the updated parameters. Typically, if quantization is performed with a fixed step size, data loss may occur if the parameter update is smaller than the quantization step size. However, in the method provided in this application, when the parameter update is small, the step size can be adaptively calculated based on the updated parameters, thereby retaining the less updated parameters and reducing accuracy loss.

[0130] 603. Quantize multiple features according to the adaptive step size corresponding to each feature to obtain a low-precision embedding representation.

[0131] After determining the adaptive step size corresponding to each feature in the full-precision embedding representation, each feature can be quantized separately based on its adaptive step size to obtain a low-precision embedding representation. Therefore, the storage or transmission resources of the computing device required to store or transmit this low-precision embedding representation are lower than those required to store or transmit the full-precision embedding representation. This computing device may include a device that executes the quantization method or recommended method provided in this application.

[0132] In this embodiment, an adaptive step size is calculated for each feature in the full-precision embedding table, and quantization is performed based on the adaptive step size. Therefore, during quantization, quantization can be performed based on a matching adaptive step size. For some features whose number does not match the number of quantization bits, adaptive step size quantization can be used. Compared to using a fixed step size for quantization, using adaptive step size quantization can reduce precision loss and improve quantization accuracy.

[0133] Furthermore, if the aforementioned steps 601 to 603 are one of the iterative processes for updating the neural network, then after obtaining the low-precision embedding representation through quantization, the low-precision embedding representation vocabulary will be updated based on the low-precision embedding representation to obtain the updated low-precision embedding representation vocabulary, that is, the updated low-precision embedding representation is written back into the low-precision embedding table.

[0134] The method of this application can be applied to various aspects of model saving or model training processes. For example, when saving a model, the quantization method provided in this application can achieve lower precision quantization, or, during model training, the quantization method provided in this application can reduce the amount of data that needs to be transmitted during training and reduce the required cache space.

[0135] For scenarios involving quantization before saving the model, please refer to the aforementioned... Figure 6 The steps in the process are illustrated below using the quantization process during model training as an example.

[0136] Taking the application in a training scenario as an example, during training, all or part of the features in the embeddingtable can be quantized in each training iteration. Taking one training iteration as an example, the quantization method provided in this application can be described as follows: Figure 7 As shown.

[0137] It should be understood that during iterative training, training can typically be performed in one or more epochs, and each epoch can be divided into multiple batches. In this embodiment, one batch is used as an example for illustrative purposes.

[0138] 701. Determine the low-precision batch embedding from the low-precision embedding table.

[0139] In a batch, the input data of the neural network trained in the current batch can be used as the input of the embedding layer. The input data is mapped to a low-precision, low-dimensional embedding representation through a low-precision embedding table, which is called low-precision batch embedding.

[0140] 702. Dequantize the low-precision batch embedding to obtain the full-precision batch embedding.

[0141] After obtaining the low-precision batch embedding, the low-precision batch embedding can be dequantized, that is, the inverse operation of quantization, to obtain the full-precision batch embedding, so that the neural network can obtain the representation corresponding to the input sample based on the full-precision batch embedding.

[0142] 703. Obtain the full-precision gradient corresponding to the current batch prediction result of the neural network through full-precision batch embedding.

[0143] After obtaining the full-precision batch embedding, during the neural network training process of the current batch, the full-precision batch embedding corresponding to the training samples can be used as the input of the neural network to output the prediction result. Subsequently, based on the prediction result and the true labels of the input training samples, the value of the loss function is calculated, and the full-precision gradient of the parameters of the neural network in the current batch is calculated based on the value of the loss function.

[0144] 704. Update the weights of the neural network based on the full-precision gradient to obtain the updated neural network.

[0145] After obtaining the full-precision gradient, the weights of the neural network can be updated based on the full-precision gradient to obtain the updated neural network for the current batch.

[0146] The parameters of a neural network can be updated using the backpropagation algorithm. Typically, forward propagation of the input signal to the output incurs error loss. By backpropagating this error loss information, the parameters in the initial neural network model are updated, thereby converging the error loss.

[0147] 705. Update the full-precision batch embedding based on the full-precision gradient to obtain the new full-precision batch embedding and quantization step size.

[0148] After obtaining the full-precision gradient, the adaptive step size can be updated based on the full-precision gradient, and the full-precision batch embedding can be quantized based on the adaptive step size to obtain a new low-precision batch embedding. The updated low-precision batch embedding is then saved to the low-precision embedding table, thus realizing the low-precision saving and transmission of the embedding table and reducing the storage space required for saving and transmitting the embedding table.

[0149] Specifically, the adaptive step size can be calculated by learning, which can be combined with the weights updated in each iteration. This allows for real-time quantization of the embedding table based on the neural network update process, thereby reducing the storage space occupied during training and saving.

[0150] Of course, the adaptive step size can also be calculated using heuristic algorithms, such as calculating the adaptive step size corresponding to each feature in the full-precision batch embedding based on the absolute value of the weights of the updated full-precision batch embedding, thus efficiently and accurately calculating the adaptive step size.

[0151] 706. Quantize the new batch embedding according to the adaptive quantization step size to obtain a new low-precision batch embedding.

[0152] After obtaining the adaptive quantization step size, the updated full-precision batch embedding can be quantized based on the adaptive quantization step size to obtain a new low-precision batch embedding.

[0153] Optionally, in the specific quantization process, the discrete values, or discrete features, of each feature can be obtained according to the adaptive step size corresponding to each feature. Then, the discrete features of each feature can be truncated by a random truncation algorithm to obtain a low-precision embedding table. When truncating by the random truncation algorithm, the value of each discrete feature determines the truncation value, so that the truncation length matches the feature value update. Even if the parameter update amplitude is small, the updated part can be quantized to achieve quantization accuracy.

[0154] 707. Determine if convergence has occurred. If yes, terminate the iteration. If no, proceed to step 701.

[0155] After each batch of training, it can be determined whether the neural network has converged. If it has, the iteration can be terminated, and the neural network after training the current batch can be output. If not, the neural network has not converged, and iterative training can continue.

[0156] Determining whether the neural network has converged can be done by judging whether the number of iterations has reached a preset number, whether the change in the loss value is less than a preset value, or whether the iteration time has reached a preset time, etc. The specific determination can be made according to the application scenario, and this application does not limit it.

[0157] Therefore, in the embodiments of this application, during the training of the neural network, the adaptive step size can be updated based on the calculated gradient, and quantization can be performed according to the adaptive step size adapted to each feature. This can ensure the quantization accuracy of each feature as much as possible, achieve lower precision quantization, and reduce the loss of information during quantization.

[0158] The foregoing has described the process of applying the quantization method provided in this application to the neural network training process. For ease of understanding, the quantization method provided in this application will be described below in conjunction with a more specific recommendation scenario.

[0159] See Figure 8 The flowchart of another quantification method provided in this application is shown below.

[0160] In the forward phase, the recommendation model is input with a batch of high-dimensional sparse data. The feature IDs in the batch data are read, and the corresponding batch embeddings are read from the low-precision embedding table. Then, through dequantization, a low-precision batch embedding with full precision representation that can be used for subsequent neural network calculations is obtained. In the backward phase, the gradient of the current batch embedding is obtained from the upper layer network, and the batch embedding is updated based on the gradient. Since the embedding table stores low-precision parameters, it is necessary to obtain the low-precision batch embedding through quantization, and then finally write it into the low-precision embedding table. The specific steps may include the following.

[0161] First, read the user's log data 801, and then use this log data as the training set for the recommendation model.

[0162] The user's log data can include information generated when the user uses the client. Different clients typically generate different information. For example, when a user uses a music app, information about the music played, clicked, favorited, or searched can be saved in the user's log; when a user uses a shopping app, information about the items browsed, favorited, or purchased can be saved in the user's log; and when a user uses an app store, information about the apps clicked, downloaded, installed, or favorited can be saved in the user's log, and so on.

[0163] Subsequently, the high-dimensional sparse batch data 802 of the current batch is read from the user log data.

[0164] In each batch, a portion of the user's log data can be extracted as high-dimensional sparse data for the current batch, which can then be used as training data for the next iteration.

[0165] Subsequently, the corresponding low-precision batch embedding is read from the low-precision embedding table803.

[0166] Typically, user log data is high-dimensional and sparse. Therefore, an embedding table can be used to map this high-dimensional sparse data into low-dimensional features, allowing the model to identify and process each feature. Specifically, after reading the current batch of high-dimensional sparse batch data from the log data, a low-precision embedding table can be used to map this high-dimensional sparse batch data into a low-dimensional representation, such as represented by low-precision batch embedding.

[0167] Subsequently, inverse quantization is performed to obtain full-precision batch embedding804.

[0168] After obtaining the low-precision batch embedding, the low-precision batch embedding is dequantized using an inverse quantization algorithm to obtain the full-precision batch embedding.

[0169] The full-precision parameters of fp32 can be obtained using the de-quantization function. ,in This is the adaptive step size corresponding to this batch embedding.

[0170] Then, the full-precision batch embedding can be used as input to the recommendation model 805, and the prediction result 806 can be output.

[0171] Then, the full-precision gradient of the current batch is calculated based on the prediction result 806, and the batch embedding and quantization step size are updated based on the full-precision gradient of the current batch 807.

[0172] After obtaining the prediction results, the loss value between the prediction results and the true labels of the input samples can be calculated. Based on this loss value, backpropagation is performed to calculate the full-precision gradient of each parameter in the current batch recommendation model. According to the learning rate step size Update the fp32 full-precision parameters in batch embedding get and quantization step size .

[0173] Specifically, either heuristic computation of adaptive quantization step size or learning-based computation of adaptive quantization step size can be used.

[0174] The heuristic calculation of the adaptive step size can be represented as follows: Calculate the adaptive quantization step size based on the maximum absolute value of the weights in each embedding vector. ,in For the embedding parameter vector, To find the maximum value, the physical meaning of this method is to take the maximum value of the current vector as the maximum value. Divide into equal parts, where m is the number of bits.

[0175] For example, the steps of learning-based computation of adaptive quantization step size may include: after weight update, training the updated weights and the unupdated quantization step size in a quantization-aware manner to update the quantization step size end-to-end. This can be represented as: First, update the weight parameters:

[0176]

[0177]

[0178] Then the adaptive step size is updated, as shown below:

[0179]

[0180] Then the updated embedding parameters are output. Updated adaptive step size And the parameters of the updated recommendation model .

[0181] Then, quantization is performed to obtain low-precision batch embedding 808, which is then written back to the embedding table.

[0182] After obtaining the adaptive step size for each feature, the updated parameters can be adjusted. Quantification can be performed, and the quantification process can be represented as:

[0183] in, It's the number of bits. The rounding function is a stage function, which can typically include various methods, such as deterministic rounding or random rounding. When the weights are smaller and their amplitudes are much smaller than the quantization step size, using deterministic rounding would erase parameter updates, potentially preventing the network from being trained. Therefore, this application uses random rounding for truncation, as shown below:

[0184] The function is used when: Less than The return value is ,like Greater than Then return .

[0185] In this embodiment, a better quantization step size is selected for the embedding parameters of each feature to retain as much parameter information as possible, helping the model maintain convergence even during low-precision training. Lower-precision training reduces the memory usage and communication overhead of embeddings during training and inference, allowing more parameters to be accommodated in the same memory. Furthermore, a random truncation rounding function can be used to ensure that gradient information is not lost due to deterministic truncation during low-precision training. In addition, when updating the adaptive step size, heuristic and learned adaptive quantization step sizes are provided to adapt to different application scenarios, avoiding the need for manual selection of quantization step sizes for different features and improving model training and quantization efficiency.

[0186] To facilitate understanding, the effects of the quantization method provided in this application will be introduced below using some specific application scenarios as examples.

[0187] In numerous personalized service scenarios, users and products have interaction records based on different types of behaviors. Recommendation models will model the user's multi-behavioral interaction history, predict the products that the user may interact with based on target behaviors, and then display the products to the user in a sorted order. Click-through rate (CTR) prediction can be performed using the methods provided in this application, and the products can be sorted and displayed on the recommendation page according to the predicted CTR; or sorted according to the predicted CTR value; or sorted based on the top CTR values; or each item to be recommended can be rated, and then sorted and displayed according to the rating value, etc.

[0188] For example, the method provided in this application can be applied to APP recommendation scenarios, such as... Figure 9 As shown, recommended app icons can be displayed on the user's terminal screen, allowing the user to click or download the recommended apps, enabling the user to quickly find the apps they need and improving the user experience.

[0189] For example, the method provided in this application can be applied to product recommendation scenarios, such as... Figure 10 As shown, icons of recommended products can be displayed on the user's terminal screen, allowing the user to click, add to cart, or purchase the recommended products, thus improving the user experience.

[0190] For example, the method provided in this application can be applied to music recommendation scenarios, such as... Figure 11 As shown, recommended music icons can be displayed on the user's terminal screen, allowing the user to click, favorite, or play the recommended music, thus improving the user experience by enabling the user to view more preferred music.

[0191] Taking click-through rate (CTR) prediction in app recommendation scenarios as an example, CTR prediction models typically include two parts: embedding and MLP (Multi-Level Processing). Recommendation data is high-dimensional and sparse, with a large embedding table, leading to increased memory consumption and training latency. Commonly used pruning and AutoML methods cannot compress training memory, hash-based methods suffer from accuracy loss, and traditional low-precision training methods can only use INT16 and do not consider how to use adaptive quantization step sizes. However, the quantization method based on adaptive quantization step sizes provided in this application normalizes continuous features before automatically discretizing them during offline CTR prediction model training.

[0192] During offline training, in each batch, BatchEmbedding is retrieved from the low-precision Embedding Table; the low-precision parameters in full-precision representation are obtained through dequantization and used for MLP layer computation, ultimately outputting the predicted value; during the training phase, the loss function is calculated using the output predicted value and the full-precision gradient of the BatchEmbedding is obtained by relying on the backpropagation gradient; the Batch Embedding module is updated based on the full-precision gradient of the batch, and the quantization step size is updated adaptively; the Batch Embedding is quantized into low-precision parameters based on the adaptive quantization step size; and then the low-precision BatchEmbedding is written back to the embedding table.

[0193] During the online inference phase, the embeddings corresponding to the input data can be read from the low-precision embedding table and dequantized to obtain the full-precision embeddings. The full-precision embeddings are then used as input to the click-through rate prediction model to output the prediction results.

[0194] For example, using some publicly available datasets, we compare some existing quantization methods with the quantization method provided in this application, such as the Avazu dataset and the Criteo dataset. The statistical information of these datasets is shown in Table 1.

[0195]

[0196] Table 1

[0197] In this dataset, the training and test sets are split according to users, with 90% of users serving as the training set and 10% as the test set. Discrete features are one-hot encoded, and continuous features are discretized. Evaluation metrics include AUC (Area Under Curve).

[0198] Some existing quantization methods include: Full Precision (FP), Quantization Awareness (LSQ), Quantization Awareness Based on Dynamic Step Size (PACT), INT8 Low Precision Training (LPT), and INT16 Low Precision Training (LPT-16). The quantization method provided in this application can be based on different adaptive step size calculation methods, such as: Heuristic Adaptive Step Size INT8 Low Precision Training (ALPT_H) and Learnable Adaptive Step Size INT8 Low Precision Training (ALPT_L).

[0199] The comparison results are shown in Table 2:

[0200] Table 2

[0201] The deterministic Rounding function used in Table 2 above, as shown in Table 3, is a more effective randomized Rounding function for low-precision training.

[0202]

[0203] Table 3

[0204] Comparing Tables 2 and 3 above, existing low-precision training methods employ deterministic truncation and do not consider adaptive quantization step sizes, enabling low-precision parameter training only based on INT16, leading to difficulties in model convergence at even lower precision levels. For example, they compress embedding parameters during the inference stage and require retraining, resulting in low practicality. While some quantization methods can compress parameters using hashing, the inevitable collisions in hash functions lead to low precision. Although some quantization methods can train models using INT16, lower precision training often fails to converge. To achieve end-to-end low-precision training, this application proposes using a randomized truncation rounding function to ensure parameter updates of gradient information during training; and proposes assigning an adaptive quantization step size to each feature to better select the quantization step size and retain as much parameter information as possible.

[0205] Furthermore, based on the aforementioned quantification method, this application also provides a recommended method, such as... Figure 12 As shown, it can specifically include: 1201. Obtain input data.

[0206] The input data may include data generated by at least one user action on the terminal.

[0207] For example, when a user clicks on or plays a piece of music, information about that click can be collected; or when a user downloads or installs an app, information about that download or installation can be collected.

[0208] 1202. Retrieve the low-precision embedding corresponding to the input data from the low-precision embedding table.

[0209] After obtaining the input data, the embedding table can be used to convert the input data into features that the neural network can recognize. The low-precision embedding table usually stores the mapping relationship between the original data and the representation. After obtaining the input data, it can be mapped to a low-precision embedding based on this mapping relationship.

[0210] 1203. Perform inverse quantization on multiple features according to the adaptive step size corresponding to each feature to obtain the full-precision embedding.

[0211] After obtaining the low-precision embedding, each feature can be dequantized according to the adaptive step size corresponding to each feature, thereby obtaining the full-precision embedding.

[0212] The inverse quantization step can be found in the preceding text. Figure 7 Step 702 or the aforementioned Figure 8 Step 804 in the previous section will not be repeated here.

[0213] 1204. Using full-precision embedding as input to the neural network, output recommendation information.

[0214] After obtaining the full-precision embedding, it can be used as the input to the recommendation network to output the corresponding recommendation information.

[0215] In this embodiment of the application, during the inference process of the neural network, an adaptive step size can be used to dequantize the low-precision embedding representation to obtain a full-precision embedding representation. Therefore, the low-precision representation can be saved or transmitted during inference, and then losslessly restored using the adaptive step size to obtain the full-precision embedding representation. This reduces the storage space occupied by the embedding representation vocabulary and allows for lossless restoration during use.

[0216] The method flow provided in this application has been described above. The apparatus provided in this application will now be described based on the aforementioned method flow.

[0217] See Figure 13 This application provides a schematic diagram of the structure of a quantization device, including: The acquisition module 1301 is used to acquire the full-precision embedding representation, which includes multiple features. The determination module 1302 is used to determine the adaptive step size for each feature among multiple features; The quantization module 1303 is used to quantize multiple features according to the adaptive step size corresponding to each feature to obtain a low-precision embedding representation. The precision of the features in the low-precision embedding representation is lower than that of the features in the full-precision embedding representation.

[0218] In one possible implementation, a low-precision embedded representation vocabulary is applied to the neural network. The acquisition module 1301 is specifically used to acquire the representation corresponding to the input data of the current iteration from the low-precision embedding representation vocabulary, and obtain the low-precision embedding representation of the current iteration; and to dequantize the low-precision embedding representation of the current iteration to obtain the full-precision embedding representation of the current iteration.

[0219] In one possible implementation, the determining module 1302 is specifically used to: take the full-precision embedding representation of the current iteration as the input of the neural network to obtain the full-precision gradient corresponding to the prediction result of the current iteration; obtain the updated full-precision embedding representation based on the full-precision gradient to obtain the updated full-precision embedding representation; and obtain the adaptive step size corresponding to each feature in the updated full-precision embedding representation based on the full-precision gradient.

[0220] In one possible implementation, the quantization module 1303 is specifically used to quantize multiple features in the full-precision low-dimensional representation of the current iteration according to the adaptive step size corresponding to each feature, so as to obtain a low-precision embedded representation.

[0221] In one possible implementation, the acquisition module is further configured to update the low-precision embedding representation vocabulary based on the low-precision embedding representation, thereby obtaining the updated low-precision embedding representation vocabulary.

[0222] In one possible implementation, the determining module 1302 is specifically used to calculate the adaptive step size corresponding to each feature using a heuristic algorithm.

[0223] In one possible implementation, the determining module 1302 is specifically used to calculate the adaptive step size corresponding to each feature based on the absolute value of the weight in each feature.

[0224] In one possible implementation, the quantization module 1303 is specifically used to: obtain the discrete features of each feature according to the adaptive step size corresponding to each feature; and truncate the discrete features of each feature using a random truncation algorithm to obtain a low-precision embedding representation.

[0225] In one possible implementation, a low-precision embedded representation vocabulary is applied to a language model or a recommendation model. The language model is used to obtain semantic information from the corpus, and the recommendation model is used to generate recommendation information based on the user's information.

[0226] See Figure 14 The present application provides a schematic diagram of the structure of a recommended device, comprising: Input module 1401 is used to acquire input data, which includes data generated by the user's at least one action on the terminal; The acquisition module 1402 is used to acquire the low-precision embedding representation corresponding to the input data from the low-precision embedding representation vocabulary, and the low-precision embedding representation includes a variety of features; The dequantization module 1403 is used to dequantize multiple features according to the adaptive step size corresponding to each feature in multiple features to obtain a full-precision embedded representation. The recommendation module 1404 is used to output recommendation information based on the full-precision embedded representation as input to the neural network. The recommendation information is used to make recommendations based on at least one user behavior.

[0227] In one possible implementation, the neural network includes a language model or a recommendation model, wherein the language model is used to acquire semantic information from the corpus, and the recommendation model is used to generate recommendation information based on user information.

[0228] Please see Figure 15 The structural schematic diagram of another quantization device provided in this application is as follows.

[0229] The recommended device may include a processor 1501 and a memory 1502. The processor 1501 and the memory 1502 are interconnected via a circuit. The memory 1502 stores program instructions and data.

[0230] The aforementioned are stored in memory 1502 Figures 6-8 The steps in the code include the corresponding program instructions and data.

[0231] Processor 1501 is used to perform the aforementioned Figures 6-8 The method steps performed by the quantization device shown in any of the embodiments.

[0232] Optionally, the recommended device may also include a transceiver 1503 for receiving or sending data.

[0233] This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the aforementioned actions. Figures 6-8 The steps in the method described in the illustrated embodiment.

[0234] Alternatively, the aforementioned Figure 15 The recommended device shown is a chip.

[0235] Please see Figure 16 The structural schematic diagram of another recommended device provided in this application is as follows.

[0236] The recommended device may include a processor 1601 and a memory 1602. The processor 1601 and the memory 1602 are interconnected via a circuit. The memory 1602 stores program instructions and data.

[0237] The aforementioned are stored in memory 1602 Figure 12 The steps in the code include the corresponding program instructions and data.

[0238] Processor 1601 is used to execute the aforementioned Figure 12 The recommended apparatus shown represents the method steps performed.

[0239] Optionally, the recommended device may also include a transceiver 1603 for receiving or sending data.

[0240] This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the aforementioned actions. Figure 12 The steps in the method described in the illustrated embodiment.

[0241] Alternatively, the aforementioned Figure 16 The recommended device shown is a chip.

[0242] This application embodiment also provides a recommendation device, which can also be referred to as a digital processing chip or a chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform the aforementioned... Figure 11 The method and steps.

[0243] This application embodiment also provides a recommendation device, which can also be referred to as a digital processing chip or a chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform the aforementioned... Figure 12 The method and steps.

[0244] This application also provides a digital processing chip. This digital processing chip integrates circuitry for implementing the functions of the processors 1501, 1601, or 1501 and 1601 described above, and one or more interfaces. When the digital processing chip integrates a memory, it can complete the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the actions of the recommended device or the recommended device performed in the foregoing embodiments based on the program code stored in the external memory.

[0245] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned actions. Figures 6-12 The steps of the method described in the illustrated embodiment.

[0246] The recommended device or recommended apparatus provided in this application embodiment can be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip in the server to perform the above-mentioned operations. Figures 6-12 The method steps described in the illustrated embodiment are as follows. Optionally, the storage unit is a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0247] Specifically, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0248] For example, please refer to Figure 17 , Figure 17 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 170. The NPU 170 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1703, which is controlled by the controller 1704 to extract matrix data from the memory and perform multiplication operations.

[0249] In some implementations, the arithmetic circuit 1703 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 1703 is a two-dimensional pulsating array. The arithmetic circuit 1703 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1703 is a general-purpose matrix processor.

[0250] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1702 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1701 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1708.

[0251] Unified memory 1706 is used to store input and output data. Weight data is directly transferred to weight memory 1702 via direct memory access controller (DMAC) 1705. Input data is also transferred to unified memory 1706 via DMAC.

[0252] The bus interface unit (BIU) 1710 is used for interaction between the AXI bus and the DMAC and instruction fetch buffer (IFB) 1709.

[0253] The bus interface unit 1710 (BIU) is used by the instruction fetch memory 1709 to fetch instructions from external memory, and also by the memory access controller 1705 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0254] The DMAC is mainly used to move input data from external memory DDR to unified memory 1706, or to weight data to weight memory 1702, or to input data to input memory 1701.

[0255] The vector computation unit 1707 includes multiple processing units that further process the output of the computation circuits as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is primarily used for computations in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0256] In some implementations, the vector computation unit 1707 can store the processed output vector in the unified memory 1706. For example, the vector computation unit 1707 can apply linear and / or nonlinear functions to the output of the computation circuit 1703, such as performing linear interpolation on feature planes extracted from convolutional layers, or accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1707 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 1703, for example, for use in subsequent layers of the neural network.

[0257] The instruction fetch buffer 1709 connected to the controller 1704 is used to store the instructions used by the controller 1704; The unified memory 1706, input memory 1701, weighted memory 1702, and instruction fetch memory 1709 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0258] The operations of each layer in the recurrent neural network can be performed by the operation circuit 1703 or the vector calculation unit 1707.

[0259] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more processors used to control the above. Figures 6-12 The method of program execution of integrated circuits.

[0260] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0261] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0262] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0263] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0264] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

Claims

1. A quantification method, characterized in that, include: Obtain the full-precision embedding representation for the current iteration, wherein the embedding representation includes multiple features; The full-precision embedding representation of the current iteration is used as the input of the neural network to obtain the full-precision gradient corresponding to the prediction result of the current iteration. The neural network includes a language model or a recommendation model. The language model is used to obtain the semantic information of the corpus, and the recommendation model is used to generate recommendation information based on the user's information. The full-precision embedding representation is updated based on the full-precision gradient to obtain the updated full-precision embedding representation; The adaptive step size corresponding to each feature in the updated full-precision embedding representation is obtained based on the full-precision gradient. The various features are quantized according to the adaptive step size corresponding to each feature to obtain a low-precision embedding representation. The precision of the features in the low-precision embedding representation is lower than that of the features in the full-precision embedding representation.

2. The method according to claim 1, characterized in that, The low-precision embedded representation vocabulary is applied to the neural network. The step of obtaining the full-precision embedded representation vocabulary for the current iteration includes: The low-precision embedding representation corresponding to the input data of the current iteration is obtained from the low-precision embedding representation vocabulary. The low-precision embedding representation of the current iteration is dequantized to obtain the full-precision embedding representation of the current iteration.

3. The method according to claim 1, characterized in that, The quantization of the multiple features according to the adaptive step size corresponding to each feature includes: Based on the adaptive step size corresponding to each feature, the various features in the full-precision low-dimensional representation of the current iteration are quantized to obtain the low-precision embedded representation.

4. The method according to any one of claims 2-3, characterized in that, The method further includes: The low-precision embedding representation vocabulary is updated based on the low-precision embedding representation to obtain the updated low-precision embedding representation vocabulary.

5. A quantification method, characterized in that, include: Obtain a full-precision embedding representation, which includes multiple features; Calculate the adaptive step size for each feature based on the absolute value of the weight in each feature; The multiple features are quantized according to the adaptive step size corresponding to each feature to obtain a low-precision embedding representation. The precision of the features in the low-precision embedding representation is lower than that of the features in the full-precision embedding representation. The low-precision embedding representation is used to update the low-precision embedding representation vocabulary. The low-precision embedding representation vocabulary is applied to a language model or a recommendation model. The language model is used to obtain the semantic information of the corpus, and the recommendation model is used to generate recommendation information based on the user's information.

6. The method according to claim 5, characterized in that, The step of quantizing the multiple features according to the adaptive step size corresponding to each feature to obtain a low-precision embedding representation includes: Based on the adaptive step size corresponding to each feature, the discrete features of each feature are obtained; The discrete features of each feature are truncated using a random truncation algorithm to obtain the low-precision embedding representation.

7. A recommendation method, characterized in that, include: Acquire input data, which includes data generated by at least one user action on the terminal; Obtain low-precision embedding representations corresponding to the input data from a low-precision embedding representation vocabulary, wherein the low-precision embedding representations include multiple features; The various features are dequantized according to the adaptive step size corresponding to each feature to obtain a full-precision embedding representation. The adaptive step size corresponding to each feature is obtained based on the full-precision gradient corresponding to the prediction result of the current iteration. The full-precision gradient corresponding to the prediction result of the current iteration is generated by the neural network with the full-precision embedding representation of the current iteration as the input of the neural network. Alternatively, the adaptive step size corresponding to each feature is calculated based on the absolute value of the weights in each feature. The neural network outputs recommendation information based on the full-precision embedded representation as input, and the recommendation information is used to make recommendations for the user’s at least one behavior.

8. The method according to claim 7, characterized in that, The neural network includes a language model or a recommendation model. The language model is used to acquire semantic information from the corpus, and the recommendation model is used to generate recommendation information based on user information.

9. A quantization device, characterized in that, include: The acquisition module is used to acquire the full-precision embedding representation of the current iteration, wherein the embedding representation includes multiple features; The determination module is used to take the full-precision embedding representation of the current iteration as the input of the neural network to obtain the full-precision gradient corresponding to the prediction result of the current iteration. The neural network includes a language model or a recommendation model. The language model is used to obtain the semantic information of the corpus, and the recommendation model is used to generate recommendation information based on the user's information. The full-precision embedding representation is updated based on the full-precision gradient to obtain the updated full-precision embedding representation; The adaptive step size corresponding to each feature in the updated full-precision embedding representation is obtained based on the full-precision gradient. The quantization module is used to quantize the multiple features according to the adaptive step size corresponding to each feature to obtain a low-precision embedding representation, wherein the precision of the features in the low-precision embedding representation is lower than the precision of the features in the full-precision embedding representation.

10. The apparatus according to claim 9, characterized in that, The low-precision embedded representation vocabulary is applied to the neural network. The acquisition module is specifically used for: The low-precision embedding representation corresponding to the input data of the current iteration is obtained from the low-precision embedding representation vocabulary. The low-precision embedding representation of the current iteration is dequantized to obtain the full-precision embedding representation of the current iteration.

11. The apparatus according to claim 9, characterized in that, The quantization module is specifically used to quantize multiple features in the full-precision low-dimensional representation of the current iteration according to the adaptive step size corresponding to each feature, so as to obtain the low-precision embedded representation.

12. The apparatus according to any one of claims 10-11, characterized in that, The acquisition module is further configured to update the low-precision embedding representation vocabulary based on the low-precision embedding representation to obtain the updated low-precision embedding representation vocabulary.

13. A quantization device, characterized in that, include: An acquisition module is used to acquire a full-precision embedding representation, which includes multiple features; The determination module is used to calculate the adaptive step size corresponding to each feature based on the absolute value of the weight in each feature; The quantization module is used to quantize the multiple features according to the adaptive step size corresponding to each feature to obtain a low-precision embedding representation. The precision of the features in the low-precision embedding representation is lower than that of the features in the full-precision embedding representation. The low-precision embedding representation is used to update the low-precision embedding representation vocabulary. The low-precision embedding representation vocabulary is applied to a language model or a recommendation model. The language model is used to obtain the semantic information of the corpus, and the recommendation model is used to generate recommendation information based on the user's information.

14. The apparatus according to claim 13, characterized in that, The quantization module is specifically used for: Based on the adaptive step size corresponding to each feature, the discrete features of each feature are obtained; The discrete features of each feature are truncated using a random truncation algorithm to obtain the low-precision embedding representation.

15. A recommended device, characterized in that, include: An input module is used to acquire input data, which includes data generated by the user's actions on the terminal at least once; The acquisition module is used to acquire low-precision embedding representations corresponding to the input data from a low-precision embedding representation vocabulary, wherein the low-precision embedding representations include multiple features; The dequantization module is used to dequantize the multiple features according to the adaptive step size corresponding to each feature to obtain a full-precision embedding representation. The adaptive step size corresponding to each feature is obtained based on the full-precision gradient corresponding to the prediction result of the current iteration. The full-precision gradient corresponding to the prediction result of the current iteration is generated by the neural network when the full-precision embedding representation of the current iteration is used as the input of the neural network. Alternatively, the adaptive step size corresponding to each feature is calculated based on the absolute value of the weights in each feature. The recommendation module is used to output recommendation information based on the full-precision embedded representation as input to the neural network, the recommendation information being used to make recommendations for the user's at least one behavior.

16. The apparatus according to claim 15, characterized in that, The neural network includes a language model or a recommendation model. The language model is used to acquire semantic information from the corpus, and the recommendation model is used to generate recommendation information based on user information.

17. A quantization device, characterized in that, The quantization device includes: a processor coupled to a memory; The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory, so that the quantization device performs the quantization method as described in any one of claims 1 to 6.

18. A recommendation device, characterized in that, The recommended device includes: a processor coupled to a memory; The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory, such that the recommendation device performs the recommendation method as described in any one of claims 7 to 8.

19. A computer program product containing instructions, characterized in that, When it is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 8.

20. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 8.

Images