Data processing method, and training method and device of neural network model

Abstract

The application discloses a data processing method and a neural network model training method and device in the field of artificial intelligence. The data processing method comprises: processing data to be processed by using a target neural network quantization model. The target neural network quantization model comprises multiple sets of fusion parameters. The target neural network quantization model is obtained by quantizing a target neural network model. An activation function of the target neural network model comprises a piecewise linear function (PWL). The PWL comprises multiple intervals. The multiple sets of fusion parameters have a corresponding relationship with the multiple intervals. The method can quantize a model using a PWL as an activation function, and improve the inference speed of the model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a data processing method, a method for training a neural network model, and an apparatus. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0003] Deep neural networks (DNNs) are feedforward neural networks with a deep structure and are one of the representative algorithms of deep learning. DNNs possess powerful feature representation capabilities and are widely used in processing and analyzing various media signals such as images, videos, and audio. Activation functions are a crucial component of DNNs. Activation functions are typically placed after convolutional or fully connected layers, introducing nonlinear operations into the DNN, enabling it to better fit nonlinear functions and solve complex real-world problems.

[0004] The Rectifier Linear Unit (ReLU) activation function is one of the most widely used activation functions in deep neural networks. However, for input features with values ​​less than 0, the output of ReLU is always 0, causing some neurons to never be activated, resulting in the corresponding parameters never being updated and affecting model performance. Using a piecewise linear function (PWL) as the activation function, by adjusting multiple trainable parameters of the PWL, various function forms can be fitted, which is beneficial for obtaining a better activation function and improving model performance.

[0005] With the development of deep neural networks, the number of parameters in network models has also increased, consuming significant computational resources and memory. Model quantization can reduce the memory usage of models, improve the inference efficiency of deep neural networks, and reduce power consumption. However, existing model quantization methods are typically designed for the ReLU activation function. For models using PWL as the activation function, existing model quantization methods cannot directly quantize the PWL parameters, resulting in the model being unable to perform quantized inference and affecting its inference speed.

[0006] Therefore, improving the processing efficiency of neural network models that use PWL as the activation function has become an urgent problem to be solved. Summary of the Invention

[0007] This application provides a data processing method, a neural network model training method, and an apparatus that can quantize models using PWL as the activation function, thereby improving the inference speed of the model.

[0008] Firstly, a data processing method is provided, which includes: acquiring data to be processed, including image data, audio data, or text data; processing the data to be processed using a target neural network quantization model, wherein the target neural network quantization model includes multiple sets of fusion parameters, the target neural network quantization model is obtained by quantizing a target neural network model, the activation function of the target neural network model includes PWL, PWL includes multiple intervals, and there is a corresponding relationship between the multiple sets of fusion parameters and the multiple intervals.

[0009] Existing solutions can only quantize the weight parameters in models using PWL as the activation function, but cannot quantize the PWL parameters themselves. Therefore, the inference process requires calculating the activation function based on the PWL parameters, meaning existing solutions cannot achieve full-process quantization inference. According to the solution of this application, the target neural network quantization model can process data based on multiple sets of fusion parameters corresponding to multiple intervals of PWL, realizing quantization inference of the model. These multiple sets of fusion parameters are obtained by fusing the quantization process and the PWL calculation process in the target neural network model. During inference, target indication information can be directly determined based on the data to be processed, and the target fusion parameter corresponding to the data to be processed can be selected from the multiple sets of fusion parameters according to the target indication information. Simple calculations are then performed on the data to be processed based on the target fusion parameter to obtain the model's processing result, without performing complex calculation processes such as PWL quantization and dequantization. This achieves quantization inference in all stages of the model, reducing computational load and improving processing efficiency.

[0010] Furthermore, since the multiple sets of fusion parameters are obtained by fusing the quantization process and PWL calculation process in the target neural network model, the target neural network quantization model provided in this application can theoretically achieve inference accuracy consistent with the target neural network model. That is, the solution provided in this application can achieve fast inference in a neural network with a PWL activation function without sacrificing the accuracy of the inference results.

[0011] The activation function of the target neural network model includes one or more PWLs (Programmable Wires). Each PWL comprises multiple intervals. In other words, one PWL corresponds to multiple sets of fusion parameters. These multiple sets of fusion parameters are calculated based on the parameters of the multiple intervals of the PWL.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the target neural network model is used to process the data to be processed, including: determining target indication information based on the data to be processed, the target indication information being used to indicate fusion parameters related to the data to be processed; obtaining target fusion parameters corresponding to the target indication information from multiple sets of fusion parameters based on the target indication information; and processing the data to be processed based on the target fusion parameters.

[0013] For example, the target indication information can be an index of a target interval; that is, the target indication information can be used to indicate a target interval. This target interval belongs to multiple intervals of a PWL.

[0014] Compared to quantization inference using ReLU as the activation function, quantization inference using PWL as the activation function only requires additional determination of target indication information and calculation of corresponding fusion parameters based on that information. Using PWL as the activation function improves model performance. In other words, using PWL as the activation function can enhance model performance without introducing excessive computation.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, determining the target indication information based on the data to be processed includes: processing the input features of the first network layer based on the weight parameters of the first network layer in the target neural network quantization model, wherein the input features of the first network layer are determined based on the data to be processed; and determining the target indication information based on the processing result of the first network layer.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the weight parameters of the first network layer are obtained by quantizing the weight parameters of the second network layer in the target neural network model. The second network layer is the network layer in the target neural network model corresponding to the first network layer. The target indication information is determined based on the processing result of the first network layer, including: processing the processing result of the first network layer based on the fusion bias parameter; calculating the target indication information based on the equivalent length of the middle interval among multiple intervals of the PWL corresponding to the first result and the second network layer. The equivalent length of the middle interval is determined based on the length of the middle interval and the quantization parameters of the second network layer.

[0017] The second network layer can be any network layer in the target neural network model, as long as the activation function corresponding to that network layer is PWL.

[0018] If we consider PWL as a standalone network layer, then the PWL corresponding to the second network layer refers to the next network layer after the second network layer, which is the PWL layer. If we consider PWL as one step in the multi-step computation of network layers such as convolutional layers, then the PWL corresponding to the second network layer refers to the activation operation performed by the second network layer, and the activation function used in this activation operation is PWL.

[0019] In this case, the lengths of the intermediate intervals can be equal.

[0020] In this way, the first result can be fully utilized to determine the target indication information without the need for other calculations to obtain the target indication information, thus avoiding unnecessary calculations and improving processing efficiency.

[0021] In conjunction with the first aspect, in some implementations of the first aspect, the equivalent length of the intermediate interval is an integer power of 2.

[0022] In this way, the index of the target interval, i.e., the target indication information, can be obtained through shifting. Specifically, shifting the data one position to the right is equivalent to dividing by 2. For example, the equivalent length of the middle interval is 2. n Where n is an integer, when calculating the index of the target interval, it is necessary to divide by 2. n The operation can be performed by shifting by n bits, avoiding the need for a divider. This eliminates the need for a divider in the hardware, reducing costs. Furthermore, compared to a divider, using shifting to obtain the index of the target interval reduces computational load, lowers power consumption, and improves inference efficiency.

[0023] In conjunction with the first aspect, in some implementations of the first aspect, target fusion parameters corresponding to the target indication information are obtained from multiple sets of fusion parameters based on the target indication information, and the data to be processed is processed based on the target fusion parameters, including: obtaining target fusion scaling parameters from the target fusion parameters based on the target indication information, processing the first result based on the target fusion scaling parameters to obtain a second result; obtaining target fusion offset parameters from the target fusion parameters based on the target indication information, processing the second result based on the target fusion offset parameters to obtain a third result.

[0024] The third result can be used as the input to the next network layer after the first network layer.

[0025] In conjunction with the first aspect, in some implementations of the first aspect, the method also includes: performing integer processing on the third result.

[0026] In this case, the result after rounding can be used as the input to the next network layer after the first network layer.

[0027] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: performing a rounding operation on the second result; processing the second result based on the target fusion offset parameter to obtain the rounded second result, including: processing the rounded second result based on the target fusion offset parameter to obtain the third result.

[0028] In this case, the third result can be used as the input to the next network layer after the first network layer.

[0029] In conjunction with the first aspect, in some implementations of the first aspect, the target indication information satisfies the following formula:

[0030]

[0031] This represents the quantized weight parameters of the second network layer. This represents the quantized input features of the second network layer, where i represents the target indication information. This represents the equivalent length of the intermediate interval, and bias represents the fusion bias parameter.

[0032] The equivalent length of the intermediate interval satisfies the following formula:

[0033]

[0034] d represents the length of the intermediate interval, S W S represents the weight quantization parameter in the quantization parameters of the second network layer. X This represents the feature quantization parameter in the quantization parameters of the second network layer.

[0035] In conjunction with the first aspect, in some implementations of the first aspect, the fusion bias parameter is determined based on at least one of the following: the weight parameters of the second network layer, the parameters of the PWL corresponding to the second network layer, or the quantization parameters of the second network layer.

[0036] In conjunction with the first aspect, in some implementations of the first aspect, the fusion bias parameter satisfies the following formula:

[0037]

[0038] Where bias represents the fusion bias parameter, and LB represents the left boundary of the PWL parameters corresponding to the second network layer. S represents the quantized weight parameters of the second network layer, b represents the bias parameters of the second network layer, and S represents the quantized weight parameters of the second network layer. W The weight quantization parameters β and S represent the quantization parameters of the second network layer. X This represents the feature quantization parameter in the quantization parameters of the second network layer.

[0039] In conjunction with the first aspect, in some implementations of the first aspect, the target fusion scaling parameter is determined based on at least one of the following: the PWL parameter corresponding to the second network layer, the quantization parameter of the second network layer, or the quantization parameter of the next network layer after the second network layer.

[0040] In conjunction with the first aspect, in some implementations of the first aspect, the target fusion scaling parameter satisfies the following formula:

[0041]

[0042] Among them, scale i This represents the target fusion scaling parameter, where i represents the target indication information, and S... W S represents the weight quantization parameter in the quantization parameters of the second network layer. X S represents the feature quantization parameter in the quantization parameters of the second network layer. Z K represents the feature quantization parameter of the next network layer after the second network layer. i This represents the slope of the i-th interval in the PWL corresponding to the second network layer.

[0043] In conjunction with the first aspect, in some implementations of the first aspect, the target fusion offset parameter is determined based on at least one of the following: the parameters of the PWL corresponding to the second network layer or the quantization parameters of the next network layer after the second network layer.

[0044] In conjunction with the first aspect, in some implementations of the first aspect, the target fusion offset parameter satisfies the following formula:

[0045]

[0046] Among them, offset i This represents the target fusion offset parameter, where i represents the target indication information, and S... Z γ and K represent the feature quantization parameters of the next network layer after the second network layer. i B represents the slope of the i-th interval in the PWL corresponding to the second network layer. i This represents the left endpoint of the i-th interval in the PWL corresponding to the second network layer.

[0047] Secondly, a method for training a neural network model is provided, comprising: acquiring a neural network model, wherein the activation function of the neural network model includes a piecewise linear function (PWL), and the PWL includes multiple intervals; training the neural network model based on training data to obtain a target neural network model, wherein the training data includes image data, audio data, or text data; and processing the target neural network model to obtain a target neural network quantization model, wherein the target neural network quantization model includes multiple sets of fusion parameters, and the multiple sets of fusion parameters have a corresponding relationship with the multiple intervals of the PWL in the target neural network model.

[0048] Existing solutions can only quantize the weight parameters in models using PWL as the activation function, but cannot quantize the PWL parameters themselves. Therefore, during inference, the activation function calculation must be performed based on the PWL parameters, meaning that models obtained through existing training methods cannot achieve quantized inference. According to the solution of this application, a quantized target neural network model is obtained by processing the target neural network model. This quantized target neural network model includes multiple sets of fusion parameters, allowing the quantized target neural network model to be processed during inference based on these multiple sets of fusion parameters corresponding to multiple intervals of PWL, effectively achieving complete quantization of the target neural network model. Specifically, these multiple sets of fusion parameters are obtained by fusing the quantization process and the PWL calculation process in the target neural network model. This allows the processing results to be obtained based on these multiple sets of fusion parameters during inference, eliminating the need for additional PWL calculations. This facilitates quantized inference of the model, reduces computational load, and improves processing efficiency.

[0049] Furthermore, the target neural network quantization model is obtained by fusing the quantization process and PWL calculation process in the target neural network model. Theoretically, the target neural network quantization model can achieve the same inference accuracy as the target neural network model.

[0050] The activation functions of this neural network model include one or more PWLs. In other words, at least one activation function in this neural network model is a PWL.

[0051] In conjunction with the second aspect, in some implementations of the second aspect, the target neural network model is processed to obtain a target neural network quantization model, including: obtaining multiple sets of fusion parameters based on the parameters of multiple intervals of PWL in the target neural network model, the weight quantization parameters and feature quantization parameters of the neural network model, so as to obtain the target neural network quantization model. The weight quantization parameters and feature quantization parameters of the neural network model are obtained through quantization training.

[0052] Quantization parameters include weight quantization parameters and feature quantization parameters. Weight quantization parameters are used to quantize the weights in the model, obtaining quantized weights. Feature quantization parameters are used to quantize the input features, obtaining quantized features.

[0053] In conjunction with the second aspect, in some implementations of the second aspect, the multiple sets of fusion parameters include multiple sets of fusion scaling parameters, which are determined based on at least one of the following: parameters of multiple intervals of PWL in the target neural network model, quantization parameters of network layers preceding PWL in the target neural network model, or quantization parameters of network layers following PWL in the target neural network model.

[0054] In conjunction with the second aspect, in some ways of the second aspect, the multiple sets of fusion parameters include multiple sets of fusion offset parameters, which are determined based on at least one of the following: parameters of multiple intervals of PWL in the target neural network model or quantization parameters of network layers after PWL in the target neural network model.

[0055] In conjunction with the second aspect, in some implementations of the second aspect, the target neural network model is obtained based on the trained neural network model, including: adjusting the length of the intermediate interval of PWL in the trained neural network model to obtain the adjusted neural network model; and obtaining the target neural network model based on the adjusted neural network model.

[0056] According to the solution of this application embodiment, the length of the intermediate interval can be adjusted to obtain the required length of the intermediate interval, thereby improving the flexibility of the model; at the same time, the model is trained to ensure the accuracy of the model.

[0057] In conjunction with the second aspect, in some implementations of the second aspect, training a neural network model based on training data to obtain a target neural network model includes: training a neural network model based on training data; adjusting the length of the intermediate interval of PWL in the neural network model to obtain an adjusted neural network model; and training the adjusted neural network model based on training data.

[0058] According to the solution of this application embodiment, the length of the intermediate interval can be adjusted to obtain the required length of the intermediate interval, thereby improving the flexibility of the model; at the same time, the model is trained to ensure the accuracy of the model.

[0059] In conjunction with the second aspect, in some implementations of the second aspect, adjusting the length of the intermediate interval of the PWL in the trained neural network model to obtain the adjusted neural network model includes: determining the target scaling factor of the PWL in the trained neural network model based on the equivalent length of the intermediate interval of the PWL in the trained neural network model, wherein the equivalent length of the intermediate interval is determined based on the length of the intermediate interval of the PWL in the trained neural network model, the feature quantization parameters and weight quantization parameters of the neural network model; adjusting the parameters of the target network layer in the trained neural network model and the length of the intermediate interval of the PWL in the trained neural network model based on the target scaling factor to obtain the adjusted neural network model, wherein the target network layer corresponds to the PWL in the trained neural network model.

[0060] Alternatively, the PWL in the trained neural network is the PWL corresponding to the target network layer. The PWL in the trained neural network model can be located within the target network layer, or it can be located after the target network layer, with the target network layer adjacent to the PWL in the trained neural network model.

[0061] Specifically, the equivalent length of the intermediate interval is obtained by dividing the length of the intermediate interval by the quantization parameter.

[0062] In conjunction with the second aspect, in some implementations of the second aspect, the target scaling factor of the PWL in the trained neural network model is determined based on the equivalent length of the intermediate interval of the PWL in the trained neural network model, including: determining the target scaling factor based on the equivalent length of the intermediate interval and the nearest integer power of 2 that is closest to the equivalent length of the intermediate interval.

[0063] For example, the nearest power of 2 that is closest to the equivalent length of the middle interval satisfies the following formula:

[0064]

[0065] in, d' represents the equivalent length of the middle interval, and d' represents the nearest power of 2 that is closest to the equivalent length of the middle interval. This indicates the floor operation.

[0066] According to the scheme of this application embodiment, by adjusting the length of the middle interval of the PWL (Programmable Logic Buffer), for example, by adjusting the boundaries of the PWL, the equivalent length of the middle interval of the PWL is made close to an integer power of 2. This helps to constrain the equivalent length of the middle interval of the PWL to an integer power of 2, thus achieving the quantization of the PWL parameters. Simultaneously, model training ensures model accuracy. In this way, during subsequent quantization inference of the model, the index of the target interval can be obtained through shifting, avoiding the need for division operations using a divider. No divider needs to be set in the hardware, reducing costs. Furthermore, compared to a divider, obtaining the index of the target interval through shifting reduces computational load, lowers power consumption, and improves inference efficiency.

[0067] In conjunction with the second aspect, in some implementations of the second aspect, determining the target scaling factor based on the equivalent length of the intermediate interval and the nearest power of 2 to the equivalent length of the intermediate interval includes: calculating a first scaling factor based on the equivalent length of the intermediate interval and the nearest power of 2 to the equivalent length of the intermediate interval; if the first scaling factor is within the target range, the target scaling factor is the first scaling factor; if the first scaling factor is outside the target range, the target scaling factor is the boundary value of the target range.

[0068] In conjunction with the second aspect, in some implementations of the second aspect, the first scaling factor satisfies the following formula:

[0069]

[0070] Where s represents the first scaling factor, and d' represents the nearest power of 2 that is closest to the equivalent length of the middle interval. This indicates the equivalent length of the intermediate interval.

[0071] In conjunction with the second aspect, in some implementations of the second aspect, the equivalent length of the intermediate interval satisfies the following formula:

[0072]

[0073] in, Let LB represent the equivalent length of the intermediate intervals, RB represent the left boundary of the PWL, RB represent the right boundary of the PWL, N represent the number of intermediate intervals in the PWL, and S represent the number of intermediate intervals in the PWL. W S represents the weight quantization parameter of the target network layer. X This represents the feature quantization parameters of the target network layer.

[0074] Thirdly, a data processing apparatus is provided, the apparatus comprising a module or unit for performing the methods described in the first aspect and any implementation thereof.

[0075] Fourthly, a training apparatus for a neural network model is provided, the apparatus comprising a module or unit for performing the methods of the second aspect and any implementation thereof.

[0076] It should be understood that the extensions, limitations, interpretations and descriptions of the relevant content in the first aspect above also apply to the same content in the second, third and fourth aspects.

[0077] Fifthly, a data processing apparatus is provided, comprising: a memory for storing a program; and a processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method of the first aspect and any implementation thereof.

[0078] The processor mentioned in the fifth aspect above can be a central processing unit (CPU) or a combination of a CPU and a neural network processing processor. The neural network processing processor can include a graphics processing unit (GPU), a neural network processing unit (NPU), and a tensor processing unit (TPU), etc. The TPU is a dedicated integrated circuit from Google, a fully custom-designed AI accelerator for machine learning.

[0079] In a sixth aspect, a training apparatus for a neural network model is provided, the apparatus comprising: a memory for storing a program; and a processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method of the second aspect and any implementation thereof.

[0080] The processor mentioned in the sixth aspect above can be a central processing unit (CPU) or a combination of a CPU and a neural network processing processor. The neural network processing processor can include a graphics processing unit (GPU), a neural network processor (NNP), and a tensor processor, among others. The TPU is a Google-designed application-specific integrated circuit (ASIC) for a fully customized AI accelerator designed for machine learning.

[0081] In a seventh aspect, a computer-readable storage medium is provided that stores program code for execution by a device, the program code including methods for performing any implementation of the first or second aspect.

[0082] Eighthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the method in any one of the implementations of the first or second aspect described above.

[0083] Ninth aspect, a chip is provided, the chip including a processor and a data interface, the processor reading instructions stored in a memory through the data interface and executing the method in any one of the implementations of the first or second aspect described above.

[0084] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in either the first aspect or the second aspect.

[0085] The aforementioned chip can be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). Attached Figure Description

[0086] Figure 1 This is a schematic diagram of an artificial intelligence main framework provided in an embodiment of this application;

[0087] Figure 2 A schematic diagram of a partial structure of a neural network model;

[0088] Figure 3 A schematic diagram of a piecewise linear function provided in an embodiment of this application;

[0089] Figure 4 A schematic flowchart illustrating the processing procedure of the neural network model provided in the embodiments of this application;

[0090] Figure 5 A schematic diagram of a system architecture provided in an embodiment of this application;

[0091] Figure 6 A schematic diagram of the hardware structure of a chip provided in an embodiment of this application;

[0092] Figure 7 A schematic diagram of a system architecture provided for an embodiment of this application;

[0093] Figure 8 A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0094] Figure 9A schematic flowchart illustrating a training method for a neural network model provided in an embodiment of this application;

[0095] Figure 10 A schematic flowchart illustrating another method for training a neural network model provided in an embodiment of this application;

[0096] Figure 11 A schematic flowchart illustrating the forward propagation process of a neural network model provided in an embodiment of this application;

[0097] Figure 12 A schematic flowchart illustrating the data processing method provided in the embodiments of this application;

[0098] Figure 13 A schematic block diagram illustrating the data processing method provided in the embodiments of this application;

[0099] Figure 14 This is a schematic block diagram of the training apparatus for the neural network model provided in the embodiments of this application;

[0100] Figure 15 This is a schematic block diagram of the data processing apparatus provided in the embodiments of this application;

[0101] Figure 16 This is a schematic block diagram of the training apparatus for the neural network model provided in the embodiments of this application;

[0102] Figure 17 This is a schematic block diagram of the data processing apparatus provided in the embodiments of this application. Detailed Implementation

[0103] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0104] Figure 1 A schematic diagram of an artificial intelligence framework is shown, which describes the overall workflow of an artificial intelligence system and is applicable to general artificial intelligence domain needs.

[0105] The above-mentioned artificial intelligence framework will be elaborated in detail from two dimensions: the "intelligent information chain" (horizontal axis) and the "information technology (IT) value chain" (vertical axis).

[0106] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom."

[0107] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology) to the industrial ecosystem of systems.

[0108] (1) Infrastructure:

[0109] Infrastructure provides computing power to support artificial intelligence systems, enables them to communicate with the outside world, and provides support through basic platforms.

[0110] Infrastructure can communicate with the outside world through sensors, and its computing power can be provided by smart chips.

[0111] The intelligent chips here can be hardware acceleration chips such as central processing units (CPUs), neural network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0112] The basic platform of the infrastructure can include distributed computing frameworks and related platform guarantees and support, such as cloud storage and computing, and interconnected networks.

[0113] For example, for infrastructure, data can be acquired through sensors and external communication, and then this data can be provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0114] (2) Data:

[0115] The data at the next layer of infrastructure is used to represent data sources in the field of artificial intelligence. This data includes graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0116] (3) Data processing:

[0117] The aforementioned data processing typically includes data training, machine learning, deep learning, search, reasoning, and decision-making.

[0118] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0119] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0120] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0121] (4) General abilities:

[0122] After the data processing mentioned above, the results of the data processing can be used to develop some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, and so on.

[0123] (5) Smart products and industry applications:

[0124] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, smart cities, or intelligent terminals, etc.

[0125] The embodiments of this application can be applied to many fields of artificial intelligence, such as intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, smart city, or intelligent terminal.

[0126] Specifically, the embodiments of this application can be applied to fields that require the use of (deep) neural networks, such as autonomous driving, image classification, image retrieval, image semantic segmentation, image quality enhancement, image super-resolution, and natural language processing.

[0127] The following is a brief introduction to two application scenarios: photo album categorization and monitoring.

[0128] Photo album categories:

[0129] When users store a large number of pictures on their terminal devices (such as mobile phones) or cloud storage, recognizing the images in the album can make it easier for users or the system to classify and manage the album, thus improving the user experience.

[0130] The data processing method described in this application can improve the inference speed of neural networks. Using this method to classify images increases classification speed and facilitates real-time tagging of images of different categories, making it easier for users to view and find them. Furthermore, these image category tags can be provided to a photo album management system for categorized management, saving users' management time, improving album management efficiency, and enhancing the user experience.

[0131] monitor:

[0132] Monitoring scenarios include smart cities, field surveillance, indoor surveillance, outdoor surveillance, and vehicle surveillance. In smart city scenarios, multiple attribute recognition is required, such as pedestrian and cyclist attribute recognition. Deep neural networks, with their powerful capabilities, play a crucial role in multi-attribute recognition.

[0133] The data processing method described in this application can improve the inference speed of neural networks. Processing the input road images using the data processing method described in this application facilitates the real-time identification of different attribute information within the road images.

[0134] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts of neural networks that may be involved in the embodiments of this application will be introduced below.

[0135] (1) Neural Network

[0136] Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit that takes an intercept of 1 as input can output the following:

[0137]

[0138] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weights are denoted by b, where b is the bias of the neural unit.

[0139] f represents the activation function of a neural network, used to introduce nonlinear characteristics and convert the input signal into the output signal. The output signal of this activation function can be used as the input to the next layer. The activation function is typically a scalar function from the real number domain to the real number domain, containing a certain nonlinear component. For example, the activation function can be ReLU, tanh, or sigmoid.

[0140] A neural network is a network formed by connecting multiple individual neural units, meaning that the output of one neural unit can be the input of another. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from the local receptive field, which can be a region composed of several neural units.

[0141] (2) Deep Neural Networks

[0142] A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple hidden layers. Based on the position of the layers, the internal neural network of a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer.

[0143] Although DNNs seem complex, the operation of each layer is actually not complicated. Simply put, it involves the following linear relationship expression: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is also relatively large. The definitions of these parameters in DNNs are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W is located, while the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input.

[0144] In summary, the coefficient from the k-th neuron in layer L-1 to the j-th neuron in layer L is defined as...

[0145] Activation functions are typically placed after convolutional or fully connected layers. Figure 2 A fully connected layer and the activation function following it are shown. Figure 2 input vector in Given (x1, x2, x3), output vector for The weight matrix W is as follows Figure 2The weights of the fully connected layer shown ( Figure 2 The matrix formed by omitting the superscript of the weight matrix is ​​called a matrix, where f represents the activation function. The input vector (x1, x2, x3) is processed by the fully connected layer to output (y1, y2), and then passed through the activation function f to obtain the output vector.

[0146] It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by vectors from many layers).

[0147] (3) Convolutional Neural Network

[0148] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are called the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be formalized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is reducing the connections between layers in the CNN, while also reducing the risk of overfitting.

[0149] (4) Loss Function

[0150] In training deep neural networks, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value and update the weight vector of each layer based on the difference. (Of course, there's usually a pre-configuration process before the first update, where parameters are pre-configured for each layer.) For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network can predict the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss. Generally, a smaller loss indicates higher training quality, while a larger loss indicates lower training quality. Similarly, smaller loss fluctuations result in more stable training, while larger loss fluctuations lead to less stable training.

[0151] (5) Backpropagation algorithm

[0152] Neural networks can employ backpropagation (BP) to correct the parameters of the neural network model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0153] For example, the loss value generated during each training iteration of a neural network model is passed layer by layer from back to front. At each layer, the update amount of that layer's parameters is calculated (partial derivative operation), and this update amount is related to the gradient. The gradient value and the loss value are linearly and positively correlated.

[0154] (6)PWL

[0155] A piecewise function is a function whose domain is represented by different expressions at different intervals or points. The common endpoints of two adjacent intervals are called the boundary points of the piecewise function. In other words, a piecewise function comprises multiple functions with different domains. The domain of a piecewise function is the union of the domains of the multiple piecewise functions, and the range of a piecewise function is the union of the ranges of the multiple piecewise functions.

[0156] A piecewise linear function is a linear function with different slopes in different intervals of its domain. That is, a piecewise linear function is linear in each interval of its domain, but the piecewise linear function as a whole is a nonlinear function.

[0157] A piecewise linear function can be defined by a variety of parameter combinations. By setting different values ​​for the parameters, different activation functions can be obtained.

[0158] For example, a piecewise linear function includes the following parameters: the boundary point, the slope over multiple intervals, and the function value corresponding to any boundary point.

[0159] The slope over multiple intervals refers to the slope of the piecewise linear function over those intervals.

[0160] The domain of a piecewise linear function is divided into multiple intervals based on its boundary points. The piecewise linear function is linear within each of these intervals. In other words, a piecewise linear function comprises multiple linear functions within multiple intervals. The slope of the piecewise linear function on any of these intervals is the slope of the linear function within that interval. The function value corresponding to a boundary point refers to the function value of that boundary point within the piecewise linear function. Alternatively, in the coordinate system containing the piecewise linear function, with the boundary point value as the x-coordinate, the y-coordinate corresponding to that x-coordinate in the piecewise linear function is the function value at that boundary point.

[0161] For example, a piecewise linear function includes the following parameters: the boundary point, the slope of the piecewise linear function on the interval with the largest domain, the slope of the piecewise linear function on the interval with the smallest domain, and the function value corresponding to the boundary point.

[0162] For example, a piecewise linear function includes the following parameters: the number of boundary points, the right boundary, the left boundary, the slope of the interval with the largest domain, the slope of the interval with the smallest domain, and the function value corresponding to each boundary point. In this case, the lengths of the intermediate intervals can be the same. The intermediate interval refers to the interval between the left and right boundaries among the multiple intervals of the piecewise linear function. Thus, the values ​​of the remaining boundary points can be determined based on the right boundary RB and the left boundary LB.

[0163] The right boundary refers to the maximum value at the dividing point, and the left boundary refers to the minimum value at the dividing point.

[0164] The number of dividing points can also be replaced by the number of intermediate intervals. That is, a piecewise linear function includes the following parameters: the number of intermediate intervals, the right boundary, the left boundary, the slope of the interval with the largest domain, the slope of the interval with the smallest domain, and the function value corresponding to the dividing point.

[0165] The interval with the largest domain and the interval with the smallest domain can also be referred to as the two ends of these multiple intervals. The function values ​​YP corresponding to multiple boundary points can be represented as an array, where each element of the array represents a function value, corresponding to a boundary point.

[0166] Figure 3 A schematic diagram of a piecewise linear function is shown below. (Followed by...) Figure 3 The above parameters are explained. Figure 3 The number of boundary points in the middle is 9, with the right boundary RB and the left boundary LB as follows: Figure 9 As shown, based on these 9 dividing points, the domain of the piecewise linear function is divided into 10 intervals, with N being the number of intermediate intervals. The interval with the largest domain is the rightmost interval in the coordinate system, and the interval with the smallest domain is the leftmost interval. The slope RK of the piecewise linear function in the interval with the largest domain is the same as the slope RK of the piecewise linear function in the rightmost interval. The slope RK of the piecewise linear function in the interval with the smallest domain is the same as the slope LK of the piecewise linear function in the leftmost interval. The function values ​​corresponding to these 9 dividing points can be represented as an array YP of the function values ​​of the x-coordinates corresponding to these 9 dividing points in the piecewise linear function. Figure 9 The diagram also shows the x-coordinate Bi corresponding to the left endpoint of the i-th interval and the y-coordinate Pi corresponding to the piecewise linear function, where the slope of the i-th interval is ki. i is an integer greater than or equal to 1 and less than or equal to N.

[0167] In the neural network model of this application embodiment, PWL is used as the activation function. By adjusting multiple trainable parameters of PWL, various function forms can be fitted, which is beneficial for obtaining a better activation function and can effectively improve the fitting ability of the neural network model, thereby improving the model's performance. Furthermore, during model training, multiple trainable parameters of PWL are updated through parameter gradients, reducing runtime overhead and improving the efficiency of activation function construction and model training. This allows for the search of activation functions for different models, obtaining activation functions suitable for different models and improving model performance.

[0168] (7) Model quantization

[0169] Model quantization is one way to improve the inference speed of a model.

[0170] Typically, the full-precision neural network model obtained after training is a floating-point model, and the parameters in the floating-point model include floating-point parameters. Model quantization refers to the process of quantizing the floating-point model to obtain a quantized model. Specifically, by quantizing the data type of the model's parameters from floating-point to integer, the model's computation is correspondingly converted from floating-point computation to integer computation, which can significantly improve computational efficiency. Hardware-wise, computational units can be set up specifically for integer data to achieve efficient inference of the quantized model.

[0171] Taking 8-bit quantization as an example, quantizing a 32-bit floating-point model results in an 8-bit quantized model. The parameters in the 32-bit floating-point model are of 32-bit floating-point data type, while the parameters in the 8-bit quantized model are of 8-bit integer data type. In this case, compared to the 32-bit floating-point model, the 8-bit quantized model reduces memory usage by a factor of four, while also improving inference speed and reducing power consumption.

[0172] (8) Learned step size quantization (LSQ)

[0173] To obtain a quantized model, the floating-point model can be trained using quantization. In this way, the quantized model obtained after quantization training can approximate the accuracy of the floating-point model before quantization to the greatest extent possible, thus ensuring the accuracy of the quantized model.

[0174] LSQ is a typical quantization training method. By inserting quantization and dequantization operations into the floating-point model, the network adapts to quantization computation during training.

[0175] Figure 4 A flowchart of LSQ is shown. During quantization training, the convolution parameters of the current convolutional layer are quantized to obtain quantized convolution parameters. The feature map input to the current convolutional layer is also quantized to obtain a quantized feature map. A convolution operation is performed based on the quantized convolution parameters and the quantized feature map, and an inverse quantization operation is performed on the result of the convolution operation. The result of the inverse quantization operation is processed through an activation function, and the processed result can be used as the input feature map to the next convolutional layer. Figure 4 As shown in (a), the activation function can be ReLU.

[0176] Convolution parameters refer to convolution weights. For example, Figure 4 The quantization operation of the convolution parameters can satisfy the following formula:

[0177]

[0178] in, S represents the quantized convolution weights, W represents the unquantized convolution weights, and S represents the unquantized convolution weights. W This represents the quantized training parameters of the convolution weights, and round() represents the rounding operation.

[0179] For example, Figure 4 The quantization operation of the feature map can satisfy the following formula:

[0180]

[0181] in, X represents the feature map after quantization, and S represents the feature map before quantization. X β and β represent the quantization training parameters of the feature map.

[0182] For example, Figure 4 The convolution operation can satisfy the following formula:

[0183]

[0184] in, It can represent the result after a convolution operation.

[0185] For example, Figure 4 The inverse quantization operation can satisfy the following formula:

[0186]

[0187] in, It can represent the result after the dequantization operation.

[0188] like Figure 4 As shown in Figure (a), the activation function used is ReLU. The output of ReLU has only two cases: y = x and y = 0. x represents the data input to ReLU, and y represents the output data after ReLU processing. That is, ReLU is linear with respect to multiplication operations, i.e., ReLU(x*A) = ReLU(x)*A. Figure 4 As shown in Figure (a), after quantization training is complete, the parameters of the dequantization operation can be fused with the parameters of the feature map quantization operation in the next layer, making the dequantization operation and the feature map quantization operation in the next layer a single operation. After quantization training is complete, the quantized model obtained after quantization training can be deployed for inference. The data type of the parameters of the quantized model is all quantized integer data, reducing memory usage. During the inference process of the quantized model, there is no need to perform dequantization operations, that is, there is no need to perform floating-point calculations. Integer calculations are used, which greatly improves computational efficiency and reduces power consumption.

[0189] like Figure 5As shown, this application embodiment provides a system architecture 100. In Figure 5 In this embodiment, the data acquisition device 160 is used to acquire training data. For example, in the data processing method of this application embodiment, if the data is image data, the training data may include training images and the classification results corresponding to the training images, wherein the classification results of the training images may be manually pre-annotated results.

[0190] After collecting the training data, the data acquisition device 160 stores the training data in the database 130, and the training device 120 trains the target model / rule 101 based on the training data maintained in the database 130.

[0191] The following describes how the training device 120 obtains the target model / rule 101 based on the training data. The training device 120 processes the input raw data and compares the output value with the target value until the difference between the output value of the training device 120 and the target value is less than a certain threshold, thereby completing the training of the target model / rule 101.

[0192] The aforementioned target model / rule 101 can be used to implement the data processing method of this application embodiment. Specifically, the target model / rule 101 in this application embodiment can be a neural network model, such as a convolutional neural network. It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices. Furthermore, it should be noted that the training device 120 may not necessarily train the target model / rule 101 entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0193] The target model / rule 101 trained using training device 120 can be applied to different systems or devices, such as... Figure 5 The execution device 110 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server or cloud service. Figure 5 In this embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. The input data may include data to be processed input by the client device.

[0194] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processes, the execution device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0195] Finally, the I / O interface 112 returns the processing result, such as the data processing result obtained above, to the client device 140, thereby providing it to the user.

[0196] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different objectives or tasks. The corresponding target models / rules 101 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0197] exist Figure 5 In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0198] It is worth noting that, Figure 5 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 5 In this context, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed within the execution device 110.

[0199] like Figure 5As shown, the target model / rule 101 is obtained by training the training device 120. The target model / rule 101 can be the neural network in this application embodiment. Specifically, the neural network constructed in this application embodiment can be CNN, etc.

[0200] Figure 6 The present application provides a hardware structure for a chip, which includes a neural network processor 50. This chip can be configured as follows: Figure 5 The execution device 110 shown is used to perform the calculations of the calculation module 111. This chip can also be located in, for example... Figure 5 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rule 101. The method in this embodiment can be used as follows: Figure 6 This is achieved in the chip shown.

[0201] The Neural Processing Unit (NPU) 50 is mounted as a coprocessor on the main central processing unit (CPU) (host CPU), and tasks are assigned by the host CPU. The core of the NPU is the arithmetic circuit 503, and the controller 504 controls the arithmetic circuit 503 to retrieve data from the memory (weight memory or input memory) and perform calculations.

[0202] In some implementations, the arithmetic circuit 503 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 503 is a two-dimensional pulsating array. The arithmetic circuit 503 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 503 is a general-purpose matrix processor.

[0203] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 502 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 501 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 508.

[0204] The vector computation unit 507 can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, the vector computation unit 507 can be used for network computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization (BN), local response normalization, etc.

[0205] In some implementations, the vector computation unit 507 can store the processed output vector into a unified buffer 506. For example, the vector computation unit 507 can apply a nonlinear function to the output of the arithmetic circuit 503, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 507 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit 503, for example, for use in subsequent layers of a neural network.

[0206] The unified memory 506 is used to store input data and output data.

[0207] The weight data is directly accessed through the memory access controller 505 (DMAC) to move input data from external memory to input memory 501 and / or unified memory 506, store weight data from external memory into weight memory 502, and store data from unified memory 506 into external memory.

[0208] The bus interface unit (BIU) 510 is used to enable interaction between the main CPU, DMAC and instruction fetch memory 509 via a bus.

[0209] The instruction fetch buffer 509, connected to the controller 504, is used to store the instructions used by the controller 504.

[0210] The controller 504 is used to call the instructions cached in the instruction memory 509 to control the operation of the computing accelerator.

[0211] Generally, the unified memory 506, input memory 501, weight memory 502, and instruction fetch memory 509 are all on-chip memories, while the external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0212] The above-mentioned Figure 5 The execution device 110 or Figure 6 The chip in the application is capable of executing the various steps of the data processing method described above. Figure 5 Training equipment 120 or Figure 6 The chip in the application is capable of executing the various steps of the training method for the neural network model of the present application embodiment.

[0213] like Figure 7 As shown, this application embodiment provides a system architecture 300. The system architecture includes a local device 301, a local device 302, an execution device 310, and a data storage system 350, wherein the local devices 301 and 302 are connected to the execution device 310 through a communication network.

[0214] The execution device 310 can be implemented by one or more servers. Optionally, the execution device 310 can be used in conjunction with other computing devices, such as data storage devices, routers, load balancers, etc. The execution device 310 can be deployed on a single physical site or distributed across multiple physical sites. The execution device 310 can use data in the data storage system 350 or call program code in the data storage system 350 to implement the neural network model training method of the embodiments of this application.

[0215] Specifically, in one implementation, the execution device 110 can perform the following process:

[0216] Obtain the neural network model. The activation function of the neural network model includes the piecewise linear function PWL, which includes multiple intervals.

[0217] The neural network model is trained based on the training data, and the target neural network model is obtained based on the trained neural network model. The training data includes: image data, audio data, or text data.

[0218] The target neural network model is processed to obtain a target neural network quantization model. The target neural network quantization model includes multiple sets of fusion parameters, and these multiple sets of fusion parameters have a corresponding relationship with multiple intervals of PWL in the target neural network model.

[0219] The device 110 can acquire a target neural network quantization model through the above process, which can be used for image classification, image processing, audio processing, or text processing, etc.

[0220] Users can interact with execution device 310 by operating their respective user devices (e.g., local device 301 and local device 302). Each local device can represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, smart camera, smart car or other type of cellular phone, media consumption device, wearable device, set-top box, game console, etc.

[0221] Each user's local device can interact with the execution device 310 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0222] In one implementation, local devices 301 and 302 obtain relevant parameters of the target neural network model from execution device 310, deploy the target neural network model on local devices 301 and 302, and use the target neural network model to perform image classification, image processing, audio processing, or text processing, etc.

[0223] In another implementation, the target neural network model can be directly deployed on the execution device 310. The execution device 310 obtains the data to be processed from the local devices 301 and 302 and processes the data using the target neural network model.

[0224] The aforementioned execution device 310 can also be a cloud device, in which case the execution device 310 can be deployed in the cloud; or, the aforementioned execution device 310 can also be a terminal device, in which case the execution device 310 can be deployed on the user terminal side. This application embodiment does not limit this.

[0225] like Figure 8As shown, the PWL model is quantized to obtain the quantized PWL model, which is the target neural network model in this embodiment. The quantized target neural network model can be deployed in the execution device 610 to process the user-provided input data and obtain the processing result. Here, the PWL model refers to a neural network model that uses PWL as the activation function. The quantized target neural network model can be obtained through the method in this embodiment. The execution device 610 can be a terminal device or a cloud device. For example, the execution device 610 can be... Figure 5 The execution device 110 in Figure 7 The execution device 310 or local device, etc.

[0226] Existing quantization training methods are generally designed for neural network models using ReLU as the activation function and cannot be applied to neural network models using more complex activation functions. For neural network models using PWL as the activation function (i.e., PWL models), if existing methods are used for quantization training, only the convolutional parameters and fully connected layer parameters of the resulting quantized model can be quantized; the PWL parameters cannot be quantized. Specifically, in the forward propagation of the quantized model, when using PWL as the activation function, a dequantization operation needs to be performed before the output of the convolutional or fully connected layers is input to the activation function, and then the activation function processes the result of the dequantization operation. For example... Figure 4 As shown in Figure (b), since PWL is not a linear function, the dequantization operation before the activation function cannot be directly fused with the feature map quantization operation of the next layer. Therefore, the parameters of PWL cannot be quantized, and the model cannot perform quantized inference. This affects the model's inference efficiency.

[0227] Therefore, improving the inference efficiency of neural network models that use PWL as the activation function has become an urgent problem to be solved.

[0228] This application proposes a data processing method that can improve the inference efficiency of neural network models that use PWL as the activation function.

[0229] The method provided in this application is described below from the perspectives of model training and model application.

[0230] The neural network model training method provided in this application embodiment involves symbolic and formal intelligent information modeling, extraction, preprocessing, and training of training data (such as image data, audio data, or text data in this application) to ultimately obtain a trained quantized model. Furthermore, the data processing method provided in this application embodiment can utilize the aforementioned trained quantized model to input input data (such as image data, audio data, or text data to be processed in this application) into the trained quantized model to obtain output data. To better describe the data processing method in this application embodiment, the model training method will be explained first below.

[0231] The following is combined with Figures 9 to 11 The training method of the neural network model in the embodiments of this application is described in detail.

[0232] Figure 9 The present application illustrates a method 900 for training a neural network model according to an embodiment of the present application. Figure 9 The method shown can be executed by a training device for a neural network model. This device can be a cloud service device or a terminal device, such as a computer, server, or other device with sufficient computing power to execute the training method for the neural network model. It can also be a system composed of a cloud service device and a terminal device. For example, method 900 can be executed by... Figure 5 Training equipment 120 Figure 6 The neural network processor 50 or Figure 7 The execution device 310 or the local device executes the execution.

[0233] Method 900 can also be understood as a quantitative training method for neural network models.

[0234] Method 900 includes steps S910 to S930. Steps S910 to S930 are described in detail below.

[0235] S910, Obtain a neural network model whose activation function includes PWL, which includes multiple intervals.

[0236] In other words, this neural network model uses PWL as its activation function. A neural network that uses PWL as its activation function can also be called a PWL model.

[0237] A PWL (Plan-Write-Like) can include multiple boundary points, which divide the domain into multiple intervals. The smallest boundary point among these boundary points is the left boundary, and the largest boundary point is the right boundary. The intermediate interval refers to the interval between the left and right boundaries of these multiple intervals.

[0238] For example, if the domain of a PWL is (-∞, +∞) and there are 4 dividing points, these 4 points divide the domain into 5 intervals. The intervals between the left and right boundaries of these 4 dividing points are 3 intervals, which are the middle intervals.

[0239] For example, if the domain of a PWL is (-∞, a], and there are 4 dividing points, the right boundary of these 4 dividing points is a. These 4 points divide the domain into 4 intervals. The left and right boundaries of these 4 dividing points include 3 intervals, which are the middle intervals.

[0240] For example, if the domain of a PWL is [a', a], and there are 4 dividing points, the right boundary of these 4 dividing points is a, and the left boundary is a'. These 4 points divide the domain into 3 intervals. The intervals between the left and right boundaries of these 4 dividing points are the middle intervals.

[0241] A PWL can include one or more intermediate intervals. If a PWL includes multiple intermediate intervals, all of these intermediate intervals must be of the same length. The number of intermediate intervals in a PWL can also be referred to as the number of segments in the PWL.

[0242] The activation functions of this neural network model include one or more PWLs. In other words, at least one activation function in this neural network model is a PWL.

[0243] If the neural network model includes multiple PWLs, the parameters of these multiple PWLs can be the same, meaning they can share parameters. Alternatively, the parameters of these multiple PWLs can also be different, meaning their parameters can be independent of each other. In this case, a change in the parameters of one PWL does not affect the parameters of the other PWLs.

[0244] In this application embodiment, each PWL in the model can be processed in the same way. For ease of description, this application embodiment only uses one PWL for illustration, and does not constitute a limitation on the solution of this application embodiment.

[0245] S920 trains the neural network model based on the training data to obtain the target neural network model.

[0246] Training data can include image data, audio data, or text data.

[0247] The type of training data depends on the model's task. For example, if the model is used for image processing tasks, the training data can be images. Specifically, image processing tasks include image classification, image detection, image segmentation, or image generation. As another example, if the neural network model is used for text processing tasks, the training data can be text. Specifically, text processing tasks include text recognition or text translation. Similarly, if the neural network model is used for audio processing tasks, the training data can be audio data. Specifically, audio processing tasks include audio recognition. This application does not limit the type of training data.

[0248] For example, the training data may be pre-stored. For instance, the training data could be... Figure 5 The training data maintained in the database 130 shown.

[0249] Alternatively, the training data can be user-inputted. For example, method 900 is executed by a device providing AutoML services, and the training data can be user-inputted.

[0250] S930 processes the target neural network model to obtain a quantized target neural network model. This quantized model includes multiple sets of fusion parameters, which correspond to multiple intervals of the PWL (Programmable Logic Scale) in the target neural network model.

[0251] Processing the target neural network model can also be understood as quantizing the target neural network model.

[0252] In other words, the parameters in the target neural network model are quantized. For example, the weight parameters in the target neural network model are quantized by converting floating-point weight parameters into integer weight parameters. These integer weight parameters are the weight parameters in the target neural network model. Specifically, the parameters in the target neural network model can be quantized based on the quantized parameters of the target neural network model.

[0253] Specifically, step S930 includes: fusing the quantization process during the forward propagation of the target neural network model with the PWL calculation to obtain a quantized model of the target neural network. The quantization process includes quantization operations and dequantization operations, etc.

[0254] Alternatively, step S930 includes: fusing the operations of the quantized network layers, the dequantization operation, the calculation of the PWL of the target neural network model, and the quantization operation of the features of the next network layer, which are executed sequentially during the forward propagation of the target neural network model, based on multiple intervals of the PWL, to obtain the quantized model of the target neural network.

[0255] By fusing the parameters from the quantization process with the parameters from PWL, multiple sets of fused parameters can be obtained.

[0256] Optionally, the multiple sets of fusion parameters are obtained based on the parameters of multiple intervals of PWL in the target neural network model, the weight parameters, the weight quantization parameters in the quantization parameters, and the feature quantization parameters in the quantization parameters.

[0257] Alternatively, step S930 includes: obtaining multiple sets of fusion parameters based on the parameters of multiple intervals of PWL in the target neural network model, the weight quantization parameters in the quantization parameters of the neural network model, and the feature quantization parameters in the quantization parameters, in order to obtain the target neural network quantization model.

[0258] Specifically, the multiple sets of fusion parameters include multiple sets of fusion scaling parameters, which are determined based on at least one of the following: parameters of multiple intervals of PWL in the target neural network model, quantization parameters of network layers preceding PWL in the target neural network model, or quantization parameters of network layers following PWL in the target neural network model.

[0259] The multiple sets of fusion parameters include multiple sets of fusion offset parameters, which are determined based on at least one of the following: parameters of multiple intervals of PWL in the target neural network model or quantization parameters of network layers after PWL in the target neural network model.

[0260] For specific calculation methods, please refer to the following text. Figure 11 The description.

[0261] The target neural network quantization model can be used to perform a target task. For example, the target task can be an image processing task, such as object detection, image segmentation, instance segmentation, image denoising, image super-resolution, etc. Alternatively, the target task can be an audio processing task, such as speech recognition, etc. Or, the target task can be a text processing task, such as text recognition or text translation, etc.

[0262] Existing solutions can only quantize the weight parameters in models using PWL as the activation function, but cannot quantize the PWL parameters themselves. Therefore, during inference, the activation function needs to be calculated based on the PWL parameters; in other words, models trained using existing methods cannot achieve quantized inference.

[0263] According to the scheme of the embodiments of this application, a target neural network model is processed to obtain a target neural network quantization model. The target neural network quantization model includes multiple sets of fusion parameters, enabling the target neural network quantization model to process the data to be processed based on multiple sets of fusion parameters corresponding to multiple intervals of PWL, which is equivalent to achieving complete quantization of the target neural network model. Specifically, the multiple sets of fusion parameters are obtained by fusing the quantization process and the PWL calculation process in the target neural network model, so that the processing result can be obtained based on multiple sets of fusion parameters during the inference process without the need to perform additional PWL calculations. This is beneficial for realizing quantitative inference of the model, reducing the amount of computation, and improving processing efficiency. In effect, the target neural network quantization model achieves complete quantization, that is, it not only achieves the quantization of weight parameters, but also the quantization of PWL parameters.

[0264] Furthermore, the target neural network quantization model is obtained by fusing the quantization process and PWL calculation process in the target neural network model. Theoretically, the target neural network quantization model can achieve the same inference accuracy as the target neural network model.

[0265] In one implementation, the neural network model in step S920 can be a full-precision neural network model.

[0266] A full-precision neural network model refers to a model trained using floating-point type (usually single-precision) parameters. In other words, a full-precision neural network model is a pre-trained model. A full-precision neural network model is a floating-point model. A floating-point model is a model whose parameters are of floating-point type. That is to say, by training an initial neural network model using floating-point type data, a full-precision neural network model can be obtained.

[0267] In this case, step S920 may include: inserting quantization and dequantization operations into the full-precision neural network model, performing quantization training, and obtaining the quantized neural network model and quantization parameters. The quantized neural network model is then used as the target neural network model.

[0268] Specifically, during quantization training, the weight parameters and quantization training parameters in the full-precision neural network model are adjusted based on the training data. Quantization training parameters are the parameters involved in the quantization and dequantization operations during quantization training. Specifically, quantization training parameters include the model's weight quantization training parameters and feature quantization training parameters. During quantization training, the weight parameters in the full-precision neural network model are quantized based on the quantization training parameters, and the input features are quantized based on the feature quantization training parameters. Furthermore, dequantization is performed on the data requiring dequantization based on the quantization training parameters. The quantization training parameters obtained after quantization training are the quantization parameters.

[0269] For example, step S920 can employ existing methods for quantization training. For instance, a full-precision neural network model can be quantized using quantization-aware training (QAT) or LSQ, based on a uniform quantization scheme, to obtain the quantized neural network model and its quantization parameters.

[0270] In another implementation, the neural network model in step S920 is the quantized and trained neural network model obtained in the aforementioned implementation.

[0271] In other words, this neural network model is obtained by inserting quantization and dequantization operations into a full-precision neural network model and then performing quantization training. The quantization parameters of this neural network model are also obtained during this process.

[0272] For example, obtaining a neural network model can be achieved by receiving the model from another device. That is, the neural network model can be obtained by quantizing and training a full-precision neural network model using another device. Alternatively, obtaining a neural network model can also be achieved by quantizing and training a full-precision neural network model. This application does not limit the method of obtaining the neural network model.

[0273] Quantization parameters include weight quantization parameters and feature quantization parameters. Weight quantization parameters are used to quantize the weights in the model, obtaining quantized weights. Feature quantization parameters are used to quantize the input features, obtaining quantized features.

[0274] The weight quantization parameters of different network layers in a neural network model can be the same or different. Similarly, the feature quantization parameters of different network layers in a model can be the same or different.

[0275] For example, the weight quantization parameter of network layer 1# is used to quantize the weight parameters of network layer 1#. The feature quantization parameter of network layer 1# is used to quantize the input features of network layer 1#.

[0276] Weight parameters can include the weight parameters of convolutional layers and / or the weight parameters of fully connected layers.

[0277] In other words, if the neural network model includes convolutional layers, then the weight parameters include the parameters of the convolutional layers.

[0278] If the neural network model includes fully connected layers, then the weight parameters include the parameters of the fully connected layers.

[0279] Optionally, step S920 includes steps S921 to S922.

[0280] S921, the neural network model is trained based on the training data.

[0281] S922, adjust the length of the middle interval of PWL in the trained neural network model to obtain the adjusted target neural network model.

[0282] The target neural network model can be obtained based on the adjusted target neural network model.

[0283] For example, the adjusted target network model is used as the target neural network model.

[0284] It should be noted that during the execution of steps S921 to S922, the quantization parameters remain unchanged, and the quantization parameters of the neural network model and the target neural network model are essentially the same.

[0285] In one possible implementation, the execution order of steps S921 and S922 can be interchanged, that is:

[0286] Adjust the length of the middle interval of PWL in the neural network model; train the adjusted neural network model based on the training data.

[0287] Furthermore, the adjusted neural network model obtained in step S922 is used as the neural network model in step S921, and steps S921 to S922 are repeated until the target neural network model is obtained.

[0288] According to the solution of this application embodiment, the length of the intermediate interval can be adjusted to obtain the required length of the intermediate interval, thereby improving the flexibility of the model; at the same time, the model is trained to ensure the accuracy of the model.

[0289] It should be noted that, due to the different execution order of the steps, the model in step S921 can be the neural network model adjusted in S922, or it can be the neural network model obtained in step S910. The following description will only use the example of the model in step S921 being the neural network model obtained in step S910, and will not limit the solution of the embodiments of this application.

[0290] The neural network model is trained using training data, which means adjusting the parameters of the neural network model based on the training data. Specifically, during training, forward propagation of the neural network model is performed based on the training data to obtain the loss function value, and then back propagation is performed based on the loss function value to calculate the gradient of the model parameters, and the model parameters are adjusted according to the gradient of the model parameters.

[0291] For example, model parameters may include model weight parameters and PWL parameters.

[0292] For example, the parameters of PWL can include at least one of the following: right boundary, left boundary, slope on the interval with the largest domain, slope on the interval with the smallest domain, or function value corresponding to the boundary point.

[0293] For example, such as Figure 11 As shown, during the forward propagation of the neural network model, the weight parameters and input features of the current network layer are quantized based on the quantization parameters. The operation of the current network layer, such as convolution, is performed based on the quantized weight parameters and quantized input features. The operation result of the current network layer is dequantized, the activation function value of the dequantization operation result is calculated, and the activation function value is quantized and used as the quantized input feature of the next network layer.

[0294] The number of iterations during training can be one or multiple.

[0295] Optionally, step S922 includes steps S1 to S3.

[0296] It should be noted that, due to the different execution order of the steps, the model in step S922, or in other words, steps S1 to S3, can be the neural network model trained in S921, or it can be the neural network model obtained in step S910. The following description will only use the trained neural network model in step S922 as an example, and will not limit the solution of the embodiments of this application.

[0297] S1, calculates the equivalent length of the middle interval of PWL based on the quantization parameters of the neural network model.

[0298] For a PWL (Precision Written Line), calculating the equivalent length of the middle interval of the PWL based on the quantization parameters refers to calculating the equivalent length of the middle interval of the PWL based on the quantization parameters of the target network layer. There is a correspondence between the target network layer and the PWL. In other words, the PWL is the PWL corresponding to the target network layer. The PWL can be located within the target network layer, or it can be located after the target network layer, with the target network layer adjacent to the PWL in the trained neural network model.

[0299] If the output of a network layer is input into a PWL (Power-Write Layer), then that network layer is the network layer adjacent to the previous PWL.

[0300] For example, the output of convolutional layer 1# is input into an activation function (PWL) for processing, and the result of the activation function is output to convolutional layer 2#. In this case, the equivalent length of the middle interval of the PWL is calculated based on the weight quantization parameters and feature quantization parameters of convolutional layer 1#.

[0301] Specifically, the equivalent length of the intermediate interval is obtained by dividing the length of the intermediate interval by the quantization parameter.

[0302] In other words, the equivalent length of the intermediate interval is determined based on parameters and quantization parameters related to the length of the intermediate interval.

[0303] For example, the equivalent length of the intermediate interval Satisfy the following formula:

[0304]

[0305] Where d represents the length of the intermediate interval, S W S represents the weight quantization parameter of the target network layer. X This represents the feature quantization parameters of the target network layer.

[0306] The length d of the intermediate interval satisfies the following formula:

[0307]

[0308] Where RB represents the right boundary of the PWL, LB represents the left boundary of the PWL, and N represents the number of intermediate intervals of the PWL.

[0309] Therefore, the equivalent length of the intermediate interval The following formula can be satisfied:

[0310]

[0311] S2, determine the target scaling factor of PWL based on the equivalent length of the intermediate interval.

[0312] Specifically, the target scaling factor is determined based on the equivalent length of the intermediate interval and the target length.

[0313] The target length can be set as needed.

[0314] Optionally, the target length is the nearest power of 2 to the equivalent length of the intermediate interval.

[0315] The nearest power of 2 that is closest to the equivalent length of the middle interval satisfies the following formula:

[0316]

[0317] Where d' represents the nearest power of 2 that is closest to the equivalent length of the middle interval. This indicates the floor operation.

[0318] Optionally, the target scaling factor is a scaling factor. The scaling factor is determined based on the equivalent length of the intermediate interval and the nearest power of 2 that is closest to the equivalent length of the intermediate interval.

[0319] For example, the scaling factor is the quotient of the power of 2 closest to the equivalent length of the middle interval and the equivalent length of the middle interval. That is, the scaling factor is obtained by dividing the power of 2 closest to the equivalent length of the middle interval by the equivalent length of the middle interval.

[0320] For example, the scaling factor can satisfy the following formula:

[0321]

[0322] Where s represents the scaling factor.

[0323] For example, the scaling factor is the quotient of the equivalent length of the middle interval and the nearest power of 2 that is closest to the equivalent length of the middle interval. That is, the scaling factor is obtained by dividing the equivalent length of the middle interval by the nearest power of 2 that is closest to the equivalent length of the middle interval.

[0324] Optionally, if the scaling factor is within the target range, the target scaling factor is the scaling factor itself; if the scaling factor is outside the target range, the target scaling factor is the boundary value of the target range.

[0325] In other words, the target scaling factor is within the target range. The target range can be set as needed.

[0326] For example, when the scaling factor is less than the first boundary value and greater than the second boundary value, the target scaling factor is the scaling factor; when the scaling factor is greater than or equal to the first boundary value, the target scaling factor is the first boundary value; when the scaling factor is less than or equal to the second boundary value, the target scaling factor is the second boundary value, and the first boundary value is greater than the second boundary value.

[0327] For example, by constraining the scaling factor, a target scaling factor can be obtained. The target scaling factor can satisfy the following formula:

[0328] s' = clip(s, low, high);

[0329] Where s' represents the target scaling factor, low represents the second boundary value, high represents the first boundary value, and clip() returns the value of low when s < low, the value of high when s > high, and the value of s otherwise. low and high are hyperparameters used to constrain the range of the target scaling factor, i.e., to constrain the target scaling factor within the target range. The values ​​of high and low can be chosen to be close to 1. For example, high = 1.01, low = 0.99. This application does not limit this.

[0330] Neural network models typically include two types of parameters. One type can be learned and estimated from the training data; these are the trainable parameters, such as the weights of the neural network model. The other type of parameter is usually not obtained from the training data; these are the hyperparameters. Hyperparameters can be manually set or obtained through hyperparameter search using AutoML.

[0331] Trainable parameters can be updated based on training data during the training of a neural network model. Hyperparameters remain unchanged during the iterative training of the neural network model.

[0332] Alternatively, when the scaling factor is less than the first boundary value, the target scaling factor is the scaling factor; when the scaling factor is greater than or equal to the first boundary value, the target scaling factor is the first boundary value.

[0333] Alternatively, when the scaling factor is greater than the second boundary value, the target scaling factor is the scaling factor; when the scaling factor is less than or equal to the second boundary value, the target scaling factor is the second boundary value.

[0334] S3, adjusts the parameters of the target network layer in the trained neural network model and the length of the intermediate interval of PWL based on the target scaling factor of PWL in the trained neural network model.

[0335] The PWL in the trained neural network is the PWL corresponding to the target network layer. The PWL in the trained neural network model can be located within the target network layer, or the PWL in the trained neural network model can be located after the target network layer, with the target network layer adjacent to the PWL in the trained neural network model.

[0336] Adjusting the parameters of the target network layer and the length of the intermediate interval of the PWL based on the target scaling factor can also be understood as scaling the parameters of the target network layer and the length of the intermediate interval of the PWL based on the target scaling factor.

[0337] For example, scaling the length of the middle interval of PWL includes scaling the left and right boundaries of PWL to obtain a scaled neural network model.

[0338] The specific scaling method depends on how the target scaling factor is calculated.

[0339] The following example illustrates step S3 using the scaling factor as the quotient of the equivalent length of the middle interval and the nearest power of 2 that is closest to the equivalent length of the middle interval.

[0340] In this case, scaling the parameters of the target network layer involves dividing the parameters of the target network layer by a target scaling factor to obtain the scaled parameters of the network layer.

[0341] For example, the parameters of the scaled network layer satisfy the following formula:

[0342]

[0343]

[0344] Where w' represents the weight parameters of the network layer after scaling, w represents the weight parameters of the network layer before scaling, b' represents the bias parameters of the network layer after scaling, and b represents the bias parameters of the network layer before scaling.

[0345] The left and right boundaries of the PWL are scaled by multiplying the left and right boundaries of the PWL by the target scaling factor to obtain the scaled left and right boundaries of the PWL.

[0346] For example, the left and right boundaries of the scaled PWL can satisfy the following formula:

[0347] LB' = LB·s';

[0348] RB' = RB·s';

[0349] Where LB' represents the left boundary of the scaled PWL, LB represents the left boundary of the unscaled PWL, RB' represents the right boundary of the scaled PWL, and RB represents the right boundary of the unscaled PWL.

[0350] If the scaling factor is the quotient of the equivalent length scaling factor of the middle interval and the nearest power of 2 to the equivalent length of the middle interval, scaling the parameters of the target network layer includes multiplying the parameters of the target network layer by the target scaling factor. Scaling the left and right boundaries of the PWL includes dividing the left and right boundaries of the PWL by the target scaling factor to obtain the scaled left and right boundaries of the PWL.

[0351] As mentioned earlier, in one implementation, the scaled neural network model obtained in step S922 can be used as the neural network model in step S921. Steps S921 to S922 are repeated until training is complete, or in other words, until the training termination condition is met. After training is completed, the trained neural network model is obtained, which is the target neural network model.

[0352] The training process in step S921 can be iterated T times. In each iteration, forward propagation, back propagation and parameter update are performed based on the model parameters adjusted after the previous iteration.

[0353] In other words, the model obtained after T iterations can be used as the model in step S922.

[0354] T is a positive integer. T is a hyperparameter, and its specific value can be set as needed. For example, the order of magnitude of T can be hundreds or thousands. For example, T = 300.

[0355] Training termination conditions can be set as needed.

[0356] For example, the training termination condition includes the number of iterations being greater than or equal to the target number of iterations. When the current number of iterations is greater than or equal to the target number of iterations, training is terminated, and the target neural network model is obtained.

[0357] Alternatively, training termination conditions include the model's accuracy being greater than or equal to the target accuracy. Training terminates when the current model's accuracy is greater than or equal to the target accuracy, resulting in the target neural network model.

[0358] Alternatively, the training termination condition includes the change in the model's accuracy obtained through multiple consecutive iterations being less than or equal to the target change. In other words, training is terminated once the model's accuracy stabilizes, yielding the target neural network model.

[0359] The determination of whether the training termination condition is met can be performed during step S921, after step S921, or after step S922. This application embodiment does not limit this.

[0360] For example, after executing step S921, it is determined whether the training termination condition is met. If the training termination condition is met, the current model is output as the target neural network model. If the training termination condition is not met, step S922 is executed.

[0361] Alternatively, after each iteration of step S921, determine whether the training termination condition is met. If the training termination condition is met, output the current model as the target neural network model. If the training termination condition is not met, continue iterating until T iterations are reached, and then execute step S922.

[0362] Alternatively, after executing step S922, determine whether the training termination condition is met. If the training termination condition is met, output the current model as the target neural network model. If the training termination condition is not met, continue executing step S921.

[0363] According to the scheme of this application embodiment, by training the neural network model and continuously adjusting the boundaries and weights of PWL, the equivalent length of the middle interval of PWL is made to continuously approach an integer power of 2. Ultimately, the equivalent length of the middle interval of PWL is constrained to an integer power of 2, thus achieving the quantization of PWL parameters. In this way, the index of the target interval can be obtained through shifting during the subsequent quantization inference process of the model. It should be noted that in this application embodiment, "shifting" can also be called "displacement". Specifically, shifting data one position to the right is equivalent to dividing by 2. If the equivalent length of the middle interval is 2... n When calculating the index of the target interval, it is necessary to perform a division by 2. n The operation can be performed by shifting by n bits, avoiding the need for a divider. This eliminates the need for a divider in the hardware, reducing costs. Furthermore, compared to a divider, using shifting to obtain the index of the target interval reduces computational load, lowers power consumption, and improves inference efficiency. The model was also trained to ensure its accuracy.

[0364] Figure 10 This application illustrates a neural network model training method 1000 provided in an embodiment of the present application. Method 1000 can be regarded as a specific implementation of step S920. For a detailed description, please refer to the description in step S920. To avoid repetition, some descriptions are omitted appropriately when describing method 1000.

[0365] Method 1000 includes steps S1010 to S1030.

[0366] S1010 trains the initial PWL model based on the training data to obtain a full-precision PWL model.

[0367] The activation function of the initial PWL model includes PWL.

[0368] Specifically, the training device can acquire the model structure and training data of the initial PWL model and execute the training process to obtain a full-precision PWL model. For example, this full-precision PWL model can be a 32-bit floating-point model.

[0369] Step S1010 is an optional step; for example, the training device can receive a full-precision PWL model from another device.

[0370] S1020 performs quantization training on full-precision PWL models.

[0371] The model obtained in step S1020 can be used as the neural network model in step S920.

[0372] For example, the full-precision PWL model can be trained using quantization training methods based on uniform quantization schemes such as QAT or LSQ.

[0373] Step S1020 is an optional step. For example, the training device can receive the quantized PWL model from other devices.

[0374] S1030, fixed quantization parameter S X S W and β.

[0375] During the quantization training process in step S1020, the quantization parameters and model parameters are adjusted, while in steps S1030 to S1090, the quantization parameters remain unchanged.

[0376] S1040, Train the current model.

[0377] Specifically, forward propagation, back propagation, and parameter updates are performed on the current model based on the training data.

[0378] During training, every T iterations, steps S1050 to S1090 are performed for each PWL in the model. T is a positive integer. For example, T is 300.

[0379] S1050, calculate the equivalent length of the intermediate interval.

[0380] Equivalent length of the middle interval Satisfy the following formula:

[0381]

[0382] S1060, calculate the nearest power of 2 that is closest to the equivalent length of the intermediate interval, and calculate the scaling factor.

[0383] The scaling factor satisfies the following formula:

[0384]

[0385] d' represents the nearest power of 2 to the equivalent length of the middle interval, and s represents the scaling factor.

[0386] S1070, constrain the scaling factor to obtain the target scaling factor.

[0387] The target scaling factor s' satisfies the following formula:

[0388] s' = clip(s, low, high);

[0389] The values ​​for high and low can be chosen to be close to 1. For example, high = 1.01, low = 0.99.

[0390] S1080 scales the parameters of the target network layer.

[0391] The parameters of the scaled network layer satisfy the following formula:

[0392]

[0393]

[0394] Where w' represents the weight parameters of the network layer after scaling, w represents the weight parameters of the network layer before scaling, b' represents the bias parameters of the network layer after scaling, and b represents the bias parameters of the network layer before scaling.

[0395] S1090 scales the boundaries of the PWL.

[0396] The left and right boundaries of the scaled PWL can satisfy the following formula:

[0397] LB' = LB·s';

[0398] RB' = RB·s';

[0399] Where LB' represents the left boundary of the scaled PWL, LB represents the left boundary of the unscaled PWL, RB' represents the right boundary of the scaled PWL, and RB represents the right boundary of the unscaled PWL.

[0400] Determine if the training termination condition has been met. If the training termination condition has been met, the training is completed, and the target neural network model and quantization parameters are obtained. If the training termination condition has not been met, return to step S1040.

[0401] For example, training termination conditions may include reaching the target number of iterations in the current iteration count.

[0402] like Figure 10 As shown, determine whether the current iteration count has reached the target iteration count. If the target iteration count has been reached, the training is complete, and the target neural network model and quantization parameters are obtained. Otherwise, return to continue executing step S1040.

[0403] It should be understood that Figure 10 The training termination condition in the example is only for reference. Other training termination conditions can also be set. For details, please refer to the description in step S920. It will not be repeated here.

[0404] According to the scheme of this application embodiment, the quantized weights are obtained by quantizing and training a full-precision PWL model. Then, the boundaries of the PWL and the weight parameters of the model are continuously adjusted so that the equivalent length of the middle interval of the PWL continuously approaches an integer power of 2. Finally, the equivalent length of the middle interval of the PWL is constrained to an integer power of 2, thus realizing the quantization of the PWL parameters. In this way, the index of the middle interval can be obtained by displacement during the subsequent quantization inference process of the model, avoiding division operations. There is no need to set up a divider in the hardware, which can reduce costs. At the same time, compared with a divider, using displacement to obtain the index of the target interval can reduce the amount of computation, reduce power consumption, and improve inference efficiency.

[0405] Figure 11 A schematic flowchart illustrating the forward propagation process of a PWL model's quantization training is shown. Figure 11 This example only uses a convolutional layer as an example and does not limit the type of network layer in the PWL model. Figure 11 The PWL model in this application can be the target neural network model obtained in step S920 or the model trained by method 1000, or it can be a model trained by other training methods. This application does not limit this.

[0406] like Figure 11 As shown, the forward propagation process of the current network layer includes the following steps.

[0407] (1) Quantize the input feature X of the current network layer to obtain the quantized input feature map of the current network layer. The weight parameters W of the current network layer are quantized to obtain the quantized weight parameters of the current network layer. The current network layer in Figure 11 is a convolutional layer, and W can also be called the convolution weight parameter.

[0408] For example, the quantized input feature map of the current network layer The following formula can be satisfied:

[0409]

[0410] S X The feature quantization parameters for the current network layer can be determined using existing quantization training methods, such as LSQ. round() means calculating the integer closest to the value in parentheses.

[0411] Alternatively, the quantized input features of the current network layer The following formula can be satisfied:

[0412]

[0413] Wherein, β and S X The feature quantization parameters for the current network layer can be determined using existing quantization training methods, such as LSQ. For ease of description, this embodiment only uses this method as an example to illustrate the quantization inference process. If β is not set during the quantization process, the value of β in the following text can be set to 0.

[0414] For example, the quantized weight parameters of the current network layer The following formula can be satisfied:

[0415]

[0416] Among them, S W The weight quantization parameters for the current network layer can be determined using existing quantization training methods, such as LSQ.

[0417] (2) Based on the quantized weight parameters W and quantized input features of the current network layer Perform a convolution operation to obtain the result of the convolution operation.

[0418] For example, the result of a convolution operation Satisfy the following formula:

[0419]

[0420] It should be understood that in the embodiments of this application, * represents matrix multiplication or convolution operation. For example, if the current network layer is a convolutional layer, then * represents convolution operation; if the current network layer is a fully connected layer, then * represents matrix multiplication operation.

[0421] (3) Perform dequantization and convolution bias operations on the result of the convolution operation to obtain the operation result.

[0422] For example, the operation result Satisfy the following formula:

[0423]

[0424] Where b represents the bias of the current network layer. If the current network layer is a convolutional layer, then b is the bias of the convolution.

[0425] It should be noted that if the current network layer does not have a convolution bias set, step (3) may include: performing an inverse quantization operation on the result of the convolution operation to obtain the operation result.

[0426] (4) Apply the PWL activation function to the operation result obtained in step (3). The processing is performed, and the result becomes the input feature Z of the next network layer.

[0427] For example, the input features Z of the next network layer can satisfy the following formula:

[0428]

[0429] Among them, B i K represents the left endpoint of the i-th interval in PWL. i Let P represent the slope of the i-th interval in PWL. i This represents the function value corresponding to the left endpoint of the i-th interval in PWL.

[0430] It should be noted that this formula is only an example; other formulas can also be used to obtain the PWL processing result, for example, by using K... i This is expressed as the result of calculations using other parameters in PWL. The embodiments in this application are merely illustrative examples of the above-described formulas and do not constitute a limitation on the solutions presented in this application.

[0431] (5) Quantize the input features Z of the next network layer to obtain the quantized input features of the next network layer.

[0432] For example, the quantized input features of the next network layer The following formula can be satisfied:

[0433]

[0434] S ZThe quantization parameters for the features of the next network layer can be determined using existing quantization training methods, such as LSQ.

[0435] Alternatively, the quantized input features of the next network layer The following formula can be satisfied:

[0436]

[0437] Wherein, γ and S Z The feature quantization parameters for the next network layer can be determined using existing quantization training methods, such as LSQ. For ease of description, this embodiment only uses this method as an example to illustrate the quantization inference process. If γ is not set during the quantization process, the value of γ in the following text can be set to 0.

[0438] like Figure 11 As shown, during the forward propagation process, before the output of the network layer is input into PWL, the output of the network layer needs to be dequantized. Then, PWL processes the result of the dequantization operation. This process requires a lot of floating-point operations and has a large computational load.

[0439] Figure 11 The convolution, dequantization, and quantization operations in PWL are all linear operations, and PWL is a linear operation across all intervals. Therefore, in this embodiment, the weight parameters in the quantized PWL model are quantized based on the weight quantization parameters, and the operations of the quantized network layers, dequantization operations, PWL calculations, and the quantization operations of the features of the next network layer are sequentially executed during the forward propagation of the target neural network model to obtain the quantized model of the target neural network. Alternatively, it can be understood as fusing the weight parameters, quantization parameters, PWL parameters, and quantization parameters of the next network layer based on multiple intervals of PWL to obtain the quantized model of the target neural network.

[0440] The following is about Figure 11 The process of performing operations on the quantized network layers, dequantization, PWL calculation, and fusion of the quantization operations of the features of the next network layer during the forward propagation of the target neural network model is explained.

[0441] As shown earlier, the quantized input features of the next network layer The following formula can be satisfied:

[0442]

[0443] Therefore, the quantized input feature map of the next network layer can be obtained. The following formula can be satisfied:

[0444]

[0445] Where bias represents the bias after fusion, and scale i The offset represents the blending scaling parameter for the i-th interval. i This represents the fusion offset parameter for the i-th interval.

[0446] Wherein, bias satisfies the following formula:

[0447]

[0448] scale i Satisfy the following formula:

[0449]

[0450] offset i Satisfy the following formula:

[0451]

[0452] The method for determining the range to which the data input into PWL belongs can be set as needed.

[0453] For example, the data input into the PWL is compared with the values ​​of each boundary point of the PWL, and the interval to which the data input into the PWL belongs is determined based on the comparison results.

[0454] Specifically, the interval to which the data input into the PWL belongs is the interval between the two boundary points that are closest to the data input into the PWL.

[0455] Alternatively, when the length of the middle intervals of the PWL is the same, it can be determined by the distance between the data input into the PWL and the left boundary. Or, it can be determined by the distance between the data input into the PWL and the right boundary.

[0456] For example, the index i of the interval to which the data input into PWL belongs satisfies the following formula:

[0457]

[0458] Therefore, the interval number i to which the data input into PWL belongs satisfies the following formula:

[0459]

[0460] in, Satisfy the following formula:

[0461]

[0462] Furthermore, the equivalent length of the middle interval in the trained PWL model is an integer power of 2. Thus, when calculating i, the division by the equivalent length of the middle interval can be implemented using a shift operation, avoiding the need for a divider. This eliminates the need for a divider in the hardware, reducing costs. Simultaneously, compared to a divider, using shifting to obtain the index of the target interval reduces computational load, lowers power consumption, and improves inference efficiency.

[0463] The bias and scale in the embodiments of this application i and offset i It is pre-stored before the start of quantization inference. During the quantization inference process, the corresponding parameters can be selected according to the index i of the target interval without the need to execute the PWL calculation process, which reduces the amount of computation, realizes the complete quantization inference of the PWL model, and improves the inference efficiency.

[0464] It should be understood that the above derivation process is only illustrated with the example of a convolutional layer in the current network. If the current network layer is a fully connected layer, the convolution operation in the above derivation process can be replaced with matrix multiplication operation.

[0465] By fusing the quantization, dequantization, and PWL calculations in the target neural network model, or in other words, fusing the weight parameters, quantization parameters, PWL parameters, and quantization parameters of the next network layer, a fused quantization parameter is obtained, which is equivalent to quantizing the PWL parameters. This fusion of quantization, dequantization, and PWL calculations integrates the PWL calculation process into the quantization process during the inference of the PWL quantization model, eliminating the need for additional PWL calculations, thus simplifying hardware implementation and reducing hardware costs. Furthermore, the parameters in the target neural network quantization model are all derived from the parameters in the target neural network model, theoretically enabling the target neural network model to achieve the same accuracy as the target neural network model.

[0466] The following is combined Figures 12 to 13 The data processing methods in the embodiments of this application are described in detail.

[0467] Figure 12 The data processing method 1200 provided in the embodiments of this application is shown. Figure 12 The method shown can be executed by an execution device of a neural network model. This device can be a cloud service device or a terminal device, such as a computer, server, vehicle, drone, or robot, with sufficient computing power to execute the data processing method. It can also be a system composed of cloud service devices and terminal devices. For example, method 1200 can be executed by... Figure 5 Execution device 110 in Figure 6 The neural network processor 50 or Figure 7 The execution is performed by the execution device 310 or a local device. Method 1200 can be implemented by software or by hardware.

[0468] For example, the solutions of this application embodiment can be built into a computing device, which can be a cloud service device or a terminal device. For instance, the solutions of this application embodiment can be built into a terminal, that is, the terminal executes the methods of this application embodiment. Alternatively, the solutions of this application embodiment can be deployed in an application (APP) on a computing device, for example, deployed in an APP on a terminal, that is, the APP calls and executes the methods of this application embodiment.

[0469] In the embodiments of this application, method 1200 can also be understood as a quantitative inference method for a neural network model.

[0470] Method 1200 includes steps S1210 to S1220. Steps S1210 to S1220 are described in detail below.

[0471] S1210, Obtain the data to be processed. The data to be processed includes image data, audio data, or text data, etc.

[0472] The type of data to be processed is related to the task of the neural network model. For example, if the neural network model is used for image processing tasks, the data to be processed can be images. Specifically, image processing tasks include image classification, image detection, image segmentation, image recognition, or image generation. As another example, if the neural network model is used for text processing tasks, the data to be processed can be text. Specifically, text processing tasks include text recognition or text translation. Similarly, if the neural network model is used for audio processing tasks, the data to be processed can be audio data. Specifically, audio processing tasks include speech recognition. The embodiments of this application do not limit the type of data to be processed.

[0473] For example, the data to be processed is an image. The image to be processed can be an image captured by a camera of a terminal device (or a computer, server, or other device or equipment), or the image to be processed can be an image obtained from within the terminal device (or a computer, server, or other device or equipment) (e.g., an image stored in the photo album of the terminal device, or an image obtained by the terminal device from the cloud). This application embodiment does not limit this.

[0474] S1220: The target neural network quantization model is used to process the data to be processed. The target neural network quantization model includes multiple sets of fusion parameters. The target neural network quantization model is obtained by quantizing the target neural network model. The activation function of the target neural network model includes PWL, which includes multiple intervals. There is a correspondence between the multiple sets of fusion parameters and the multiple intervals.

[0475] Specifically, step S1220 includes: determining target indication information based on the data to be processed; obtaining target fusion parameters corresponding to the target indication information from multiple sets of fusion parameters based on the target indication information; and processing the data to be processed based on the target fusion parameters.

[0476] The activation function of the target neural network model includes one or more PWLs (Programmable Wires). Each PWL comprises multiple intervals. In other words, one PWL corresponds to multiple sets of fusion parameters. These multiple sets of fusion parameters are calculated based on the parameters of the multiple intervals of the PWL.

[0477] The target neural network model can be obtained through quantization training. That is, quantization and dequantization operations are inserted into the model for quantization training to improve the performance of the quantized model.

[0478] For example, the target neural network model may be obtained by method 900 or method 1000. Alternatively, the target neural network model may also be trained by other methods.

[0479] For example, the target indication information can be an index of a target interval; that is, the target indication information can be used to indicate a target interval. This target interval belongs to multiple intervals of a PWL.

[0480] The target indication information is determined based on the data to be processed. This can be understood as either being determined by the data itself, or by the processed data. For example, the data to be processed may undergo processing at one or more network layers, and the target indication information may be determined based on the result.

[0481] Optionally, determining target indication information based on the data to be processed includes: processing the input features of the first network layer based on the weight parameters of the first network layer in the target neural network quantization model; and determining the target indication information based on the processing result of the first network layer. The input features of the first network layer are determined based on the data to be processed.

[0482] Multiple sets of fusion parameters can be pre-stored. For example, these multiple sets of fusion parameters can be stored in a buffer, and during quantization inference, the corresponding fusion parameters can be retrieved from the buffer based on the target indication information.

[0483] According to the scheme of this application embodiment, the target neural network quantization model can be processed based on multiple sets of fusion parameters corresponding to multiple intervals of PWL, realizing the quantization inference of the model. Specifically, the multiple sets of fusion parameters are obtained by fusing the quantization process and the PWL calculation process in the target neural network model. During the inference process, the processing result of the model can be obtained based on the multiple sets of fusion parameters without additional PWL calculation, thus realizing the quantization inference of the model, reducing the amount of computation, and improving the processing efficiency. In effect, the target neural network quantization model achieves complete quantization, that is, it not only realizes the quantization of weight parameters, but also the quantization of PWL parameters. If method 1200 is implemented in hardware, it can reduce the amount of computation, reduce power consumption, and improve inference efficiency.

[0484] Compared to quantization inference using ReLU as the activation function, quantization inference using PWL as the activation function only requires additional determination of target indication information and calculation of the corresponding fusion parameters based on that information. Furthermore, using PWL as the activation function improves model performance. In other words, using PWL as the activation function can enhance model performance without introducing excessive computation.

[0485] Optionally, the multiple sets of fusion parameters are obtained by fusing the quantized network layer operations, dequantization operations, PWL calculations, and feature quantization operations of the next network layer, which are sequentially performed during the forward propagation of the target neural network model, based on these multiple intervals.

[0486] For example, a network layer may include a convolutional layer or a fully connected layer, etc.

[0487] For specific integration methods, please refer to... Figure 11 The relevant descriptions will not be repeated here.

[0488] For example, the target neural network model is obtained through quantization training. During the forward propagation of the target neural network model, the operations of the quantized network layer 1#, the dequantization operation, the calculation of the PWL corresponding to network layer 1#, and the quantization operation of the features of the next network layer after network layer 1# need to be executed sequentially. For example, Figure 11 As shown. The above operations are fused together, or in other words, the parameters of the above operations are fused together to obtain multiple sets of fused parameters. These multiple sets of fused parameters correspond one-to-one with multiple intervals of the PWL.

[0489] During the forward propagation of the target neural network model, after performing the operation of the quantized network layer 1#, the dequantization operation is performed. The target interval of PWL is determined based on the result of the dequantization operation, that is, which interval in the current PWL domain the result of the dequantization operation falls into. Then, the PWL calculation is performed, and the calculation result of PWL is used as the input feature of the next network layer. The quantization operation of the feature of the next network layer is then performed on the input feature.

[0490] During the inference process of the target neural network quantization model, after performing the operation of the quantized network layer 1#, the target indication information is determined based on the operation result. This target indication information indicates the index of the target interval. Then, based on this target indication information, the target fusion parameter is obtained from the multiple sets of fusion parameters, and processing is performed based on this target fusion parameter. In other words, the target fusion parameter is the fusion parameter corresponding to the target interval. Theoretically, for the same data to be processed, the processing result of the target neural network quantization model is consistent with the processing result of the target neural network model.

[0491] In the embodiments of this application, the target neural network quantization model is obtained by fusing the quantization operations, dequantization operations, and PWL calculation process in the inference process of the target neural network model. In other words, the inference process of the target neural network quantization model is equivalent to fusing the quantization process and PWL calculation process in the target neural network model, reducing computational load and improving processing efficiency. It also simplifies hardware implementation and reduces implementation costs. Furthermore, the multiple sets of fusion parameters of the target neural network quantization model are achieved by fusing the operations in the target neural network model. Theoretically, the inference accuracy of the target neural network quantization model is consistent with the accuracy of the target neural network model. If the target neural network model is obtained by quantizing and training a full-precision neural network model, then the target neural network quantization model can achieve inference accuracy similar to that of the full-precision neural network model.

[0492] In one possible implementation, step S1220 includes steps S1221 to S1225. Figure 12 Not shown in the image.

[0493] S1221, The features input to the first network layer are processed based on the weight parameters of the first network layer in the target neural network quantization model.

[0494] The weight parameters of the first network layer in the target neural network quantization model are obtained by quantizing the weight parameters of the second network layer in the same model. The second network layer corresponds to the first network layer in the target neural network quantization model, and the input features of the first network layer are the quantized features. The input features of the first network layer are determined based on the data to be processed.

[0495] The activation function for the second network layer is PWL. In other words, the second network layer can be any network layer in the target neural network model, as long as the activation function for that network layer is PWL.

[0496] If we consider PWL as a standalone network layer, then the PWL corresponding to the second network layer refers to the next network layer after the second network layer, which is the PWL layer. If we consider PWL as one step in the multi-step computation of network layers such as convolutional layers, then the PWL corresponding to the second network layer refers to the activation operation performed by the second network layer, and the activation function used in this activation operation is PWL.

[0497] It should be understood that the terms "first" and "second" in the "first network layer" and "second network layer" in the embodiments of this application are only used to distinguish the network layers in the target neural network quantization model and the network layers in the target neural network model, and do not have any other limiting function.

[0498] Specifically, the quantization parameters of the second network layer include the weight quantization parameters and the feature quantization parameters of the second network layer. The weight quantization parameters are used to quantize the weight parameters of the second network layer, and the feature quantization parameters are used to quantize the input features of the second network layer.

[0499] The weight parameters of the first network layer are obtained by quantizing the weight parameters of the second network layer based on the weight quantization parameters of the second network layer.

[0500] For example, the weight parameters of the first network layer satisfy the following formula:

[0501]

[0502] S represents the weight parameters of the first network layer, which are the quantized weight parameters of the second network layer. W represents the weight parameters of the second network layer. W This represents the weight quantization parameters of the second network layer. If the second network layer is a convolutional layer, then W can also be called the convolution weight parameters of the second network layer. It can also be referred to as the convolution weight parameters of the first network layer.

[0503] The input features of the first network layer can be obtained by quantizing the input features of the second network layer based on the feature quantization parameters of the second network layer.

[0504] The input features of the first network layer are determined based on the data to be processed. This can be understood as the input features of the first network layer being obtained after quantizing the data, or as the input features of the first network layer being determined based on the processed data. For example, if the data to be processed is subjected to one or more network layers, the result of this processing can be understood as the input features of the second network layer. Based on the feature quantization parameters of the second network layer, the processed result is quantized to obtain the input features of the first network layer. These one or more network layers are the network layers preceding the first network layer.

[0505] For example, the input features of the first network layer satisfy the following formula:

[0506]

[0507] in, S represents the input features of the first network layer, which is the quantized input features of the second network layer. X This represents the feature quantization parameters of the second network layer.

[0508] Alternatively, the input features of the first network layer can satisfy the following formula:

[0509]

[0510] Wherein, β and S X This refers to the feature quantization parameters for the second network layer. For ease of description, this embodiment only uses this method as an example to illustrate the quantization inference process. If β is not set during the quantization process, the value of β in the following text can be set to 0.

[0511] For example, the processing result of the first network layer satisfies the following formula:

[0512]

[0513] in, This represents the processing result of the first network layer.

[0514] S1222, The processing result of the first network layer is processed based on the fusion bias parameter to obtain the first result.

[0515] For ease of description, in the embodiments of this application, the processing based on the fusion bias parameter is referred to as the operation of performing bias.

[0516] In other words, step S1222 can be understood as performing a bias operation on the processing result of the first network layer based on the fusion bias parameters to obtain a first result. For ease of understanding and description, the first result will be referred to as the result of the bias operation in the following text.

[0517] Specifically, step S1222 includes performing addition or subtraction operations on the fusion bias parameters and the processing result of the first network layer.

[0518] Optionally, the fusion bias parameter is determined based on at least one of the following: the weight parameters of the second network layer, the parameters of the PWL corresponding to the second network layer, or the quantization parameters of the second network layer.

[0519] Furthermore, if the second network layer includes bias parameters, the fusion bias parameters can also be determined based on the bias parameters of the second network layer.

[0520] Optionally, the fusion bias parameter satisfies the following formula:

[0521]

[0522] Where bias represents the fusion bias parameter, LB represents the left boundary of the PWL parameters corresponding to the second network layer, and b represents the bias parameter of the second network layer.

[0523] For example, the result of the bias operation can satisfy the following formula:

[0524]

[0525] This represents the result of the bias operation.

[0526] Or, such as Figure 13 As shown in (b), the results of steps S1121 and S1122 can also satisfy the following formula:

[0527]

[0528] Both bias and bias can be pre-stored in the buffer and retrieved from the buffer during steps S1121 and S1122. And bias, so that the corresponding operations can be performed.

[0529] S1223, Determine target indication information.

[0530] As shown earlier, the target indication information can indicate the target interval in PWL, that is, the interval in which the input PWL data falls.

[0531] Optionally, target indication information can be obtained based on the result of the bias calculation in step S1222.

[0532] In this way, the calculation results in step S1222 can be fully utilized, eliminating the need for additional calculations to obtain target indication information, thus avoiding unnecessary calculations and improving processing efficiency.

[0533] Optionally, obtaining target indication information based on the result of the bias operation includes: calculating the target indication information based on the equivalent length of the intermediate interval in the PWL corresponding to the second network layer, where the equivalent length of the intermediate interval is determined based on the length of the intermediate interval and the quantization parameters of the second network layer. In this case, the lengths of the intermediate intervals can be equal.

[0534] Optionally, the equivalent length of the intermediate interval satisfies the following formula:

[0535]

[0536] in, Let S represent the equivalent length of the intermediate interval, d represent the length of the intermediate interval, and S represent the equivalent length of the intermediate interval. W S represents the weight quantization parameter of the second network layer. X This represents the feature quantization parameters of the second network layer.

[0537] The length of the intermediate interval can be determined based on the left and right boundaries of the PWL and the number of intermediate intervals. The specific calculation method can be found in the previous description, and will not be repeated here.

[0538] Optionally, the target indication information satisfies the following formula:

[0539]

[0540] i represents the target indication information, or it can also be called the index of the target interval, that is, the i-th interval is the target interval.

[0541] For example, such as Figure 13 As shown in (b), target indication information is calculated based on the result of the bias operation. It should be noted that... Figure 13 The value of i in (b) is only an example, and the specific range of values ​​can be set as needed.

[0542] The target indication information satisfies the following formula:

[0543]

[0544] Optionally, the equivalent length of the intermediate interval is an integer power of 2.

[0545] In this way, the index of the target interval, i.e., the target indication information, can be obtained through shifting. Specifically, shifting the data one position to the right is equivalent to dividing by 2. If the equivalent length of the middle interval is 2... n Where n is an integer, when calculating the index of the target interval, it is necessary to divide by 2. n The operation can be performed by shifting by n bits, avoiding the need for a divider. This eliminates the need for a divider in the hardware, reducing costs. Furthermore, compared to a divider, using shifting to obtain the index of the target interval reduces computational load, lowers power consumption, and improves inference efficiency.

[0546] For example, the target neural network model can be the target neural network model obtained by method 900, or it can be the target neural network model obtained by method 1000.

[0547] Alternatively, the target indication information can be determined based on the processing results of the first network layer and the boundary points in the PWL corresponding to the second network layer. The equivalent value of each boundary point can be determined based on the value of each boundary point and the quantization parameters of the second network layer.

[0548] For example, the processing result of the first network layer is dequantized, and the result of the dequantization operation is compared with each boundary point to obtain target indication information.

[0549] S1224, Obtain the target fusion scaling parameter from the target fusion parameters according to the target indication information, and process the first result based on the target fusion scaling parameter to obtain the second result.

[0550] In other words, the target fusion scaling parameter is determined by the target indication information, which corresponds to which range.

[0551] For ease of description, in the embodiments of this application, the processing based on the target fusion scaling parameters is referred to as scaling processing.

[0552] In other words, processing the first result based on the target fusion scaling parameters can be understood as performing scaling processing on the first result based on the target fusion scaling parameters to obtain the second result. For ease of description and understanding, the second result will be referred to as the result of the scaling processing in the following text.

[0553] Specifically, scaling processing is performed on the first result based on the target fusion scaling parameters, including performing multiplication or division operations on the target fusion scaling parameters and the first result.

[0554] The blending parameters include blending scaling parameters and blending offset parameters.

[0555] The target fusion parameter is one of multiple sets of fusion parameters, and correspondingly, the target fusion scaling parameter is one of multiple fusion scaling parameters. The corresponding fusion scaling parameter, i.e., the target fusion scaling parameter, can be obtained from these multiple fusion scaling parameters based on the target indication information.

[0556] For example, target indication information can be the index of the target interval. There is a correspondence between the indices of multiple intervals in PWL and multiple fusion scaling parameters. The target fusion scaling parameters can be obtained based on the index of the target interval and this correspondence.

[0557] For example, such as Figure 13 As shown in (b), the target fusion scaling parameter scale is obtained based on the index i of the target interval. i .

[0558] Optionally, the target fusion scaling parameter is determined based on at least one of the following: the PWL parameter corresponding to the second network layer, the quantization parameter of the second network layer, or the quantization parameter of the network layer adjacent to the second network layer.

[0559] The quantization parameters of the network layers adjacent to the second network layer include the feature quantization parameters of the network layers adjacent to the second network layer.

[0560] Optionally, the target fusion scaling parameters satisfy the following formula:

[0561]

[0562] Among them, scale i Indicates the target fusion scaling parameter. S Z K represents the feature quantization parameter of the next network layer after the second network layer. i This represents the slope of the i-th interval in the PWL corresponding to the second network layer.

[0563] S1224, Obtain the target fusion offset parameter from the target fusion parameters according to the target indication information, and process the second result based on the target fusion offset parameter to obtain the third result.

[0564] For ease of description, in the embodiments of this application, the processing based on the target fusion offset parameter is referred to as offset processing.

[0565] In other words, processing the second result based on the target fusion offset parameters can be understood as performing offset processing on the second result based on the target fusion offset parameters to obtain the third result. For ease of description and understanding, the third result will be referred to as the result of the offset processing in the following text.

[0566] Specifically, the first result is offset based on the target fusion offset parameter, including performing addition or subtraction operations on the target fusion offset parameter and the second result.

[0567] In other words, the target fusion offset parameter is determined by the target indication information, which corresponds to which interval.

[0568] The target fusion parameter is one of multiple sets of fusion parameters, and correspondingly, the target fusion offset parameter is one of multiple fusion offset parameters. The corresponding fusion offset parameter, i.e., the target fusion offset parameter, can be obtained from these multiple fusion offset parameters based on the target indication information.

[0569] For example, target indication information can be the index of the target interval. There is a correspondence between the indices of multiple intervals in PWL and multiple fusion offset parameters. The target fusion offset parameters can be obtained based on the index of the target interval and this correspondence.

[0570] For example, such as Figure 13 As shown in (b), the target fusion offset parameter offset is obtained based on the index i of the target interval. i .

[0571] The fusion offset parameter is determined based on at least one of the following: the PWL parameter corresponding to the second network layer or the quantization parameter of the network layer adjacent to the second network layer.

[0572] Optionally, the target fusion offset parameter satisfies the following formula:

[0573]

[0574] Among them, offset i This represents the target fusion offset parameter, where i represents the target indication information. Z γ and K represent the feature quantization parameters of the next network layer after the second network layer. i This represents the slope of the i-th interval in the PWL corresponding to the second network layer. B i This represents the left endpoint of the i-th interval in the PWL corresponding to the second network layer.

[0575] Furthermore, step S1220 also includes step S1226.

[0576] S1226, the result after offset processing is rounded down. The result of the rounding process is the input feature of the next network layer after the first network layer.

[0577] For example, the input features of the next network layer after the first network layer satisfy the following formula:

[0578]

[0579] This represents the input features of the next network layer after the first network layer.

[0580] Alternatively, the input features of the next network layer after the first network layer can also satisfy the following formula:

[0581]

[0582] Alternatively, step S1224 may further include: rounding the result after scaling.

[0583] In this case, step S1125 includes: performing offset processing on the rounded result based on the target fusion offset parameter, and the result obtained is the input feature of the next network layer of the first network layer.

[0584] For example, the input features of the network layers after the first network layer satisfy the following formula:

[0585]

[0586] Or, such as Figure 13 As shown in (b), the input features of the next network layer after the first network layer can satisfy the following formula:

[0587]

[0588] The results obtained by performing rounding after scaling and rounding after offsetting are not significantly different. In other words, in this embodiment, rounding can be performed after scaling or after offsetting, and this embodiment does not limit the specific execution of the rounding.

[0589] In the inference process of the target neural network quantization model, the parameters can be pre-calculated using the methods described above. For example, bias, scale... i offset i Both d and d are pre-calculated based on the parameters in the target neural network model.

[0590] For comparison, Figure 13(a) also illustrates the quantization inference process using ReLU as the activation function. Compared to the quantization inference process using ReLU, the quantization inference process using PWL as the activation function requires determining the target indication information and obtaining the corresponding fusion parameters for calculation based on the target indication information. When the equivalent length of the intermediate interval of PWL is an integer power of 2, the index of the target intermediate interval can be obtained through shifting without increasing the computational load excessively. Furthermore, using PWL as the activation function can improve the model's performance. In other words, using PWL as the activation function can improve the model's performance without introducing excessive computational load.

[0591] It should be understood that Figure 13 The quantization inference process of only one convolutional layer is used as an example. The target neural network quantization model may also include other network layers, and this application does not limit this.

[0592] Alternatively, step S1220 can be implemented in other ways, depending on the specific form of the fusion parameters.

[0593] Method 900 can be understood as the quantization training phase of the PWL model (e.g.) Figure 5 The training device 120 shown is performing the training phase, specifically using the neural network model provided in method 900; while method 1200 can be understood as the application phase of the model (such as...). Figure 5 The execution phase of the execution device 110 shown can specifically involve quantizing the target neural network model trained by method 900 and obtaining the output result based on the input data to be processed. It should be noted that method 1200 may also not use the target neural network model trained by method 900.

[0594] The following is combined with Figures 14 to 17 The apparatus of the embodiments of this application will be described below. It should be understood that the apparatus described below is capable of performing the methods of the foregoing embodiments of this application. To avoid unnecessary repetition, repeated descriptions will be appropriately omitted when describing the apparatus of the embodiments of this application below.

[0595] Figure 14 This is a schematic block diagram of a training apparatus for a neural network model according to an embodiment of this application. Figure 14 The training device 3000 for the neural network model shown includes an acquisition unit 3010 and a processing unit 3020.

[0596] The acquisition unit 3010 and the processing unit 3020 can be used to execute the training method of the neural network model of the present application embodiment. Specifically, they can be used to execute method 900 or method 1000.

[0597] The acquisition unit 3010 is used to acquire a neural network model. The activation function of the neural network model includes a piecewise linear function PWL, which includes multiple intervals.

[0598] The processing unit 3020 is used to: train a neural network model based on training data to obtain a target neural network model, the training data including image data, audio data, or text data; process the target neural network model to obtain a target neural network quantization model, the target neural network quantization model including multiple sets of fusion parameters, the multiple sets of fusion parameters having a corresponding relationship with multiple intervals of PWL in the target neural network model.

[0599] Optionally, as an embodiment, the processing unit 3020 is specifically used to: obtain multiple sets of fusion parameters based on the parameters of multiple intervals of PWL in the target neural network model, the weight quantization parameters and feature quantization parameters of the neural network model, so as to obtain the target neural network quantization model, wherein the weight quantization parameters and feature quantization parameters of the neural network model are obtained through quantization training.

[0600] Optionally, as an embodiment, the processing unit 3020 is specifically used for: training a neural network model based on training data; adjusting the length of the middle interval of PWL in the trained neural network model to obtain an adjusted neural network model; and obtaining a target neural network model based on the adjusted neural network model.

[0601] Optionally, as an embodiment, the processing unit 3020 is specifically used to: determine the target scaling factor of the PWL in the trained neural network model based on the equivalent length of the intermediate interval, wherein the equivalent length of the intermediate interval is determined based on the length of the intermediate interval of the PWL in the trained neural network model, the weight quantization parameters and feature quantization parameters of the neural network model; adjust the parameters of the target network layer in the trained neural network model and the length of the intermediate interval of the PWL in the trained neural network model based on the target scaling factor to obtain the adjusted neural network model, wherein the target network layer corresponds to the PWL in the trained neural network model.

[0602] Optionally, as an embodiment, the processing unit 3020 is specifically used to: determine the target scaling factor based on the equivalent length of the intermediate interval and the nearest power of 2 to the equivalent length of the intermediate interval.

[0603] Figure 15 This is a schematic block diagram of the data processing apparatus 4000 provided in the embodiments of this application. Figure 15 The data processing apparatus 4000 shown includes an acquisition unit 4010 and a processing unit 4020.

[0604] The acquisition unit 4010 and the processing unit 4020 can be used to execute the data processing method of the embodiments of this application, for example, they can be used to execute method 1200.

[0605] The acquisition unit 4010 is used to acquire data to be processed, including image data, audio data, or text data.

[0606] The processing unit 4020 is used to process the data to be processed using the target neural network quantization model. The target neural network quantization model includes multiple sets of fusion parameters. The target neural network quantization model is obtained by quantizing the target neural network model. The activation function of the target neural network model includes the piecewise linear function PWL. PWL includes multiple intervals. There is a corresponding relationship between the multiple sets of fusion parameters and the multiple intervals.

[0607] Optionally, as an embodiment, the processing unit 4020 is specifically used to: determine target indication information based on the data to be processed; obtain target fusion parameters corresponding to the target indication information from multiple sets of fusion parameters based on the target indication information; and process the data to be processed based on the target fusion parameters.

[0608] Optionally, as an embodiment, the processing unit 4020 is specifically used to: process the input features of the first network layer based on the weight parameters of the first network layer in the target neural network quantization model, wherein the input features of the first network layer are determined according to the data to be processed; and determine the target indication information based on the processing result of the first network layer.

[0609] Optionally, as an embodiment, the weight parameters of the first network layer are obtained by quantizing the weight parameters of the second network layer in the target neural network model. The second network layer is the network layer in the target neural network model corresponding to the first network layer. The processing unit 4020 is specifically used to: process the processing result of the first network layer based on the fusion bias parameter to obtain a first result; calculate the target indication information based on the equivalent length of the middle interval among multiple intervals of the PWL corresponding to the second network layer and the first result. The equivalent length of the middle interval is determined based on the length of the middle interval and the quantization parameters of the second network layer.

[0610] Alternatively, as an example, the equivalent length of the intermediate interval is an integer power of 2.

[0611] Optionally, as an embodiment, the processing unit 4020 is specifically used to: obtain the target fusion scaling parameter in the target fusion parameters according to the target indication information, process the first result based on the target fusion scaling parameter to obtain the second result; obtain the target fusion offset parameter in the target fusion parameters according to the target indication information, process the second result based on the target fusion offset parameter to obtain the third result.

[0612] Optionally, as an embodiment, the processing unit 4020 is further configured to: perform rounding processing on the third result.

[0613] Optionally, as an embodiment, the processing unit 4020 is further configured to: perform rounding processing on the second result to obtain a rounded second result; specifically, the processing unit 4020 is configured to: process the rounded second result based on the target fusion offset parameter to obtain a third result.

[0614] It should be noted that the training device 3000 and device 4000 mentioned above are embodied in the form of functional units. The term "unit" here can be implemented in software and / or hardware, and there is no specific limitation on this.

[0615] For example, a "unit" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include an application-specific integrated circuit (ASIC), electronic circuitry, a processor (e.g., a shared processor, a proprietary processor, or a group processor) and memory for executing one or more software or firmware programs, integrated logic circuitry, and / or other suitable components that support the described functions.

[0616] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0617] Figure 16 This is a schematic diagram of the hardware structure of the training device for the neural network model provided in this application embodiment. Figure 16 The training device 5000 for the neural network model shown (specifically, the device 5000 can be a computer device) includes a memory 5001, a processor 5002, a communication interface 5003, and a bus 5004. The memory 5001, processor 5002, and communication interface 5003 are interconnected via the bus 5004.

[0618] The memory 5001 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 5001 can store programs. When the program stored in the memory 5001 is executed by the processor 5002, the processor 5002 executes the various steps of the neural network model training method of this application embodiment. Specifically, the processor 5002 can execute the steps described above... Figure 9 Steps S920 to S930 in the method shown, or performing the above steps Figure 10 Steps S1010 to S1090 are shown.

[0619] The processor 5002 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute related programs to implement the neural network model training method of the method embodiment of this application.

[0620] The processor 5002 can also be an integrated circuit chip with signal processing capabilities; for example, it could be... Figure 6 The chip shown. In the implementation process, each step of the training method for the neural network model of this application can be completed by the integrated logic circuit of the hardware in the processor 5002 or by software instructions.

[0621] The processor 5002 described above can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 5001, and processor 5002 reads information from memory 5001 and combines it with its hardware to complete... Figure 14 The training apparatus shown includes units that are required to perform functions, or to perform the methods described in this application. Figure 9 or Figure 10 The training method for the neural network model shown.

[0622] The communication interface 5003 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between the device 5000 and other devices or communication networks. For example, training data or neural network models can be acquired through the communication interface 5003.

[0623] Bus 5004 may include a pathway for transmitting information between various components of device 5000 (e.g., memory 5001, processor 5002, communication interface 5003).

[0624] Figure 17 This is a schematic diagram of the hardware structure of the data processing device according to an embodiment of this application. Figure 17 The data processing device 6000 shown includes a memory 6001, a processor 6002, a communication interface 6003, and a bus 6004. The memory 6001, processor 6002, and communication interface 6003 are interconnected via the bus 6004.

[0625] The memory 6001 can be a ROM, static storage device, or RAM. The memory 6001 can store programs, and when the program stored in the memory 6001 is executed by the processor 6002, the processor 6002 and the communication interface 6003 are used to execute various steps of the data processing method of this application embodiment. Specifically, the processor 6002 can execute the steps described above... Figure 12Step S1220 in the method shown.

[0626] The processor 6002 may be a general-purpose CPU, microprocessor, ASIC, GPU, or one or more integrated circuits, used to execute relevant programs to achieve the functions required by the units in the data processing apparatus of this application embodiment, or to execute the data processing method of this application method embodiment.

[0627] The processor 6002 can also be an integrated circuit chip with signal processing capabilities; for example, it could be... Figure 6 The chip shown. In implementation, each step of the data processing method in this embodiment can be completed by the integrated logic circuitry in the processor 6002 or by software instructions.

[0628] The processor 6002 described above can also be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 6001. The processor 6002 reads the information in memory 6001 and, in conjunction with its hardware, completes the functions required by the units included in the data processing apparatus of the embodiments of this application, or executes the data processing method of the method embodiments of this application.

[0629] The communication interface 6003 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between the device 6000 and other devices or communication networks. For example, data to be processed can be obtained through the communication interface 6003.

[0630] Bus 6004 may include a pathway for transmitting information between various components of device 6000 (e.g., memory 6001, processor 6002, communication interface 6003).

[0631] It should be noted that although only the memory, processor, and communication interface are shown in the above-described devices 5000 and 6000, those skilled in the art should understand that in specific implementations, devices 5000 and 6000 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that devices 5000 and 6000 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that devices 5000 and 6000 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 16 and Figure 17 All the devices shown.

[0632] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0633] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0634] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0635] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0636] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0637] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0638] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0639] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0640] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0641] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0642] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0643] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0644] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, include: Acquire the data to be processed, which includes: image data, audio data, or text data; The data to be processed is processed using a target neural network quantization model. The target neural network quantization model includes multiple sets of fusion parameters, including fusion scaling parameters and fusion offset parameters. The target neural network quantization model is obtained by quantizing a target neural network model. The activation function of the target neural network model includes a piecewise linear function (PWL), and the PWL includes multiple intervals. The multiple sets of fusion parameters have a corresponding relationship with the multiple intervals. The fusion scaling parameter corresponding to the i-th interval of the plurality of intervals is obtained by multiplying the weight quantization parameter of the second network layer, the feature quantization parameter of the second network layer, and the slope of the i-th interval of the PWL corresponding to the second network layer, and then dividing by the feature quantization parameter of the next network layer of the second network layer. The fusion offset parameter corresponding to the i-th interval is: multiply the difference between the left boundary of the PWL corresponding to the second network layer and the left endpoint of the i-th interval in the PWL corresponding to the second network layer by the slope of the i-th interval in the PWL corresponding to the second network layer; add the resulting product to the function value corresponding to the left endpoint of the i-th interval in the PWL corresponding to the second network layer, subtract the feature quantization parameter of the next network layer of the second network layer, and divide the final result by the feature quantization parameter of the next network layer of the second network layer. Wherein, the second network layer is the network layer in the target neural network model that corresponds to the first network layer, and the first network layer is the network layer in the target neural network quantization model.

2. The method according to claim 1, characterized in that, The process of processing the data to be processed using the target neural network quantization model includes: Determine the target indication information based on the data to be processed; Based on the target indication information, the target fusion parameter corresponding to the target indication information is obtained from the multiple sets of fusion parameters, and the data to be processed is processed based on the target fusion parameter.

3. The method according to claim 2, characterized in that, The step of determining the target indication information based on the data to be processed includes: The input features of the first network layer are processed based on the weight parameters of the first network layer in the target neural network quantization model. The input features of the first network layer are determined based on the data to be processed. The target indication information is determined based on the processing result of the first network layer.

4. The method according to claim 3, characterized in that, The weight parameters of the first network layer are obtained by quantizing the weight parameters of the second network layer in the target neural network model. The second network layer is the network layer in the target neural network model corresponding to the first network layer. The step of determining the target indication information based on the processing result of the first network layer includes: The processing result of the first network layer is processed based on the fusion bias parameter to obtain the first result; The target indication information is calculated based on the equivalent length of the middle interval among the multiple intervals of the PWL corresponding to the first result and the second network layer. The equivalent length of the middle interval is determined based on the length of the middle interval and the quantization parameters of the second network layer.

5. The method according to claim 4, characterized in that, The equivalent length of the intermediate interval is an integer power of 2.

6. A method for training a neural network model, characterized in that, include: Obtain a neural network model, wherein the activation function of the neural network model includes a piecewise linear function PWL, and the PWL includes multiple intervals; The neural network model is trained based on the training data to obtain the target neural network model. The training data includes image data, audio data, or text data. The target neural network model is processed to obtain a target neural network quantization model. The target neural network quantization model includes multiple sets of fusion parameters, including fusion scaling parameters and fusion offset parameters. The multiple sets of fusion parameters have a corresponding relationship with multiple intervals of PWL in the target neural network model. The fusion scaling parameter corresponding to the i-th interval of the plurality of intervals is obtained by multiplying the weight quantization parameter of the second network layer, the feature quantization parameter of the second network layer, and the slope of the i-th interval in the PWL corresponding to the second network layer, and then dividing by the feature quantization parameter of the next network layer of the second network layer. The fusion offset parameter corresponding to the i-th interval is: multiply the difference between the left boundary of the PWL corresponding to the second network layer and the left endpoint of the i-th interval in the PWL corresponding to the second network layer by the slope of the i-th interval in the PWL corresponding to the second network layer; add the resulting product to the function value corresponding to the left endpoint of the i-th interval in the PWL corresponding to the second network layer, subtract the feature quantization parameter of the next network layer of the second network layer, and divide the final result by the feature quantization parameter of the next network layer of the second network layer. Wherein, the second network layer is the network layer in the target neural network model that corresponds to the first network layer, and the first network layer is the network layer in the target neural network quantization model.

7. The training method according to claim 6, characterized in that, The process of processing the target neural network model to obtain a target neural network quantization model includes: The multiple sets of fusion parameters are obtained based on the parameters of each interval of PWL in the target neural network model, the weight quantization parameters and feature quantization parameters of the neural network model, to obtain the target neural network quantization model. The weight quantization parameters and feature quantization parameters of the neural network model are obtained through quantization training.

8. The training method according to claim 6 or 7, characterized in that, The step of training the neural network model based on training data to obtain the target neural network model includes: The neural network model is trained based on the training data; Adjusting the length of the middle interval of the PWL in the trained neural network model yields the adjusted neural network model; The target neural network model is obtained based on the adjusted neural network model.

9. The training method according to claim 8, characterized in that, The step of adjusting the length of the intermediate interval of the PWL in the trained neural network model to obtain the adjusted neural network model includes: The target scaling factor of the PWL in the trained neural network model is determined based on the equivalent length of the middle interval of the PWL in the trained neural network model. The equivalent length of the middle interval is determined based on the length of the middle interval of the PWL in the trained neural network model, the weight quantization parameters and feature quantization parameters of the neural network model. The parameters of the target network layer in the trained neural network model and the length of the intermediate interval of the PWL in the trained neural network model are adjusted based on the target scaling factor to obtain the adjusted neural network model, wherein the target network layer corresponds to the PWL in the trained neural network model.

10. The training method according to claim 9, characterized in that, Determining the target scaling factor of the PWL in the trained neural network model based on the equivalent length of the middle interval of the PWL in the trained neural network model includes: The target scaling factor is determined based on the equivalent length of the intermediate interval and the nearest power of 2 to the equivalent length of the intermediate interval.

11. A data processing apparatus, characterized in that, include: An acquisition unit is used to acquire data to be processed, the data including: image data, audio data, or text data; The processing unit is used to process the data to be processed using a target neural network quantization model. The target neural network quantization model includes multiple sets of fusion parameters, including fusion scaling parameters and fusion offset parameters. The target neural network quantization model is obtained by quantizing a target neural network model. The activation function of the target neural network model includes a piecewise linear function (PWL), and the PWL includes multiple intervals. The multiple sets of fusion parameters have a corresponding relationship with the multiple intervals. The fusion scaling parameter corresponding to the i-th interval of the plurality of intervals is obtained by multiplying the weight quantization parameter of the second network layer, the feature quantization parameter of the second network layer, and the slope of the i-th interval in the PWL corresponding to the second network layer, and then dividing by the feature quantization parameter of the next network layer of the second network layer. The fusion offset parameter corresponding to the i-th interval is: multiply the difference between the left boundary of the PWL corresponding to the second network layer and the left endpoint of the i-th interval in the PWL corresponding to the second network layer by the slope of the i-th interval in the PWL corresponding to the second network layer; add the resulting product to the function value corresponding to the left endpoint of the i-th interval in the PWL corresponding to the second network layer, subtract the feature quantization parameter of the next network layer of the second network layer, and divide the final result by the feature quantization parameter of the next network layer of the second network layer. Wherein, the second network layer is the network layer in the target neural network model that corresponds to the first network layer, and the first network layer is the network layer in the target neural network quantization model.

12. The apparatus according to claim 11, characterized in that, The processing unit is specifically used for: Determine the target indication information based on the data to be processed; Based on the target indication information, the target fusion parameter corresponding to the target indication information is obtained from the multiple sets of fusion parameters, and the data to be processed is processed based on the target fusion parameter.

13. The apparatus according to claim 12, characterized in that, The processing unit is specifically used for: The input features of the first network layer are processed based on the weight parameters of the first network layer in the target neural network quantization model. The input features of the first network layer are determined based on the data to be processed. The target indication information is determined based on the processing result of the first network layer.

14. The apparatus according to claim 13, characterized in that, The weight parameters of the first network layer are obtained by quantizing the weight parameters of the second network layer in the target neural network model. The second network layer is the network layer in the target neural network model corresponding to the first network layer. The processing unit is specifically used for: The processing result of the first network layer is processed based on the fusion bias parameter to obtain the first result; The target indication information is calculated based on the equivalent length of the middle interval among the multiple intervals in the PWL corresponding to the first result and the second network layer. The equivalent length of the middle interval is determined based on the length of the middle interval and the quantization parameters of the second network layer.

15. The apparatus according to claim 14, characterized in that, The equivalent length of the intermediate interval is an integer power of 2.

16. A training device for a neural network model, characterized in that, include: An acquisition unit is used to acquire a neural network model, wherein the activation function of the neural network model includes a piecewise linear function (PWL), and the PWL includes multiple intervals; Processing unit, used for: The neural network model is trained based on the training data to obtain the target neural network model. The training data includes image data, audio data, or text data. The target neural network model is processed to obtain a target neural network quantization model. The target neural network quantization model includes multiple sets of fusion parameters, including fusion scaling parameters and fusion offset parameters. The multiple sets of fusion parameters have a corresponding relationship with multiple intervals of PWL in the target neural network model. The fusion scaling parameter corresponding to the i-th interval of the plurality of intervals is obtained by multiplying the weight quantization parameter of the second network layer, the feature quantization parameter of the second network layer, and the slope of the i-th interval in the PWL corresponding to the second network layer, and then dividing by the feature quantization parameter of the next network layer of the second network layer. The fusion offset parameter corresponding to the i-th interval is: multiply the difference between the left boundary of the PWL corresponding to the second network layer and the left endpoint of the i-th interval in the PWL corresponding to the second network layer by the slope of the i-th interval in the PWL corresponding to the second network layer; add the resulting product to the function value corresponding to the left endpoint of the i-th interval in the PWL corresponding to the second network layer, subtract the feature quantization parameter of the next network layer of the second network layer, and divide the final result by the feature quantization parameter of the next network layer of the second network layer. Wherein, the second network layer is the network layer in the target neural network model that corresponds to the first network layer, and the first network layer is the network layer in the target neural network quantization model.

17. The training device according to claim 16, characterized in that, The processing unit is specifically used to: obtain the multiple sets of fusion parameters based on the parameters of multiple intervals of PWL in the target neural network model, the weight quantization parameters and feature quantization parameters of the neural network model, so as to obtain the target neural network quantization model, wherein the weight quantization parameters and feature quantization parameters of the neural network model are obtained through quantization training.

18. The training apparatus according to claim 16 or 17, characterized in that, The processing unit is specifically used for: The neural network model is trained based on the training data; Adjusting the length of the middle interval of the PWL in the trained neural network model yields the adjusted neural network model; The target neural network model is obtained based on the adjusted neural network model.

19. The training device according to claim 18, characterized in that, The processing unit is specifically used for: The target scaling factor of the PWL in the trained neural network model is determined based on the equivalent length of the middle interval of the PWL in the trained neural network model. The equivalent length of the middle interval is determined based on the length of the middle interval of the PWL in the trained neural network model, the weight quantization parameters and feature quantization parameters of the neural network model. The parameters of the target network layer in the trained neural network model and the length of the intermediate interval of the PWL in the trained neural network model are adjusted based on the target scaling factor to obtain the adjusted neural network model, wherein the target network layer corresponds to the PWL in the trained neural network model.

20. The training device according to claim 19, characterized in that, The processing unit is specifically used for: The target scaling factor is determined based on the equivalent length of the intermediate interval and the nearest power of 2 to the equivalent length of the intermediate interval.

21. A data processing apparatus, characterized in that, It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to invoke the program instructions to perform the method of any one of claims 1 to 5.

22. A training device for a neural network model, characterized in that, It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to invoke the program instructions to perform the method of any one of claims 6 to 10.

23. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code for execution by the device, the program code including methods for performing any one of claims 1 to 5 or 6 to 10.

24. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 5 or 6 to 10.

25. A chip, characterized in that, The chip includes a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute the method as described in any one of claims 1 to 5 or 6 to 10.

Images