Model training method and device

Abstract

The present application discloses a model training method that can be applied to the field of artificial intelligence. The method includes: obtaining a first neural network model, the first neural network model including a convolutional BN layer and a first quantization operator, the convolutional BN layer is used to perform convolution processing on the input Nth batch of data according to a first weight, normalize the convolution processing result according to the BN coefficient, and update the BN coefficient based on the normalization processing result, update the first weight of the updated BN coefficient, the first quantization operator is used to quantize and dequantize the updated first weight to obtain a second weight, and the convolutional BN layer is also used to perform convolution processing on the input N+1th batch of data according to the second weight. The present application can reduce the amount of data calculation of the convolution layer in the neural network.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a model training method and device. Background Art

[0002] Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and develop new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0003] Neural network quantization is a model compression technology that converts floating-point storage (operations) into integer storage (operations). For example, the model parameters of a model originally represented by float32 (32-bit floating point) are represented by int8 (8-bit fixed point) after quantization. Through the quantization operation of the model, the model's operation speed is improved at the expense of a small loss of accuracy.

[0004] Quantization aware training (QAT) uses training data to compensate for the accuracy loss caused by quantization. Its main process is: 1. Inserting a quantization operator before model training. 2. During training, the minimum and maximum values ​​of each layer (weights and activations) of the model are counted to calculate the quantization factor.

[0005] During the model training phase, QAT needs to insert a pseudo-quantization node SimQuant (also called a quantization operator in this embodiment) into the weight input and activation output of the original model. In addition, for the convolutional neural network (CNN) and batch normalization (BN) structure, another CNN is required to implement BN folding to achieve the fusion of BN coefficients and CNN weights. During the training process, SimQuant will count the min and max values ​​in the corresponding data stream (Tensor) for the subsequent calculation of the scale quantization factor. Figure 2As shown in the figure, when QAT folds CNN and BN, it needs to build another CNN to perform convolution operations on the data of the current batch. BN uses the results of the convolution operation to update the BN coefficients, and then uses the updated BN coefficients to construct weights. The quantization operator can quantize and dequantize the constructed weights, and CNN can perform convolution operations on the data of the current batch based on the weights obtained after dequantization. However, because another CNN is required to implement BN folding, two CNNs will perform convolution operations on the same batch of data during the training process, which increases the amount of calculations required by the CNN during training and thus reduces the training speed. Summary of the Invention

[0006] In a first aspect, the present application provides a model training method, the method comprising:

[0007] Obtain a first neural network model, wherein the first neural network model includes a convolutional BN layer and a first quantization operator, the convolutional BN layer is used to perform convolution processing on the input Nth batch data according to a first weight, normalize the convolution processing result according to the BN coefficient, and update the BN coefficient based on the normalization processing result, and update the first weight of the updated BN coefficient. The first quantization operator is used to quantize and dequantize the updated first weight to obtain a second weight, and the convolutional BN layer is also used to perform convolution processing on the input N+1th batch data according to the second weight; perform model training on the first neural network model to obtain a trained first neural network model.

[0008] The convolution processing result refers to the result obtained after convolution processing on the Nth batch of data.

[0009] The convolutional BN layer can be used as an independent layer. In one implementation, the convolutional BN layer can still be distinguished into portions corresponding to the first convolutional layer and the first batch normalization BN layer. For convenience of description, the portion of the convolutional BN layer corresponding to the first convolutional layer is still referred to as the first convolutional layer, and the portion of the convolutional BN layer corresponding to the first batch normalization BN layer is referred to as the first batch normalization BN layer. The first convolutional layer is configured to convolve the Nth batch of input data according to a first weight to obtain a first output (i.e., the convolution result mentioned above). The first BN layer is configured to normalize the first output according to the BN coefficient and update the BN coefficient based on the normalization result. The convolutional BN layer is configured to update the first weight of the updated BN coefficient. The first quantization operator is configured to quantize and dequantize the updated first weight to obtain a second weight. The first convolutional layer is further configured to convolve the N+1th batch of input data according to the second weight. The first neural network model is trained to obtain a trained first neural network model.

[0010] In which, the first neural network model can be obtained by performing BN folding processing on a pre-trained model and adding a quantization operator (also known as a pseudo quantization node SimQuant). The first output can be used as the input of the first BN layer. The first BN layer can normalize the first output and update the BN coefficient based on the normalization processing result. In the training process, the BN layer performs BN operation based on the mean and standard deviation of the output features of the convolution layer in the feedforward process. Exemplarily, the first BN layer is connected to the first convolution layer, and the first BN layer is used to perform BN operation on the first output according to the mean and standard deviation of the first output of the first convolution layer. Afterwards, the training device can update the BN coefficient based on the operation result. In which, the BN coefficient may include but is not limited to at least one of the mean μ, variance σ, scale parameter γ and offset parameter β, or the operation result between any two or more of them.

[0011] In the prior art, the first convolutional layer determines the weights used for the current batch by updating the BN coefficients after the BN layer processes the data of the current batch. Therefore, in addition to the first convolutional layer, a separate convolutional layer is required to process the data so that the BN layer can update the BN coefficients based on the data of the current batch. In the embodiment of the present application, since the first convolutional layer determines the weights used for the current batch by updating the BN coefficients after the BN layer processes the data of the previous batch, there is no need to set up another separate convolutional layer. On the one hand, the model size can be reduced, and on the other hand, the amount of data calculations of the convolutional layer in the neural network is reduced. Since the training process is a process that requires a large number of iterations, the available computing resources of the training equipment are limited. In this embodiment, the convolutional layer in the neural network is reduced by one convolution operation during the training process. In a large number of training processes, the computing resource consumption of the training equipment can be greatly reduced, thereby improving the training speed.

[0012] In one possible implementation, the method further includes:

[0013] Obtain a second neural network model, which can be a pre-trained model. The second neural network model includes a first convolutional layer and a first BN layer; the first convolutional layer can be used to perform convolution processing on the input data according to a target weight to obtain a first output, and the target weight is the weight included in the convolution kernel in the first convolutional layer; the first BN layer is used to normalize the first output according to the BN coefficient, and update the BN coefficient based on the normalization result; perform BN folding processing on the first convolutional layer and the first BN layer to obtain the first neural network model, and the first neural network model includes the convolutional BN layer obtained after folding the first convolutional layer and the first BN layer.

[0014] Specifically, in order to identify the convolutional layers and BN layers that need to be BN folded in the second neural network model, the structure of the convolutional layers and BN layers that need to be BN folded in the model can be judged according to the operator type in the computational flow graph of the second neural network model (this embodiment can also be described as a CNN+BN structure), and the identified CNN+BN structure can be combined into a block (that is, the convolutional BN layer in the above embodiment); then the combined convolutional BN layer can replace the original CNN+BN structure.

[0015] In one possible implementation, the convolutional BN layer is obtained by folding the convolutional layer and the BN layer, the first weight is obtained according to the product of the BN coefficient and the target weight, the updated first weight is obtained by multiplying the updated BN coefficient and the target weight, and the target weight is the weight included in the convolutional layer.

[0016] In which, the second neural network model may include the convolution layer and BN layer described above (the convolution layer may also be referred to as the first convolution layer in the second neural network model described above, and the BN layer may also be referred to as the first BN layer in the second neural network model described above). The first convolution layer in the second neural network model is used to perform convolution processing on the input data according to the target weight. The first convolution layer in the second neural network model is used to perform convolution processing on the input data according to the target weight. The input data is the data input to the first convolution layer, that is, the input of the intermediate layer in the neural network, not the input of the neural network. The first weight is obtained by multiplying the BN coefficient by the target weight. The updated first weight is obtained by multiplying the updated BN coefficient by the target weight. The target weight is the weight included in the convolution kernel in the convolution layer. Specifically, the target weight may be the weight included in the first convolution layer in the second neural network model. The first convolution layer may perform a convolution operation on the input data based on the convolution kernel including the target weight. The convolution kernel may include the target weight and a bias.

[0017] In one possible implementation, the method further includes: multiplying the BN coefficient by the target weight to obtain a first target tensor, wherein the first target tensor includes M elements; replacing the N target elements with the largest absolute values ​​among the M elements included in the first target tensor with the largest element among the MN elements of the M elements other than the N target elements to obtain the first weight.

[0018] In an embodiment of the present application, the first weight in the first neural network model can be initialized using the weights and BN coefficients in the second neural network model (pre-trained model). Specifically, the BN coefficients can be multiplied by the target weights according to the pre-trained model to obtain a first target tensor. For example, the first target tensor can be γ / σ*W, and then the elements in the first target tensor are sorted according to size, and the main part values ​​are intercepted in a symmetrical manner (such as: intercepting 95% to 99.5%), and the remaining elements are replaced with the largest value in the main part values, so as to achieve the initialization of the first weight. Wherein, the first weight is in the form of a tensor.

[0019] The number of elements with large absolute values ​​in the first target tensor is small. During the subsequent quantization and dequantization process, the accuracy of the operation will be affected due to the large absolute values. For example, unnecessary smoothing of other elements of the quantization factor will be performed. The embodiment of the present application improves the accuracy of neural network model processing by truncating elements of the first target tensor.

[0020] In one possible implementation, the first convolutional layer in the first neural network model is used to perform convolution processing on the input N+1th batch data according to the second weight to obtain a convolution processing result, and divide the convolution processing result by the updated BN coefficient to obtain a second output.

[0021] Similar to the above embodiment, in the embodiment of the present application, in order to quantize the output of each activation layer in the second neural network, a second quantization operator can be added at the output position of the activation layer.

[0022] In one possible implementation, the second neural network model also includes a target activation layer, and the target activation layer in the second neural network model is used to process input data, wherein the input data is data input to the target activation layer, that is, the input of the intermediate layer in the neural network, rather than the input of the neural network, to obtain a third output. The first neural network model also includes the target activation layer and a second quantization operator, and the target activation layer in the first neural network model is used to process the input data to obtain a fourth output, and the second quantization operator is used to quantize and dequantize the fourth output according to a second quantization factor.

[0023] In one possible implementation, the third output is a second target tensor, where the second target tensor includes X elements. The method further includes: obtaining Y target elements having the largest absolute values ​​among the X elements; and replacing the Y target elements in the second target tensor with a largest element among XY elements in the X elements excluding the Y target elements, to obtain the second quantization factor.

[0024] Similar to the above embodiment, in the embodiment of the present application, in the process of initializing the quantization factor located at the output position of the activation layer, after arranging the elements in the second target tensor from large to small, a certain percentage of the elements can be intercepted. The percentage can be but is not limited to 95% to 99.5%, and 95% to 99.5% of the elements can be the elements of the main part of the element distribution, that is, elements whose absolute values ​​are close to 0.

[0025] In one possible implementation, the trained first neural network model includes a trained first quantization factor and a trained BN coefficient, and the method further includes:

[0026] The first neural network model is quantized according to the trained first quantization factor and the trained BN coefficient to obtain a third neural network model, wherein the third neural network model includes the quantized first convolution layer, and the first convolution layer is used to perform convolution processing on the input data according to the quantized weights, and the quantized weights are obtained according to the first quantization factor and the trained BN coefficients.

[0027] In a second aspect, the present application provides a model training device, comprising:

[0028] An acquisition module is used to acquire a first neural network model, wherein the first neural network model includes a convolutional BN layer and a first quantization operator, the convolutional BN layer is used to perform convolution processing on the input Nth batch of data according to a first weight, normalize the convolution processing result according to the BN coefficient, update the BN coefficient based on the normalization processing result, and update the first weight on the updated BN coefficient. The first quantization operator is used to quantize and dequantize the updated first weight to obtain a second weight, and the convolutional BN layer is further used to perform convolution processing on the input N+1th batch of data according to the second weight;

[0029] The model training module is used to perform model training on the first neural network model to obtain a trained first neural network model.

[0030] In one possible implementation, the acquisition module is used to obtain a second neural network model, which includes a first convolutional layer and a first BN layer; performing BN folding processing on the first convolutional layer and the first BN layer to obtain the first neural network model, which includes the convolutional BN layer obtained after folding the first convolutional layer and the first BN layer.

[0031] In one possible implementation, the convolutional BN layer is obtained by folding the convolutional layer and the BN layer, the first weight is obtained according to the product of the BN coefficient and the target weight, the updated first weight is obtained by multiplying the updated BN coefficient and the target weight, and the target weight is the weight included in the convolutional layer.

[0032] In a possible implementation, the apparatus further includes:

[0033] a product operation module, configured to perform a product operation on the BN coefficient and the target weight to obtain a first target tensor, where the first target tensor includes M elements;

[0034] An element replacement module is used to replace the N target elements with the largest absolute values ​​among the M elements included in the first target tensor with the largest element among the MN elements in the M elements excluding the N target elements, so as to obtain the first weight.

[0035] In one possible implementation, the first convolutional layer in the first neural network model is used to perform convolution processing on the input N+1th batch data according to the second weight to obtain a convolution processing result, and divide the convolution processing result by the updated BN coefficient to obtain a second output.

[0036] In one possible implementation, the second neural network model also includes a target activation layer, and the target activation layer in the second neural network model is used to process the input data to obtain a third output. The first neural network model also includes the target activation layer and a second quantization operator, and the target activation layer in the first neural network model is used to process the input data to obtain a fourth output. The second quantization operator is used to quantize and dequantize the fourth output according to a second quantization factor.

[0037] In a possible implementation, the third output is a second target tensor, the second target tensor includes X elements, and the acquisition module is used to acquire Y target elements with the largest absolute values ​​among the X elements;

[0038] The element replacement module is configured to replace the Y target elements in the second target tensor with a maximum element among XY elements in the X elements excluding the Y target elements, so as to obtain the second quantization factor.

[0039] In one possible implementation, the first quantization operator is used to perform quantization and dequantization on the updated first weight according to the first quantization factor, the trained first neural network model includes the trained first quantization factor and the trained BN coefficient, and the apparatus further includes:

[0040] In a third aspect, an embodiment of the present application provides a model training device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute the programs in the memory to perform the first aspect and any optional method thereof.

[0041] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned first aspect and any optional method thereof.

[0042] In a fifth aspect, an embodiment of the present application provides a computer program, including code, which, when executed, is used to implement the above-mentioned first aspect and any optional method thereof.

[0043] In a sixth aspect, the present application provides a chip system comprising a processor for supporting an execution device or a training device in implementing the functions described in the aforementioned aspects, such as transmitting or processing data or information described in the aforementioned methods. In one possible design, the chip system further comprises a memory for storing program instructions and data necessary for the execution device or the training device. The chip system may consist of a single chip or may include a chip and other discrete components.

[0044] The embodiment of the present application provides a model training method, which includes: obtaining a first neural network model, wherein the first neural network model includes a convolutional BN layer and a first quantization operator, the convolutional BN layer is used to perform convolution processing on the input Nth batch of data according to a first weight, normalize the convolution processing result according to the BN coefficient, and update the BN coefficient based on the normalization processing result, update the first weight of the updated BN coefficient, the first quantization operator is used to quantize and dequantize the updated first weight to obtain a second weight, the convolutional BN layer is also used to perform convolution processing on the input N+1th batch of data according to the second weight, and perform model training on the first neural network model to obtain the trained first neural network model. In the above manner, since the first convolution layer determines the weight used for the current batch by the BN coefficient updated after the BN layer processes the data of the previous batch, there is no need to set up a separate convolution layer. On the one hand, the model size can be reduced, and on the other hand, the amount of data calculation of the convolution layer in the neural network is also reduced. Since the training process requires a large number of iterations, the available computing resources of the training device are limited. In this embodiment, the convolution operation of the convolution layer in the neural network is reduced by one during the training process. During a large number of training processes, the computing resource consumption of the training device can be greatly reduced, thereby improving the training speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A structural diagram of the main framework of artificial intelligence;

[0046] Figure 2 Schematic diagram of folding CNN and BN for QAT;

[0047] Figure 3 A schematic diagram of a convolutional neural network provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of a convolutional neural network provided in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of a system architecture provided in an embodiment of the present application;

[0050] Figure 6 This is an example of a model training method provided in an embodiment of the present application;

[0051] Figure 7 A schematic diagram of BN folding provided in an embodiment of the present application;

[0052] Figure 8 A schematic diagram of the structure of a convolutional BN layer provided in an embodiment of the present application;

[0053] Figure 9 A schematic diagram of an element cutout provided in an embodiment of the present application;

[0054] Figure 10 A schematic diagram of a model training device 1000 provided in an embodiment of the present application;

[0055] Figure 11 A schematic diagram of the structure of an execution device provided in an embodiment of the present application;

[0056] Figure 12 This is a structural diagram of a training device provided in an embodiment of the present application;

[0057] Figure 13 A schematic diagram of the structure of the chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following describes the embodiments of the present invention in conjunction with the accompanying drawings. The terms used in the embodiments of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0059] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0060] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0061] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1 The following diagram illustrates a structural diagram of the AI ​​framework. This framework is explained below from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it encompasses the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed progression from "data-information-knowledge-wisdom." The "IT value chain," encompassing the entire process from the underlying infrastructure of human intelligence, information (provided and processed by technology), to the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.

[0062] (1) Infrastructure

[0063] Infrastructure provides computing power for AI systems, enabling communication with the outside world and supporting this through a foundational platform. External communication occurs through sensors; computing power is provided by intelligent chips (CPUs, NPUs, GPUs, ASICs, FPGAs, and other hardware accelerators). The foundational platform includes a distributed computing framework and network-related platform guarantees and support, including cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then fed into the intelligent chips within the distributed computing system provided by the foundational platform for computation.

[0064] (2) Data

[0065] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0066] (3) Data processing

[0067] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0068] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0069] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0070] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0071] (4) General ability

[0072] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0073] (5) Smart products and industry applications

[0074] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, safe cities, etc.

[0075] The embodiments of the present application can be applied in scenarios such as image classification, object detection, semantic segmentation, room layout, image completion or automatic encoding, etc.

[0076] The following is a brief introduction to the application scenarios of this application using the two application scenarios of ADAS / ADS visual perception system and mobile phone beautification as examples.

[0077] Application Scenario 1: ADAS / ADS Visual Perception System

[0078] ADAS and ADS require real-time 2D object detection for multiple types of objects, including dynamic obstacles (pedestrians, cyclists, tricycles, cars, trucks, and buses), static obstacles (traffic cones, traffic sticks, fire hydrants, motorcycles, and bicycles), and traffic signs (traffic signs, guide signs, billboards, red / yellow / green / black traffic lights, and road signs). Furthermore, to accurately determine the area occupied by dynamic obstacles in 3D space, 3D estimation is required to output a 3D bounding box. To fuse the data with the LiDAR, it is necessary to obtain a mask for dynamic obstacles, thereby filtering out the laser point cloud that hits them. To accurately park a car, it is necessary to simultaneously detect the four key points of the parking space. To perform composition positioning, it is necessary to detect the key points of static targets. The neural network model trained using the technical solution provided in the embodiments of this application can complete all or part of the functions of the ADAS / ADS visual perception system described above.

[0079] Application scenario 2: mobile phone beauty function

[0080] In a mobile phone, the neural network model trained by the technical solution provided in the embodiment of the present application (for example, the trained first neural network model, the second neural network model, and the third neural network model) can detect the mask and key points of the human body, and can enlarge and reduce the corresponding parts of the human body, such as performing waist-tightening and buttock-beautifying operations, thereby outputting a beauty image.

[0081] Application scenario 3: Image classification scenario:

[0082] After acquiring an image to be classified, the categories of objects in the image to be classified can be obtained based on a neural network, and then the image to be classified can be classified according to the categories of the objects in the image to be classified. Photographers take many photos every day, including animals, people, and plants. Using the method of this application, photos can be quickly classified according to their content into photos containing animals, photos containing people, and photos containing plants.

[0083] When there are a large number of images, manual classification is inefficient, and people are prone to fatigue when dealing with the same thing for a long time, and the classification results will have large errors. However, the neural network model trained by the technical solution provided in the embodiments of the present application (for example, the trained first neural network model, the second neural network model, and the third neural network model) can quickly classify images.

[0084] The embodiments of the present application can train a neural network, and the trained neural network can process tasks in the above-mentioned scenarios.

[0085] Neural network quantization is a model compression technology that converts floating-point storage (operations) into integer storage (operations). For example, the model parameters of a model originally represented by float32 (32-bit floating point) are represented by int8 (8-bit fixed point) after quantization. Through the quantization operation of the model, the model's operation speed is improved at the expense of a small loss of accuracy.

[0086] The essence of model quantization is the conversion / mapping between two data types. In one implementation of converting floating-point data (data of floating-point data type) to fixed-point data (data of fixed-point data type), the following formula can be used:

[0087]

[0088] Among them, R is the input floating-point data, Q is the fixed-point data after the floating-point data R is quantized, Z represents the zero point value (ZeroPoint), and S represents the ratio. It can be seen that after determining S and Z, the conversion between the two data can be performed. There are many ways to determine S and Z, for example:

[0089]

[0090] Z=Q max -R max / S;

[0091] Among them, Rmax represents the maximum value of the input floating-point data, Rmin represents the minimum value of the input floating-point data, Qmax represents the maximum value of the fixed-point data, and Rmin represents the minimum value of the fixed-point data.

[0092] Among them, the conversion between fixed-point data with different bit numbers (bits, 1 bit = 1 position) can refer to the conversion method between floating-point data and fixed-point data mentioned above, or it can be other conversion methods in the prior art, which will not be repeated here.

[0093] In one implementation, 4-bit and 8-bit conversions can be performed using the above-mentioned conversion method, while floating-point data and 2-bit (1-bit) conversions can be performed using the following formula:

[0094]

[0095] The 2-bit data can be represented by three numbers: -1, 0, and 1. T is the threshold. When the floating-point data is greater than or equal to T, the converted 2-bit fixed-point data is 1. When the floating-point data is less than -T, its value is converted to -1. For other floating-point values, its value is converted to 0. The conversion method for 1-bit data is similar to that for 2-bit data, but its fixed-point values ​​are only -1 and 1, and the value of T is 0.

[0096] Quantization aware training (QAT) uses training data to compensate for the accuracy loss caused by quantization. Its main process is: 1. Inserting a quantization operator before model training. 2. During training, the minimum and maximum values ​​of each layer (weights and activations) of the model are counted to calculate the quantization factor.

[0097] During the model training phase, QAT needs to insert a pseudo-quantization node SimQuant (also called a quantization operator in this embodiment) into the weight input and activation output of the original model. In addition, for the convolutional neural network (CNN) and batch normalization (BN) structure, another CNN is required to implement BN folding to achieve the fusion of BN coefficients and CNN weights. During the training process, SimQuant will count the min and max values ​​in the corresponding data stream (Tensor) for the subsequent calculation of the scale quantization factor. Figure 2 As shown in the figure, when QAT folds CNN and BN, it needs to build another CNN to perform convolution operations on the data of the current batch. BN uses the results of the convolution operation to update the BN coefficients, and then uses the updated BN coefficients to construct weights. The quantization operator can quantize and dequantize the constructed weights, and CNN can perform convolution operations on the data of the current batch based on the weights obtained after dequantization. However, because another CNN is required to implement BN folding, two CNNs will perform convolution operations on the same batch of data during the training process, which increases the amount of calculations required by the CNN during training and thus reduces the training speed.

[0098] In the upper neural network provided in the embodiment of the present application, when folding CNN and BN, the computational complexity of CNN can be reduced.

[0099] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0100] (1) Neural Network

[0101] A neural network can be composed of neural units. A neural unit can refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input. The output of the operation unit can be:

[0102] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal of the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0103] (2) Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or convolution feature plane (feature map). The convolution layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolution layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolution layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are the convolution kernels. Shared weights can be understood as the way of extracting image information is independent of position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. This means that the image information learned in a part can also be used in another part. Therefore, for all positions on the image, we can use the same learned image information. In the same convolutional layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0104] Convolution kernels can be initialized as matrices of random size, and during the training process of the convolutional neural network, the convolution kernels can be learned to obtain reasonable weights. In addition, the direct benefit of shared weights is that they reduce the number of connections between the layers of the convolutional neural network, while also reducing the risk of overfitting.

[0105] (5) Backpropagation algorithm

[0106] Convolutional neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during training, reducing the reconstruction error loss of the super-resolution model. Specifically, the forward propagation of the input signal to the output generates an error loss. This error loss information is then backpropagated to update the parameters of the initial super-resolution model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by the error loss, aiming to obtain the optimal super-resolution model parameters, such as the weight matrix.

[0107] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. It is a deep learning architecture, which uses machine learning algorithms to perform multiple levels of learning at different levels of abstraction. As a deep learning architecture, a CNN is a feed-forward artificial neural network in which individual neurons respond to overlapping regions in an input image.

[0108] like Figure 3 As shown, a convolutional neural network (CNN) 100 may include an input layer 110 , a convolutional layer / pooling layer 120 , wherein the pooling layer is optional, and a neural network layer 130 .

[0109] Among them, the structure composed of the convolution layer / pooling layer 120 and the neural network layer 130 can be the first convolution layer and the second convolution layer described in this application, the input layer 110 is connected to the convolution layer / pooling layer 120, the convolution layer / pooling layer 120 is connected to the neural network layer 130, the output of the neural network layer 130 can be input to the activation layer, and the activation layer can perform nonlinear processing on the output of the neural network layer 130.

[0110] Convolutional layer / pooling layer 120:

[0111] Convolutional layer:

[0112] like Figure 3The convolutional layer / pooling layer 120 shown may include layers 121-126, for example. In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of a convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0113] Taking convolution layer 121 as an example, convolution layer 121 can include many convolution operators, also known as kernels. Their role in image processing is equivalent to a filter that extracts specific information from the input image matrix. The convolution operator can essentially be a weight matrix, which is usually predefined. During the convolution operation on the image, the weight matrix is ​​usually processed horizontally on the input image one pixel at a time (or two pixels at a time... depending on the value of the stride), thereby completing the task of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It is important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. During the convolution operation, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a convolution output with a single depth dimension. However, in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same dimension are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolved image. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unwanted noise in the image... The multiple weight matrices have the same dimensions, and the feature maps extracted by the multiple weight matrices with the same dimensions also have the same dimensions. The multiple feature maps with the same dimensions extracted are then merged to form the output of the convolution operation.

[0114] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0115] When the convolutional neural network 100 has multiple convolutional layers, the initial convolutional layer (for example, 121) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by the later convolutional layers (for example, 126) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0116] Pooling layer:

[0117] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer, such as Figure 3 The layers 121-126 in the example 120 can be a convolution layer followed by a pooling layer, or multiple convolution layers can be followed by one or more pooling layers.

[0118] Neural Network Layer 130:

[0119] After being processed by the convolution layer / pooling layer 120, the convolution neural network 100 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 120 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 100 needs to use the neural network layer 130 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 130 may include multiple hidden layers (such as Figure 3 131, 132 to 13n) and the output layer 140 shown, the parameters contained in the multiple hidden layers can be pre-trained based on relevant training data of specific task types, for example, the task types may include image recognition, image classification, image super-resolution reconstruction, etc.

[0120] After the multiple hidden layers in the neural network layer 130, that is, the last layer of the entire convolutional neural network 100 is the output layer 140, which has a loss function similar to the classification cross entropy, specifically for calculating the prediction error. Once the forward propagation of the entire convolutional neural network 100 (such as Figure 3 The propagation from 110 to 140 is forward propagation), and the reverse propagation (such as Figure 3 The propagation from 140 to 110 is back propagation) and then starts to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network 100 and the error between the result output by the convolutional neural network 100 through the output layer and the ideal result.

[0121] It should be noted that if Figure 3The convolutional neural network 100 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models, such as Figure 4 The multiple convolutional layers / pooling layers shown are operated in parallel, and the features extracted from each layer are input to the full neural network layer 130 for processing.

[0122] (3) BN: Through mini-batch normalization, the differences in parameter optimization caused by inputs at different levels are eliminated, the possibility of overfitting in a certain layer of the model is reduced, and training can proceed more smoothly. The BN coefficients can be: mean μ, variance σ, scale parameter γ, and offset parameter β.

[0123] (4) BN-folding: The main purpose is to fuse BN and CNN calculations to reduce the amount of computation. This method is mainly used in QAT, so that training quantization can simulate the inference BN fusion process, so that BN and CNN can be fused in the model conversion (the relevant coefficient roots are merged into one coefficient according to the calculation rules), accelerating the model inference efficiency.

[0124] (5) Convolutional BN (ConvBn) represents the fusion operator of convolution and BN. This operator implements the functions of both CNN and BN. Since the coefficients of BN are visible to CNN, it is easy to implement BN folding, so that the CNN convolution weights and related BN coefficients are fused.

[0125] Figure 5 This is a schematic diagram of a system architecture 100 provided in an embodiment of the present application. Figure 5 In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. A user can input data to the I / O interface 112 through a client device 140 .

[0126] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing (such as implementing the functions of the neural network in this application), the execution device 110 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 150.

[0127] Finally, the I / O interface 112 returns the processing result to the client device 140 so as to provide it to the user.

[0128] Optionally, the client device 140 may be, for example, a control unit in an autonomous driving system or a functional algorithm module in a mobile phone terminal. For example, the functional algorithm module may be used to implement related tasks.

[0129] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different goals or different tasks. The corresponding target models / rules can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0130] exist Figure 5 In the case shown, the user can manually input data, which can be operated through the interface provided by I / O interface 112. In another case, client device 140 can automatically send input data to I / O interface 112. If the automatic transmission of input data by client device 140 requires user authorization, the user can set the corresponding permissions in client device 140. The user can view the results output by execution device 110 on client device 140, which can be displayed, sounded, or displayed in a specific form. Client device 140 can also serve as a data acquisition terminal, collecting input data input into I / O interface 112 and output results from I / O interface 112 as new sample data, and storing them in database 130. Of course, the collection can also be performed without going through client device 140, and instead the input data input into I / O interface 112 and output results from I / O interface 112 as new sample data can be directly stored in database 130 by I / O interface 112.

[0131] It is worth noting that Figure 5 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, Figure 5 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110.

[0132] First, the model training method provided in the embodiment of the present application is described by taking the model training stage as an example.

[0133] Reference Figure 6 , Figure 6 An example of a model training method provided in the present application is shown in FIG. Figure 6 As shown, a model training method provided in an embodiment of the present application includes:

[0134] 601. Obtain a first neural network model, wherein the first neural network model includes a convolutional BN layer and a first quantization operator, the convolutional BN layer is used to perform convolution processing on the input Nth batch data according to a first weight, normalize the convolution processing result according to the BN coefficient, and update the BN coefficient based on the normalization processing result, and update the first weight of the updated BN coefficient. The first quantization operator is used to quantize and dequantize the updated first weight to obtain a second weight. The convolutional BN layer is also used to perform convolution processing on the input N+1th batch data according to the second weight.

[0135] The first neural network model may be obtained by performing BN folding processing on a pre-trained model and adding a quantization operator (also referred to as a pseudo-quantization node SimQuant).

[0136] In an embodiment of the present application, the training device can obtain a second neural network model, which is a pre-trained model, and the second neural network model includes the first convolutional layer and the first BN layer; the first convolutional layer and the first BN layer in the second neural network model are folded to obtain the first neural network model, and the first neural network model includes a convolutional BN layer obtained by folding the first convolutional layer and the first BN layer, and the convolutional BN layer includes the first convolutional layer and the first BN layer.

[0137] In the embodiment of the present application, the second neural network model is a pre-trained model, and the second neural network model is trained so that it has high data processing accuracy for a specific task. In order to quantize the weights of the second neural network model (more specifically, the weights in the convolutional layer can be quantized), a quantization operator can be inserted into the second neural network model, and the convolutional layer and the BN layer can be folded.

[0138] For details, please refer to Figure 7 , Figure 7 A schematic diagram of BN folding provided in an embodiment of the present application is shown as follows: Figure 7 As shown, the second neural network model includes the first convolutional layer and the first BN layer. The training device can fold the first convolutional layer and the first BN layer in the second neural network model to obtain the first neural network model, and the first neural network model includes the convolutional BN layer obtained by folding the first convolutional layer and the first BN layer.

[0139] In one possible implementation, the first convolutional layer in the second neural network model is configured to perform convolution processing on input data based on target weights. To quantize the target weights and implement BN folding, the target weights are multiplied by BN coefficients, and the product is quantized and dequantized using a quantization operator. The dequantized result is then used as the weight of the first convolutional layer.

[0140] Specifically, the first neural network model includes a convolutional BN layer and a first quantization operator. The convolutional BN layer may include a first convolutional layer and a first batch normalization BN layer. The first convolutional layer is used to perform convolution processing on the input data according to the first weight to obtain a first output. The first BN layer is used to normalize the first output according to the BN coefficient and update the BN coefficient based on the normalization result. The first quantization operator is used to quantize and dequantize the updated first weight according to the first quantization factor to obtain a second weight. The updated first weight is obtained according to the updated BN coefficient. The first convolutional layer is used to perform convolution processing on the input data according to the second weight.

[0141] In an embodiment of the present application, for multiple batches of data, the BN coefficient can be updated by the data of the previous batch, and the weight of the current convolution layer can be updated based on the BN coefficient updated by the data of the previous batch. Specifically, the first neural network model includes a first convolution layer, a first batch normalization BN layer, and a first quantization operator. The first convolution layer is used to perform convolution processing on the input Nth batch of data according to the first weight to obtain a first output. The first BN layer is used to normalize the first output according to the BN coefficient and update the BN coefficient based on the normalization result. The first quantization operator is used to quantize and dequantize the updated first weight according to the first quantization factor to obtain a second weight. The updated first weight is obtained according to the updated BN coefficient. The first convolution layer is used to perform convolution processing on the input N+1th batch of data according to the second weight to obtain a second output. Wherein, the first weight is obtained according to the product result of the BN coefficient and the target weight, and the updated first weight is obtained by multiplying the updated BN coefficient and the target weight.

[0142] For details, please refer to Figure 8 , Figure 8 A structural diagram of a convolutional BN layer provided in an embodiment of the present application is shown as follows: Figure 8As shown in the figure, conv represents a convolutional layer, bn represents a batch normalization layer, div represents division, and mul represents multiplication. The first convolutional layer conv can perform convolution processing on the data of the previous batch (that is, the data of the Nth batch) to obtain a first output. It should be understood that the first output is the result of dividing the convolution processing result obtained by the first convolutional layer conv on the data of the previous batch by the batch normalization coefficient.

[0143] The first output can be used as input to the first BN layer, and the first BN layer can normalize the first output and update the BN coefficient based on the normalization result. During the training process, the BN layer performs a BN operation based on the mean and standard deviation of the output features of the convolution layer in the feedforward process. Exemplarily, the first BN layer is connected to the first convolution layer, and the first BN layer is used to perform a BN operation on the first output according to the mean and standard deviation of the first output of the first convolution layer. Afterwards, the training device can update the BN coefficient based on the operation result, wherein the BN coefficient may include but is not limited to at least one of the mean μ, variance σ, scale parameter γ and offset parameter β, or the operation result between any two or more of them.

[0144] In the embodiment of the present application, the convolutional BN layer is obtained by folding the convolutional layer and the BN layer, the first weight is obtained according to the product of the BN coefficient and the target weight, the updated first weight is obtained by multiplying the updated BN coefficient and the target weight, and the target weight is the weight included in the convolutional layer. The updated BN coefficient (such as Figure 8 The scale parameter γ is shown new and variance σ new ), then we can adjust the scale parameter γ new and variance σ new Divide to obtain γ / σ. In order to perform BN folding, the updated BN coefficient (for example, γ / σ) can be multiplied by the target weight W, and the product result (γ / σ*W) is input into the first quantization operator. The first quantization operator is used to quantize and dequantize the updated first weight according to the first quantization factor to obtain a second weight. The first convolution layer can perform convolution on the next batch of data (that is, the N+1th batch of data) according to the second weight to obtain a second output. It should be understood that the number of bits to be quantized in the quantization operator also needs to be set.

[0145] Specifically, in order to identify the convolutional layers and BN layers that need to be BN folded in the second neural network model, the structure of the convolutional layers and BN layers that need to be BN folded in the model can be judged according to the operator type in the computational flow graph of the second neural network model (this embodiment can also be described as a CNN+BN structure), and the identified CNN+BN structure can be combined into a block (that is, the convolutional BN layer in the above embodiment); then the combined convolutional BN layer can replace the original CNN+BN structure.

[0146] In the prior art, the first convolutional layer determines the weights used for the current batch by updating the BN coefficients after the BN layer processes the data of the current batch. Therefore, in addition to the first convolutional layer, a separate convolutional layer is required to process the data so that the BN layer can update the BN coefficients based on the data of the current batch. In the embodiment of the present application, since the first convolutional layer determines the weights used for the current batch by updating the BN coefficients after the BN layer processes the data of the previous batch, there is no need to set up another separate convolutional layer. On the one hand, the model size can be reduced, and on the other hand, the amount of data calculations of the convolutional layer in the neural network is reduced. Since the training process is a process that requires a large number of iterations, the available computing resources of the training equipment are limited. In this embodiment, the convolutional layer in the neural network is reduced by one convolution operation during the training process. In a large number of training processes, the computing resource consumption of the training equipment can be greatly reduced, thereby improving the training speed.

[0147] In addition, the forward graph for training and the inference graph are the same, which reduces the complexity of model storage and conversion.

[0148] In one possible implementation, the first convolutional layer in the first neural network model is used to perform convolution processing on the input N+1th batch data according to the second weight to obtain a convolution processing result, and divide the convolution processing result by the updated BN coefficient to obtain a second output.

[0149] In an embodiment of the present application, the first weight in the first neural network model can be initialized. Specifically, the BN coefficient can be multiplied by the target weight to obtain a first target tensor, which includes M elements; the N target elements with the largest absolute values ​​among the M elements included in the first target tensor are replaced with the largest element among the MN elements in the M elements other than the N target elements to obtain the first weight.

[0150] In an embodiment of the present application, the weights in the second neural network model (pre-trained model) and the coefficients of BN can be used to initialize the first weights in the first neural network model. Specifically, the BN coefficients can be multiplied by the target weights according to the pre-trained model to obtain a first target tensor. For example, the first target tensor can be γ / σ*W, and then the elements in the first target tensor are sorted by size, and the trunk values ​​are intercepted in a symmetrical manner (such as 95% to 99.5%), and the remaining elements are replaced with the largest value in the trunk values, so as to achieve the initialization of the first weight.

[0151] For example, you can Figure 9 As shown, Figure 9 This is an example of an element interception provided in an embodiment of the present application. After arranging the elements in the first target tensor from large to small, the following can be obtained: Figure 9 The distribution shown in , wherein a certain percentage of elements can be intercepted, the percentage can be but not limited to 95% to 99.5%, and 95% to 99.5% of the elements can be the elements of the main part of the element distribution, that is, the elements with absolute values ​​close to 0.

[0152] In an embodiment of the present application, the number of elements with large absolute values ​​in the first target tensor is small. During the subsequent quantization and dequantization process, the accuracy of the operation will be affected due to the large absolute values. For example, unnecessary smoothing of other elements of the quantization factor will be performed. The embodiment of the present application improves the accuracy of neural network model processing by truncating elements of the first target tensor.

[0153] In one possible implementation, the second neural network model also includes a target activation layer, and the target activation layer in the second neural network model is used to process the input data to obtain a third output. The first neural network model also includes the target activation layer and a second quantization operator, and the target activation layer in the first neural network model is used to process the input data to obtain a fourth output. The second quantization operator is used to quantize and dequantize the fourth output according to a second quantization factor.

[0154] Similar to the above embodiment, in an embodiment of the present application, in order to quantize the outputs of each activation layer in the second neural network, a second quantization operator can be added at the output position of the activation layer. Specifically, the second neural network model also includes a target activation layer, and the target activation layer in the second neural network model is used to process the input data to obtain a third output. The first neural network model also includes the target activation layer and a second quantization operator. The target activation layer in the first neural network model is used to process the input data to obtain a fourth output. The second quantization operator is used to quantize and dequantize the fourth output according to a second quantization factor.

[0155] In one possible implementation, the third output is a second target tensor, where the second target tensor includes X elements. The method further includes obtaining Y target elements having the largest absolute values ​​among the X elements, and replacing the Y target elements in the second target tensor with a largest element among XY elements in the X elements excluding the Y target elements, to obtain the second quantization factor.

[0156] Similar to the above embodiment, in the embodiment of the present application, in the process of initializing the quantization factor located at the output position of the activation layer, after arranging the elements in the first and second target tensors from large to small, a certain percentage of the elements can be intercepted. The percentage can be but is not limited to 95% to 99.5%, and 95% to 99.5% of the elements can be the elements of the main part of the element distribution, that is, elements whose absolute values ​​are close to 0.

[0157] 602. Perform model training on the first neural network model to obtain a trained first neural network model.

[0158] In an embodiment of the present application, after obtaining the first neural network model, model training can be performed on the first neural network model to obtain a trained first neural network model.

[0159] Specifically, the model can be quantized and trained according to the set epoch. During the training process, if the freeze-bn operation is performed in the current epoch, the quantized model is obtained by training in the current epoch, and the current quantized model is verified by reasoning in the current epoch. If the freeze-bn operation is not performed in the current epoch, the quantized model is obtained by training in the current epoch, and the current quantized model is verified by reasoning in the current epoch.

[0160] In one possible implementation, the trained first neural network model includes a trained first quantization factor and a trained BN coefficient. The training device can also quantize the first neural network model according to the trained first quantization factor and the trained BN coefficient to obtain a third neural network model. The third neural network model includes the quantized first convolution layer, and the first convolution layer is used to perform convolution processing on the input data according to the quantized weights, and the quantized weights are obtained according to the first quantization factor and the trained BN coefficients.

[0161] For example, if the input X of the first convolutional layer is of UINT type and the weight W of the first convolutional layer is of INT type, to convert it into UINT type reasoning, you can load the third neural network model into the converter and perform the following operations on each layer of the model: The weights will be quantized and saved as UINT type, where bits is the number of quantized bits. For example, after 8 bits, the scale quantization factor values ​​and weight quantization of each layer will be saved to the inference model.

[0162] The embodiment of the present application provides a model training method, which includes: obtaining a first neural network model, wherein the first neural network model includes a convolutional BN layer and a first quantization operator, the convolutional BN layer is used to perform convolution processing on the input Nth batch of data according to a first weight, normalize the convolution processing result according to the BN coefficient, and update the BN coefficient based on the normalization processing result, update the first weight of the updated BN coefficient, the first quantization operator is used to quantize and dequantize the updated first weight to obtain a second weight, the convolutional BN layer is also used to perform convolution processing on the input N+1th batch of data according to the second weight, and perform model training on the first neural network model to obtain the trained first neural network model. In the above manner, since the first convolution layer determines the weight used for the current batch by the BN coefficient updated after the BN layer processes the data of the previous batch, there is no need to set up a separate convolution layer. On the one hand, the model size can be reduced, and on the other hand, the amount of data calculation of the convolution layer in the neural network is also reduced. Since the training process requires a large number of iterations, the available computing resources of the training device are limited. In this embodiment, the convolution operation of the convolution layer in the neural network is reduced by one during the training process. During a large number of training processes, the computing resource consumption of the training device can be greatly reduced, thereby improving the training speed.

[0163] Next, the model training method in the embodiment of the present application is described with reference to a specific example.

[0164] In this embodiment, two types of convolutional batch normalization (BN) layers (ConvBn) can be defined: 1. ConvBnV1, which inserts only weight quantization nodes; 2. ConvBnV2, which inserts both weight quantization nodes and activation quantization nodes. For a [CNN+BN+activation operator] structure, directly replace it with ConvBnV1. However, for a [CNN+BN] structure (i.e., direct output after BN without an activation operator), directly replace it with ConvBnV2.

[0165] In the training stage, the activation quantization bit number is set to 8, the first layer weight quantization bit number is set to 8, and the remaining weight quantization bit number is set to 4 as an example. First, according to the MobileNetV2 structure, an activation quantization node can be inserted after the activation operator ReLU6 (quantization range: 0~255, quantization bit number is set to 8), and a quantization node can be inserted after the Add operator of the residual structure (quantization range: -127~127, quantization bit number is set to 8). A quantization node is inserted after the fully connected (FC) operator (quantization range: -7~7, quantization bit number is set to 4). Then, the [CNN+BN+ReLU6] and [CNN+BN] structures in MobileNetV2 are scanned and replaced with ConvBnV1 and ConvBnV2 respectively to realize BN folding; the activated quantization in the ConvBnV2 operator is The number of bits is set to 8, and the number of weight quantization bits in ConvBnV1 and ConvBnV2 is set according to the situation (for example, the number of weight quantization bits of the first layer of the model is set to 8, and the number of quantization bits of the weights of the remaining layers is set to 4); then the pre-trained model is loaded, the model structure and the corresponding weights are mapped one by one layer by layer, and the corresponding scale quantization factor values ​​are initialized using the weights and BN coefficients in the pre-trained model, and the proportion of backbone truncation is set to 95%; the pre-trained model and arbitrarily selected 256 training set data are used for inference to obtain the activation X of each layer, and the scale quantization factor corresponding to the initial X of each layer is used, and the proportion of backbone truncation is 99.5%; the model is quantized and trained according to epoch = 20, and the current quantization model is verified by inference at the current epoch.

[0166] In the model conversion stage, taking full 8-bit quantization as an example, the model weight quantization range is -127 to 127, the ReLU6 quantization range is 0 to 255, the quantization range after BN without activation is -127 to 127, and the quantization range after the residual structure is added is -127 to 127. To convert this model to UINT reasoning, first load the quantized model into the converter and perform the quantization of each layer of the model according to The weights will be quantized and saved as UINT type, where bits is the number of quantized bits. For example, after 8 bits, the scale quantization factor values ​​and weight quantization of each layer will be saved to the inference model.

[0167] Reference Figure 10 , Figure 10 A schematic diagram of a model training device 1000 provided in an embodiment of the present application is shown as follows: Figure 10 As shown in FIG, the model training device 1000 provided by the present application includes:

[0168] An acquisition module 1001 is configured to acquire a first neural network model, wherein the first neural network model includes a convolutional BN layer and a first quantization operator, the convolutional BN layer being configured to perform convolution processing on the input Nth batch of data according to a first weight, normalize the convolution processing result according to the BN coefficient, update the BN coefficient based on the normalization processing result, and update the first weight on the updated BN coefficient. The first quantization operator is configured to perform quantization and dequantization processing on the updated first weight to obtain a second weight, and the convolutional BN layer is further configured to perform convolution processing on the input N+1th batch of data according to the second weight;

[0169] The model training module 1002 is used to perform model training on the first neural network model to obtain a trained first neural network model.

[0170] In the prior art, the first convolutional layer determines the weights used for the current batch by updating the BN coefficients after the BN layer processes the data of the current batch. Therefore, in addition to the first convolutional layer, a separate convolutional layer is required to process the data so that the BN layer can update the BN coefficients based on the data of the current batch. In the embodiment of the present application, since the first convolutional layer determines the weights used for the current batch by updating the BN coefficients after the BN layer processes the data of the previous batch, there is no need to set up another separate convolutional layer. On the one hand, the model size can be reduced, and on the other hand, the amount of data calculations of the convolutional layer in the neural network is reduced. Since the training process is a process that requires a large number of iterations, the available computing resources of the training equipment are limited. In this embodiment, the convolutional layer in the neural network is reduced by one convolution operation during the training process. In a large number of training processes, the computing resource consumption of the training equipment can be greatly reduced, thereby improving the training speed.

[0171] In one possible implementation, the acquisition module 1001 is used to obtain a second neural network model, which is a pre-trained model and includes a first convolutional layer and a first BN layer; performing BN folding processing on the first convolutional layer and the first BN layer to obtain the first neural network model, and the first neural network model includes the convolutional BN layer obtained after folding the first convolutional layer and the first BN layer.

[0172] In one possible implementation, the convolutional BN layer is obtained by folding the convolutional layer and the BN layer, the first weight is obtained according to the product of the BN coefficient and the target weight, the updated first weight is obtained by multiplying the updated BN coefficient and the target weight, and the target weight is the weight included in the convolutional layer.

[0173] In a possible implementation, the apparatus further includes:

[0174] a product operation module, configured to perform a product operation on the BN coefficient and the target weight to obtain a first target tensor, where the first target tensor includes M elements;

[0175] An element replacement module is used to replace the N target elements with the largest absolute values ​​among the M elements included in the first target tensor with the largest element among the MN elements in the M elements excluding the N target elements, so as to obtain the first weight.

[0176] In an embodiment of the present application, the first weight in the first neural network model can be initialized using the weights and BN coefficients in the second neural network model (pre-trained model). Specifically, the BN coefficients can be multiplied by the target weights according to the pre-trained model to obtain a first target tensor. For example, the first target tensor can be γ / σ*W, and then the elements in the first target tensor are sorted according to size, and the main part values ​​are intercepted in a symmetrical manner (such as: intercepting 95% to 99.5%), and the remaining elements are replaced with the largest value in the main part values, so as to achieve the initialization of the first weight. Wherein, the first weight is in the form of a tensor.

[0177] The number of elements with large absolute values ​​in the first target tensor is small. During the subsequent quantization and dequantization process, the accuracy of the operation will be affected due to the large absolute values. For example, unnecessary smoothing of other elements of the quantization factor will be performed. The embodiment of the present application improves the accuracy of neural network model processing by truncating elements of the first target tensor.

[0178] In one possible implementation, the first convolutional layer in the first neural network model is used to perform convolution processing on the input N+1th batch data according to the second weight to obtain a convolution processing result, and divide the convolution processing result by the updated BN coefficient to obtain the second output.

[0179] In one possible implementation, the second neural network model also includes a target activation layer, and the target activation layer in the second neural network model is used to process the input data to obtain a third output. The first neural network model also includes the target activation layer and a second quantization operator, and the target activation layer in the first neural network model is used to process the input data to obtain a fourth output. The second quantization operator is used to quantize and dequantize the fourth output according to a second quantization factor.

[0180] In a possible implementation, the third output is a second target tensor, the second target tensor includes X elements, and the acquisition module is used to acquire Y target elements with the largest absolute values ​​among the X elements;

[0181] The element replacement module is configured to replace the Y target elements in the second target tensor with a maximum element among XY elements in the X elements excluding the Y target elements, so as to obtain the second quantization factor.

[0182] Similar to the above embodiment, in the embodiment of the present application, in the process of initializing the quantization factor located at the output position of the activation layer, after arranging the elements in the second target tensor from large to small, a certain percentage of the elements can be intercepted. The percentage can be but is not limited to 95% to 99.5%, and 95% to 99.5% of the elements can be the elements of the main part of the element distribution, that is, elements whose absolute values ​​are close to 0.

[0183] In a possible implementation, the trained first neural network model includes a trained first quantization factor and a trained BN coefficient, and the apparatus further includes:

[0184] A quantization module is used to quantize the first neural network model according to the trained first quantization factor and the trained BN coefficient to obtain a third neural network model, wherein the third neural network model includes the quantized first convolution layer, and the quantized first convolution layer is used to perform convolution processing on the input data according to the quantized weights, and the quantized weights are obtained according to the first quantization factor and the trained BN coefficients.

[0185] The relevant description of the product operation module can refer to the above embodiment, which describes how to multiply the BN coefficient and the target weight to obtain a first target tensor, and the first target tensor includes M elements, which will not be repeated here.

[0186] The relevant description of the element replacement module can refer to the above embodiment, which describes how to replace the N target elements with the largest absolute values ​​among the M elements included in the first target tensor with the largest element among the MN elements in the M elements except the N target elements to obtain the first weight, which will not be repeated here.

[0187] The quantization module can refer to the above embodiment on how to quantize the first neural network model according to the trained first quantization factor and the trained BN coefficient to obtain a third neural network model, and the third neural network model includes the quantized first convolution layer, and the quantized first convolution layer is used to perform convolution processing on the input data according to the quantized weights. The quantized weights are obtained according to the first quantization factor and the trained BN coefficient, and will not be repeated here.

[0188] Next, we will introduce an execution device provided by the embodiment of the present application. Figure 11 , Figure 11 This is a schematic diagram of the structure of the execution device provided in the embodiment of the present application. The execution device 1200 can be specifically manifested as a mobile phone, tablet, laptop, smart wearable device, server, etc., which is not limited here. Among them, the execution device 1200 can be deployed with Figure 10 The data processing device described in the corresponding embodiment is used to implement Figure 10 The data processing function in the corresponding embodiment. Specifically, the execution device 1200 includes: a receiver 1201, a transmitter 1202, a processor 1203 and a memory 1204 (wherein the number of the processor 1203 in the execution device 1200 can be one or more, Figure 11 (taking one processor as an example), the processor 1203 may include an application processor 12031 and a communication processor 12032. In some embodiments of the present application, the receiver 1201, the transmitter 1202, the processor 1203 and the memory 1204 may be connected via a bus or other means.

[0189] The memory 1204 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1203. A portion of the memory 1204 may also include non-volatile random access memory (NVRAM). The memory 1204 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0190] Processor 1203 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0191] The method disclosed in the above embodiment of the present application can be applied to the processor 1203, or implemented by the processor 1203. The processor 1203 can be an integrated circuit chip with signal processing capabilities. During the implementation process, the steps of the above method can be completed by the integrated logic circuit of the hardware in the processor 1203 or the instructions in the form of software. The above-mentioned processor 1203 can be a general-purpose processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and a vision processor (vision processing unit, VPU), a tensor processing unit (tensorprocessing unit, TPU) and other processors suitable for AI computing, and can further include an application specific integrated circuit (ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 1203 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of this application can be directly implemented as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 1204, and processor 1203 reads the information in memory 1204 and, in conjunction with its hardware, completes the steps of the above method.

[0192] Receiver 1201 can be used to receive input digital or character information and generate signal input related to executing device-related settings and function control. Transmitter 1202 can be used to output digital or character information through the first interface. Transmitter 1202 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 1202 can also include a display device such as a display screen.

[0193] The execution device can obtain Figure 6 The model obtained by training the model training method in the corresponding embodiment is used to perform model inference.

[0194] The present application also provides a training device. Figure 12 , Figure 12 This is a structural diagram of a training device provided in an embodiment of the present application. Specifically, the training device 1300 is implemented by one or more servers. The training device 1300 may have relatively large differences due to different configurations or performances. It may include one or more central processing units (CPUs) 1313 (for example, one or more processors) and memory 1332, and one or more storage media 1330 (for example, one or more mass storage devices) storing application programs 1342 or data 1344. Among them, the memory 1332 and the storage medium 1330 can be short-term storage or persistent storage. The program stored in the storage medium 1330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processing unit 1313 can be configured to communicate with the storage medium 1330 to execute a series of instruction operations in the storage medium 1330 on the training device 1300.

[0195] The training device 1300 may also include one or more power supplies 1326, one or more wired or wireless network interfaces 1350, one or more input and output interfaces 1358; or, one or more operating systems 1341, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0196] Specifically, the training device can perform Figure 6 The model training method in the corresponding embodiment.

[0197] Figure 10 The model training apparatus 1000 described in the specification may be a module in a training device, and the processor in the training device may execute the model training method performed by the model training apparatus 1000 .

[0198] An embodiment of the present application also provides a computer program product, which, when running on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0199] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0200] The execution device, training device or terminal device provided in the embodiments of the present application can specifically be a chip, and the chip includes: a processing unit and a communication unit, the processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0201] For details, please refer to Figure 13 , Figure 13 A schematic diagram of the structure of a chip provided in an embodiment of the present application, which can be represented as a neural network processor NPU 1400. NPU 1400 is mounted on the host CPU as a coprocessor and is assigned tasks by the host CPU. The core of the NPU is arithmetic circuit 1403, which is controlled by controller 1404 to extract matrix data from memory and perform multiplication operations.

[0202] NPU 1400 can achieve Figure 6 The model training method provided in the described embodiments, or reasoning on the trained model.

[0203] Among them, the computing circuit 1403 in the NPU 1400 can execute the steps of obtaining a first neural network model and performing model training on the first neural network model.

[0204] More specifically, in some implementations, the arithmetic circuit 1403 in the NPU 1400 includes multiple processing units (PEs). In some implementations, the arithmetic circuit 1403 is a two-dimensional systolic array. The arithmetic circuit 1403 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1403 is a general-purpose matrix processor.

[0205] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 1402 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 1401 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 1408.

[0206] Unified memory 1406 is used to store input and output data. Weight data is directly transferred to weight memory 1402 through the Direct Memory Access Controller (DMAC) 1405. Input data is also transferred to unified memory 1406 through the DMAC.

[0207] BIU stands for Bus Interface Unit, i.e., bus interface unit 1410 , which is used for interaction between the AXI bus, DMAC, and instruction fetch buffer (IFB) 1409 .

[0208] The bus interface unit 1410 (BIU) is used for the instruction fetch memory 1409 to obtain instructions from the external memory, and is also used for the storage unit access controller 1405 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0209] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1406 or transfer weight data to the weight memory 1402 or transfer input data to the input memory 1401.

[0210] The vector calculation unit 1407 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit 1403, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0211] In some implementations, the vector calculation unit 1407 can store the processed output vector to the unified memory 1406. For example, the vector calculation unit 1407 can apply a linear function or a nonlinear function to the output of the operation circuit 1403, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values ​​to generate an activation value. In some implementations, the vector calculation unit 1407 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1403, for example, for use in subsequent layers in a neural network.

[0212] An instruction fetch buffer 1409 connected to the controller 1404 is used to store instructions used by the controller 1404;

[0213] Unified memory 1406, input memory 1401, weight memory 1402, and instruction fetch memory 1409 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0214] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0215] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0216] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0217] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0218] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A model training method, characterized in that: The method comprises: Obtain a first neural network model, wherein the first neural network model includes a convolutional BN layer and a first quantization operator, the convolutional BN layer is used to perform convolution processing on the input Nth batch of data according to a first weight, normalize the convolution processing result according to the BN coefficient, update the BN coefficient based on the normalization processing result, and update the first weight on the updated BN coefficient. The first quantization operator is used to quantize and dequantize the updated first weight to obtain a second weight. The convolutional BN layer is further used to perform convolution processing on the input N+1th batch of data according to the second weight, where the data includes at least one of the following: image, speech, and text; Performing model training on the first neural network model to obtain a trained first neural network model; The first quantization operator is used to perform quantization and dequantization on the updated first weight according to the first quantization factor, the trained first neural network model includes the trained first quantization factor and the trained BN coefficient, and the method further includes: The first neural network model is quantized according to the trained first quantization factor and the trained BN coefficient to obtain a third neural network model, wherein the third neural network model includes a quantized first convolution layer, and the quantized first convolution layer is used to perform convolution processing on the input data according to the quantized weights, and the quantized weights are obtained according to the first quantization factor and the trained BN coefficient.

2. The method according to claim 1, characterized in that The method further comprises: Obtain a second neural network model, where the second neural network model includes a first convolutional layer and a first BN layer; The first convolutional layer and the first BN layer are subjected to BN folding processing to obtain the first neural network model, wherein the first neural network model includes the convolutional BN layer obtained by folding the first convolutional layer and the first BN layer.

3. The method according to claim 1, characterized in that The convolutional BN layer is obtained by folding the convolutional layer and the BN layer, the first weight is obtained according to the product of the BN coefficient and the target weight, the updated first weight is obtained by multiplying the updated BN coefficient and the target weight, and the target weight is the weight included in the convolutional layer.

4. The method according to claim 3, characterized in that The method further comprises: The BN coefficient is multiplied by the target weight to obtain a first target tensor, where the first target tensor includes M elements; the N target elements with the largest absolute values ​​among the M elements included in the first target tensor are replaced by the largest element among the MN elements of the M elements excluding the N target elements to obtain the first weight.

5. The method according to claim 3 or 4, characterized in that The first convolutional layer in the first neural network model is used to perform convolution processing on the input N+1th batch data according to the second weight to obtain a convolution processing result, and divide the convolution processing result by the updated BN coefficient to obtain a second output.

6. The method according to claim 2, characterized in that The second neural network model also includes a target activation layer, and the target activation layer in the second neural network model is used to process the input data to obtain a third output. The first neural network model also includes the target activation layer and a second quantization operator, and the target activation layer in the first neural network model is used to process the input data to obtain a fourth output. The second quantization operator is used to quantize and dequantize the fourth output according to a second quantization factor.

7. The method according to claim 6, characterized in that The third output is a second target tensor, the second target tensor including X elements, and the method further includes: Get the Y target elements with the largest absolute values ​​among the X elements; The Y target elements in the second target tensor are replaced by a maximum element among XY elements in the X elements excluding the Y target elements to obtain the second quantization factor.

8. A model training device, characterized in that: The device comprises: An acquisition module is configured to acquire a first neural network model, wherein the first neural network model includes a convolutional BN layer and a first quantization operator, the convolutional BN layer is configured to perform convolution processing on the input Nth batch of data according to a first weight, normalize the convolution processing result according to the BN coefficient, update the BN coefficient based on the normalization processing result, and update the first weight on the updated BN coefficient. The first quantization operator is configured to perform quantization and inverse quantization on the updated first weight to obtain a second weight. The convolutional BN layer is further configured to perform convolution processing on the input N+1th batch of data according to the second weight, where the data includes at least one of the following: image, speech, and text; A model training module, configured to perform model training on the first neural network model to obtain a trained first neural network model; The first quantization operator is used to perform quantization and dequantization on the updated first weight according to the first quantization factor, the trained first neural network model includes the trained first quantization factor and the trained BN coefficient, and the device further includes: A quantization module is used to quantize the first neural network model according to the trained first quantization factor and the trained BN coefficient to obtain a third neural network model, wherein the third neural network model includes a quantized first convolution layer, and the quantized first convolution layer is used to perform convolution processing on the input data according to the quantized weights, and the quantized weights are obtained according to the first quantization factor and the trained BN coefficients.

9. The device according to claim 8, characterized in that The acquisition module is used to obtain a second neural network model, which includes a first convolutional layer and a first BN layer; performing BN folding processing on the first convolutional layer and the first BN layer to obtain the first neural network model, which includes the convolutional BN layer obtained after folding the first convolutional layer and the first BN layer.

10. The device according to claim 8, characterized in that The convolutional BN layer is obtained by folding the convolutional layer and the BN layer, the first weight is obtained according to the product of the BN coefficient and the target weight, the updated first weight is obtained by multiplying the updated BN coefficient and the target weight, and the target weight is the weight included in the convolutional layer.

11. The device according to claim 10, characterized in that The device further comprises: a product operation module, configured to perform a product operation on the BN coefficient and the target weight to obtain a first target tensor, where the first target tensor includes M elements; An element replacement module is used to replace the N target elements with the largest absolute values ​​among the M elements included in the first target tensor with the largest element among the MN elements in the M elements excluding the N target elements, so as to obtain the first weight.

12. The device according to claim 10 or 11, characterized in that The first convolutional layer in the first neural network model is used to perform convolution processing on the input N+1th batch data according to the second weight to obtain a convolution processing result, and divide the convolution processing result by the updated BN coefficient to obtain a second output.

13. The device according to claim 9, characterized in that The second neural network model also includes a target activation layer, and the target activation layer in the second neural network model is used to process the input data to obtain a third output. The first neural network model also includes the target activation layer and a second quantization operator, and the target activation layer in the first neural network model is used to process the input data to obtain a fourth output. The second quantization operator is used to quantize and dequantize the fourth output according to a second quantization factor.

14. The device according to claim 13, characterized in that The third output is a second target tensor, the second target tensor includes X elements, and the acquisition module is used to obtain Y target elements with the largest absolute values ​​among the X elements; The device further comprises: An element replacement module is configured to replace the Y target elements in the second target tensor with a maximum element among XY elements in the X elements excluding the Y target elements, so as to obtain the second quantization factor.

15. A model training device, characterized in that: The device includes a memory and a processor; the memory stores codes, and the processor is configured to obtain the codes and execute the method according to any one of claims 1 to 7.

16. A computer storage medium, characterized in that The computer storage medium stores one or more instructions, which, when executed by one or more computers, enable the one or more computers to implement the method of any one of claims 1 to 7.

17. A computer product comprising code, characterized in that When the code is executed, it is used to implement the method according to any one of claims 1 to 7.

Images