A service data processing method, a model training method and related equipment

Abstract

The application discloses a service data processing method, a model training method and related equipment, including: a cloud device determines an equivalent convolution operator based on a first GAP layer of a first neural network model; the cloud device replaces the first GAP layer with the equivalent convolution operator to obtain a second neural network model; in the case that the equivalent convolution operator and a first operator meet fusion conditions, the cloud device fuses the equivalent convolution operator and the first operator to obtain a third neural network model; and the cloud device determines a target neural network model based on the third neural network model. The first operator is connected adjacent to the equivalent convolution operator. Embodiments of the application can accelerate the inference speed of a neural network model.

Description

TECHNICAL FIELD

[0001] The present solution relates to the field of computer processing, and in particular, to a service data processing method, a model training method and related equipment. BACKGROUND

[0002] At present, a terminal device needs to use a neural network model to complete a related output result. For example, an image recognition algorithm, an image classification algorithm, an image deblurring algorithm, and the like. However, the above processing is a test for the computing and storage performance of the device. For example, in the inference process of the neural network model, the terminal device involves specific level calculation and storage of calculation results. After storage, the next layer continues to read the stored results as input to perform calculation of the next layer. Due to the complex structure of the neural network and the large number of levels, the model inference speed of the terminal device is slow. SUMMARY

[0003] The present application provides a service data processing method, a model training method and related equipment, which can accelerate the inference speed of the neural network model.

[0004] In a first aspect, the present application provides a service data processing method, comprising: inputting first data into a target neural network model by the terminal device to obtain an output result; the first data comprises image or text type service data; wherein the target neural network model is a neural network model obtained by fusing a first GAP layer in a first neural network model; the target neural network model does not include the first GAP layer.

[0005] The target neural network model is a model obtained by processing the first neural network model. The first neural network model includes a first GAP layer; and the equivalent convolution operator is a calculation unit corresponding to the first GAP layer.

[0006] In the embodiments of the present application, the target neural network model reduces the first GAP layer compared with the first neural network model. The GAP layer changes from a non-fusible level to a fusible operator, and after fusion, the level of the neural network becomes smaller, and the number of calculations and the number of read and write times of the memory are reduced, thereby accelerating the inference speed of the neural network model. That is, it can be understood that the inference speed of the target neural network model is faster than that of the first neural network model, which can reduce the performance requirements of the model on the terminal device, make the network processing more universal, and have a wider range of use.

[0007] In a possible implementation, the target neural network model is obtained by fusing a first GAP layer in a first neural network model, and the target neural network model comprises a fusion operator; the fusion operator is an operator obtained by fusing an equivalent convolution operator in a second neural network model and a first operator, the equivalent convolution operator being a convolution calculation unit determined based on the first GAP layer; the equivalent convolution operator is connected adjacent to the first operator; and the equivalent convolution operator is of a conv operator type.

[0008] In the embodiment of the application, the equivalent convolution operator is a convolution calculation unit determined based on the first GAP layer, which indicates that the result of the equivalent convolution operator has the ability to be fused after conversion, and the GAP layer before conversion cannot be fused, thereby creating a premise for fusing the equivalent convolution operator and the first operator and providing a basis and premise for the acceleration effect of the model.

[0009] In a possible implementation, the equivalent convolution operator is a convolution calculation unit determined based on the first GAP layer, and the convolution kernel size of the equivalent convolution operator is the output feature map size W*H of a previous layer operator of the first GAP layer; and the weight of each pixel point in the equivalent convolution operator is the reciprocal 1 / (W*H) of the total number of pixel points. In this way, the GAP layer is converted into the equivalent convolution operator in the network model, which has a universal conversion mode, guarantees the same output of the first GAP layer and the equivalent convolution operator, provides conditions for fusion, guarantees the reliability of the fusion result, and further provides support for the acceleration of the model.

[0010] In a possible implementation, the equivalent convolution operator in the second neural network model and the first operator satisfy a fusion condition; and the two operators satisfying the fusion condition have a condition for being fused into one operator. In this way, in the case where the equivalent convolution operator and the first operator adjacent thereto have a fusion premise, the fusion processing can be performed according to the corresponding operator type of the fusion, and the fusion has a related premise, which indicates that the conv operator can be fused with the adjacent operator.

[0011] In a second aspect, the present application provides a model training method, which is applied to a cloud device, and includes: determining, by the cloud device, an equivalent convolution operator based on a first GAP layer of a first neural network model; replacing, by the cloud device, the first GAP layer with the equivalent convolution operator to obtain a second neural network model; in a case where the equivalent convolution operator and a first operator satisfy a fusion condition, fusing, by the cloud device, the equivalent convolution operator and the first operator in the second neural network model to obtain a third neural network model; and determining, by the cloud device, a target neural network model based on the third neural network model, the target neural network model being used for model inference; wherein the first operator is connected to the equivalent convolution operator adjacently.

[0012] In the embodiments of the present application, the target neural network model has fewer first GAP layers than the first neural network model, the GAP layer is changed from a non-fusible layer to a fusible operator, and after fusion, the level of the neural network is smaller, the number of calculations and the number of read and write operations on the memory are reduced, thereby accelerating the inference speed of the neural network model. That is, it can be understood that the inference speed of the target neural network model is faster than that of the first neural network model, which can reduce the performance requirements of the model on the terminal device, make the network processing more universal, and have a wider range of use.

[0013] In a possible implementation, the cloud device determines an equivalent convolution layer operator based on the first GAP layer of the first neural network model, including: determining, by the cloud device, the size of the equivalent convolution layer operator as the output feature map size W*H of an operator of a previous layer of the first GAP layer; and determining, by the cloud device, the weight of each pixel point in the equivalent convolution operator as the reciprocal 1 / (W*H) of the total number of pixel points. In this way, in the network model, the GAP layer is converted into an equivalent convolution operator, which has a universal conversion mode, guarantees the same output of the first GAP layer and the equivalent convolution operator, provides conditions for fusion, guarantees the reliability of the fusion result, and further provides support for model acceleration.

[0014] In a possible implementation, the two operators satisfying the fusion condition have a condition of being fused into one operator. In this way, in a case where the equivalent convolution operator and the surrounding first operator exist a fusion premise, the fusion processing can be performed according to the corresponding operator type of fusion, and the fusion exists a related premise, indicating that the conv operator can be fused with the adjacent operator.

[0015] In a possible implementation, the type of the equivalent convolution operator is a conv operator; and the type of the first operator is one of a conv operator, a ReLU operator, an add operator, and a BN operator. In this way, since the operator fusion has the problem of universality, the types of the equivalent convolution operator and the first operator meet the fusion condition, so that the fusion is determined as a prerequisite, and the fusion is ensured to meet the fusion prerequisite.

[0016] In a possible implementation, the cloud device determines a target neural network model based on the third neural network model, including: the cloud device inserts a pseudo quantization node into the third neural network model, and performs quantization perception training to obtain the target neural network model; the target neural network model outputs floating-point data, and the third neural network model outputs fixed-point data. In this way, the third neural network model can also be subjected to quantization perception training, so as to convert the stored operators and the calculated feature maps from floating-point numbers to fixed-point numbers, reduce the storage data amount, simplify the calculation process, and further speed up the network inference.

[0017] In a possible implementation, the method further includes: the cloud device sends target neural network model data to a terminal device; the target neural network model data includes the target neural network model, and the target neural network model is used for model inference by the terminal device. In this way, after processing the neural network model, the cloud device can send the processed model to the terminal, and the terminal device processes the model, so that the processed model has less calculation amount and less memory read-write times than the model before processing, thereby speeding up the model processing.

[0018] In a third aspect, the present application provides a terminal device, including: one or more processors and one or more memories, the one or more memories are used to store computer program codes, the computer program codes include computer instructions, when the one or more processors execute the computer instructions, the terminal device is caused to execute: inputting first data into a target neural network model to obtain an output result; the first data includes image or text type service data; wherein the target neural network model is a neural network model obtained by fusing a first GAP layer in a first neural network model; and the target neural network model does not include the first GAP layer.

[0019] The target neural network model is a model obtained by processing a first neural network model. The first neural network model includes a first GAP layer.

[0020] In the embodiments of the present application, the target neural network model reduces the first GAP layer compared with the first neural network model, the GAP layer changes from a fusion level that cannot be fused to an operator that can be fused, and after fusion, the level of the neural network is smaller, the number of calculations and the number of read and write times to the memory are less, thereby accelerating the inference speed of the neural network model. That is, it can be understood that the inference speed of the target neural network model is faster than that of the first neural network model, which can reduce the performance requirements of the model on the terminal device, make the network processing more universal, and have a wider range of use.

[0021] In a possible implementation, the target neural network model is obtained by fusing the first GAP layer in the first neural network model, and includes: the target neural network model includes a fusion operator; the fusion operator is an operator obtained by fusing an equivalent convolution operator in a second neural network model and a first operator, the equivalent convolution operator is a convolution calculation unit determined based on the first GAP layer; the equivalent convolution operator is connected adjacent to the first operator; and the operator type of the equivalent convolution operator is a conv operator.

[0022] In the embodiments of the present application, the equivalent convolution operator is a convolution calculation unit determined based on the first GAP layer, indicating that the result of the equivalent convolution operator has the ability to be fused after conversion, and the GAP layer before conversion cannot be fused, thus creating the premise of fusing the equivalent convolution operator and the first operator, providing a basis and premise for the acceleration effect of the model.

[0023] In a possible implementation, the equivalent convolution operator is a convolution calculation unit determined based on the first GAP layer, including: the convolution kernel size of the equivalent convolution operator is the output feature map size W*H of the operator of the previous layer of the first GAP layer; and the weight of each pixel point in the equivalent convolution operator is the reciprocal 1 / (W*H) of the total number of pixel points. In this way, in the network model, the GAP layer is converted into an equivalent convolution operator, which has a universal conversion mode, ensures that the outputs of the first GAP layer and the equivalent convolution operator are the same, provides conditions for fusion, ensures the reliability of the fusion result, and further provides support for the acceleration of the model.

[0024] In a possible implementation, the equivalent convolution operator in the second neural network model and the first operator satisfy a fusion condition; and the two operators satisfying the fusion condition have a condition for being fused into one operator. In this way, in the case that the equivalent convolution operator and the surrounding first operator have a fusion premise, the fusion processing can be performed according to the corresponding operator type of the fusion, and the fusion has a related premise, indicating that the conv operator can be fused with the adjacent operator.

[0025] In a fourth aspect, the present application provides a cloud device, comprising: one or more processors and one or more memories, the one or more memories being configured to store computer program code, the computer program code comprising computer instructions that, when executed by the one or more processors, cause the cloud device to perform the following steps:

[0026] determining an equivalent convolution operator based on the first GAP layer of the first neural network model; replacing the first GAP layer with the equivalent convolution operator to obtain a second neural network model; in a case where the equivalent convolution operator and the first operator satisfy a fusion condition, fusing the equivalent convolution operator and the first operator in the second neural network model to obtain a third neural network model; determining a target neural network model based on the third neural network model, the target neural network model being used for model inference; wherein the first operator is connected adjacent to the equivalent convolution operator.

[0027] In the embodiments of the present application, the target neural network model reduces the first GAP layer compared with the first neural network model, the GAP layer changes from a non-fusible layer to a fusible operator, and after fusion, the level of the neural network becomes smaller, the number of calculations and the number of read and write times of the memory are reduced, thereby accelerating the inference speed of the neural network model. That is, it can be understood that the inference speed of the target neural network model is faster than that of the first neural network model, which can reduce the performance requirements of the model on the terminal device, make the network processing more universal, and have a wider range of use.

[0028] In a possible implementation, the cloud device determines an equivalent convolution layer operator based on the first GAP layer of the first neural network model, and specifically performs: determining that the size of the equivalent convolution layer operator is the output feature map size W*H of the previous layer operator of the first GAP layer; determining that the weight of each pixel point in the equivalent convolution operator is the reciprocal 1 / (W*H) of the total number of pixel points. In this way, in the network model, the GAP layer is converted into an equivalent convolution operator, which has a universal conversion mode, guarantees the same output of the first GAP layer and the equivalent convolution operator, provides conditions for fusion, guarantees the reliability of the fusion result, and further provides support for model acceleration.

[0029] In a possible implementation, the two operators satisfying the fusion condition have a condition for being fused into one operator. In this way, in a case where the equivalent convolution operator and the surrounding first operator exist a fusion premise, the fusion processing can be performed according to the corresponding operator type of fusion, and the fusion exists a related premise, indicating that the conv operator can be fused with the adjacent operator.

[0030] In a possible implementation, the type of the equivalent convolution operator is a conv operator; and the type of the first operator is one of a conv operator, a ReLU operator, an add operator, and a BN operator. In this way, since the operator fusion has the problem of universality, the equivalent convolution operator and the first operator can be fused only when the types of the two operators meet the fusion condition, thereby ensuring that the fusion meets the fusion premise.

[0031] In a possible implementation, the cloud device determines a target neural network model based on the third neural network model, and specifically performs: inserting a pseudo-quantization node into the third neural network model, and performing quantization-aware training to obtain the target neural network model; the target neural network model outputs floating-point data, and the third neural network model outputs fixed-point data. In this way, the third neural network model can also be subjected to quantization-aware training, so as to convert the stored operators and the calculated feature maps from floating-point numbers to fixed-point numbers, thereby reducing the storage data volume and simplifying the calculation process, and further speeding up the network inference.

[0032] In a possible implementation, the cloud device further performs: sending target neural network model data to the terminal device; the target neural network model data includes the target neural network model, and the target neural network model is used for model inference by the terminal device. In this way, after processing the neural network model, the cloud device can send the processed model to the terminal, and the terminal device can process the model, so as to ensure that the processed model has less calculation amount and less memory read-write times than the model before processing, thereby speeding up the model processing.

[0033] In a fifth aspect, an embodiment of the present application provides a computer storage medium, including computer instructions, when the computer instructions run on a terminal device, causing the apparatus to perform the service data processing method in any possible implementation manner of the first aspect.

[0034] In a sixth aspect, an embodiment of the present application provides a computer storage medium, including computer instructions, when the computer instructions run on a terminal device, causing the apparatus to perform the model training method in any possible implementation manner of the second aspect.

[0035] In a seventh aspect, an embodiment of the present application provides a computer program product, when the computer program product runs on a computer, causing the computer to perform the service data processing method in any possible implementation manner of the first aspect.

[0036] Eighthly, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to execute the model training method in any possible implementation of the second aspect described above. Attached Figure Description

[0037] The accompanying drawings used in the embodiments of this application are described below.

[0038] Figure 1 This is a schematic diagram of the structure of a convolutional neural network provided in an embodiment of this application;

[0039] Figure 2 This is a schematic diagram of a global average pooling method provided in an embodiment of this application;

[0040] Figure 3 This is a schematic diagram of a quantization mapping provided in an embodiment of this application;

[0041] Figure 4A and Figure 4B This is a schematic flowchart of a method for fusing the conv operator and the add operator provided in an embodiment of this application;

[0042] Figure 5 This is a schematic flowchart of a neural network training method provided in an embodiment of this application;

[0043] Figure 6A and Figure 6B This is a schematic flowchart of a method for determining the equivalent convolution operator of a GAP layer according to an embodiment of this application;

[0044] Figure 7 This is a schematic flowchart of a method for determining the equivalent convolution operator of a GAP layer according to an embodiment of this application;

[0045] Figure 8A and Figure 8B These are schematic diagrams of the network structure before and after quantization training, provided in the embodiments of this application.

[0046] Figure 9 This is a schematic diagram of a network architecture provided in an embodiment of this application;

[0047] Figure 10 This is a schematic diagram of the hardware structure of a cloud device 901 provided in an embodiment of this application;

[0048] Figure 11 This is a schematic diagram of the hardware structure of a terminal device 902 provided in an embodiment of this application. Detailed Implementation

[0049] The terminology used in the following embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the description of the embodiments of the present application and the appended claims, the singular forms "a," "an" and "the" are intended to include both singular and plural forms, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or," as used in the present application, refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0050] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments.

[0051] First, some terms and related technologies involved in the present application are explained in conjunction with the accompanying drawings, so as to facilitate the understanding of those skilled in the art.

[0052] 1. Neural network:

[0053] The neural network can be composed of neurons, which can refer to an operation unit with x s and intercept 1 as input. The output of the operation unit can be:

[0054]

[0055] wherein s = 1, 2, … n, n is a natural number greater than 1, W s is the weight of x s , b is the bias or bias of the neuron. f is the activation function of the neuron, which is used to introduce non-linear characteristics into the neural network to convert the input signal in the neuron into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. The neural network is a network formed by connecting many single neurons as described above, that is, the output of one neuron can be the input of another neuron. The input of each neuron can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of several neurons.

[0056] 2. Deep neural network (DNN):

[0057] Deep neural network, also called multi-layer neural network, can be understood as a neural network with many hidden layers, where "many" is not a special measure. From the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the number of layers in between is the hidden layer. The layers are fully connected, that is, any neuron in the i-th layer is connected to any neuron in the i+1-th layer. Although DNN looks very complex, in terms of the work of each layer, it is actually not complex. In simple terms, it is expressed as the following linear relationship expression: wherein, is the input vector, is the output vector, b is the offset vector, W is the weight matrix (also called coefficient), and a() is the activation function. Each layer only obtains the output vector from the input vector by such a simple operation. Since the number of layers of DNN is large, the number of coefficients W and offset vectors b is also large. These parameters are defined in the DNN as follows: taking the coefficient W as an example: assuming in a three-layer DNN, the linear coefficient from the fourth neuron of the second layer to the second neuron of the third layer is defined as The superscript 3 represents the layer number of the coefficient W, and the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficient from the k-th neuron of the L-1-th layer to the j-th neuron of the L-th layer is defined as It should be noted that the input layer has no W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. In theory, the more parameters a model has, the higher its complexity and "capacity", which means it can complete more complex learning tasks. Training a deep neural network is a process of learning the weight matrix, and the ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vectors W).

[0058] 3. Convolutional neural network (CNN)

[0059] A convolutional neural network is a deep neural network with a convolutional structure. The convolutional neural network includes a feature extractor composed of convolutional layers and subsampling layers. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or a convolutional feature map. The convolutional layer refers to a layer of neurons in the convolutional neural network that performs convolution processing on an input signal. In the convolutional layer of the convolutional neural network, a neuron can be connected only to part of the adjacent layer neurons. In a convolutional layer, there are usually several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights are convolution kernels. The shared weights can be understood as being irrelevant to the way and position of extracting image information. The implicit principle is that the statistical information of a part of the image is the same as that of other parts, which means that the learned image information of a part can also be used on another part. Therefore, the same learned image information can be used for all positions on the image. In the same convolutional layer, multiple convolution kernels can be used to extract different image information. Generally, the more the number of convolution kernels, the richer the image information reflected by the convolution operation.

[0060] The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning in the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between layers of the convolutional neural network, and to reduce the risk of overfitting.

[0061] Figure 1 is an exemplary structure diagram of a convolutional neural network disclosed by the embodiments of the present application. As shown in Figure 1 , the convolutional neural network can generally include an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer.

[0062] The input layer can input training data into the neural network; the convolutional layer extracts features from the data; the pooling layer performs down-sampling processing and expands the receptive field; the fully connected layer performs activation classification, and can expand the feature map obtained by the last convolutional layer into a one-dimensional vector and provide input for the classifier. The fully connected layer can be regarded as a "classifier" in the entire network. The output layer can output the training result.

[0063] Although the full connection layer can reduce the dimension of data, the parameters are too large, which is easy to cause overfitting. Global average pooling (GAP) can replace the full connection layer to reduce the number of parameters, reduce the size of the model and prevent overfitting. Assuming that the feature map has N channels, the left full connection layer reduces the N feature maps generated by the convolution layer to a one-dimensional vector (N*1), and then converts the one-dimensional vector into a specified number of unit values.

[0064] Figure 2 Figure 1 is a schematic diagram of global average pooling according to an embodiment of the present application. Global average pooling can calculate the average value of the feature map, simplify the calculation, retain the information of the previous convolution layers, and improve the application effect, and remove the size limitation of the feature map.

[0065] 4. Types of neural networks:

[0066] (1) Super resolution convolutional neuron network (SRCNN):

[0067] SRCNN can be a deblurring network, and the task of super resolution is to convert the input low-resolution image into a high-resolution image, which is consistent with image denoising and image deblurring. Super resolution focuses on how to fill new pixels from small size to large size images; image denoising focuses on replacing the pixels contaminated by noise with correct pixels without changing the size of the image.

[0068] (2) The classification network model can usually be divided into two parts: feature extraction and classification. The feature extraction part uses convolution for feature extraction, and the classification part uses full connection for classification, such as scene classification, whether there is a face, whether there is an animal, and other binary classification determination.

[0069] (3) The detection network model can include target detection, for example, finding which objects (classification) and their positions (coordinates) in the picture, and framing the position of a specific object. The detection network can also include key point detection, such as face key point positioning, which is the positioning of facial features and contours, and hand key point positioning, which is the positioning of various joints of the human body and the joints of the hand.

[0070] (4) Attention Model (AM):

[0071] Attention Model has been widely used in natural language and machine vision in recent years. Traditional channel attention is a weight adjustment in the channel dimension.

[0072] First, these models have achieved excellent performance across numerous tasks, such as machine translation, question answering systems, sentiment analysis, part-of-speech tagging, voter analysis, and question answering systems. While improving model performance, the attention mechanism increases the interpretability of the neural network architecture. Since traditional neural networks are black-box models, improving their interpretability is crucial for enhancing the fairness, reliability, and transparency of machine learning models.

[0073] The attention module automatically learns weights to capture the correlation between the encoder and decoder hidden states. These learned attention weights are used to construct a context vector as input to the decoder. At each position in the decoder, the context vector is obtained by weighted summation of the attention weights over all encoder hidden states. Thus, this context vector provides the decoder with a mechanism to access the entire input sequence and focus on specific relevant positions within it; this mechanism is called the attention mechanism.

[0074] 5. Quantification:

[0075] Quantization is a technique for compressing parameters in deep learning models to reduce computational cost. Neural networks are approximate computational processes, not requiring absolute precision in every calculation. Therefore, in some cases, model parameters that require more bits of storage can be converted to those requiring fewer bits without affecting the model's accuracy.

[0076] Quantization typically involves processing a floating-point model (the Tensor data type in common neural networks is generally float32) into a quantized model (the Tensor data type is int8, etc.).

[0077] Quantization can be seen as converting floating-point real numbers into fixed-point integers, essentially approximating a finite number of discrete values ​​for a continuously variable value. For example, the range of a uint8 integer is 0–255, and the range of a float32 integer is 0.0–1.0. Adjusting a float32 value to a uint8 value can be understood as quantization. The essence of quantization is a readjustment of the numerical range, which can be roughly understood as a linear mapping. Quantization can also be a method of information compression, effectively reducing the large amount of memory consumed in neural network models, addressing storage priorities, and reducing the number of data reads.

[0078] Figure 3 This is an exemplary quantization mapping diagram disclosed in an embodiment of this application. For example... Figure 3 As shown, let r represent a floating-point real number and q represent a quantized fixed-point integer. The conversion process between floating-point real numbers and fixed-point integers can be as follows:

[0079] r = S(qZ)

[0080] q = round(r / S + Z)

[0081] Where S is the scale, representing the proportional relationship between real numbers and integers; Z is the zero point, representing the integer corresponding to 0 after scaling. S = (r max -r min ) / (q max -q min Z = round(q) max -r max / S). Where r max The maximum value of r; r min q is the minimum value of r; max The maximum value of q; q min This is the minimum value of q. The `round` function calculates and returns a value rounded to a specified number of digits.

[0082] The meaning of inverse quantization:

[0083] The process opposite to quantization is dequantization. Dequantization generally does not result in information loss, while quantization typically involves a loss of precision. The dequantization process is as follows:

[0084] In one possible scenario: the calculation of dequantization can be in, It can represent fixed-point numbers; It can represent floating-point numbers. Dequantization is the process of converting a fixed-point number into a floating-point number. S refers to the scale mentioned above.

[0085] In another possible scenario, the dequantization calculation can be XS(Xint-Z); X = q(X; s, z, b) = s[clamp(|X / s|+z; 0, 2]); b -1)-z]; where X can represent a floating-point number and Z can represent a zero point. The meaning of clamp can be understood as a truncation process, specifically calculated as follows:

[0086]

[0087] Here, a and b can represent the range of floating-point values ​​output by each feature map layer, and b can be the range of r mentioned above. max ;a can be the r mentioned above min Divide this range by the bit width (2^3). b -1) can be used to obtain the value of S.

[0088] Quantization-aware-training (QAT):

[0089] Pseudo-quantization is actually a combination of quantization and dequantization, which simulates the error caused by the quantization round operation.

[0090] The input and output of the pseudo-quantization operation do not change, but the round operation is simulated, and this error is treated as a kind of training noise. During the QAT fine-tuning, this noise is clocked, so that when the final quantization is INT8, the precision loss is reduced.

[0091] Pseudo-quantization process: taking FP32 quantization to INT8 as an example. First, train the model in FP32 precision on the dataset to obtain the trained baseline model; second, insert the pseudo-quantization node into the baseline model to obtain the QAT model, and fine-tune the QAT model on the dataset; third, the pseudo-quantization node will simulate the quantization process during inference and save the quantization parameters calculated during the fine-tuning process; after the fine-tuning is completed, the QAT model is quantized using the quantization parameters to obtain the INT8 model, and is deployed to the inference framework for inference.

[0092] Quantization processing of operators: the convolution kernel is changed from a floating-point matrix to a fixed-point matrix.

[0093] 6. Operator types in neural network models:

[0094] Operator: a deep learning algorithm is composed of a plurality of calculation units, which can be referred to as operators (OP). In a neural network model, an operator corresponds to the calculation logic in a layer, for example, a convolution layer is an operator, which is abbreviated as conv; the weight summation process in a fully connected layer is also an operator. An activation function can also be an operator, for example, ReLU.

[0095] Name of operator: identifies a certain operator in the network, and the names of operators in the same network need to be kept unique.

[0096] Type of operator: each operator in the network is implemented according to the matching of the operator type, and the types of operators can include conv operators, ReLU operators, Add operators, and pool operators, etc. There can be multiple operators of the same type in a network. For example, conv1, conv2, conv3; pool1, pool2, etc.

[0097] A neural network model can be described in the form of an operator graph. The operator graph can describe the direction and type of operation, and the operator graph includes two types of nodes and edges. The node can represent the calculation type of the operator, and the edge can represent the input and output direction of the node data.

[0098] The ReLU operator is processed according to a determined ReLU function, and the ReLU function is:

[0099]

[0100] The conv operator is used for convolution calculation, and the calculation method can refer to the calculation method of the convolution layer in the neural network described above, and details are not repeated. The Pool operator is used for pooling calculation, and the Add operator is used for addition operation, etc.

[0101] 7. Tensor:

[0102] The tensor is a container for operator calculation data, including input data and output data, and can represent a multidimensional array or a vector. The tensor descriptor (TensorDesc) is a description of the input data and the output data. The tensor descriptor can include the following attributes: name, data type, data arrangement format, etc. Among them, the data arrangement format is that the data can be stored by a multidimensional array, such as the feature map of the convolutional neural network, which is usually saved by four-dimensional data, that is, 4D format: N: batch number, number of images; H: Height, height of the feature map, that is, the number of pixels in the vertical height direction. W: Width, the width of the feature map, that is, the number of pixels in the horizontal width direction. C: Channels, the feature map channel, for example, the Channels of the color RGB image is 64.

[0103] 8. Structure fusion of neural network model:

[0104] The operator fusion in the neural network refers to combining multiple neural network operators (such as convolution, pooling, normalization, etc.) together to improve the calculation efficiency and performance. Combining multiple convolution kernels into a larger convolution kernel to reduce the calculation amount and storage amount. For example, the fusion of the conv operator and the BN (batch normalization) operator into one operator for calculation, and the fusion of the conv operator and the Add operator into one operator.

[0105] Exemplarily, Figure 4A and Figure 4B is a method flowchart of the fusion of a conv operator and an Add operator exemplarily shown by an embodiment of the present application.

[0106] As Figure 4A shown, in the case that the conv operator and the Add operator have the same input and output size, the conv operator and the Add operator are calculated according to the calculation flow of the order, and if the time spent by the conv operator is A and the time spent by the Add operator is B, the total time spent is A+B.

[0107] As Figure 4BAs shown, two operators can be distributed through ping-pong distribution. At the same time, an input data is cut into two parts, which are generally operated separately. Ping-pong memory is used to realize the calculation pipeline in each operation. Add1 and conv2 are calculated in parallel. The left conv1+Add1 calculates the first half of the picture using ping memory, and the right conv2 and Add2 calculate the second half of the picture using pong memory. The time used in the above process is A+B / 2, which saves B / 2 time. Of course, further segmentation and fusion can be further divided. The fusion mode is not limited here.

[0108] However, the types and structures of operators that can be fused are very limited, which requires high properties of the operator for the fusion transformation of the operator, misses an optimization opportunity that can be fused, and does not fully utilize the hardware performance. That is, it can be understood that the operator fusion faces the problem of generalization, and only specific operator combinations can be fused, and some operator structures cannot support operator fusion for more complex neural network structures.

[0109] In the shooting scene of the user's use terminal device, in order to obtain better image effect, the terminal device can use various neural network models to process the image. For example, during the photographing process, the terminal device can use super-resolution neural network and classification network model, detection network model, attention model, face recognition algorithm, object recognition algorithm, beautifying algorithm, image classification algorithm, image fusion algorithm, image deblurring algorithm, etc. Therefore, the terminal device uses one or more neural network models to form an image processing model after stacking, and the structure of the stacked network model is complex, has multiple levels, and has high requirements for the computing resources and storage resources of the terminal device.

[0110] In the neural network model, the terminal side chip needs to process various types of operators. Operators can be divided into two categories according to processing speed: one is a compute-intensive operator, such as a conv operator. The other is an access-intensive operator, such as GAP calculation and ReLU operator. Among them, the terminal device supports compute-intensive operators well and has high processing speed; while access-intensive operators have low efficiency due to the need to read memory.

[0111] Further, in the structure of the neural network model, the operators are connected adjacent to each other, which means that the output of a previous operator is taken as the input of a next operator. Therefore, the output feature map of the previous operator needs to be stored, and then the next operator reads the output feature map of the previous operator for calculation. If the number of levels of the operator is larger, the number of memory reading times is also larger. The more the levels of the neural network model, the more complex the structure, and the greater the test on the calculation and storage capabilities of the chip. The terminal product with poor chip capability cannot support the performance of the algorithm model (neural network model) to meet the standard.

[0112] In the face of the above problems of the neural network model, such as a large number of operators, a complex structure, and a large number of calculations, which consume huge computing resources and storage resources, the number of memory reading times can be reduced through the operator fusion method, so as to accelerate the calculation speed and save the computing and storage resources. For example, the fusion between the ReLU operator and the conv operator. However, many operators cannot be fused, which limits the fusion effect of the network model, resulting in that the inference speed of the fusion result of the neural network model is still slow. For example, the GAP operator cannot be fused.

[0113] In some application scenarios, the terminal device has high requirements on the inference performance and storage performance of the neural network model. For example, when the terminal device enters the mobile phone shooting interface to capture a portrait or an animal, since the body posture of the person is instantaneous, the terminal device needs to infer the image effect of the result according to the neural network model, and therefore, the real-time requirement of the capture is high, which is a great test on the calculation performance and storage performance of the terminal device.

[0114] In the implementation manner, the embodiment of the present application provides a model training method. The first neural network model includes a first GAP layer. The first device can determine an equivalent convolution layer of the first GAP layer, replace the first GAP layer in the first neural network model with the equivalent convolution layer (equivalent convolution operator), and obtain a second neural network model. Then, the equivalent convolution operator and the first operator can be fused to obtain a third neural network model. The terminal device can quantitatively adjust the third neural network model to obtain a fourth neural network model. The fourth neural network model can be sent to the terminal device for use. The first operator and the equivalent convolution operator are two adjacent operators.

[0115] In the above process, the GAP layer is replaced by the convolution operator of the equivalent convolution layer, which changes the non-fusible operator into a fusible operator. Further, the first device can fuse the convolution operator of the equivalent convolution layer with the first operator connected thereto to obtain a fused network model. In this way, the operator fusion range of the neural network model is widened, the fusion effect is further improved, and the inference speed of the network on the end side can be improved, and the calculation resources and storage resources of the end side can be saved.

[0116] The following neural network model training process can be completed by a cloud device, i.e., the first device is a cloud device, such as a server, a computer, etc.

[0117] In the process of training the neural network model, the cloud device needs to build the network structure of the neural network model. The network structure can include various operators, and the connection mode of the operators constitutes the network structure of the neural network. In the embodiments of the present application, after the neural network model is built, the method of Figure 5 is executed. Of course, the order of executing the method of Figure 5 and the order of training the neural network model are not limited. The method of Figure 5 may be executed first, and then the weight of the neural network is adjusted; or the weight of the neural network model can be adjusted first, and then the method of Figure 5 is executed.

[0118] Figure 5 is a method flow diagram of a neural network training provided by the embodiments of the present application. As shown in Figure 5 , the neural network training method can include but is not limited to the following steps:

[0119] S501: The first device determines a first equivalent convolution operator of a first GAP layer of a first neural network model, and replaces the first GAP layer with the first equivalent convolution operator to obtain a second neural network model.

[0120] The operator parameters of the first equivalent convolution operator can include: channel, convolution kernel size, and convolution kernel weight.

[0121] The convolution kernel weight can represent the weight size of each pixel position in the convolution kernel. In the embodiments of the present application, the output result obtained by the equivalent convolution operator processing is equal to the GAP layer calculation output result.

[0122] The first device can determine the channel number of the equivalent convolution operator as the number N of channels of an operator before the GAP layer. In this way, the channel number can be aligned, and the correctness of the number of output results can be ensured. Wherein, N is a positive integer.

[0123] The first device can determine the operator size (convolution kernel size) of the equivalent convolution operator as the output feature map size W*H of the previous layer operator of a GAP. In this way, it can be ensured that the output matrix after the output feature map of the previous layer operator is convolved with the equivalent convolution operator is a point value of 1*1 size, ensuring that the output result of each channel is a numerical value, corresponding to the calculation result of the GAP.

[0124] The first device can set the weight of each pixel point of the equivalent convolution operator as 1 / (W*H), that is, the inverse of the number of convolution kernel pixels. In this way, it can be ensured that the calculation result of each channel after the convolution of the equivalent convolution operator is the average value in the output feature map of the previous layer operator.

[0125] Figure 6A And Figure 6B is an example of a GAP layer equivalent convolution operator determination method flowchart disclosed by the embodiments of the present application. As Figure 6A indicated, it can represent the method flowchart of the calculation of the second operator and the GAP layer in the first neural network model. The first neural network model can include the second operator and the GAP layer. Among them, the second operator is the last connected operator level before the GAP layer. The first neural network model can include at least one GAP layer, and the number and position of the GAP layer in the first neural network model are not limited by the present application. The number of channels in the second operator is N, and the size of the output feature map is 8*8. After the output feature map is calculated by the GAP layer, the output result of each channel can be obtained, and each channel corresponds to a numerical value.

[0126] As Figure 6B indicated, the GAP layer calculation process in Figure 6A is calculated by the convolution process of the equivalent convolution operator. Given the relevant parameters of the second operator and the output feature map of the second operator, the equivalent convolution operator can be obtained as N (corresponding to N channels), and the convolution kernel size of each channel is 8*8, and the weight of the convolution kernel pixel is 1 / 64. Figure 6A And Figure 6B have the same output result.

[0127] After the terminal device determines the equivalent convolution operator, the GAP layer calculation process can be replaced by the equivalent convolution operator. The results before and after the replacement can refer to the above Figure 6A and Figure 6B indicated, the first neural network model (including the network structure of Figure 6A ) can be obtained after the above replacement process, and the second neural network model can include the network structure of Figure 6B .

[0128] The equivalent change of the GAP layer above, the convolution operator conv from being unable to be fused to being an operator that can be fused, provides a premise for the operator fusion process in S502, and widens the range of operator fusion.

[0129] In addition, in the embodiment of the present application, the first neural network model can include a super-resolution network, a classification network, a detection network, a network with an attention mechanism, a frequency domain expansion, a natural language processing NLP model, etc. The present application does not limit the specific type of the first neural network. If the first neural network model is used for a shooting scene, in the case of displaying a shooting main interface on the terminal device, after the user clicks the shooting control and obtains a shooting image, the shooting image can be input to the neural network model for processing and output the result. The output result of the first neural network can be an image classification result, a face recognition result, a beautified image, a deblurred image, a classification result, etc. The present application does not limit this.

[0130] S502: The first device fuses the equivalent convolution operator in the second neural network model with the first operator to obtain a third neural network model.

[0131] It should be noted that the equivalent convolution operator in the second neural network model is connected with the second operator and the third operator before and after, respectively, that is, the second operator is the operator that is connected most recently before the equivalent convolution operator, and the third operator is the operator that is connected most recently after the equivalent convolution operator. The first operator can be one of the second operator or the third operator.

[0132] The first operator can be a conv operator, a ReLU operator, an add operator, a BN operator, etc. The present application does not limit the type of the operator. Among them, the add operator performs addition operation; the ReLU operator performs calculation of an activation function, and the conv operator performs convolution calculation. The BN (Batch Normlization) operator performs batch normalization calculation.

[0133] The terminal device can fuse the equivalent convolution operator with the first operator. The fusion manner and result are different for different connection relationships between the equivalent convolution operator and the first operator and different types of the first operator. The following examples illustrate several possible operator fusion cases:

[0134] Case 1: Fusion of the BN operator (the first operator) and the conv operator (the equivalent convolution operator).

[0135] In the case of the BN operator in front and the conv operator behind, the position and information between the two operators are fused to obtain necessary enhancement of the position. After fusion, the weight of the operator is: W=W BN *W conv The bias b=W BN *b conv +bBN wherein, W BN is the convolution kernel weight matrix of the BN operator; W conv is the convolution kernel weight matrix of the conv operator; b BN is the convolution kernel bias matrix of the BN operator; b conv is the convolution kernel bias matrix of the conv operator. The convolution kernel weight matrix of the fusion operator is W, and the bias matrix is b, which can be calculated according to the above formula.

[0136] Case 2: conv1 (the first operator) is fused with the conv2 operator (the equivalent convolution operator). Fusion can also be performed between two conv operators.

[0137] Case 3: the ReLU operator (the first operator) is fused with the conv operator (the equivalent convolution operator).

[0138] Case 4: the add operator (the first operator) is fused with the conv operator (the equivalent convolution operator).

[0139] Wherein, the fusion of the add operator and the conv operator (the equivalent convolution operator) can refer to the description of the above related ping-pong, which is not described herein.

[0140] In addition, the order and structure of the above operator fusion process can not be limited, and the specific fusion method is not limited. The above operator fusion is only illustratively described, and is not limited.

[0141] Figure 7 is a flowchart of an equivalent convolution operator determination method of a GAP layer disclosed by the embodiments of the present application. As shown in Figure 7 from left to right, the partial operator graph of the first neural network model changes to the partial operator graph of the second neural network, and then changes to the partial operator graph of the third neural network model. The partial operator graph of the first neural network model can include four operators in series, conv1, GAP, ReLU1 and Add, and the equivalent convolution operator conv2 is determined for the GAP layer; conv2 is replaced in the original GAP layer to obtain the partial operator graph of the second neural network. Then, the first device can fuse the two operators of conv2 and ReLU1 to obtain the fusion operator conv2+ReLU1, and then the first device can replace the positions of the two operators of conv2 and ReLU1 in the second neural network model with the conv2+ReLU1 operator to obtain the partial operator graph of the third neural network model. It should be noted that the above fusion method can also be the fusion of conv1 and conv2, and can also be the fusion of conv2, ReLU1 and Add, which is not limited by the present application.

[0142] Given that the first device already knows the operator graph of the second neural network model, the operator graph needs to be optimized. Operators can be fused with upstream and downstream operators to form a super operator (super OP). The purpose of operator fusion is to combine several small OPs into one large OP, thereby reducing the amount of data moved from memory / GPU memory. For example: Assume that the data volume (data length) of two operators X and Y is L. In TensorFlow, this corresponds to two OPs: Add and GAP. That is, the first device stores the parameters of these two operators. When performing GAP calculation, X and Y must first be read from memory / GPU memory, and then the calculation result must be written back to memory / GPU memory. Therefore, the amount of data read and written from memory for GAP calculation is 2L. When performing Add calculation, the input data must also be read from memory / GPU memory, and then the result must be written back to memory / GPU memory. Therefore, the amount of data read and written from memory for Add calculation is 3L. The total amount of data read and written for Add and GAP combined is 5L. If X and Y are merged into a single operation, after reading X and Y from memory / GPU memory, a gap is first performed, followed by an add operation, and then the final result is saved back to memory / GPU memory. This reduces the total data read / write volume to only 3L, a 40% reduction compared to before merging. In this process, the gap is equivalent to a convolution, and this computation can be merged. Furthermore, conv+add can be piped ping-pong between convolution and addition instructions through instruction scheduling, allowing convolution computation to mask addition computation. Moreover, for some hardware, if intermediate results can only be written back to the last level of storage, the time spent reading and writing data from memory / GPU memory accounts for a significant portion of the overall operation computation, making this speedup effect particularly noticeable.

[0143] Before executing S502, the first device also needs to determine whether the equivalent convolution operator and the first operator meet the fusion conditions. Meeting the fusion conditions means that the first operator and the equivalent convolution operator meet the conditions of a preset fusion operator type. Meeting the preset fusion operator type means that the two operators preset by the first device have two operator types that can be fused into one operator. The preset fusion operator types can include at least the types corresponding to the above four cases. Among them, the type of the equivalent convolution operator is fixed as the conv operator; the type of the first operator can be one of the conv operator, ReLU operator, add operator, and BN operator. This application does not specifically limit the type of the first operator.

[0144] S503: The first device obtains the convolution kernel of the fusion operator in the third neural network model, inserts pseudo-quantization nodes, adjusts the third neural network model for quantization training, and obtains the fourth neural network model.

[0145] After the processing of S502 above, the operator weights and input / output results in the third neural network model are all floating-point numbers.

[0146] In the process of quantization training of the network structure, the pseudo-quantization node can be inserted in the third neural network model to obtain a QAT model. Then, the QAT model can be fine-tuned according to the input and output of each node in the third neural network model. After that, the fine-tuned QAT model can be quantized to change from a float model to a fixed-point model, i.e., a fourth neural network model.

[0147] The third neural network model can include many operators. Taking an operator as an example, the operator can include a weight w and a bias b. Both w and b are stored in float32. float32 occupies 32 bits in computer storage, and int8 occupies 8 bits. It is assumed that the third neural network model in the embodiment of the present application is a float32 model, and the fourth neural network model is an int8 network model. The storage size is reduced, and the data storage and reading speed during the inference of the neural network model is also accelerated, thereby accelerating the model inference speed.

[0148] Figure 8A and Figure 8B are a set of network structure diagrams before and after quantization training adjustment disclosed in the embodiment of the present application. As shown in Figure 8A , taking a conv operator as an example, the third neural network model can include a calculation unit (operator) and a data unit. The calculation unit can include a conv and an addition operation "+". The data unit can include weights, conv, bias, and input data.

[0149] Before quantization adjustment, the third neural network model trained is known. The pseudo-quantization node is inserted in the third neural network model.

[0150] As shown in Figure 8B , based on the network structure in Figure 8A , the position where the pseudo-quantization node needs to be inserted is determined. The pseudo-quantization operator can include a quant1 operator and a quant2 operator. The first device inserts the quant1 operator after the weights; and inserts the quant2 operator after the "+" operator. The two operators can perform pseudo-quantization calculation on the data. Then, the network structure based on the inserted quant1 operator and quant2 operator can be adjusted. Further, the first device can adjust the output data in the network structure in Figure 8A Figure 8A ​The operator in the network model is adjusted (for example, the data in the weight, bias bit, and pseudo quantization node is adjusted to maintain the same input and output). After the adjustment is completed, the obtained network model is of a floating point type. That is, the data of the data units such as input data, weight, and bias is a floating point number. Then the first device can quantize the network model to obtain a fourth neural network model. The value quantization processing of the above-mentioned data unit makes the data such as input data, weight, and bias be fixed point type.

[0151] The pseudo quantization adjustment process described above can reduce the problem of data deviation caused by the quantization-dequantization process, reduce the amount of data, reduce the storage and reading times in the model inference process, and ensure the accuracy of the model as much as possible, and reduce the deviation.

[0152] In addition, Figure 8A and Figure 8B The pseudo quantization adjustment process and the quantization process are only illustrated by taking the convolution operator as an example. Other types of operators can also be quantized and trained, which is not limited by the present application.

[0153] S504: The first device sends fourth neural network model data to the second device.

[0154] In the case where the first device trains to obtain the fourth neural network model, the first device can send the fourth neural network model data to the second device. Correspondingly, the second device can receive the fourth neural network model data from the first device. The fourth neural network model data can include the fourth neural network model.

[0155] The second device is a terminal device, that is, various types of devices on the user terminal side, which is not limited by the present application.

[0156] In the embodiments of the present application, the fourth neural network model can be a target neural network model, and of course, the third neural network model can also be a target neural network model. In the target neural network model, the second device uses the target neural network model to perform model inference to obtain an output result, and S505 takes the fourth neural network model as the target neural network model as an example.

[0157] S505: The second device uses the fourth neural network model data to perform inference to obtain an output result.

[0158] After the second device receives the fourth neural network model data from the first device, the second device can use the fourth neural network model to perform inference (calculation) to obtain an output result.

[0159] Specifically, the second device inputs the first data into the fourth neural network model data to obtain an output result. The first data is image or text type business data. The business data can be data in a user shooting process, a business of classifying or analyzing user input text, a business of identifying an image, a business of image deblurring, and the like, which are not limited in the present application.

[0160] It should be noted that the first device sends the target neural network model to the second device in S504, and the second device uses the target neural network model data to perform inference to obtain an output result in S505. The target neural network model can be the fourth neural network model, or the third neural network model, that is, the process of S503 can not be executed, which is not limited in the present application. Specifically, the first device can obtain the target neural network model based on the third neural network model. The target neural network model can be the third neural network model or the fourth neural network model.

[0161] Figure 9 is an exemplary network structure architecture provided by an embodiment of the present application. As shown in Figure 9 , the network structure can include a cloud device 901 and a terminal device 902.

[0162] After determining the fourth neural network model, the cloud device 901 can send the algorithm of the fourth neural network model to the terminal device 902. The result of the model can be

[0163] The first device can be the cloud device 901, which can be a super multi-core server, a computer deployed with a graphics processing unit (GPU) cluster, a large distributed computer, a cluster computer with hardware resource pooling, and the like. The server can also be a cloud device deployed with an edge computing node, a cloud computing node, and the like, which is not limited herein. The cloud device 901 can include a memory, a processor, and a transceiver. The memory can be used to store programs and data related to the object reminding; the memory can also be used to store neural network models related to the model. The processor can be used to train the neural network model. The transceiver communicates with the terminal device 902.

[0164] The second device can be a terminal device 902. This application does not specifically limit the type of terminal device 902. In some embodiments, the terminal device 902 in this application can be a mobile phone, wearable device (e.g., smart bracelet), tablet computer, laptop computer, handheld computer, computer, ultra-mobile personal computer (UMPC), cellular phone, personal digital assistant (PDA), augmented reality (AR) / virtual reality (VR) device, or other portable device. It can also be a speaker, smart screen, television, refrigerator, air conditioner, in-vehicle equipment, printer, projector, smart door lock, and smart doorbell, etc. Exemplary embodiments of the terminal device include, but are not limited to, devices equipped with… Or terminal devices using other operating systems.

[0165] Figure 10 This is a schematic diagram of the hardware structure of a cloud device 901, as exemplarily proposed in an embodiment of this application.

[0166] Cloud device 901 can provide Figure 10 The server shown is 1000.

[0167] like Figure 10 As shown, server 1000 may include: one or more processors 1001, memory 1002, communication interface 1003, transmitter 1005, receiver 1006, etc. These components can be connected via bus 1004 or other means. Figure 10 Taking a bus connection as an example:

[0168] The communication interface 1003 can be used by the server 1000 to communicate with other communication devices, such as the terminal device 902. Specifically, the communication interface 1003 can be a 3G communication interface, a Long Term Evolution (LTE) (4G) communication interface, a 5G communication interface, a WLAN communication interface, a WAN communication interface, and future evolution communication interfaces, etc. Not limited to wireless communication interfaces, the server 1000 can also be configured with a wired communication interface 1003 to support wired communication. In this embodiment, the server 1000 can send fourth neural network model data to a second device through the communication interface 1003.

[0169] In some embodiments of the present application, the transmitter 1005 and the receiver 1006 can be regarded as a wireless modem. The transmitter 1005 can be used for transmitting processing of signals output by the processor 1001. The receiver 1006 can be used for receiving signals. In the server 1000, the number of the transmitter 1005 and the receiver 1006 can be one or more.

[0170] The memory 1002 is coupled to the processor 1001 and is used to store various software programs and / or sets of instructions. Specifically, the memory 1002 can include a high-speed random access memory, and can also include a non-volatile memory such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices.

[0171] The memory 1002 can store an operating system (hereinafter referred to as a system) such as an embedded operating system uCOS, VxWorks, RTLinux, etc. The memory 1002 can also store a network communication program which can be used for communication with the server 1000 and one or more terminal devices.

[0172] In the embodiments of the present application, the processor 1001 can be used to read and execute computer readable instructions. Specifically, the processor 1001 can be used to call programs stored in the memory 1002, such as the implementation program of the method for sharing data provided by one or more embodiments of the present application on the server 1000 side, and execute the instructions contained in the program. The processor 1001 can execute Figure 5 the training task performed by the first device.

[0173] It should be noted that Figure 10 The server 1000 shown is only one implementation of the embodiments of the present application, and in actual applications, the server 1000 can also include more or fewer components, which are not limited here.

[0174] Figure 11 is an exemplary hardware structure of a terminal device 902 provided by the embodiments of the present application.

[0175] As Figure 11 shown, the terminal device 1100 can include a processor 1101, a memory 1102, a wireless communication module 1103, a USB communication processing module 1104, a power switch 1105, a display screen 1106, etc. Among them:

[0176] The processor 1001 can be configured to read and execute computer-readable instructions. In a specific implementation, the processor 1101 can mainly include a controller, an arithmetic unit, and a register. Among them, the controller is mainly responsible for instruction decoding and issuing control signals for the operation corresponding to the instruction. The arithmetic unit is mainly responsible for saving the register operands and intermediate operation results temporarily stored during the execution of instructions.

[0177] In some embodiments, the processor 1101 can perform inference calculation on the neural network model.

[0178] The memory 1102 is coupled with the processor 1101, and is configured to store various software programs and / or groups of instructions. In a specific implementation, the memory 1102 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more disk storage devices, flash devices, or other non-volatile solid-state storage devices. The memory 1102 can store an operating system, and the memory 1102 can also store a communication program that can be used to communicate with the terminal device 1100, one or more servers, or accessory devices.

[0179] In some embodiments, the memory 1102 can store neural network model data.

[0180] The wireless communication module 1103 can be configured to communicate with other devices, such as the cloud device 901. The wireless communication module 1103 can include one or more of a cellular mobile communication module, a UWB communication module, a Bluetooth communication module, a WLAN communication module, and a GPS communication module.

[0181] In some embodiments, the wireless communication module 1103 can receive neural network model data from a server.

[0182] The power switch 1105 can be configured to control the power supply to the terminal device 1100.

[0183] The USB communication processing module 1104 can be configured to communicate with other devices through a USB interface (not shown).

[0184] The display screen 1106 can be used to display images, videos, etc. The display screen 1106 can be a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode (AMOLED) display, a flexible light-emitting diode (FLED) display, a quantum dot light emitting diodes (QLED) display, etc.

[0185] It can be understood that Figure 11 The schematic structure does not constitute a specific limitation on the terminal device 1100. In some other embodiments of the present application, the terminal device 1100 can include more or fewer components than those shown, or combine certain components, or split certain components, or different arrangement of components. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0186] In the above embodiments, all or part of the functions can be implemented by software, hardware, or a combination of software and hardware. When implemented by software, it can be implemented in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available media can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0187] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be instructed by a computer program to relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes ROM or random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. A business data processing method, characterized in that, The method is applied to a terminal device and includes: The terminal device inputs the first data into the target neural network model and obtains the output result; the first data includes image-type business data; the target neural network model is used for the scene capture process; The target neural network model is a neural network model obtained by fusing the first GAP layer in the first neural network model and performing quantitative perception training. The step of fusing the first GAP layer in the first neural network model and performing quantized perceptual training includes: The number of channels of the equivalent convolution operator is determined to be the number of channels N of the operator preceding the first GAP layer; The operator size of the equivalent convolution operator is determined to be the output feature map size W*H of the operator of the layer preceding the first GAP layer; The kernel weight of each pixel in the equivalent convolution operator is determined to be 1 / (W*H); The equivalent convolution operator of the first GAP layer is determined based on the number of channels, the operator size, and the kernel weights. The first GAP layer in the first neural network model is replaced with an equivalent convolution operator to obtain the second neural network model; the first neural network model is a super-resolution neural network SRCNN; SRCNN is a network whose task objective is to convert low-resolution input images into high-resolution images for deblurring. Determine whether the equivalent convolution operator and the first operator in the second neural network model satisfy the fusion condition; the fusion condition is that the first operator and the equivalent convolution operator meet the conditions of a preset fusion operator type; the type of the equivalent convolution operator in the preset fusion operator type is the conv operator; the type of the first operator is one of the conv operator, ReLU operator, add operator, and BN operator; the first operator and the equivalent convolution operator are adjacent and connected; If the equivalent convolution operator and the first operator satisfy the fusion condition, the equivalent convolution operator and the first operator in the second neural network model are fused to obtain the third neural network model. A pseudo-quantization node is inserted into the third neural network model, and quantization-aware training is performed to obtain the target neural network model; the output data of the target neural network model is floating-point data, and the output data of the third neural network model is fixed-point data; the target neural network model does not include the first GAP layer.

2. A model training method, characterized in that, The method is applied to a cloud device, and the method includes: The number of channels of the equivalent convolution operator is determined to be the number of channels N of the operator before the first GAP layer; The operator size of the equivalent convolution operator is determined to be the output feature map size W*H of the operator of the layer preceding the first GAP layer; The kernel weight of each pixel in the equivalent convolution operator is determined to be 1 / (W*H); The equivalent convolution operator of the first GAP layer is determined based on the number of channels, the operator size, and the convolution kernel weights; the first neural network model is a super-resolution neural network SRCNN; SRCNN is a network whose task objective is to convert a low-resolution input image into a high-resolution image deblurring network; The cloud device replaces the first GAP layer with the equivalent convolution operator to obtain a second neural network model; the second neural network model is a neural network model in which the first GAP layer in the first neural network model is replaced with the equivalent convolution operator; The cloud device determines whether the equivalent convolution operator and the first operator in the second neural network model meet the fusion condition. The fusion condition is that the first operator and the equivalent convolution operator meet the conditions of a preset fusion operator type. The type of the equivalent convolution operator in the preset fusion operator type is the conv operator. The type of the first operator is one of the conv operator, ReLU operator, add operator, and BN operator. The first operator and the equivalent convolution operator are adjacent and connected. When the equivalent convolution operator and the first operator satisfy the fusion condition, the cloud device fuses the equivalent convolution operator and the first operator in the second neural network model to obtain a third neural network model. The cloud device inserts pseudo-quantization nodes into the third neural network model and performs quantization-aware training to obtain the target neural network model; the target neural network model outputs floating-point data, while the third neural network model outputs fixed-point data. A target neural network model is determined based on the third neural network model, and the target neural network model is used for model inference; the target neural network model includes a fusion operator, and the target neural network model does not include the first GAP layer; the target neural network model is used for the scene capture process; the target neural network model is used to process first data, and the first data includes image-type business data.

3. The method according to claim 2, characterized in that, The method further includes: The cloud device sends target neural network model data to the terminal device; the target neural network model data includes the target neural network model, which is used by the terminal device for model inference.

4. A terminal device, characterized in that, include: One or more processors and one or more memories, the one or more memories being used to store computer program code, the computer program code including computer instructions, which, when executed by the one or more processors, cause the terminal device to perform the method as described in claim 1.

5. A cloud device, characterized in that, include: One or more processors and one or more memories, the one or more memories being used to store computer program code, the computer program code including computer instructions, which, when executed by the one or more processors, cause the cloud device to perform the method as described in claim 2 or 3.

6. A computer-readable storage medium comprising instructions, characterized in that, When the instruction is executed on a terminal device, the terminal device causes the terminal device to perform the method as described in claim 1.

7. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on a cloud device, the cloud device causes the cloud device to perform the method as described in claim 2 or 3.

Images