Neural network model training method, storage medium and electronic device

Abstract

The present invention discloses a training method, storage medium, and electronic device for a neural network model. The method includes: performing a quantization operation on training data during the neural network model training process, using the quantized small-bit data to train the neural network model; and, to ensure the accuracy of the neural network model, performing a target operation on the quantized data and then performing a dequantization operation on the data output after the target operation. That is, during the quantization training of the neural network model, while ensuring that the loss of quantization accuracy is not significant, the capacity of the neural network model can be reduced to a certain value, and the memory used during operation can also be reduced. This reduces data handling and significantly reduces model power consumption, thereby resolving the technical problem in the prior art of requiring a large number of resources during the training of the neural network model.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a training method, storage medium and electronic device for a neural network model. Background Art

[0002] With the development of artificial intelligence, neural network models are increasingly being applied to various scenarios, such as identifying facial key points and facial expressions in images. Before a neural network model can be used, it must be trained to meet the requirements. During the training process, it is important to consider the training duration and resources required.

[0003] In related art, during the training of a neural network model, the input data of each computing module in the neural network model often occupies a large amount of storage space. For example, when the input data is a floating-point type (such as float32), the input data occupies 32 bits of storage space. Furthermore, because the input data occupies a large amount of storage space, each computing module consumes more resources when processing the input data, such as computing time or memory usage during operation.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] Embodiments of the present invention provide a training method, storage medium, and electronic device for a neural network model to at least solve the technical problem in the prior art that a large number of resources are required during the training of a neural network model.

[0006] According to one aspect of an embodiment of the present invention, a training method for a neural network model is provided, comprising: obtaining a first input value of a first data type, wherein the first input value is set as an input value of a target operation in the neural network model to be trained, and the number of bits occupied by the first input value of the first data type is a first number of bits; performing a quantization operation on the first input value to obtain a second input value of a second data type, wherein the number of bits occupied by the second input value of the second data type is a second number of bits, and the second number of bits is smaller than the first number of bits; performing the target operation on the second input value to obtain a first output value of the second data type, and the number of bits occupied by the first output value of the second data type is the second number of bits; performing an inverse quantization operation on the first output value to obtain a second output value of the first data type, and the number of bits occupied by the second output value of the first data type is the first number of bits; and training the neural network model according to the second output value.

[0007] According to another aspect of an embodiment of the present invention, a training device for a neural network model is also provided, including: a first acquisition unit, used to acquire a first input value of a first data type, wherein the first input value is set as the input value of a target operation in the neural network model to be trained, and the number of bits occupied by the first input value of the first data type is a first number of bits; a quantization unit, used to perform a quantization operation on the first input value to obtain a second input value of a second data type, wherein the number of bits occupied by the second input value of the second data type is a second number of bits, and the second number of bits is smaller than the first number of bits; a first operation unit, used to perform the target operation on the second input value to obtain a first output value of the second data type, and the number of bits occupied by the first output value of the second data type is the second number of bits; an inverse quantization unit, used to perform an inverse quantization operation on the first output value to obtain a second output value of the first data type, and the number of bits occupied by the second output value of the first data type is the first number of bits; and a training unit, used to train the neural network model according to the second output value.

[0008] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned training method of the neural network model when running.

[0009] According to another aspect of an embodiment of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the training method of the neural network model through the computer program.

[0010] In an embodiment of the present invention, during the training of a neural network model, a quantization operation is performed on the input values ​​of a target operation in the neural network model to reduce the number of bits occupied by the input values. In this way, when the target operation is performed on the input values, the resources consumed by performing the target operation can be reduced, thereby improving the operation speed, for example, reducing the duration of the operation, or reducing the memory occupied during operation. Furthermore, during the training of the neural network model, a dequantization operation is performed on the output values ​​of the target operation in the neural network model to convert the data type of the output values ​​into the data type before the quantization operation is performed. This reduces the resources consumed by performing the target operation while ensuring that the accuracy of the data in the neural network model is not lost (i.e., the loss of quantization accuracy is not large). BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0012] Figure 1 is a schematic diagram of an application environment of an optional neural network model training method according to an embodiment of the present invention;

[0013] Figure 2 is a flowchart of an optional training method for a neural network model according to an embodiment of the present invention;

[0014] Figure 3 is a schematic diagram of a training process of an optional neural network model according to an embodiment of the present invention;

[0015] Figure 4 is a schematic diagram of an optional facial key point neural network model training process according to an embodiment of the present invention;

[0016] Figure 5 is a schematic diagram of an optional facial expression category recognition neural network model training process according to an embodiment of the present invention;

[0017] Figure 6 1 is a schematic diagram of an optional training process of quantization and dequantization operations in a facial key point neural network model according to an embodiment of the present invention (I);

[0018] Figure 7 is a schematic diagram of an optional training process of quantization and dequantization operations in a convolutional layer according to an embodiment of the present invention;

[0019] Figure 8 2 is a schematic diagram of an optional training process of quantization and dequantization operations in a facial key point neural network model according to an embodiment of the present invention (II);

[0020] Figure 9 Schematic diagram (3) of an optional training process of quantization and dequantization operations in a facial key point neural network model according to an embodiment of the present invention;

[0021] Figure 10 Schematic diagram (four) of an optional training process of quantization and dequantization operations in a facial key point neural network model according to an embodiment of the present invention;

[0022] Figure 11 Schematic diagram (V) of an optional training process of quantization and dequantization operations in a facial key point neural network model according to an embodiment of the present invention;

[0023] Figure 12is a block diagram of an optional pseudo-quantization-based facial key point quantization perception training method according to an embodiment of the present invention;

[0024] Figure 13 is a schematic structural diagram of an optional neural network model training device according to an embodiment of the present invention;

[0025] Figure 14 FIG. 4 is a schematic structural diagram of an optional electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] According to one aspect of an embodiment of the present invention, a training method for a neural network model is provided. Optionally, as an optional implementation, the training method for the neural network model can be applied to, but is not limited to, Figure 1 In the environment shown, there is a server 202 and a user terminal 204.

[0029] The user terminal 202 is used to display a sample of the neural network model; the server 204 executes the following steps S101-S104: obtaining a first input value of a first data type, wherein the first input value is set as an input value of a target operation in the neural network model to be trained, and the number of bits occupied by the first input value of the first data type is a first number of bits; performing a quantization operation on the first input value to obtain a second input value of a second data type, wherein the number of bits occupied by the second input value of the second data type is a second number of bits, and the second number of bits is less than the first number of bits; performing a target operation on the second input value to obtain a first output value of the second data type, The number of bits occupied by the first output value of the second data type is the second number of bits; an inverse quantization operation is performed on the first output value to obtain a second output value of the first data type, and the number of bits occupied by the second output value of the first data type is the first number of bits; based on the second output value, the neural network model is trained, that is, in the process of training the neural network model, the input value of the target operation in the neural network model is quantized to reduce the number of bits occupied by the input value, so that when the target operation is performed on the input value, the resources consumed by performing the target operation can be reduced, and the operation speed is improved, for example, the operation time is reduced, or the memory occupied during operation is reduced. Further, in the process of training the neural network model, an inverse quantization operation is performed on the output value of the target operation in the neural network model to convert the data type of the output value into the data type before the quantization operation is performed, thereby reducing the resources consumed by performing the target operation while ensuring that the accuracy of the data in the neural network model is not lost (that is, the quantization accuracy loss is not large).

[0030] The server 202 applies the trained neural network model to the user terminal 204 .

[0031] Optionally, in this embodiment, the above-mentioned user terminal can be a terminal device configured with a target client, which can include but is not limited to at least one of the following: a mobile phone (such as an Android phone, an iOS phone, etc.), a laptop computer, a tablet computer, a PDA, an MID (Mobile Internet Devices), a PAD, a desktop computer, a smart TV, etc. The target client can be a video client, an instant messaging client, a browser client, an education client, etc. The above-mentioned network can include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network and a wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that realize wireless communication. The above-mentioned server can be a single server, or it can be a server cluster composed of multiple servers, or a cloud server. The above is only an example, and this embodiment does not impose any limitation on this.

[0032] The training method of the above-mentioned neural network may include but is not limited to use in the field of artificial intelligence, and may not include artificial intelligence-based image recognition, identification of facial key points in images, categories of facial expressions in images, and category recognition of organisms in images, etc.

[0033] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0034] Alternatively, as an optional implementation, Figure 2 As shown, the training method of the above neural network model includes:

[0035] Step S202: Obtain a first input value of a first data type, wherein the first input value is set as an input value of a target operation in a neural network model to be trained, and the number of bits occupied by the first input value of the first data type is a first number of bits.

[0036] Step S204 : performing a quantization operation on the first input value to obtain a second input value of a second data type, wherein the second input value of the second data type occupies a second number of bits, which is smaller than the first number of bits.

[0037] Step S206 , performing a target operation on the second input value to obtain a first output value of the second data type, where the number of bits occupied by the first output value of the second data type is the second number of bits.

[0038] Step S208 : performing an inverse quantization operation on the first output value to obtain a second output value of the first data type, where the number of bits occupied by the second output value of the first data type is the first number of bits.

[0039] Step S210: training the neural network model according to the second output value.

[0040] Optionally, in this embodiment, the training method of the neural network model may include but is not limited to applying the training process of the neural network model.

[0041] like Figure 3 As shown in the figure, the training process of the neural network model is shown in the figure. Figure 3As shown, the training process of the neural network model to be trained includes inputting sample data into the neural network model to be trained, the neural network model to be trained performs calculations on the sample data to obtain prediction data of the neural network model to be trained for the sample data, and determining the loss function value of the neural network model to be trained based on the prediction data and the actual data of the sample data. It is determined whether the training of the neural network to be trained is completed based on the loss function value, that is, when the loss function value meets the predetermined threshold, the training of the neural network to be trained is terminated to obtain the target neural network model, and the target neural network model is used in the actual recognition process.

[0042] In this embodiment, when the neural network model to be trained is being trained, quantization operations and dequantization operations are performed on the training data. Through the quantization operation, the neural network model to be trained can be trained with smaller bit data during training, and the calculation process of the training data occupies less system memory, saving system computing memory, and small bit data operations can increase the computing speed. In addition, through the dequantization operation, the trained neural network model can be more easily deployed on different terminals, and the trained neural network model has better mobility between different terminals.

[0043] In an embodiment, the data type may include but is not limited to float32 type, float64 type, int8 type, int16 type, int32 type, and the like.

[0044] In this embodiment, a quantization operation is performed on a first input value of a floating-point type to obtain a second input value of an integer type; wherein the first data type is float32 type, and the second data type is int8 type, or int16 type; or the first data type is float64 type, and the second data type is int8 type, or int16 type, or int32 type. During the training of the neural network model, a quantization operation is performed on the input value of the target operation in the neural network model to reduce the number of bits occupied by the input value. In this way, when the target operation is performed on the input value, the resources consumed by performing the target operation can be reduced, thereby improving the operation speed, for example, reducing the operation time, or reducing the memory occupied during operation.

[0045] It should be noted that the data type may include but is not limited to the floating-point data type, and the float data type is used to store single-precision floating-point numbers or double-precision floating-point numbers. Floating-point numbers use the IEEE (Institute of Electrical and Electronics Engineers) format. The single-precision value of the floating-point type has 4 bytes, including a sign bit, an 8-bit excess-127 binary exponent, and a 23-bit mantissa. The mantissa represents a number between 1.0 and 2.0. Since the high-order bit of the mantissa is always 1, it is not stored in digital form. That is, the value of the float type is represented by 4 bytes, a total of 32 bits. The entire floating-point number can be expressed as: f = s × t × 2i, where s is the sign bit 0 or 1, t is the mantissa, and i is the exponent.

[0046] Optionally, in this embodiment, the above-mentioned neural network model may include but is not limited to a neural network model for facial key point recognition, a neural network model for facial expression recognition, and the like.

[0047] like Figure 4 As shown in the figure, the schematic diagram of the training process of the facial key point neural network model is as follows: Figure 4 As shown, a face sample image and key point data annotated by a face sample icon are obtained, the face sample image is input into a face key point recognition neural network model to be trained, and the key point data predicted by the face key point recognition neural network model to be trained is output, and a loss function value is determined according to the annotated key point data and the predicted key point data. When the loss function value meets a preset threshold, the training of the face key point recognition neural network model is terminated to obtain a target face key point recognition neural network model for face key point recognition. In the training process of the neural network model for face key point recognition, quantization and dequantization operations are performed on each layer of input data in the model of the neural network for face key point recognition, so that each layer of data operation uses data with fewer bytes for operation, thereby reducing the memory occupied by the system during the operation of the neural network model for face key point recognition training.

[0048] like Figure 5 As shown in the figure, the training process of the neural network model for facial expression category recognition is shown in the figure. Figure 5As shown, a face sample image and key point data annotated by a face sample icon are obtained, the face sample image is input into a neural network model for facial expression category recognition to be trained, the key point data predicted by the neural network model for facial expression category recognition to be trained and the facial expression category information are output, the loss function value is determined according to the annotated key point data and the predicted key point data, and when the loss function value meets a preset threshold, the training of the neural network model for facial expression category recognition is terminated to obtain a target face key point recognition neural network model for facial expression category recognition, wherein, during the training process of the neural network model for facial key point recognition, quantization and dequantization operations are performed on each layer of input data in the model of the neural network for facial key point recognition, so that each layer of data operation uses data with fewer bytes for operation, thereby reducing the memory occupied by the system during the training operation of the neural network model for facial key point recognition.

[0049] It should be noted that a neural network model generally includes a convolutional layer, a pooling layer, and a fully connected layer, wherein a convolutional layer may include multiple, a pooling layer may include multiple, and a fully connected layer may include multiple.

[0050] In this embodiment, the neural network model includes a convolutional layer, a pooling layer, and a fully connected layer, wherein the target operation can be understood as an operator in each layer of the neural network model, which is used to perform a quantization operation or a dequantization operation on the input numerical value.

[0051] In this embodiment, during the training of the neural network model, the first input value of the first data type float32 can be quantized into an 8-bit second input value according to the 8-bit standard. Since the number of bits of the quantized second input value is known, and the target quantization parameter can be calculated based on the maximum and minimum values ​​determined by statistics of the first input value.

[0052] Optionally, determining a target quantization parameter used for the quantization operation according to a maximum value and a minimum value corresponding to the second data type and a maximum value and a minimum value of the first input value may include:

[0053] When an absolute value of a maximum value and an absolute value of a minimum value corresponding to the second data type are the same, determining a first quantization parameter used in a quantization operation according to the maximum value and the minimum value corresponding to the second data type and the maximum value and the minimum value of the first input value, wherein the target quantization parameter includes the first quantization parameter:

[0054]

[0055] zero_point=0

[0056] Among them, the target quantization parameters include scale and zero_point, min_val represents the minimum value of the first input value, max_val represents the maximum value of the first input value, qmax represents the maximum value corresponding to the second data type, and qmin represents the minimum value corresponding to the second data type.

[0057] For example, if the second data type is 8 bits, then qmax is 2^7-1=127, and qmin is -2^7=-128. Counting the first input value, max_val is 255 and min_val is -255, then:

[0058]

[0059] Optionally, when the absolute value of the maximum value and the absolute value of the minimum value corresponding to the second data type are different, a second quantization parameter used in the quantization operation is determined according to the maximum value and the minimum value corresponding to the second data type and the maximum value and the minimum value of the first input value, wherein the target quantization parameter is determined by the following formula:

[0060]

[0061]

[0062] The target quantization parameters may include scale and zero_point, min_val represents the minimum value of the first input value, max_val represents the maximum value of the first input value, qmax represents the maximum value corresponding to the second data type, qmin represents the minimum value corresponding to the second data type, and the round function is used to round Perform rounding calculations.

[0063] For example, if the second data type is 8 bits, then qmax is 2^7-1=127, and qmin is -2^7=-128. Counting the first input value, max_val is 255 and min_val is -255, then:

[0064]

[0065]

[0066] Optionally, in this embodiment, performing a quantization operation on the first input value using the target quantization parameter to obtain a second input value of the second data type may include:

[0067]

[0068] Among them, the target quantization parameters include scale and zero_point, X Q Represents the second input value, X F Represents the first input value, the round function is used to Rounding calculation is performed, qmax represents the maximum value corresponding to the second data type, and qmin represents the minimum value corresponding to the second data type.

[0069] It should be noted that the maximum value and minimum value corresponding to the second data type are related to the data type, that is, when the second data type is determined, the maximum value and minimum value corresponding to the second data type are determined. For example, if the second data type is 8 bits, the corresponding maximum value is 2^7-1=127, and the corresponding minimum value is -2^7=-128. If the second data type is 7 bits, the corresponding maximum value is 2^6-1=63, and the corresponding minimum value is -2^6=-64. Similarly, when the second data type is 5 bits, the corresponding maximum value is 2^5-1=31, and the corresponding minimum value is -2^5=-32.

[0070] It should also be noted that, to facilitate engineering acceleration, each operation requiring quantization can be aligned with the Tensorflow Lite quantization standard. The Tensorflow Lite quantization standard primarily refers to the 8-bit quantization standard proposed by Google. This quantization standard specifies the type of quantization operations in weights and activations, the type, range, and constraints of quantization parameters (scale / zero_point), and the subtypes of neural network operations that support quantization. Quantization training schemes must reference and align with Google's quantization standard to facilitate instruction-level acceleration of edge devices. Quantization training methods that meet Google's standard can be deployed on a variety of edge devices, demonstrating a certain degree of versatility.

[0071] Through the embodiments provided by the present application, a first input value of a first data type is obtained, wherein the first input value is set as the input value of a target operation in a neural network model to be trained, and the number of bits occupied by the first input value of the first data type is a first number of bits; a quantization operation is performed on the first input value to obtain a second input value of a second data type, wherein the number of bits occupied by the second input value of the second data type is a second number of bits, and the second number of bits is less than the first number of bits; a target operation is performed on the second input value to obtain a first output value of the second data type, and the number of bits occupied by the first output value of the second data type is the second number of bits; an inverse quantization operation is performed on the first output value to obtain a second output value of the first data type, and the number of bits occupied by the second output value of the first data type is the first number of bits; based on the second output value, the neural network model is trained, that is, in the process of training the neural network model, the input value of the target operation in the neural network model is quantized to reduce the number of bits occupied by the input value, so that when the target operation is performed on the input value, the resources consumed by performing the target operation can be reduced, thereby improving the operation speed, for example, reducing the operation time, or reducing the memory occupied during runtime. Furthermore, in the process of training the neural network model, an inverse quantization operation is performed on the output value of the target operation in the neural network model to convert the data type of the output value into the data type before the quantization operation is performed, thereby reducing the resources consumed in executing the target operation while ensuring that there is no loss in the accuracy of the data in the neural network model (that is, ensuring that the quantization accuracy loss is not large).

[0072] Optionally, performing a quantization operation on the first input value to obtain a second input value of the second data type may include: determining a target quantization parameter used for the quantization operation based on the maximum value and the minimum value corresponding to the second data type and the maximum value and the minimum value of the first input value; and performing a quantization operation on the first input value using the target quantization parameter to obtain a second input value of the second data type.

[0073] In this embodiment, a quantization operation is performed on a first input value to obtain a second input value of a second data type. For example, if the first input value is a FLAT32 value, an INT8 value is obtained through a quantization operation, i.e., a 4-byte value is quantized to a 1-byte value, and the operators of each layer in the neural network model are calculated using the 1-byte value. Since the 4-byte value is quantized to a 1-byte value, the memory occupied by the value storage is reduced, thereby reducing the number of bits occupied by the input value. In this way, when a target operation is performed on the input value, the resources consumed by performing the target operation can be reduced, thereby improving the operation speed, for example, reducing the duration of the operation, or reducing the memory occupied during operation.

[0074] In this embodiment, the maximum value and minimum value corresponding to the second data type are related to the second data type. For example, if the second data type is 8 bits, the corresponding maximum value is 2^7-1=127, and the corresponding minimum value is -2^7=-128. If the second data type is 7 bits, the corresponding maximum value is 2^6-1=63, and the corresponding minimum value is -2^6=-64. Similarly, when the second data type is 5 bits, the corresponding maximum value is 2^5-1=31, and the corresponding minimum value is -2^5=-32.

[0075] Optionally, determining the target quantization parameter used for the quantization operation based on the maximum value and minimum value corresponding to the second data type and the maximum value and minimum value of the first input value may include: when the absolute value of the maximum value and the absolute value of the minimum value corresponding to the second data type are the same, determining the first quantization parameter used for the quantization operation based on the maximum value and the minimum value corresponding to the second data type and the maximum value and the minimum value of the first input value, wherein the target quantization parameter includes the first quantization parameter; when the absolute value of the maximum value and the absolute value of the minimum value corresponding to the second data type are different, determining the second quantization parameter used for the quantization operation based on the maximum value and the minimum value corresponding to the second data type and the maximum value and the minimum value of the first input value, wherein the target quantization parameter includes the second quantization parameter, and the first quantization parameter is different from the second quantization parameter.

[0076] Among them, when the target operation includes a convolution function executed on a convolution layer in a neural network model, the absolute value of the maximum value and the absolute value of the minimum value corresponding to the second data type are set to the same; when the target operation includes a weight function executed on a fully connected layer in a neural network model, the absolute value of the maximum value and the absolute value of the minimum value corresponding to the second data type are set to the same; when the target operation includes an activation function in a neural network model, the absolute value of the maximum value and the absolute value of the minimum value corresponding to the second data type are set to different.

[0077] In this embodiment, the quantization operation in the neural network model includes a symmetric quantization operation and an asymmetric quantization operation, wherein the symmetric quantization operation means that when calculating the target quantization parameter, the absolute values ​​of the maximum value and the minimum value corresponding to the second data type are the same. The asymmetric quantization operation means that when calculating the target quantization parameter, the absolute values ​​of the maximum value and the minimum value corresponding to the second data type are different.

[0078] In this embodiment, each layer of the neural network model corresponds to one or more operators, and one operator corresponds to one target operation. The neural network model includes multiple operators, each layer corresponds to one operator, or some layers correspond to multiple operators. For example, a convolutional layer can correspond to multiple operators, and a pooling layer corresponds to one operator. Due to the same or different operators, the multiple target operations included in the neural network model can be the same or different. The target operation is a quantization operation performed on the first input data according to the operator.

[0079] Operators are responsible for establishing the correspondence between fixed-point and floating-point numbers. Quantization benefits can only be realized by converting floating-point calculations to fixed-point calculations during neural network training. The following demonstrates how fixed-point calculations simulate floating-point calculations using quantization operators, and also provides a classification of quantization operators.

[0080] X_q=(int)(X_f*2^k)

[0081] X_f=((float)(X_f*2^k)) / 2^k

[0082] 10 = (int)(1.25*2^3)

[0083] 3 = (int)(0.4*2^3)

[0084] 1.25+0.4=1.65 is approximately equal to (10+3) / 2^3==1.625

[0085] 1.25*0.4=0.5 is approximately equal to 10*3 / 2^6==0.46875

[0086] From the above, we can see that the fixed-point representations of scales 2^3, 1.25, and 0.4 are 10 and 3 respectively, and floating-point calculations can be simulated through fixed-point calculations.

[0087] Optionally, determining a target quantization parameter used for the quantization operation according to the maximum value and the minimum value corresponding to the second data type and the maximum value and the minimum value of the first input numerical value may include: determining the target quantization parameter using the following formula:

[0088]

[0089] zero_point=0

[0090] Among them, the target quantization parameters include scale and zero_point, min_val represents the minimum value of the first input value, max_val represents the maximum value of the first input value, qmax represents the maximum value corresponding to the second data type, and qmin represents the minimum value corresponding to the second data type.

[0091] For example, if the second data type is 8 bits, then qmax is 2^7-1=127, qmin is -2^7=-128, and the first input value is counted, and max_val is 255 and min_val is -510. Then:

[0092]

[0093] Optionally, determining a target quantization parameter used for the quantization operation according to the maximum value and the minimum value corresponding to the second data type and the maximum value and the minimum value of the first input numerical value includes: determining the target quantization parameter using the following formula:

[0094]

[0095]

[0096] Among them, the target quantization parameters include scale and zero_point, min_val represents the minimum value of the first input value, max_val represents the maximum value of the first input value, qmax represents the maximum value corresponding to the second data type, qmin represents the minimum value corresponding to the second data type, and the round function is used to round Perform rounding calculations.

[0097] For example, if the second data type is 8 bits, then qmax is 2^7-1=127, qmin is -2^7=-128, and the first input value is counted, and max_val is 255 and min_val is -510. Then:

[0098]

[0099]

[0100] Optionally, performing a quantization operation on the first input value using the target quantization parameter to obtain a second input value of the second data type may include:

[0101]

[0102] Among them, the target quantization parameters include scale and zero_point, X Q Represents the second input value, X F Represents the first input value, the round function is used to Rounding calculation is performed, qmax represents the maximum value corresponding to the second data type, and qmin represents the minimum value corresponding to the second data type.

[0103] Optionally, performing an inverse quantization operation on the first output value to obtain a second output value of the first data type may include: performing an inverse quantization operation on the first output value using a target quantization parameter to obtain the second output value of the first data type.

[0104] In this embodiment, the quantization operation and the inverse quantization operation in the neural network model appear in pairs. The quantization operator that performs the quantization operation and the inverse quantization operator that performs the inverse quantization operation are regarded as a pseudo-quantization operator in the process of neural network training. Therefore, after the first input value of the neural network model is quantized, the second input value is obtained. After the target operation is performed on the second input value, the first output value is obtained. The inverse quantization operation is performed on the first output value. The quantization operation and the inverse quantization operation appear in pairs. The target quantization parameter of the quantization operation can be used to perform the inverse quantization operation on the first output value to obtain the second output value, and the quantized data can be restored. Figure 3 As shown, the first input value of float32 is quantized using the target quantization parameter to obtain the second input value of int8, the target operation is performed on the second input value (the input convolution layer performs convolution function operation, the pooling layer performs function operation, and the fully connected layer performs function operation), and the first output value of int8 is output. The first output value of int8 is dequantized to obtain the second output value of float32.

[0105] Optionally, performing a dequantization operation on the first output value using the target quantization parameter to obtain a second output value of the first data type may include obtaining the second output value of the first data type using the following formula:

[0106] X T =(X P -zero_point)*scale

[0107] Among them, the target quantization parameters include scale and zero_point, X T represents the second output value, and X represents the first output value.

[0108] Optionally, performing a quantization operation on the first input value to obtain a second input value of the second data type, performing a target operation on the second input value to obtain a first output value of the second data type, and performing a dequantization operation on the first output value to obtain a second output value of the first data type may include:

[0109] In a case where the target operation includes multiple operations in the neural network model and the first input value of the first data type includes input values ​​of the multiple operations, the following steps are performed for each operation of the multiple operations, wherein each operation is considered as a current operation when performing the following steps:

[0110] Performing a quantization operation on the input value of the current operation to obtain an input value of a second data type;

[0111] Performing the current operation on the input value of the second data type to obtain an output value of the second data type;

[0112] An inverse quantization operation is performed on the output value of the second data type to obtain an output value of the first data type.

[0113] In this embodiment, a neural network model to be trained of facial key points is taken as an example to illustrate a training process in which the target operation includes multiple operations in the neural network model.

[0114] like Figure 6 As shown in FIG1 , a schematic diagram of the training process of quantization and dequantization operations in the face key point neural network model (I) is shown, in which a quantization operation is performed on the input data of the convolutional layer in the face key point neural network model, and a dequantization operation is performed on the output data of the convolutional layer.

[0115] like Figure 7 As shown in Figure 2, a schematic diagram of the training process of quantization and dequantization operations in the convolutional layer. Figure 7 As shown in the figure, when the convolution layer includes multiple operators OP, multiple quantization and dequantization operations may appear in the convolution layer, that is, the input of the input OP1 is quantized, and the data of the output OP1 is dequantized. The data of the dequantized operation can be quantized again and input to OP2. The dequantization operation is performed on the data output by OP2, and then the dequantized data is quantized and output to the next pooling layer.

[0116] like Figure 8 As shown in Figure 2, a schematic diagram of the quantization and dequantization training process in the facial landmark neural network model is shown. Quantization is performed on the input data of the pooling layer in the facial landmark neural network model, and dequantization is performed on the output data of the pooling layer. After the dequantization operation is performed on the output data of the pooling layer, it is input to the fully connected layer, and quantization is performed on the dequantized data.

[0117] It should be noted that the target operation performed in the convolution layer and the target operation performed in the pooling layer can be the same or different, that is, when the target operation is performed in the convolution layer, the operation operator and the target quantization parameter used can be different from or the same as when the target operation is performed in the pooling layer.

[0118] in, Figures 6 and 7 As shown in Figure 1, some computational layers in the facial key point neural network model perform quantization and dequantization operations during operation.

[0119] like Figure 9 As shown in Figure 3, the schematic diagram of the training process of quantization and dequantization operations in the face key point neural network model is as follows: Figure 9 As shown in the figure, the convolutional layers, pooling layers, and fully connected layers in the facial key point neural network model all perform quantization and dequantization operations.

[0120] like Figure 10 Figure 4 shows the schematic diagram of the training process for quantization and dequantization operations in the facial landmark neural network model. The pooling layer and fully connected layer in the facial landmark neural network model are considered as a whole. Quantization and dequantization operations can be performed on the pooling layer and the fully connected layer once. The data input to the pooling layer is quantized, and the data output by the pooling layer is directly input to the fully connected layer. Dequantization is then performed on the data output by the fully connected layer.

[0121] like Figure 11 As shown in FIG5 , a schematic diagram of the training process of quantization and dequantization operations in the facial key point neural network model is shown. During the training process of the facial key point neural network model, a quantization operation and a dequantization operation can be performed. That is, a quantization operation is performed on the data input to the facial key point neural network model for the first time, and a dequantization operation is performed on the data output by the last layer of the facial key point neural network model. That is, a quantization operation is performed on the data input to the facial key point neural network model, and a dequantization operation is performed on the data output by the facial key point neural network model for the last time.

[0122] Optionally, performing the target operation on the second input value to obtain the first output value of the second data type may include at least one of the following:

[0123] Where the target operation comprises a convolution function performed on a convolutional layer in the neural network model, inputting the second input value to the convolution function to obtain a first output value of the second data type output by the convolution function;

[0124] In a case where the target operation includes a weight function executed on a fully connected layer in the neural network model, inputting the second input value into the weight function to obtain a first output value of the second data type output by the weight function;

[0125] In a case where the target operation includes an activation function in a neural network model, the second input value is input to the activation function to obtain a first output value of the second data type output by the activation function.

[0126] In this embodiment, the target operation of each training layer in the neural network model can be the same or different. For a convolutional layer, quantized data is input to a convolution function to obtain the output data of the convolution function. That is, the target operation of the convolutional layer is the convolution function. For a fully connected layer, quantized data is input to a weight function to obtain the output data of the weight function. That is, the target operation of the fully connected layer is the weight function.

[0127] In this embodiment, the target operation may include performing convolution function calculation on the input data, performing weight function calculation, and performing activation function calculation.

[0128] Optionally, training the neural network model according to the second output value may include:

[0129] When the input sample of the neural network model is a target training sample, determining a predicted recognition result output by the neural network model according to the second output value;

[0130] When the loss value between the predicted recognition result and the actual recognition result does not meet the preset loss condition, adjusting the parameters in the neural network model, wherein the actual recognition result is a pre-acquired recognition result, and the actual recognition result is used to represent the recognition result of the target training sample;

[0131] When the loss value between the predicted recognition result and the actual recognition result meets the preset loss condition, the training of the neural network model is terminated, wherein the neural network model at the time of the termination of the training of the neural network model is determined as the target neural network model.

[0132] Taking the training of the facial key point neural network model as an example, the above target neural network model is used to identify facial key points in the input image.

[0133] Taking the training of a neural network model for facial expression recognition as an example, the target neural network model is used to identify facial expression categories in input images.

[0134] Taking the training of facial key point neural network model as an example, a pseudo-quantization-based quantitative perception training method for facial key points is proposed.

[0135] In this embodiment, the FakeQuantize operator is inserted before and after each operation layer that needs to be quantized. Its main function is to first quantize the activation or weight, and then dequantize it to map the value to a discrete value. This ensures that the operation layer is still implemented in floating point while allowing the neural network to perceive the loss caused by quantization, and through training and fine-tuning, the neural network is made more robust to quantization loss. Figure 12 As shown in FIG, a block diagram of the face key point quantization perception training method based on pseudo quantization.

[0136] like Figure 12 As shown in the figure, for a quantized operator (Op) (relative to the target operation in the training layer), a pseudo-quantization (FakeQuantize) operator is inserted before and after it. The specific operation is as follows: assuming that the convolution operation is to be quantized, a pseudo-quantization convolution operation is defined, and the dequantization operator (Dequantize) is inserted before the convolution and then the quantization operator (Quantize) is inserted after the convolution. The quantization operator after the previous operator and the dequantization operator before this convolution are merged into a pseudo-quantization operator. The pseudo-quantization operator is quantized first and then dequantized. The input is a floating-point 32-bit value, and the number domain is a real number. After pseudo-quantization, the output is a 32-bit value, but it will be quantized to a discrete floating-point number. Through the pseudo-quantization operator, the floating-point simulated quantization training process can be implemented, so that quantitative training can be implemented using major open source training frameworks. Each pseudo-quantization operator needs to perform pseudo-quantization operations and also needs to count the fluctuation range of the numerical value, namely the maximum value (max_val) and the minimum value (min_val). During quantization training, the maximum and minimum values ​​of the pseudo-quantization operator input are counted through the Observer, using a moving average strategy.

[0137] To facilitate engineering acceleration, each quantized operator needs to be aligned with the Tensorflow Lite quantization standard.

[0138] Commonly used operators are implemented as follows: 1. The quantization of convolution and fully connected weights uses symmetric quantization, and the quantization value range is [-127, 127]. Bias is not quantized; 2. The quantization of activation output uses asymmetric quantization, and the quantization value range is [-128, 127]; 3. The pooling layer needs to limit the input quantization parameters (scale / zero_point) to be the same as the output. By introducing a SharedFakeQuantize operator, this operator shares the quantization parameters with the FakeQuantize operator before pooling, but does not update the quantization parameters; 4. The Concat operation counts the maximum and minimum values ​​of all inputs to calculate the quantization parameters, and the output quantization parameters are consistent with the input.

[0139] Among them, the quantization operator and the dequantization operator are calculated as follows:

[0140] Considering that the range of input data x is [min_val, max_val] and the range of quantization is [qmin, qmax], quantization can be expressed as follows:

[0141]

[0142] Among them, X Q is the quantized output, X Fis a floating point input, scale and zero_point are quantization parameters. Based on this, dequantization can be expressed as follows:

[0143] X F =(X Q -zero_point)*scale

[0144] Symmetric quantization can calculate the quantization parameter as follows:

[0145]

[0146] zero_point=0

[0147] Asymmetric quantization can calculate the quantization parameter as follows:

[0148]

[0149]

[0150] It should be noted that the quantization parameters scale and zero_point in this embodiment are obtained by counting the value range of each layer. Other alternatives can be achieved by embedding the quantization parameter into the neural network as a learnable parameter. The quantization parameter optimized with the convolution parameter plays a better role of supervisory information than the statistical parameter, thereby guiding better quantization parameters and convolution parameters.

[0151] During the training process, the back propagation of pseudo quantization: Since the pseudo quantization operation is a non-differentiable function, it needs to be approximated by a differentiable function:

[0152]

[0153] In this embodiment, the facial key point quantization perception training method based on pseudo-quantization can include but is not limited to video image traversal apps, short video apps, video calls and other projects or products that require face processing.

[0154] 8-bit quantization technology can reduce model size to 1 / 4, significantly reducing runtime memory usage, and offering faster computation speeds and lower power consumption. Based on these advantages, model quantization is widely used in edge devices such as mobile devices. It is particularly important for computationally intensive tasks such as special effects live streaming.

[0155] This embodiment uses an easy-to-implement quantization-aware training method that is aligned with the Tensorflow Lite standard (easy to accelerate engineering). The FakeQuantize operator is introduced to make the neural network aware of quantization during training, making the network less sensitive to quantization. The training process in this embodiment is easy to implement in major frameworks (Tensorflow\Pytorch, etc.) and can be easily migrated to lower-bit quantization training.

[0156] Through the solution in this embodiment, the facial key points are aligned to a small block of 128x128. By calculating the distance loss between the predicted facial key points and the true value as the MSE indicator of the network performance function, it can be seen that the 8-bit quantization model has little loss on the facial key point model FaceLmk.

[0157] Among them, the quantization training in this embodiment can almost reduce the model capacity to 1 / 4 of the floating point while ensuring that the loss of quantization accuracy is not large, and the memory during runtime will also be reduced to 1 / 4 of the floating point. While reducing data transportation, the model power consumption can be greatly reduced. After engineering acceleration, the current 8-bit quantization operation acceleration can reach more than 20%.

[0158] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0159] According to another aspect of the present invention, a training device for a neural network model is provided for implementing the training method of the neural network model. Figure 13 As shown, the training device of the neural network model includes: an acquisition unit 1301, a quantization unit 1303, an operation unit 1305, an inverse quantization unit 1307 and a training unit 1309.

[0160] The acquisition unit 1301 is used to obtain a first input value of a first data type, wherein the first input value is set as the input value of the target operation in the neural network model to be trained, and the number of bits occupied by the first input value of the first data type is a first number of bits.

[0161] The quantization unit 1303 is configured to perform a quantization operation on the first input value to obtain a second input value of a second data type, wherein the second input value of the second data type occupies a second number of bits, which is smaller than the first number of bits.

[0162] The operation unit 1305 is configured to perform a target operation on the second input value to obtain a first output value of the second data type, where the number of bits occupied by the first output value of the second data type is the second number of bits.

[0163] The inverse quantization unit 1307 is configured to perform an inverse quantization operation on the first output value to obtain a second output value of the first data type, where the number of bits occupied by the second output value of the first data type is the first number of bits.

[0164] The training unit 1309 is used to train the neural network model according to the second output value.

[0165] Through the embodiment provided by the present application, the acquisition unit 1301 acquires a first input value of a first data type, wherein the first input value is set as the input value of the target operation in the neural network model to be trained, and the number of bits occupied by the first input value of the first data type is a first number of bits; the quantization unit 1303 performs a quantization operation on the first input value to obtain a second input value of a second data type, wherein the number of bits occupied by the second input value of the second data type is a second number of bits, and the second number of bits is less than the first number of bits; the operation unit 1305 performs a target operation on the second input value to obtain a first output value of the second data type, and the first output value of the second data type occupies a first number of bits. The number of bits is the second number of bits; the inverse quantization unit 1307 performs an inverse quantization operation on the first output value to obtain a second output value of the first data type, and the number of bits occupied by the second output value of the first data type is the first number of bits; the training unit 1309 trains the neural network model based on the second output value, that is, in the process of training the neural network model, the input value of the target operation in the neural network model is quantized to reduce the number of bits occupied by the input value, so that when the target operation is performed on the input value, the resources consumed by performing the target operation can be reduced, and the operation speed is improved, for example, the operation time is reduced, or the memory occupied during operation is reduced. Further, in the process of training the neural network model, the output value of the target operation in the neural network model is inversely quantized to convert the data type of the output value into the data type before the quantization operation is performed, thereby reducing the resources consumed by performing the target operation while ensuring that the accuracy of the data in the neural network model is not lost (that is, the quantization accuracy loss is not large).

[0166] Optionally, the above-mentioned quantization unit 1303 may include: a first determination module, used to determine the target quantization parameter used for the quantization operation based on the maximum value and minimum value corresponding to the second data type, and the maximum value and minimum value of the first input value; a quantization module, used to perform a quantization operation on the first input value using the target quantization parameter to obtain a second input value of the second data type.

[0167] Among them, the above-mentioned quantization module is used to perform a quantization operation on a first input value of a floating-point type to obtain a second input value of an integer type; wherein the first data type is float32 type, and the second data type is int8 type, or int16 type; or the first data type is float64 type, and the second data type is int8 type, or int16 type, or int32 type.

[0168] Among them, the above-mentioned first determination module may include: a first determination sub-module, used to determine the first quantization parameter used for the quantization operation according to the maximum value and minimum value corresponding to the second data type, and the maximum value and minimum value of the first input value when the absolute value of the maximum value corresponding to the second data type is the same as the absolute value of the minimum value, wherein the target quantization parameter includes the first quantization parameter.

[0169] The second determination submodule is used to determine the second quantization parameter used for the quantization operation according to the maximum value and the minimum value corresponding to the second data type and the maximum value and the minimum value of the first input value when the absolute value of the maximum value corresponding to the second data type is different from the absolute value of the minimum value, wherein the target quantization parameter includes the second quantization parameter, and the first quantization parameter is different from the second quantization parameter.

[0170] Optionally, the above-mentioned device may further include: a first setting submodule, which is used to set the absolute value of the maximum value and the absolute value of the minimum value corresponding to the second data type to be the same when the target operation includes a convolution function executed on a convolution layer in a neural network model. A second setting submodule, which is used to set the absolute value of the maximum value and the absolute value of the minimum value corresponding to the second data type to be the same when the target operation includes a weight function executed on a fully connected layer in a neural network model. A third setting submodule, which is used to set the absolute value of the maximum value and the absolute value of the minimum value corresponding to the second data type to be different when the target operation includes an activation function in a neural network model.

[0171] Optionally, the quantization unit 1303 may also be configured to determine a target quantization parameter using the following formula:

[0172]

[0173] zero_point=0

[0174] Among them, the target quantization parameters include scale and zero_point, min_val represents the minimum value of the first input value, max_val represents the maximum value of the first input value, qmax represents the maximum value corresponding to the second data type, and qmin represents the minimum value corresponding to the second data type.

[0175] Optionally, the quantization unit 1303 may also be configured to determine a target quantization parameter using the following formula:

[0176]

[0177]

[0178] Among them, the target quantization parameters include scale and zero_point, min_val represents the minimum value of the first input value, max_val represents the maximum value of the first input value, qmax represents the maximum value corresponding to the second data type, qmin represents the minimum value corresponding to the second data type, and the round function is used to round Perform rounding calculations.

[0179] 7 Optionally, the quantization unit 1303 may further perform the following operations:

[0180]

[0181] Among them, the target quantization parameters include scale and zero_point, X Q Represents the second input value, X F Represents the first input value, the round function is used to Rounding calculation is performed, qmax represents the maximum value corresponding to the second data type, and qmin represents the minimum value corresponding to the second data type.

[0182] Optionally, the above-mentioned dequantization unit 1307 may include: a dequantization module, configured to perform a dequantization operation on the first output value using a target quantization parameter to obtain a second output value of the first data type.

[0183] The dequantization module is further configured to obtain a second output value of the first data type using the following formula:

[0184] X T =(X P -zero_point)*scale

[0185] Among them, the target quantization parameters include scale and zero_point, X T Represents the second output value, X P Indicates the first output value.

[0186] Optionally, the quantization unit 1303 and the inverse quantization unit 130 are further configured to perform the following operations:

[0187] In a case where the target operation includes multiple operations in the neural network model and the first input value of the first data type includes input values ​​of the multiple operations, the following steps are performed for each operation of the multiple operations, wherein each operation is considered as a current operation when performing the following steps:

[0188] Performing a quantization operation on the input value of the current operation to obtain an input value of a second data type;

[0189] Performing the current operation on the input value of the second data type to obtain an output value of the second data type;

[0190] An inverse quantization operation is performed on the output value of the second data type to obtain an output value of the first data type.

[0191] Optionally, the operating unit 1305 may include at least one of the following:

[0192] 1) A first operation module, configured to input a second input value into the convolution function to obtain a first output value of a second data type output by the convolution function when the target operation includes a convolution function performed on a convolution layer in the neural network model.

[0193] 2) A second operation module, used to input a second input value into the weight function when the target operation includes a weight function executed on a fully connected layer in the neural network model, to obtain a first output value of a second data type output by the weight function.

[0194] 3) A third operation module, configured to input the second input value into the activation function to obtain a first output value of the second data type output by the activation function when the target operation includes an activation function in the neural network model.

[0195] Optionally, the above-mentioned training unit 1309 may include: a third determination module, used to determine the predicted recognition result output by the neural network model according to the second output value when the input sample of the neural network model is the target training sample; an adjustment module, used to adjust the parameters in the neural network model when the loss value between the predicted recognition result and the actual recognition result does not meet the preset loss condition, wherein the actual recognition result is a pre-acquired recognition result, and the actual recognition result is used to represent the recognition result of the target training sample; an end training module, used to end the training of the neural network model when the loss value between the predicted recognition result and the actual recognition result meets the preset loss condition, wherein the neural network model at the end of the training of the neural network model is determined as the target neural network model, and the target neural network model is used to recognize facial key points in the input image.

[0196] According to another aspect of the embodiment of the present invention, an electronic device for implementing the training method of the above-mentioned neural network model is also provided. The electronic device can be Figure 1 The terminal device or server shown in FIG. This embodiment is described by taking the electronic device as a server as an example. Figure 14 As shown, the electronic device includes a memory 1402 and a processor 1404. The memory 1402 stores a computer program, and the processor 1404 is configured to execute the steps in any of the above method embodiments through the computer program.

[0197] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0198] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0199] S1, obtaining a first input value of a first data type, wherein the first input value is set as an input value of a target operation in a neural network model to be trained, and the number of bits occupied by the first input value of the first data type is a first number of bits;

[0200] S2, performing a quantization operation on the first input value to obtain a second input value of a second data type, wherein the second input value of the second data type occupies a second number of bits, and the second number of bits is less than the first number of bits;

[0201] S3, performing a target operation on the second input value to obtain a first output value of the second data type, where the number of bits occupied by the first output value of the second data type is the second number of bits;

[0202] S4, performing an inverse quantization operation on the first output value to obtain a second output value of the first data type, where the number of bits occupied by the second output value of the first data type is the first number of bits;

[0203] S5: Training the neural network model according to the second output value.

[0204] Alternatively, those skilled in the art will appreciate that Figure 14 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 14 It does not limit the structure of the electronic device. For example, the electronic device may also include Figure 14 More or fewer components (such as network interfaces, etc.) as shown in, or with Figure 14 Different configurations shown.

[0205] Among them, the memory 1402 can be used to store software programs and modules, such as the program instructions / modules corresponding to the training method and device of the neural network model in the embodiment of the present invention. The processor 1404 executes various functional applications and data processing by running the software programs and modules stored in the memory 1402, that is, realizes the above-mentioned training method of the neural network model. The memory 1402 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1402 may further include a memory remotely located relative to the processor 1404, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks and combinations thereof. Among them, the memory 1402 can be specifically, but not limited to, used to store information such as a first input value of a first data type, a second input value of a second data type, a first output value of a second data type, a second output value of a first data type, and a neural network model. As an example, if Figure 14 As shown, the memory 1402 may include, but is not limited to, the acquisition unit 1301, quantization unit 1303, operation unit 1305, inverse quantization unit 1307, and training unit 1309 in the training device for the neural network model. In addition, it may also include, but is not limited to, other module units in the training device for the neural network model, which will not be repeated in this example.

[0206] Optionally, the transmission device 1406 is configured to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one embodiment, the transmission device 1406 includes a network interface controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In one embodiment, the transmission device 1406 is a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0207] In addition, the electronic device further includes: a display 1408 for displaying sample data of the neural network model; and a connection bus 1410 for connecting the various module components in the electronic device.

[0208] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting multiple nodes through network communication. The nodes may form a peer-to-peer (P2P) network, and any computing device, such as a server, terminal, or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0209] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions, the computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the training aspects of the neural network model or the training methods of the neural network model provided in various optional implementations of the training aspects of the neural network model. The computer program is configured to execute the steps of any of the above method embodiments when running.

[0210] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0211] S1, obtaining a first input value of a first data type, wherein the first input value is set as an input value of a target operation in a neural network model to be trained, and the number of bits occupied by the first input value of the first data type is a first number of bits;

[0212] S2, performing a quantization operation on the first input value to obtain a second input value of a second data type, wherein the second input value of the second data type occupies a second number of bits, and the second number of bits is less than the first number of bits;

[0213] S3, performing a target operation on the second input value to obtain a first output value of the second data type, where the number of bits occupied by the first output value of the second data type is the second number of bits;

[0214] S4, performing an inverse quantization operation on the first output value to obtain a second output value of the first data type, where the number of bits occupied by the second output value of the first data type is the first number of bits;

[0215] S5: Training the neural network model according to the second output value.

[0216] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing the hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0217] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0218] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above-mentioned computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for causing one or more computer devices (such as personal computers, servers, or network devices) to execute all or part of the steps of the methods described in various embodiments of the present invention.

[0219] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0220] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, and can be electrical or other forms.

[0221] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0222] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0223] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A training method for a neural network model, characterized in that: include: Obtaining a first input value of a first data type based on the sample face image and the associated point data corresponding thereto, wherein the first input value is set as an input value of a target operation in the neural network model to be trained, and the number of bits occupied by the first input value of the first data type is a first number of bits; Determining a target quantization parameter used for a quantization operation based on a maximum value and a minimum value corresponding to the second data type and a maximum value and a minimum value of the first input value, including: when an absolute value of the maximum value and an absolute value of the minimum value corresponding to the second data type are the same, determining a first quantization parameter used for the quantization operation based on the maximum value and the minimum value corresponding to the second data type and the maximum value and the minimum value of the first input value, wherein the target quantization parameter includes the first quantization parameter; when an absolute value of the maximum value and an absolute value of the minimum value corresponding to the second data type are different, determining a second quantization parameter used for the quantization operation based on the maximum value and the minimum value corresponding to the second data type and the maximum value and the minimum value of the first input value, wherein the target quantization parameter includes the second quantization parameter, and the first quantization parameter is different from the second quantization parameter; performing a quantization operation on the first input value using the target quantization parameter to obtain a second input value of the second data type, wherein the second input value of the second data type occupies a second number of bits, and the second number of bits is less than the first number of bits; Performing the target operation on the second input value to obtain a first output value of the second data type, where the number of bits of the first output value of the second data type is the second number of bits; Performing an inverse quantization operation on the first output value to obtain a second output value of the first data type, where the number of bits occupied by the second output value of the first data type is the first number of bits; The neural network model is trained according to the second output value.

2. The method according to claim 1, characterized in that The method further comprises at least one of the following: In a case where the target operation includes a convolution function performed on a convolutional layer in the neural network model, setting the absolute value of the maximum value and the absolute value of the minimum value corresponding to the second data type to be the same; In a case where the target operation includes a weight function performed on a fully connected layer in the neural network model, setting the absolute value of the maximum value and the absolute value of the minimum value corresponding to the second data type to be the same; In a case where the target operation includes an activation function in the neural network model, the absolute value of the maximum value and the absolute value of the minimum value corresponding to the second data type are set to be different.

3. The method according to claim 1, characterized in that The determining, according to the maximum value and the minimum value corresponding to the second data type and the maximum value and the minimum value of the first input value, a target quantization parameter used in the quantization operation includes: The target quantization parameter is determined by the following formula: ， ， Wherein, the target quantization parameters include the and stated , represents the minimum value of the first input value, represents the maximum value of the first input value, Indicates the maximum value corresponding to the second data type, Indicates the minimum value corresponding to the second data type.

4. The method according to claim 1, wherein The determining, according to the maximum value and the minimum value corresponding to the second data type and the maximum value and the minimum value of the first input value, a target quantization parameter used in the quantization operation includes: The target quantization parameter is determined by the following formula: ， ， Wherein, the target quantization parameters include the and stated , represents the minimum value of the first input value, represents the maximum value of the first input value, Indicates the maximum value corresponding to the second data type, Indicates the minimum value corresponding to the second data type, Function is used to Perform rounding calculations.

5. The method according to claim 1, wherein The performing a quantization operation on the first input value using the target quantization parameter to obtain the second input value of the second data type includes: ， Wherein, the target quantization parameters include the and stated , represents the second input value, represents the first input value, the Used for Rounding calculation is performed, the Indicates the maximum value corresponding to the second data type, Indicates the minimum value corresponding to the second data type.

6. The method according to claim 1, characterized in that The performing a dequantization operation on the first output value to obtain a second output value of the first data type includes: An inverse quantization operation is performed on the first output value using the target quantization parameter to obtain the second output value of the first data type.

7. The method according to claim 6, characterized in that The performing a dequantization operation on the first output value using the target quantization parameter to obtain the second output value of the first data type includes: The second output value of the first data type is obtained by the following formula: * ， Wherein, the target quantization parameters include the and stated , represents the second output value, Indicates the first output value.

8. The method according to any one of claims 1 to 7, characterized in that Performing a quantization operation on the first input value to obtain a second input value of a second data type, performing the target operation on the second input value to obtain a first output value of the second data type, and performing a dequantization operation on the first output value to obtain a second output value of the first data type, including: In a case where the target operation includes a plurality of operations in the neural network model and the first input value of the first data type includes input values ​​of the plurality of operations, the following steps are performed on each of the plurality of operations, wherein each operation is considered as a current operation when performing the following steps: Performing a quantization operation on the input value of the current operation to obtain an input value of the second data type; Performing the current operation on the input value of the second data type to obtain an output value of the second data type; An inverse quantization operation is performed on the output value of the second data type to obtain an output value of the first data type.

9. The method according to any one of claims 1 to 7, characterized in that The performing the target operation on the second input value to obtain a first output value of the second data type includes at least one of the following: In a case where the target operation comprises a convolution function performed on a convolutional layer in the neural network model, inputting the second input value to the convolution function to obtain the first output value of the second data type output by the convolution function; In a case where the target operation includes a weight function executed on a fully connected layer in the neural network model, inputting the second input value into the weight function to obtain the first output value of the second data type output by the weight function; In a case where the target operation includes an activation function in the neural network model, the second input value is input to the activation function to obtain the first output value of the second data type output by the activation function.

10. The method according to any one of claims 1 to 7, characterized in that The step of training the neural network model according to the second output value includes: When the input sample of the neural network model is a target training sample, determining a predicted recognition result output by the neural network model according to the second output value; If a loss value between the predicted recognition result and the actual recognition result does not satisfy a preset loss condition, adjusting parameters in the neural network model, wherein the actual recognition result is a pre-acquired recognition result, and the actual recognition result is used to represent the recognition result of the target training sample; When the loss value between the predicted recognition result and the actual recognition result satisfies the preset loss condition, the training of the neural network model is terminated, wherein the neural network model at the end of the training of the neural network model is determined as the target neural network model, and the target neural network model is used to identify facial key points in the input picture.

11. The method according to any one of claims 1 to 7, characterized in that Performing a quantization operation on the first input value to obtain a second input value of a second data type includes: Performing a quantization operation on the first input value of a floating-point type to obtain the second input value of an integer type; The first data type is float32 type, and the second data type is int8 type, or int16 type; or The first data type is float64 type, and the second data type is int8 type, or int16 type, or int32 type.

12. A computer-readable storage medium comprising a stored program, wherein: When the program is executed, the method described in any one of claims 1 to 11 is executed.

13. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 11 through the computer program.

Images