A model training method, device, storage medium and equipment

Abstract

The present specification discloses a model training method, apparatus, storage medium and device, which can group and fuse the activation values ​​output by each network layer of the model to be trained during the forward propagation process of model training, and can bit-compress the activation value derivative mask tensor of each network layer, thereby significantly reducing the memory usage during the model training process, thereby improving the model training efficiency and reducing the model training cost.

Description

Technical Field

[0001] The present invention relates to the field of neural network model technology, and in particular to a model training method, apparatus, storage medium and device. Background Art

[0002] At present, the wide application of deep learning technology in many fields has demonstrated its excellent efficiency. The two cores in the training process of deep neural network models are tensor multiplication and activation function. Among them, tensor multiplication is used to achieve linear transformation, and activation function is used to introduce nonlinear features to the linear transformation achieved by tensor multiplication. Together, the two enable deep learning models to learn and perform complex tasks.

[0003] However, with the increase of model parameters and the deepening of network layers of deep neural network models, the memory consumption and computational complexity in the training process of deep neural network models have increased significantly, becoming the main bottleneck limiting the flexibility of model expansion and hardware deployment. Especially in the back-propagation stage, it is necessary to combine the activation function and tensor multiplication operation to calculate the gradient of the loss function with respect to the model parameters. In this process, it is necessary to store the activation value and gradient information of each network layer of the deep neural network model, which takes up a lot of storage space and increases the complexity of memory access, resulting in a high cost of training deep neural network models.

[0004] Therefore, how to reduce the training cost of deep neural network models is an urgent problem to be solved. Summary of the invention

[0005] This specification provides a model training method, device, storage medium and equipment to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides a model training method, including:

[0008] Acquire a model to be trained and training sample data, wherein the training sample data is at least one of text data, image data, video data, and audio data;

[0009] Input the training sample data into the model to be trained, so as to determine the groups of neuron nodes contained in the network layer according to the preset grouping relationship between the neuron nodes for each network layer of the model to be trained, and determine the fusion activation value of each group according to the activation value output by the neuron nodes contained in each neuron node group, and determine the activation value derivative mask tensor of the network layer according to the activation value output by each neuron node contained in the network layer, wherein the activation value derivative mask tensor is used to reflect the importance of each neuron node in the gradient calculation;

[0010] According to the group fusion activation value, determining the gradient of a preset loss function relative to each neuron node group of the network layer as a fusion gradient;

[0011] Determine, according to the activation value derivative mask tensor and the fused gradient, the gradient of the loss function relative to each neuron node of the network layer as each target gradient;

[0012] The weight parameters of the network layer are adjusted according to the target gradients to obtain a trained model, and business execution is performed according to the trained model.

[0013] Optionally, determining the activation value derivative mask tensor of the network layer according to the activation value output by each neuron node included in the network layer specifically includes:

[0014] Determine an activation value derivative mask tensor of the network layer according to the activation value output by each neuron node included in the network layer, and perform a bit compression operation on the activation value derivative mask tensor to obtain a compressed activation value derivative mask tensor;

[0015] According to the activation value derivative mask tensor and the fused gradient, the gradient of the loss function relative to each neuron node of the network layer is determined as each target gradient, specifically including:

[0016] According to the compressed activation value derivative mask tensor and the fused gradient, the gradient of the loss function relative to each neuron node of the network layer is determined as each target gradient.

[0017] Optionally, the training sample data is input into the model to be trained, so as to determine the groups of neuron nodes contained in the network layer according to the preset grouping relationship between the neuron nodes for each network layer of the model to be trained, and to determine the fusion activation value of each group according to the activation value output by the neuron nodes contained in each neuron node group, and to determine the activation value derivative mask tensor of the network layer according to the activation value output by each neuron node contained in the network layer, specifically including:

[0018] The training sample data is input into the model to be trained to determine the output value of each neuron node contained in each network layer of the model to be trained, and the activation value of each neuron node contained in the network layer is determined according to a preset specified activation function and the output value, and the neuron node groups contained in the network layer are determined according to the preset grouping relationship between the neuron nodes, and the fusion activation value of each group is determined according to the activation values ​​output by the neuron nodes contained in each neuron node group.

[0019] Optionally, determining the gradient of the loss function relative to each neuron node of the network layer as each target gradient according to the activation value derivative mask tensor and the fused gradient, specifically includes:

[0020] Determine the target dimension reduction multiple according to the number of neuron nodes contained in each neuron node group;

[0021] According to the activation value derivative mask tensor, the fused gradient and the target dimension reduction multiple, the gradient of the loss function relative to each neuron node of the network layer is determined as each target gradient.

[0022] This specification provides a service execution method, including:

[0023] Acquire target business data, where the target business data includes: at least one of text data, image data, video data, and audio data;

[0024] Inputting the target business data into a pre-trained business model to obtain a business execution result according to the target business data through the business model, wherein the business model is trained by the above-mentioned model training method;

[0025] The business is executed according to the business execution result.

[0026] This specification provides a model training device, including:

[0027] An acquisition module, used to acquire a model to be trained and training sample data, wherein the training sample data is at least one of text data, image data, video data, and audio data;

[0028] A forward propagation module, used for inputting the training sample data into the model to be trained, so as to determine the groups of neuron nodes contained in each network layer of the model to be trained according to the preset grouping relationship between the neuron nodes, and to determine the fusion activation value of each group according to the activation value output by the neuron nodes contained in each neuron node group, and to determine the activation value derivative mask tensor of the network layer according to the activation value output by each neuron node contained in the network layer, wherein the activation value derivative mask tensor is used to reflect the importance of each neuron node in the gradient calculation;

[0029] A first back propagation module is used to determine, according to the group fusion activation value, the gradient of a preset loss function relative to each neuron node group of the network layer as a fusion gradient;

[0030] A second back propagation module is used to determine the gradient of the loss function relative to each neuron node of the network layer as each target gradient according to the activation value derivative mask tensor and the fused gradient;

[0031] The execution module is used to adjust the weight parameters of the network layer according to the target gradients to obtain a trained model, and perform tasks according to the trained model.

[0032] Optionally, the forward propagation module is specifically used to determine the activation value derivative mask tensor of the network layer according to the activation value output by each neuron node contained in the network layer, and perform a bit compression operation on the activation value derivative mask tensor to obtain a compressed activation value derivative mask tensor; based on the compressed activation value derivative mask tensor and the fused gradient, determine the gradient of the loss function relative to each neuron node of the network layer as each target gradient.

[0033] This specification provides a service execution device, including:

[0034] An acquisition module, used to acquire target business data, wherein the target business data includes: at least one of text data, image data, video data, and audio data;

[0035] A data processing module, used for inputting the target business data into a pre-trained business model, so as to obtain a business execution result according to the target business data through the business model, wherein the business model is trained by the above-mentioned model training method;

[0036] An execution module is used to execute the service according to the service execution result.

[0037] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned model training and business execution methods.

[0038] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned model training and business execution methods when executing the program.

[0039] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0040] In the model training method provided in the present specification, firstly, a model to be trained and training sample data are obtained, wherein the training sample data is at least one of text data, image data, and video data, and the training sample data is input into the model to be trained, so as to determine the groups of neuron nodes contained in the network layer according to the preset grouping relationship between the neuron nodes for each network layer of the model to be trained, and determine the fusion activation value of each group according to the activation value output by the neuron nodes contained in each neuron node group, and determine the activation value derivative mask tensor of the network layer according to the activation value output by each neuron node contained in the network layer, the activation value derivative mask tensor is used to reflect the importance of each neuron node in the gradient calculation, and determine the gradient of the preset loss function relative to each neuron node group of the network layer according to the group fusion activation value as the fusion gradient, and determine the gradient of the loss function relative to each neuron node of the network layer according to the activation value derivative mask tensor and the fusion gradient as each target gradient, and adjust the weight parameter of the network layer according to each target gradient to obtain the trained model, and perform business according to the trained model.

[0041] It can be seen from the above method that during the forward propagation process of training the model to be trained, the activation values ​​output by each network layer of the model to be trained can be grouped and fused, and the importance of different neuron nodes in the gradient calculation of back propagation can be characterized by the activation value derivative mask tensor of each network layer, so that in the back propagation stage, the gradient of the loss function relative to each neuron node can be calculated only based on the grouped fused activation values ​​and the activation value derivative mask tensor, thereby significantly reducing the memory usage during the training of the model to be trained, thereby improving the training efficiency of the model to be trained and reducing the training cost of the model to be trained. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0043] Figure 1 A flowchart of a model training method provided in this specification;

[0044] Figure 2A A schematic diagram of a network layer in a neural network model provided in this specification;

[0045] Figure 2B A schematic diagram of the network layers in the model to be trained provided in this specification;

[0046] Figure 3 A schematic diagram of a process for determining the first activation value derivative mask tensor provided in this specification;

[0047] Figure 4 A schematic diagram of a process for determining a second activation value derivative mask tensor provided in this specification;

[0048] Figure 5 A schematic diagram of a process for determining a third activation value derivative mask tensor provided in this specification;

[0049] Figure 6 A flowchart of a service execution method provided in this specification;

[0050] Figure 7 A schematic diagram of a model training device provided in this specification;

[0051] Figure 8 A schematic diagram of a service execution device provided in this specification;

[0052] Fig. 9 A method corresponding to the Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0054] At present, in the training and reasoning process of neural network models, data exchange between the computing core and the memory is very frequent. Specifically, in the forward propagation stage of the neural network model, the computing core needs to store the calculated activation values ​​output by each network layer of the neural network model into the memory, and in the backward propagation stage, these activation values ​​are read from the memory to calculate the gradient of the loss function relative to each network layer, and then adjust the weight parameters of each network layer. Since the resource consumption of accessing memory is often higher than the resource consumption of performing computing operations, the training cost of the model is high. Therefore, in order to reduce the cost of using the model and improve the computing efficiency of the model, it is very necessary to optimize the forward propagation and backward propagation process of the neural network model to reduce the memory usage and storage access times of the computing hardware.

[0055] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0056] Figure 1 A flow chart of a model training method provided in this specification includes the following steps:

[0057] S101: Obtain a model to be trained and training sample data, where the training sample data is at least one of text data, image data, and video data.

[0058] In this specification, the business platform can obtain the model to be trained and the training sample data required for training the model to be trained according to the operation instructions input by the user, and then input the training sample data into the model to be trained, and perform forward propagation through each network layer inside the model to be trained to generate the output result of the model to be trained, and then use a preset loss function to determine the loss value according to the difference between the output result of the model to be trained and the true label corresponding to the preset training sample data, and based on the determined loss value, calculate the gradient of the loss function relative to each network layer inside the model to be trained, and finally adjust the weight parameters of each network layer according to the calculated gradient to obtain the trained model.

[0059] The above-mentioned training sample data may be determined according to actual needs, and the training sample data here may include: at least one of text data, image data, audio data, and video data.

[0060] For example, if the model to be trained is an image recognition model, the above-mentioned training sample data may be the image data to be recognized, and the image data to be recognized may be input into the model to be trained to obtain the recognition result corresponding to the image data to be recognized through the model to be trained.

[0061] For another example: if the model to be trained is a large language model, the above-mentioned training sample data can be question text data, and then the question text data can be input into the model to be trained to generate corresponding reply text data for the question text data through the model to be trained.

[0062] In this specification, the execution entity used to implement the model training method may refer to a designated device set up in the business platform by a server, etc., or may refer to a terminal device such as a desktop computer, a laptop computer, etc. For the sake of ease of description, the model training method provided in this specification is described below using the server as the execution entity as an example.

[0063] S102: Input the training sample data into the model to be trained, so as to determine the groups of neuron nodes contained in each network layer of the model to be trained according to the preset grouping relationship between the neuron nodes, and determine the fusion activation value of each group according to the activation value output by the neuron nodes contained in each neuron node group, and determine the activation value derivative mask tensor of the network layer according to the activation value output by each neuron node contained in the network layer, wherein the activation value derivative mask tensor is used to reflect the importance of each neuron node in the gradient calculation.

[0064] In this specification, the server can input training sample data into the model to be trained, so that for each network layer of the model to be trained, the neuron node groups contained in the network layer can be determined according to the preset grouping relationship between the neuron nodes, and the fusion activation values ​​of each group can be determined according to the activation values ​​output by the neuron nodes contained in each neuron node group, and the activation value derivative mask tensor of the network layer can be determined according to the activation values ​​output by each neuron node contained in the network layer.

[0065] Among them, the above-mentioned grouping relationship can be pre-set, for example: among the neuron nodes included in the network layer, every N neuron nodes are set to belong to the same neuron node group, where N can be set according to actual needs, such as: if the number of neuron nodes included in the network layer is 15, the 1st to 5th neuron nodes can be set as the first neuron node group, the 6th to 10th neuron nodes can be set as the second neuron node group, and the 11th to 15th neuron nodes can be set as the third neuron node group.

[0066] It is worth noting that in order to reduce the memory access overhead caused by frequent data exchange between the computing core and the memory during the model training process, at least part of the network layers in the above-mentioned network layers included in the model to be trained can be composed of hidden layers and group fusion layers. In other words, there are at least some hidden layers in each hidden layer included in the model to be trained. For each hidden layer in the at least part of the hidden layers mentioned above, a group fusion layer is further provided between the hidden layer and the next hidden layer located at the hidden layer, as the group fusion layer corresponding to the hidden layer. The combination of the hidden layer and the group fusion layer corresponding to the hidden layer can be regarded as a network layer included in the model to be trained. Specifically, Figure 2A , Figure 2B shown.

[0067] Figure 2A A schematic diagram of a network layer in a neural network model provided in this specification.

[0068] Combination Figure 2A It can be seen that in a conventional neural network model, for each neuron node contained in each hidden layer, the neuron node is connected to each node in the next hidden layer of the hidden layer where the neuron node is located. This structure requires that during the training of the conventional neural network model, the activation value output by each neuron node needs to be saved for back propagation. However, since access operations for a large number of activation values ​​need to be performed during the training process, the memory access overhead is significantly increased.

[0069] Figure 2B This is a schematic diagram of the network layers in the model to be trained provided in this specification.

[0070] Combination Figure 2B It can be seen that in the above-mentioned model to be trained, for each hidden layer in the above-mentioned at least part of the hidden layers, a group fusion layer is also provided between the hidden layer and the next hidden layer located in the hidden layer, and each specified number of neuron nodes in the hidden layer is connected to a neuron node in the group fusion layer corresponding to the hidden layer.

[0071] For example: if the hidden layer includes nine neuron nodes from 1 to 9, and the above-mentioned specified number is 3, then every 3 neuron nodes in the neuron nodes contained in the hidden layer can be divided into a neuron node group, wherein each neuron node group corresponds one-to-one to each neuron node contained in the group fusion layer corresponding to the hidden layer, and for each neuron node group, each neuron node in the neuron node group is connected to the neuron node corresponding to the neuron node group in the group fusion layer.

[0072] Combination Figure 2BIt can be seen that the preset grouping relationship between the neuron nodes included in the hidden layer and the grouping relationship between the neuron nodes included in the hidden layer corresponding to the hidden layer can be determined, and the grouping of the neuron nodes included in the network layer can be determined.

[0073] Through each group fusion layer, the activation values ​​output by each neuron node contained in each hidden layer can be grouped and fused, so that in the forward propagation process, only the group fusion activation values ​​output by the neuron nodes corresponding to each neuron node group in the group fusion layer need to be saved, and the back propagation is performed based on the group fusion activation values ​​output by each neuron node in the group fusion layer, thereby significantly reducing the amount of data that needs to be accessed, and further reducing the cost of model training. For details, please refer to the following formula:

[0074]

[0075] In the above formula, The group fusion layer The ith neuron node of the hidden layer fuses the activation values ​​output by each neuron node in the corresponding hidden layer neuron node group to obtain the group fusion activation value. is the dimension reduction factor (i.e., group fusion layer Each neuron in the corresponding hidden layer The number of inputs, in other words, is the group fusion layer Each neuron in the hidden layer The number of neuron nodes in the corresponding neuron node group is the number of each neuron node in the hidden layer after group fusion processing. Passed to the packet fusion layer The information is reduced to dimension, The group fusion layer The internal pairs of neurons in the corresponding hidden layer The sequence number of the neuron input is , For hidden layer The outputs of neurons are grouped and fused in the memory according to their proximity. No. The number of the neuron node in the neuron node group corresponding to the coding layer, The group fusion layer No. The neuron node in the coding layer corresponds to the neuron node group The activation value output by the neuron node, For group fusion layer The group fusion results output in are numerically scaled (i.e., averaged) to obtain the group fusion activation value.

[0076]

[0077] In the above formula, is the loss function relative to the group fusion layer No. The number of the neuron node in the neuron node group corresponding to the coding layer is The derivative of the activation value output by the neuron node, is the loss function relative to the group fusion layer No. The derivative of the activation value output by a neuron node.

[0078] It can be seen from the above formula that in the back propagation stage of the model to be trained, when calculating the gradient of the loss function relative to each neuron node included in the hidden layer, it can be directly obtained by dividing the gradient of the neuron activation value of the grouped fusion layer l by the dimensionality reduction multiple n and then multiplying it by the activation derivative mask tensor, so that the business platform can discard the activation value output by each neuron node included in the hidden layer in the forward propagation stage, that is, the business platform only needs to save the activation value output by each neuron node included in the grouped fusion layer in the forward propagation stage, and then perform back propagation, thereby reducing the frequency of data exchange between the computing core and the memory during the model training process, and further reducing the memory access overhead during the model training process.

[0079] In actual application scenarios, the above-mentioned server can determine the neuron node groups contained in each network layer of the model to be trained according to the preset grouping relationship between the neuron nodes, and determine the fusion activation value of each group according to the activation value output by the neuron nodes contained in each neuron node group. The method can be that the server inputs the training sample data into the model to be trained, so as to determine the neuron node groups contained in the network layer according to the connection relationship between the neuron nodes in the hidden layer of the network layer and the neuron nodes in the group fusion layer of the network layer for each network layer of the model to be trained, and determine the fusion activation value of each group according to the activation value output by the neuron nodes contained in each neuron node group through the neuron nodes corresponding to each neuron node group in the hidden layer in the group fusion layer.

[0080] Furthermore, after determining the fusion activation values ​​of each group, the server can determine the activation value derivative mask tensor of the network layer according to the activation values ​​output by each neuron node contained in the hidden layer of the network layer, wherein the activation value derivative mask tensor here can be used to reflect the importance of each neuron node in the gradient calculation.

[0081] Specifically, the server can input the training sample data into the model to be trained to determine the output value of each neuron node contained in each network layer of the model to be trained, and determine the activation value of each neuron node contained in the network layer according to a preset specified activation function and the output value of each neuron node contained in the network layer, divide the neuron nodes contained in the network layer into neuron node groups, and determine the fusion activation value of each group according to the activation value output by the neuron nodes contained in each neuron node group.

[0082] The above-mentioned specified activation function may be, for example, a rectified linear unit (ReLU), a rectified linear unit 6 (ReLU6), a Gaussian error linear unit (GELU), etc.

[0083] For ease of understanding, the following uses the activation functions ReLU, ReLU6, and GELU as examples to explain in detail how the server determines the activation value derivative mask tensor. Figure 3 , Figure 4 , Figure 5 shown.

[0084] Figure 3 This is a schematic diagram of the process of determining the first activation value derivative mask tensor provided in this specification.

[0085] Combination Figure 3 It can be seen that when the above specified activation function is the ReLU activation function, the ReLU activation function is defined as , when the input x is positive, the output activation value is equal to the input, when the input x is non-positive, the output activation value is 0. At this time, when the input x is positive, the calculated derivative of the activation value of the input x is 1, and when the input x is non-positive, the calculated derivative of the activation value of the input x is 0. For details, please refer to the following formula:

[0086]

[0087] In the above formula, That is, the output value of each neuron node contained in the hidden layer The activation value derivative mask tensor obtained after calculation, where With input The same shape.

[0088] Figure 4This is a schematic diagram of the process of determining the second activation value derivative mask tensor provided in this specification.

[0089] Combination Figure 4 It can be seen that when the above specified activation function is the ReLU6 activation function, the definition of the ReLU6 activation function is , that is, when the input x is a non-positive number, the output value is 0, and when the input x is a value greater than 6, the output value is 6. When the input x is greater than 0 and less than 6, the output value is x itself. Therefore, when the input x is a non-positive number or a value greater than 6, the output value of the ReLU6 activation function does not change with the change of the input x, that is, it is always 0, so when the input x is a non-negative integer or a value greater than 6, the calculated derivative of the activation value of the input x is 0, when the input x is greater than 0 and less than 6, the output value of the ReLU6 activation function is equal to the input x, so when the input x is greater than 0 and less than 6, the calculated derivative of the activation value of the input x is 1. For details, please refer to the following formula:

[0090]

[0091] In the above formula, That is, the output value of each neuron node contained in the hidden layer The activation value derivative mask tensor obtained after calculation, where With input The same shape.

[0092] Figure 5 This is a schematic diagram of the process of determining the third activation value derivative mask tensor provided in this specification.

[0093] Combination Figure 5 It can be seen that when the above specified activation function is the GELU activation function, the definition of the GELU activation function can refer to the following formula:

[0094]

[0095] In the above formula, That is the output value of the GELU activation function. From the above formula, it can be seen that when the input x is much greater than zero, the output value of the GELU activation function approaches the linear function y=x, and the derivative of the activation value of the calculated input x approaches 1. When the input x is less than 0, the output value of the GELU activation function approaches 0, and the derivative of the activation value of the calculated input x is a negative number close to 0.

[0096] Based on this, in practical application scenarios, thresholding can be used to approximate the derivative of the GELU activation function, that is, Figure 5In the example, it is assumed that the derivative of the activation value of the calculated input x has four values, namely, 0, 0.2, 0.6, and 1. At this time, the output values ​​of each neuron node in the calculated hidden layer can be used to calculate The approximate derivative of , as a tensor of activation value derivative masks.

[0097] In addition, in order to further reduce memory access overhead, during the forward propagation process, for each network layer of the model to be trained, after calculating the activation value derivative mask tensor corresponding to the hidden layer in the network layer, the server can also perform bit compression operations on the activation value derivative mask tensor to obtain a compressed activation value derivative mask tensor. Then, based on the compressed activation value derivative mask tensor and the fused gradient, the gradient of the loss function relative to each neuron node of the network layer can be determined as the target gradient.

[0098] The bit compression operation is used to compress the activation value derivative mask tensor, and the uncompressed operation may be, for example, a bitwise shift, a bitwise OR operation, or the like.

[0099] For ease of understanding, the above compression operation is described in detail below by taking the above activation value derivative mask tensor as the activation value derivative mask tensor calculated by the ReLU activation function as an example.

[0100] Specifically, it can be seen from the above content that each value contained in the activation value derivative mask tensor calculated by the ReLU activation function is 0 or 1. For example, the calculated activation value derivative mask tensor is [1, 0, 1, 1, 0, 0, 1, 1]. At this time, each binary bit can be processed from left to right, the result is first initialized to 0, and then each bit is left shifted so that the value of each new bit enters the target position in sequence. Finally, the current bit can be added to the result using a bitwise OR operation, so that the activation value derivative mask tensor can be converted into a byte for storage, that is, 0b10110011, and then each ReLU6 activation value derivative mask tensor can be reduced from the traditional 32-bit floating point storage to 1-bit storage, reducing the memory usage by 32 times.

[0101] In addition, when the above-mentioned specified activation value function is the GELU activation function, it can be seen from the above content that each value contained in the above-mentioned activation value derivative mask tensor can have four types. At this time, when the activation value derivative mask tensor is bit compressed, the four values ​​can be represented by two binary bits, for example: 00, 01, 10, 11, and then bit compressed. For example, if the activation value derivative mask tensor is [1, 0.2, 0.6, 1], it can be converted to [11, 01, 10, 11]. At this time, each binary bit is processed from left to right by the above method. The result is first initialized to 0, and each two bits are left shifted so that the value of each new bit enters the target position in sequence. Then, the current bit can be added to the result using a bitwise OR operation, so that the activation value derivative mask tensor can be converted to a byte for storage, that is, 0b11011011. Each GELU mask can be reduced from the traditional 32-bit floating point storage to 2-bit storage, reducing the memory usage by 16 times.

[0102] S103: According to the group fusion activation value, determine the gradient of the preset loss function relative to each neuron node group of the network layer as the fusion gradient.

[0103] S104: Determine the gradient of the loss function relative to each neuron node of the network layer according to the activation value derivative mask tensor and the fused gradient as each target gradient.

[0104] S105: adjusting the weight parameters of the network layer according to the target gradients to obtain a trained model, and performing the task according to the trained model.

[0105] In this specification, during the forward propagation process of training the model to be trained, after determining the fusion values ​​of each group, the server can input the determined group fusion values ​​into the next network layer and store the group fusion values ​​and the activation value derivative mask tensor of the network layer. Then, during the back propagation process, the group fusion activation value can be read from the memory, and based on the group fusion activation value, the gradient of the preset loss function relative to each neuron node group of the network layer (i.e., the gradient of the loss function relative to the group fusion layer) can be determined as the fusion gradient.

[0106] Furthermore, the server can read the activation value derivative mask tensor of the network layer from the memory to determine the gradient of the loss function relative to each neuron node of the network layer according to the activation value derivative mask tensor and the fused gradient as each target gradient.

[0107] Specifically, the server can determine the target dimensionality reduction multiple based on the number of neuron nodes contained in each neuron node group, and thus can determine the gradient of the loss function relative to each neuron node of the network layer based on the activation value derivative mask tensor, the fusion gradient and the target dimensionality reduction multiple as the target gradients, and then can adjust the weight parameters of the network layer according to the target gradients to obtain the trained model, and then can perform business execution according to the trained model.

[0108] It can be seen from the above method that the server can group and fuse the activation values ​​output by each network layer of the model to be trained during the forward propagation process of training the model to be trained, and can perform bit compression on the activation value derivative mask tensor of each network layer, thereby significantly reducing the memory usage during the training of the model to be trained, thereby improving the training efficiency of the model to be trained and reducing the training cost of the model to be trained.

[0109] For ease of understanding, the following describes in detail the process of executing business on the model trained by the above model training method. Figure 6 shown.

[0110] Figure 6 A flowchart of a service execution method provided in this specification includes the following steps.

[0111] S601: Acquire target business data, where the target business data includes at least one of text data, image data, video data, and audio data.

[0112] In this specification, the server can execute business through the business model trained by the above-mentioned model training method. Before that, the server can first obtain the data required to be processed when executing the business through the business model as the target business data.

[0113] The above-mentioned target business data can be determined according to the business scenario actually corresponding to the business model, and the above-mentioned target business data can be at least one of text data, image data, audio data, and video data.

[0114] S602: Input the target business data into a pre-trained business model to obtain a business execution result according to the target business data through the business model, wherein the business model is trained by the above-mentioned model training method.

[0115] S603: Execute the service according to the service execution result.

[0116] Furthermore, after acquiring the target business data, the server can input the target business data into the business model trained by the above-mentioned model training method, and then process the target business data through the business model to obtain the business execution result.

[0117] For example, if the above-mentioned business model is an image classification model, the business execution result here is the image classification result output by the business model for the target business data.

[0118] For another example: if the above-mentioned business model is an intelligent customer service model, the business execution result here is the reply text data output by the business model for the target business data.

[0119] It can be seen from the above content that the server can execute corresponding services by using the service model trained by the above model training method.

[0120] The above are one or more implementation model training methods of this specification. Based on the same idea, this specification also provides corresponding model training and business execution devices, such as Figure 7 , Figure 8 shown.

[0121] Figure 7 A schematic diagram of a model training device provided in this specification includes:

[0122] An acquisition module 701 is used to acquire a model to be trained and training sample data, wherein the training sample data is at least one of text data, image data, video data, and audio data;

[0123] A forward propagation module 702 is used to input the training sample data into the model to be trained, so as to determine the groups of neuron nodes contained in the network layer according to the preset grouping relationship between the neuron nodes for each network layer of the model to be trained, and to determine the fusion activation value of each group according to the activation value output by the neuron nodes contained in each neuron node group, and to determine the activation value derivative mask tensor of the network layer according to the activation value output by each neuron node contained in the network layer, wherein the activation value derivative mask tensor is used to reflect the importance of each neuron node in the gradient calculation;

[0124] A first back propagation module 703 is used to determine the gradient of a preset loss function relative to each neuron node group of the network layer according to the group fusion activation value as a fusion gradient;

[0125] A second back propagation module 704 is used to determine the gradient of the loss function relative to each neuron node of the network layer as each target gradient according to the activation value derivative mask tensor and the fused gradient;

[0126] The execution module 705 is used to adjust the weight parameters of the network layer according to the target gradients to obtain a trained model, and perform tasks according to the trained model.

[0127] Optionally, the forward propagation module 702 is specifically used to determine the activation value derivative mask tensor of the network layer according to the activation value output by each neuron node contained in the network layer, and perform a bit compression operation on the activation value derivative mask tensor to obtain a compressed activation value derivative mask tensor;

[0128] The second back propagation module 704 is specifically used to determine the gradient of the loss function relative to each neuron node of the network layer as each target gradient based on the compressed activation value derivative mask tensor and the fused gradient.

[0129] Optionally, the forward propagation module 702 is specifically used to input the training sample data into the model to be trained, so as to determine the output value of each neuron node contained in each network layer of the model to be trained, and determine the activation value of each neuron node contained in the network layer according to a preset specified activation function and the output value, divide the neuron nodes contained in the network layer into neuron node groups, and determine the fusion activation value of each group according to the activation value output by the neuron nodes contained in each neuron node group.

[0130] Optionally, the second back propagation module 704 is specifically used to determine the target dimensionality reduction multiple based on the number of neuron nodes contained in each neuron node group; and determine the gradient of the loss function relative to each neuron node of the network layer as each target gradient based on the activation value derivative mask tensor, the fusion gradient and the target dimensionality reduction multiple.

[0131] Figure 8 A schematic diagram of a service execution device provided in this specification includes:

[0132] The acquisition module 801 is used to acquire target business data, where the target business data includes at least one of text data, image data, video data, and audio data;

[0133] The data processing module 802 is used to input the target business data into a pre-trained business model to obtain a business execution result according to the target business data through the business model, and the business model is trained by the above-mentioned model training method;

[0134] The execution module 803 is used to execute the service according to the service execution result.

[0135] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A model training method provided.

[0136] This manual also provides Fig. 9 The one shown corresponds to Figure 1 A schematic diagram of the electronic device. Fig. 9 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The model training method described. Of course, in addition to the software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0137] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvements to the circuit structure of the graphics processing unit (GPU), diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements in the method flow today can be regarded as direct improvements in the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0138] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0139] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0140] For the convenience of description, the above device is described by dividing it into various units according to its functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0141] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0145] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0146] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0147] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0148] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0149] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0151] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0152] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A model training method, characterized in that: include: Acquire a model to be trained and training sample data, wherein the training sample data is at least one of text data, image data, video data, and audio data; Input the training sample data into the model to be trained, so as to determine the groups of neuron nodes contained in the network layer according to the preset grouping relationship between the neuron nodes for each network layer of the model to be trained, and determine the fusion activation value of each group according to the activation value output by the neuron nodes contained in each neuron node group, and determine the activation value derivative mask tensor of the network layer according to the activation value output by each neuron node contained in the network layer, wherein the activation value derivative mask tensor is used to reflect the importance of each neuron node in the gradient calculation; According to the group fusion activation value, determine the gradient of the preset loss function relative to each neuron node group of the network layer as the fusion gradient; Determine, according to the activation value derivative mask tensor and the fused gradient, the gradient of the loss function relative to each neuron node of the network layer as each target gradient; The weight parameters of the network layer are adjusted according to the target gradients to obtain a trained model, and business execution is performed according to the trained model.

2. The method according to claim 1, characterized in that According to the activation value output by each neuron node included in the network layer, determining the activation value derivative mask tensor of the network layer specifically includes: Determine an activation value derivative mask tensor of the network layer according to the activation value output by each neuron node included in the network layer, and perform a bit compression operation on the activation value derivative mask tensor to obtain a compressed activation value derivative mask tensor; According to the activation value derivative mask tensor and the fused gradient, the gradient of the loss function relative to each neuron node of the network layer is determined as each target gradient, specifically including: According to the compressed activation value derivative mask tensor and the fused gradient, the gradient of the loss function relative to each neuron node of the network layer is determined as each target gradient.

3. The method according to claim 1, characterized in that The training sample data is input into the model to be trained, so as to determine the groups of neuron nodes contained in the network layer according to the preset grouping relationship between the neuron nodes for each network layer of the model to be trained, and to determine the fusion activation value of each group according to the activation value output by the neuron nodes contained in each neuron node group, and to determine the activation value derivative mask tensor of the network layer according to the activation value output by each neuron node contained in the network layer, specifically including: The training sample data is input into the model to be trained to determine the output value of each neuron node contained in each network layer of the model to be trained, and the activation value of each neuron node contained in the network layer is determined according to a preset specified activation function and the output value, and the neuron node groups contained in the network layer are determined according to the preset grouping relationship between the neuron nodes, and the fusion activation value of each group is determined according to the activation values ​​output by the neuron nodes contained in each neuron node group.

4. The method according to claim 1, characterized in that According to the activation value derivative mask tensor and the fused gradient, the gradient of the loss function relative to each neuron node of the network layer is determined as each target gradient, specifically including: Determine the target dimension reduction multiple according to the number of neuron nodes contained in each neuron node group; According to the activation value derivative mask tensor, the fused gradient and the target dimension reduction multiple, the gradient of the loss function relative to each neuron node of the network layer is determined as each target gradient.

5. A service execution method, characterized in that: include: Acquire target business data, where the target business data includes: at least one of text data, image data, video data, and audio data; Inputting the target business data into a pre-trained business model to obtain a business execution result according to the target business data through the business model, wherein the business model is trained by the method according to any one of claims 1 to 4 above; The business is executed according to the business execution result.

6. A model training device, characterized in that: include: An acquisition module, used to acquire a model to be trained and training sample data, wherein the training sample data is at least one of text data, image data, video data, and audio data; A forward propagation module, used for inputting the training sample data into the model to be trained, so as to determine the groups of neuron nodes contained in each network layer of the model to be trained according to the preset grouping relationship between the neuron nodes, and to determine the fusion activation value of each group according to the activation value output by the neuron nodes contained in each neuron node group, and to determine the activation value derivative mask tensor of the network layer according to the activation value output by each neuron node contained in the network layer, wherein the activation value derivative mask tensor is used to reflect the importance of each neuron node in the gradient calculation; A first back propagation module is used to determine, according to the group fusion activation value, the gradient of a preset loss function relative to each neuron node group of the network layer as a fusion gradient; A second back propagation module is used to determine the gradient of the loss function relative to each neuron node of the network layer as each target gradient according to the activation value derivative mask tensor and the fused gradient; The execution module is used to adjust the weight parameters of the network layer according to the target gradients to obtain a trained model, and perform tasks according to the trained model.

7. The device according to claim 6, characterized in that The forward propagation module is specifically used to determine the activation value derivative mask tensor of the network layer according to the activation value output by each neuron node contained in the network layer, and perform bit compression operation on the activation value derivative mask tensor to obtain a compressed activation value derivative mask tensor; according to the compressed activation value derivative mask tensor and the fused gradient, determine the gradient of the loss function relative to each neuron node of the network layer as each target gradient.

8. A service execution device, characterized in that: include: An acquisition module, used to acquire target business data, wherein the target business data includes: at least one of text data, image data, video data, and audio data; A data processing module, used for inputting the target business data into a pre-trained business model, so as to obtain a business execution result according to the target business data through the business model, wherein the business model is trained by the method according to any one of claims 1 to 5 above; An execution module is used to execute the service according to the service execution result.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 5 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 5 is implemented.

Images