Hybrid-precision-based brain-inspired spiking neural network model design method and device, and computer equipment

Abstract

The present disclosure provides a kind of hybrid precision-based brain-like pulse neural network model design method, device and computer equipment.The object classification model is a pulse neural network model, and the training method of the object classification model includes: inputting the current batch training sample into the object classification model to obtain the prediction result of the current batch training sample;Based on the prediction result of the current batch training sample and the training label determined in advance, the total loss of the object classification model is calculated;Based on the total loss, the model parameters of the object classification model are updated;Wherein, the model parameters include: weight parameter, data quantization parameter related to data quantization step, data bit width parameter related to model size and calculation amount.

Description

Technical Field

[0001] The present disclosure generally relates to the field of spiking neural network technology, and more specifically, to a method, apparatus, and computer device for designing a brain-like spiking neural network model based on mixed precision. Background Art

[0002] Traditional artificial neural network (ANN) models suffer from low computational efficiency and poor biological interpretability, lacking the ability to draw inspiration from biological intelligence. Spiking neural networks (SNNs), a third-generation neural network, achieve low-power information transmission and processing through a biomimetic spike activation representation. To achieve higher model accuracy, new burst spiking neural network models introduce multi-bit spike representations within a single spike timestep. Furthermore, spiking neural networks typically use only 32-bit or 16-bit full-precision weight parameter representations. This makes model compression and edge device migration of spiking neural network models difficult, contradicting the low-power intelligent technology characteristics of spiking neural networks. Therefore, developing efficient spiking neural network models with low data bit rates has become a key research priority. However, existing technologies fail to effectively balance model accuracy, speed, and storage and computational overhead. Summary of the Invention

[0003] An exemplary embodiment of the present disclosure is to provide a method, apparatus, and computer device for designing a brain-like pulse neural network model based on mixed precision, which can improve the accuracy and classification speed of the object classification model and reduce the model storage overhead and computing overhead.

[0004] According to a first aspect of an embodiment of the present disclosure, a training method for an object classification model is provided, wherein the object classification model is a pulse neural network model, wherein the training method includes: inputting a current batch of training samples into the object classification model to obtain a prediction result of the current batch of training samples; calculating the total loss of the object classification model based on the prediction result of the current batch of training samples and predetermined training labels; and updating the model parameters of the object classification model based on the total loss; wherein the model parameters include: a weight parameter, a data quantization parameter related to the data quantization step size, and a data bit width parameter related to the model size and computational complexity.

[0005] Optionally, the data quantization parameters include: membrane potential emission threshold and weight parameter quantization step; the data bit width parameters include: pulse bit width, weight parameter bit width, and pulse sequence length.

[0006] Optionally, the step of calculating the total loss of the object classification model based on the prediction results of the current batch of training samples and the predetermined training labels includes: calculating a first loss based on the prediction results of the current batch of training samples and the training labels; calculating a second loss based on the average bit width of each weight parameter and a preset target weight bit width; calculating a third loss based on the average pulse sequence length of each feature element and a preset target pulse sequence length; calculating a fourth loss based on the average pulse bit width of each feature element and a preset target pulse bit width; and calculating the total loss of the object classification model based on the first loss, the second loss, the third loss and the fourth loss.

[0007] Optionally, the step of updating the model parameters of the object classification model based on the total loss includes: calculating a first gradient scaling coefficient based on the pulse bit width of the current layer, and updating the pulse bit width of the current layer based on the total loss and the first gradient scaling coefficient; calculating a second gradient scaling coefficient based on the weight parameter bit width of the current layer, and updating the weight parameter bit width of the current layer based on the total loss and the second gradient scaling coefficient; calculating a third gradient scaling coefficient based on the pulse bit width of the current layer, and updating the membrane potential emission threshold of the current layer based on the total loss and the third gradient scaling coefficient; calculating a fourth gradient scaling coefficient based on the weight parameter bit width of the current layer, and updating the weight parameter quantization step of the current layer based on the total loss and the fourth gradient scaling coefficient; updating the pulse sequence length of the current layer based on the total loss.

[0008] Optionally, the step of updating the pulse bit width of the current layer based on the total loss and the first gradient scaling coefficient includes: calculating the gradient of the first loss term with respect to the pulse bit width of the current layer based on the first loss term in the total loss and the first gradient scaling coefficient, wherein the first loss term is the first loss; calculating the gradient of the fourth loss term in the total loss with respect to the pulse bit width of the current layer, wherein the fourth loss term is the product of the fourth loss and its corresponding penalty coefficient; taking the sum of the gradient of the first loss term with respect to the pulse bit width of the current layer and the gradient of the fourth loss term with respect to the pulse bit width of the current layer as the gradient of the pulse bit width of the current layer; and updating the pulse bit width of the current layer based on the gradient of the pulse bit width of the current layer.

[0009] Optionally, the step of updating the weight parameter bit width of the current layer based on the total loss and the second gradient scaling coefficient includes: calculating the gradient of the first loss term with respect to the weight parameter bit width of the current layer based on the first loss term in the total loss and the second gradient scaling coefficient, wherein the first loss term is the first loss; calculating the gradient of the second loss term in the total loss with respect to the weight parameter bit width of the current layer, wherein the second loss term is the product of the second loss and its corresponding penalty coefficient; taking the sum of the gradient of the first loss term with respect to the weight parameter bit width of the current layer and the gradient of the second loss term with respect to the weight parameter bit width of the current layer as the gradient of the weight parameter bit width of the current layer; and updating the weight parameter bit width of the current layer based on the gradient of the weight parameter bit width of the current layer.

[0010] Optionally, the step of updating the membrane potential emission threshold of the current layer based on the total loss and the third gradient scaling coefficient includes: calculating the gradient of the first loss term with respect to the membrane potential emission threshold of the current layer based on the first loss term in the total loss and the third gradient scaling coefficient, wherein the first loss term is the first loss; using the gradient of the first loss term with respect to the membrane potential emission threshold of the current layer as the gradient of the membrane potential emission threshold of the current layer; and updating the membrane potential emission threshold of the current layer based on the gradient of the membrane potential emission threshold of the current layer.

[0011] Optionally, the step of updating the weight parameter quantization step of the current layer based on the total loss and the fourth gradient scaling coefficient includes: calculating the gradient of the first loss term with respect to the weight parameter quantization step of the current layer based on the first loss term in the total loss and the fourth gradient scaling coefficient, wherein the first loss term is the first loss; using the gradient of the first loss term with respect to the weight parameter quantization step of the current layer as the gradient of the weight parameter quantization step of the current layer; and updating the weight parameter quantization step of the current layer based on the gradient of the weight parameter quantization step of the current layer.

[0012] Optionally, the step of updating the pulse sequence length of the current layer based on the total loss includes: calculating the gradient of the first loss term in the total loss with respect to the pulse sequence length of the current layer, wherein the first loss term is the first loss; calculating the gradient of the third loss term in the total loss with respect to the pulse sequence length of the current layer, wherein the third loss term is the product of the third loss and its corresponding penalty coefficient; taking the sum of the gradient of the first loss term with respect to the pulse sequence length of the current layer and the gradient of the third loss term with respect to the pulse sequence length of the current layer as the gradient of the pulse sequence length of the current layer; and updating the pulse sequence length of the current layer based on the gradient of the pulse sequence length of the current layer.

[0013] Optionally, in the step of inputting the current batch of training samples into the object classification model to obtain the prediction results of the current batch of training samples, for each layer of the object classification model, if it is monitored that the integer value corresponding to the data bit width parameter to be used for the current batch of training samples is inconsistent with the integer value corresponding to the data bit width parameter used for the previous batch of training samples, then the data quantization parameter that can minimize the data quantization error is calculated and used for the current batch of training samples.

[0014] According to a second aspect of an embodiment of the present disclosure, an object classification method is provided, comprising: obtaining a sample to be classified; inputting the sample to be classified into an object classification model to obtain a classification result; wherein the object classification model is trained by executing the training method as described above.

[0015] According to a third aspect of an embodiment of the present disclosure, a training device for an object classification model is provided, wherein the object classification model is a pulse neural network model, wherein the training device includes: a first prediction unit, configured to input a current batch of training samples into the object classification model to obtain a prediction result of the current batch of training samples; a loss calculation unit, configured to calculate the total loss of the object classification model based on the prediction result of the current batch of training samples and predetermined training labels; a parameter updating unit, configured to update the model parameters of the object classification model based on the total loss; wherein the model parameters include: a weight parameter, a data quantization parameter related to the data quantization step size, and a data bit width parameter related to the model size and computational complexity.

[0016] Optionally, the data quantization parameters include: membrane potential emission threshold and weight parameter quantization step; the data bit width parameters include: pulse bit width, weight parameter bit width, and pulse sequence length.

[0017] Optionally, the loss calculation unit is configured to: calculate a first loss based on the prediction results of the current batch of training samples and the training labels; calculate a second loss based on the average bit width of each weight parameter and a preset target weight bit width; calculate a third loss based on the average pulse sequence length of each feature element and a preset target pulse sequence length; calculate a fourth loss based on the average pulse bit width of each feature element and a preset target pulse bit width; and calculate the total loss of the object classification model based on the first loss, the second loss, the third loss and the fourth loss.

[0018] Optionally, the parameter update unit is configured to: calculate a first gradient scaling coefficient based on the pulse bit width of the current layer, and update the pulse bit width of the current layer based on the total loss and the first gradient scaling coefficient; calculate a second gradient scaling coefficient based on the weight parameter bit width of the current layer, and update the weight parameter bit width of the current layer based on the total loss and the second gradient scaling coefficient; calculate a third gradient scaling coefficient based on the pulse bit width of the current layer, and update the membrane potential emission threshold of the current layer based on the total loss and the third gradient scaling coefficient; calculate a fourth gradient scaling coefficient based on the weight parameter bit width of the current layer, and update the weight parameter quantization step of the current layer based on the total loss and the fourth gradient scaling coefficient; and update the pulse sequence length of the current layer based on the total loss.

[0019] Optionally, the parameter updating unit is configured to: calculate the gradient of the first loss item with respect to the pulse bit width of the current layer based on the first loss item in the total loss and the first gradient scaling coefficient, wherein the first loss item is the first loss; calculate the gradient of the fourth loss item in the total loss with respect to the pulse bit width of the current layer, wherein the fourth loss item is the product of the fourth loss and its corresponding penalty coefficient; take the sum of the gradient of the first loss item with respect to the pulse bit width of the current layer and the gradient of the fourth loss item with respect to the pulse bit width of the current layer as the gradient of the pulse bit width of the current layer; and update the pulse bit width of the current layer based on the gradient of the pulse bit width of the current layer.

[0020] Optionally, the parameter update unit is configured to: calculate the gradient of the first loss term with respect to the weight parameter bit width of the current layer based on the first loss term in the total loss and the second gradient scaling coefficient, wherein the first loss term is the first loss; calculate the gradient of the second loss term in the total loss with respect to the weight parameter bit width of the current layer, wherein the second loss term is the product of the second loss and its corresponding penalty coefficient; take the sum of the gradient of the first loss term with respect to the weight parameter bit width of the current layer and the gradient of the second loss term with respect to the weight parameter bit width of the current layer as the gradient of the weight parameter bit width of the current layer; and update the weight parameter bit width of the current layer based on the gradient of the weight parameter bit width of the current layer.

[0021] Optionally, the parameter update unit is configured to: calculate the gradient of the first loss term with respect to the membrane potential emission threshold of the current layer based on the first loss term in the total loss and the third gradient scaling coefficient, wherein the first loss term is the first loss; use the gradient of the first loss term with respect to the membrane potential emission threshold of the current layer as the gradient of the membrane potential emission threshold of the current layer; and update the membrane potential emission threshold of the current layer based on the gradient of the membrane potential emission threshold of the current layer.

[0022] Optionally, the parameter update unit is configured to: calculate the gradient of the first loss term with respect to the weight parameter quantization step of the current layer based on the first loss term in the total loss and the fourth gradient scaling coefficient, wherein the first loss term is the first loss; use the gradient of the first loss term with respect to the weight parameter quantization step of the current layer as the gradient of the weight parameter quantization step of the current layer; and update the weight parameter quantization step of the current layer based on the gradient of the weight parameter quantization step of the current layer.

[0023] Optionally, the parameter updating unit is configured to: calculate the gradient of the first loss term in the total loss with respect to the pulse sequence length of the current layer, wherein the first loss term is the first loss; calculate the gradient of the third loss term in the total loss with respect to the pulse sequence length of the current layer, wherein the third loss term is the product of the third loss and its corresponding penalty coefficient; take the sum of the gradient of the first loss term with respect to the pulse sequence length of the current layer and the gradient of the third loss term with respect to the pulse sequence length of the current layer as the gradient of the pulse sequence length of the current layer; and update the pulse sequence length of the current layer based on the gradient of the pulse sequence length of the current layer.

[0024] Optionally, the parameter updating unit is further configured to: for each layer of the object classification model, if it is monitored that the integer value corresponding to the data bit width parameter to be used for the current batch of training samples is inconsistent with the integer value corresponding to the data bit width parameter used for the previous batch of training samples, then calculate and use the data quantization parameter that can minimize the data quantization error for the current batch of training samples.

[0025] According to a fourth aspect of an embodiment of the present disclosure, an object classification device is provided, comprising: a sample acquisition unit configured to acquire samples to be classified; a second prediction unit configured to input the samples to be classified into an object classification model to obtain a classification result; wherein the object classification model is trained by executing the training method as described above.

[0026] According to a fifth aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided, which, when executed by at least one processor, enables the at least one processor to execute the object classification model training method and / or object classification method as described above.

[0027] According to a sixth aspect of an embodiment of the present disclosure, a computer device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the object classification model training method and / or object classification method as described above.

[0028] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer-executable instructions, which, when executed by at least one processor, implement the object classification model training method and / or object classification method as described above.

[0029] Compared with the prior art that only optimizes the weight parameters of the model based on model loss, the brain-like pulse neural network model design method, device and computer equipment based on mixed precision according to the exemplary embodiments of the present disclosure, in addition to optimizing the weight parameters of the model based on model loss, also optimizes the data quantization parameters and data bit width parameters based on model loss, that is, the data quantization parameters and data bit width parameters are set as learnable parameters rather than variables with fixed values, which can be optimized through gradient optimization, thereby improving the accuracy and classification speed of the object classification model, and reducing the model storage overhead and computing overhead.

[0030] In the following description, some aspects and / or advantages of the general inventive concept of the present disclosure will be set forth, and some aspects and / or advantages will be known through the following description or implementation of the general inventive concept of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] These and / or other aspects and advantages of the present application will become more clear and easier to understand from the following detailed description of the embodiments of the present application in conjunction with the accompanying drawings, in which:

[0032] Figure 1 A flowchart illustrating a method for training an object classification model according to an exemplary embodiment of the present disclosure;

[0033] Figure 2 A flowchart illustrating a method for calculating the total loss of an object classification model according to an exemplary embodiment of the present disclosure;

[0034] Figure 3 A flowchart illustrating a method for updating model parameters of an object classification model based on a total loss according to an exemplary embodiment of the present disclosure;

[0035] Figure 4 A flowchart illustrating a method for monitoring data bit width parameters and updating data quantization parameters according to an exemplary embodiment of the present disclosure is provided;

[0036] Figure 5 An example of a training method of an object classification model according to an exemplary embodiment of the present disclosure is shown;

[0037] Figure 6 A flowchart illustrating an object classification method according to an exemplary embodiment of the present disclosure;

[0038] Figure 7 A structural block diagram illustrating a training apparatus for an object classification model according to an exemplary embodiment of the present disclosure is shown;

[0039] Figure 8 A structural block diagram showing an object classification device according to an exemplary embodiment of the present disclosure is shown;

[0040] Figure 9 A structural block diagram of a computer device according to an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0041] Reference will now be made in detail to the embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like parts throughout. The embodiments are described below with reference to the drawings in order to explain the present disclosure.

[0042] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0043] It should be noted that the phrase "at least one of the several items" in this disclosure includes three types of parallel situations: "any one of the several items", "a combination of any multiple of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. For another example, "performing at least one of step 1 and step 2" means the following three parallel situations: (1) performing step 1; (2) performing step 2; and (3) performing both step 1 and step 2.

[0044] Compared with traditional artificial neural networks, spiking neural networks have an additional time dimension, and the network as a whole has the ability to represent time series. The core difference between spiking neural networks and artificial neural networks lies in the use of spiking neurons. Artificial neural networks use nonlinear activation functions (such as ReLU activation function), and the input and output are all real numbers ( ). The spiking neural network uses a spiking neuron model, and the input is a real membrane potential sequence , the output is a binary pulse sequence Where T is the length of the pulse train, which is a positive integer. The spiking neuron model adopts the leaky-integrate-fire model (LIF), and its discrete mathematical expression is as follows:

[0045] (1)

[0046] in, Indicates the The membrane potential of the pulse neuron at the moment, the initial value is 0; and Represent the resting potential and membrane potential emission threshold respectively; τ is the pre-set membrane potential attenuation coefficient. The pulse neuron will input the real membrane potential sequence , converted into a binary pulse sequence As the activation value (or eigenvalue) output. Therefore, the binary activation value will obtain lower hardware computing overhead when participating in the matrix multiplication operator in the deep learning model, such as convolution calculation, linear calculation, etc. After the pulse sequence of the current layer passes through the matrix multiplication operator, it becomes the input membrane potential sequence of the pulse neurons in the next layer, that is, .in, l Represents the layer number of the model, w represents the model weight parameter, and j represents the number of the spiking neuron in the current layer connected to the spiking neuron in the next layer.

[0047] However, such a binary pulse sequence will greatly weaken the information representation ability of the model feature map, thereby reducing the model accuracy. By introducing the burst pulse emission mechanism, a pulse neuron model with multiple pulses in a single step (so that more pulses can be present in a single time step) is constructed. That is, a single-step pulse is represented by more bits, which can make the pulse neural network have stronger information transmission ability in a single-step pulse. Specifically, formula (1) can be rewritten as formula (2):

[0048] (2)

[0049] Among them, the floor(x) function will only retain the integer part of the input value x as the function output. At this time, the output of the pulse neuron becomes a multi-bit pulse sequence, that is, .in, Indicates the pulse data bit width in a single time step.

[0050] Based on the above content, this disclosure further proposes a high energy efficiency pulse neural network model design method based on mixed precision. Figures 1 to 9 Provide detailed explanation.

[0051] Figure 1 A flowchart of a method for training an object classification model according to an exemplary embodiment of the present disclosure is shown. The object classification model is a spiking neural network model, specifically, for example, a brain-like spiking neural network model based on mixed precision.

[0052] As an exemplary embodiment, the object classification model can be used to classify images or videos. Accordingly, the input of the object classification model can be image data or video data, and the output of the object classification model can be an image classification result or a video classification result. For example, the video data can be a multimedia stream such as a bitstream or an event stream.

[0053] Reference Figure 1 In step S101, the current batch of training samples is input into the object classification model to obtain the prediction results of the current batch of training samples.

[0054] In step S102, total loss of the object classification model is calculated based on the prediction result of the current batch of training samples and the predetermined training label.

[0055] It should be understood that the prediction result of the current batch of training samples is the classification result of the current batch of training samples by the object classification model, and the training label of the current batch of training samples is the real classification result (for example, the classification result of artificial classification) of the current batch of training samples.

[0056] In step S103, the model parameters of the object classification model are updated based on the total loss of the object classification model.

[0057] The model parameters of the object classification model include not only the weight parameters but also the data quantization parameters related to the data quantization step, and the data bit width parameters related to the model size and calculation amount (i.e., the data bit width parameters related to the model storage and calculation overhead).

[0058] In the prior art, only the weight parameters of the model are optimized based on the model loss, while in the present disclosure, the data quantization parameters and the data bit width parameters are optimized based on the model loss in addition to the optimization of the weight parameters of the model based on the model loss, in other words, in the present disclosure, the data quantization parameters and the data bit width parameters are set as learnable parameters instead of fixed numerical variables, which can be optimized by gradient.

[0059] In fact, the present disclosure considers that by quantizing the weight parameters of the model , the size compression of the weight parameters of the model can be realized. The quantization formula is as follows:

[0060] (3)

[0061] wherein the round(x) function represents the rounding calculation of the input x; the function represents the maximum and minimum value truncation of the input x, the minimum value is a, and the maximum value is b; represents the quantization step of the weight parameters of the i-th layer; l represents the quantization bit width of the weight parameters of the i-th layer; represents the quantized value of the weight parameters of the i-th layer. l l In particular, when i = 1, the round(x) function will degenerate into the sign function sign(x), and the truncation function will not be needed, i.e., formula (3) becomes: . .

[0062] From formula (2), it can be seen that is actually the current pulse membrane potential​​​ One The quantization process, the quantization step is the membrane potential emission threshold , and the quantization bit width is , The quantized value is It can be concluded that the feature map data size of the multi-bit spike neural network is and The model size, hardware computing and storage load, and hardware computing amount of the multi-bit spiking neural network are determined by 、 、 Direct decision.

[0063] On the one hand, the present disclosure proposes , , The three data width integer variables are parameterized to obtain 、 and , thereby achieving the construction of a hybrid model and the controllable model storage and computational complexity. That is, the data bit width parameters that can be updated based on model loss may include: pulse bit width (i.e., the bit width of a pulse transmitted at a single moment), weight parameter width , pulse sequence length (i.e., the pulse time step).

[0064] Since the parameterized integer variables are actually represented by continuous values, such as 、 and Therefore, in the process of forward propagation of the model, it is necessary to 、 and Mapped to the integer domain by rounding 、 and Then use it. In the process of model back propagation, 、 and The gradient is obtained by, for example, straight-throughestimation (STE), thus achieving 、 and (i.e., 、 and ) can be learned.

[0065] As an exemplary embodiment, the parameterization is as follows (based on the model l layer For example):

[0066]

[0067] (4)

[0068] in, Representing variables The preset positive integer upper bound of ; express The learnable real number parameter of . Similarly, we can get l layer The parameterization process:

[0069]

[0070] (5)

[0071] in, Representing variables The preset positive integer upper bound of ; express The learnable real number parameter of . Similarly, we can get l layer The parameterization process:

[0072]

[0073] (6)

[0074] in, Representing variables The preset positive integer upper bound of ; express learnable real parameters of .

[0075] On the other hand, the present disclosure proposes a threshold for the emission of membrane potential and quantization step size These two variables are parameterized layer by layer. However, unlike the integer variable parameterization above, , It is a real number, so no additional parameterization calculation is required. That is, the data quantization parameters that can be updated based on the model loss may include: membrane potential emission threshold and weight parameter quantization step size .

[0076] In addition, in order to facilitate the subsequent gradient calculation design, by comparing formulas (2) and (7), on the one hand, the present disclosure proposes to move the membrane potential of the spiking neuron toward zero during the pulse emission phase. The offset of Indicates thel On the other hand, to address the problem that the gradient of parameter T cannot be calculated because it does not participate in the model calculation, the present disclosure proposes to average the pulse output and multiply it with the membrane potential emission threshold and weight parameter to form the membrane potential input of the next layer. This membrane potential input will be repeatedly input at each time step in the calculation of the spiking neurons in the next layer. Therefore, the multi-variable parameterized multi-bit spiking neuron model proposed in the present disclosure is as follows:

[0077] (7)

[0078] The pulse sequence space output by the pulse neuron model is , in this case, the multi-bit spiking neuron is equivalent to a linear output. When the spiking neuron needs to shield the negative domain of the input membrane potential sequence like a nonlinear activation function (such as the ReLU activation function), the last line of formula (7) can be transformed into:

[0079] (8)

[0080] In particular, when hour, or , which degenerates into a binary pulse representation, and the last line of formula (7) becomes: or .

[0081] Furthermore, as an exemplary embodiment, the introduced parameterized variables 、 、 、 and Sharing is done at the model layer level to avoid data format being too trivial. In other words, in the same model layer, there is only one ,one ,one ,one and a .

[0082] In addition, the present disclosure takes into account: 、 and In addition to continuously updating and optimizing based on the task loss gradients received during supervised model training, additional loss terms are also required to limit the size of the pulse bit width, model weight bit width, and pulse sequence length, thereby controlling the overall storage and computational overhead of the model. Figure 2 To describe an exemplary embodiment of step S102, which will not be expanded here.

[0083] In addition, the present disclosure also takes into account that during the training process of the mixed precision model, due to the data width (i.e. 、 and ) is parameterized, which results in serious data width jumps and data offsets during the training of the spiking neural network, thus affecting the effect of model training. Therefore, the present disclosure proposes that: in the process of using gradients to update parameters, the gradients of the parameters are scaled, and then the parameters are updated based on the scaled gradients. Figure 3 To describe an exemplary embodiment of step S103, which will not be expanded here.

[0084] Figure 2 A flow chart illustrating a method for calculating the total loss of an object classification model according to an exemplary embodiment of the present disclosure is shown. It should be understood that the present disclosure does not impose a fixed restriction on the order in which steps S201 to S204 are executed.

[0085] Reference Figure 2 ,In step S201, the first loss is calculated based on the prediction results and training labels of the current batch of training samples.

[0086] The first loss is also the mission loss .

[0087] In step S202 , a second loss is calculated based on the average bit width of each weight parameter and a preset target weight bit width.

[0088] As an exemplary embodiment, step S202 may include: collecting , and calculate the average bit width of each weight parameter , then The preset target weight width Perform difference two-norm calculation to obtain the supervisory signal with limited average bit width of weight parameter (i.e., the second loss).

[0089] In step S203 , a third loss is calculated based on the average pulse sequence length of each characteristic element and a preset target pulse sequence length.

[0090] As an exemplary embodiment, step S203 may include: and the number of characteristic elements, calculate the average pulse sequence length of each characteristic element , then With the preset target pulse sequence length Perform difference two norm calculation to obtain the supervisory signal of the average pulse sequence length of the limiting feature element (i.e., third loss).

[0091] In step S204, a fourth loss is calculated based on the average pulse bit width of each characteristic element and a preset target pulse bit width.

[0092] As an exemplary embodiment, step S204 may include: and the number of characteristic elements, calculate the average pulse bit width of each characteristic element , then The preset target pulse bit width Perform difference two-norm calculation to obtain the supervisory signal of the average pulse bit width of the restricted feature element (i.e., the fourth loss).

[0093] In step S205 , the total loss of the object classification model is calculated based on the first loss, the second loss, the third loss, and the fourth loss.

[0094] During supervised training, the loss signal is directly based on the relationship between model predictions and data labels. Optimizing the data width parameters using methods such as mean square error loss or cross entropy loss does not guarantee that these data width parameters can be compressed toward the target model data width. Therefore, this disclosure proposes to directly constrain the average data width of the model, that is, to propose the following loss calculation formula:

[0095] (9)

[0096] in, represents the total loss of the object classification model, , and is the penalty coefficient.

[0097] Figure 3 A flowchart of a method for updating model parameters of an object classification model based on total loss according to an exemplary embodiment of the present disclosure is shown. It should be understood that the present disclosure does not impose a fixed restriction on the order in which steps S301 to S305 are executed.

[0098] Reference Figure 3 In step S301, a first gradient scaling factor is calculated based on the pulse bit width of the current layer, and the pulse bit width of the current layer is updated based on the total loss and the first gradient scaling factor.

[0099] Specifically, based on the total loss and , and a first gradient scaling coefficient, calculate the gradient of the pulse bit width of the current layer, and update the pulse bit width of the current layer based on the gradient.

[0100] As an example, the first loss term in the total loss (i.e., ) and the first gradient scaling factor, calculate the gradient of the first loss term with respect to the pulse bit width of the current layer; calculate the fourth loss term in the total loss (ie, ) with respect to the gradient of the pulse bit width of the current layer; taking the sum of the gradient of the first loss term with respect to the pulse bit width of the current layer and the gradient of the fourth loss term with respect to the pulse bit width of the current layer as the gradient of the pulse bit width of the current layer; based on the gradient of the pulse bit width of the current layer, updating the pulse bit width of the current layer.

[0101] In step S302, a second gradient scaling coefficient is calculated based on the weight parameter bit width of the current layer, and the weight parameter bit width of the current layer is updated based on the total loss and the second gradient scaling coefficient.

[0102] Specifically, based on the total loss and , and a second gradient scaling coefficient, calculate the gradient of the weight parameter bit width of the current layer, and update the weight parameter bit width of the current layer based on the gradient.

[0103] As an example, the first loss term in the total loss (i.e., ) and the second gradient scaling factor, calculate the gradient of the first loss term with respect to the weight parameter bit width of the current layer; calculate the second loss term in the total loss (ie, ) with respect to the gradient of the weight parameter bit width of the current layer; taking the sum of the gradient of the first loss term with respect to the weight parameter bit width of the current layer and the gradient of the second loss term with respect to the weight parameter bit width of the current layer as the gradient of the weight parameter bit width of the current layer; based on the gradient of the weight parameter bit width of the current layer, updating the weight parameter bit width of the current layer.

[0104] In step S303, a third gradient scaling coefficient is calculated based on the pulse bit width of the current layer, and the membrane potential emission threshold of the current layer is updated based on the total loss and the third gradient scaling coefficient.

[0105] Specifically, based on the total loss and a third gradient scaling coefficient, calculate the gradient of the membrane potential emission threshold of the current layer, and update the membrane potential emission threshold of the current layer based on the gradient.

[0106] As an example, the first loss term in the total loss (i.e., ) and the third gradient scaling coefficient, calculate the gradient of the first loss term with respect to the membrane potential emission threshold of the current layer; use the gradient of the first loss term with respect to the membrane potential emission threshold of the current layer as the gradient of the membrane potential emission threshold of the current layer; based on the gradient of the membrane potential emission threshold of the current layer, update the membrane potential emission threshold of the current layer.

[0107] In step S304, a fourth gradient scaling coefficient is calculated based on the weight parameter bit width of the current layer, and the weight parameter quantization step size of the current layer is updated based on the total loss and the fourth gradient scaling coefficient.

[0108] Specifically, based on the total loss and the fourth gradient scaling coefficient, calculate the gradient of the quantization step of the weight parameter of the current layer, and update the quantization step of the weight parameter of the current layer based on the gradient.

[0109] As an example, the first loss term in the total loss (i.e., ) and the fourth gradient scaling coefficient, calculate the gradient of the first loss term with respect to the quantization step of the weight parameter of the current layer; use the gradient of the first loss term with respect to the quantization step of the weight parameter of the current layer as the gradient of the quantization step of the weight parameter of the current layer; based on the gradient of the quantization step of the weight parameter of the current layer, update the quantization step of the weight parameter of the current layer.

[0110] In step S305 , the pulse sequence length of the current layer is updated based on the total loss.

[0111] Specifically, based on the total loss and , calculate the gradient of the pulse sequence length of the current layer, and update the pulse sequence length of the current layer based on the gradient.

[0112] As an example, the first loss term in the total loss (i.e., ) with respect to the gradient of the spike train length of the current layer; calculate the third loss term in the total loss (i.e., ) with respect to the gradient of the pulse sequence length of the current layer; taking the sum of the gradient of the first loss term with respect to the pulse sequence length of the current layer and the gradient of the third loss term with respect to the pulse sequence length of the current layer as the gradient of the pulse sequence length of the current layer; updating the pulse sequence length of the current layer based on the gradient of the pulse sequence length of the current layer.

[0113] In fact, some model parameters in the above steps will encounter a series of non-differentiable problems in the process of gradient back propagation. The present disclosure proposes a parameter gradient calculation method by combining the chain rule gradient solution, the direct gradient estimation method and the gradient scaling method. As an example, the corresponding gradient scaling factor is calculated, and then the gradient scaling factor is multiplied to the gradient value of the corresponding parameter. As another example, the direct gradient estimation method can be taken to solve the problem of non-differentiable gradient of integer variable parameterization, that is:

[0114] (10)

[0115] The following will describe in detail the exemplary embodiments of calculating the gradient of the pulse bit width of the first l layer, the gradient of the weight parameter bit width of the first layer, the gradient of the membrane potential firing threshold of the first layer, the gradient of the weight parameter quantization step of the first layer, and the gradient of the pulse sequence length of the first layer.

[0116] Regarding the gradient of the pulse bit width of the first l layer, as a first example, when , i.e. and the pulse neuron is a linear mapping, Regarding the gradient of the pulse bit width of the first layer, as a first example, when , i.e. and the pulse neuron is a linear mapping,

[0117] (11)

[0118] wherein j represents the index of the feature map element (i.e., the feature element) of the first l layer at the t-th time step; sign(x) is a sign function; the minimum value of the quantized integer ; the maximum value of the quantized integer (i.e., the maximum integer value that can be expressed by ); and the first gradient scaling factor .

[0119] Regarding the gradient of the pulse bit width of the first l layer, as a second example, when , i.e. and the pulse neuron is a nonlinear mapping, Regarding the gradient of the pulse bit width of the first layer, as a first example, when , i.e. and the pulse neuron is a linear mapping,

[0120] (12)

[0121] Among them, j represents the l The index of the feature map element at the tth time step of the layer; sign(x) is the sign function; the minimum value of the quantized integer ; Quantized integer maximum value ; First gradient scaling factor .

[0122] About l Pulse bit width of the layer The gradient of , as a third example, is ,Right now And when the spiking neuron is a linear mapping, about The gradient calculation formula can be:

[0123] (13)

[0124] Among them, j represents the l The index of the feature map element at the tth time step of the layer; sign(x) is the sign function; the minimum value of the quantized integer ; Quantized integer maximum value ; First gradient scaling factor .

[0125] About l Pulse bit width of the layer The gradient of , as a fourth example, in ,Right now And when the spiking neuron is a nonlinear mapping, about The gradient calculation formula can be:

[0126] (14)

[0127] Among them, j represents the l The index of the feature map element at the tth time step of the layer; sign(x) is the sign function; the minimum value of the quantized integer ; Quantized integer maximum value ; First gradient scaling factor .

[0128] About l Layer weight parameter bit width The gradient of, as a first example, hour, about The gradient calculation formula can be:

[0129] (15)

[0130] Among them, i represents the l The index of the weight parameter of the layer; sign(x) is the sign function; the minimum value of the quantized integer ; Quantized integer maximum value ; Second gradient scaling factor .

[0131] About l Layer weight parameter bit width The gradient of , as a second example, is hour, about The gradient calculation formula can be:

[0132] (16)

[0133] Among them, i represents the l The index of the layer weight parameter; sign(x) is the sign function; the minimum value of the quantized integer ; Quantized integer maximum value ; Second gradient scaling factor .

[0134] About l The membrane potential firing threshold of the layer The gradient of, as a first example, or hour, about The gradient calculation formula can be:

[0135] (17)

[0136] Among them, j represents the l The index number of the feature map element at the t-th time step of the layer; the minimum value of the quantized integer ; Quantized integer maximum value ; The third gradient scaling factor .

[0137] About l The membrane potential firing threshold of the layer The gradient of , as a second example, is or hour, about The gradient calculation formula can be:

[0138] (18)

[0139] Among them, j represents the l The index number of the feature map element at the t-th time step of the layer; the minimum value of the quantized integer ; Quantized integer maximum value ; The third gradient scaling factor .

[0140] About l Quantization step size of the layer's weight parameters The gradient of , as an example, about The gradient calculation formula can be:

[0141] (19)

[0142] Among them, i represents the l The index number of the layer weight parameter; when When quantized integer minimum , quantized integer maximum value ; Otherwise, quantize the integer minimum value , quantized integer maximum value ; Fourth gradient scaling factor .

[0143] Based on the above exemplary embodiments, it can be seen that the first gradient scaling coefficient and the third gradient scaling coefficient are equal, and the second gradient scaling coefficient and the fourth gradient scaling coefficient are equal.

[0144] About l Pulse train length of the layer The gradient of , as an example, About parameters Gradient According to formula (7), The calculation of can be done directly by the gradient automatic solver based on the chain rule, and there is no non-differentiable point. Therefore, the parameter The gradient of can be obtained normally and no special discussion is required.

[0145] In addition, considering the existence of non-differentiable floor(x) and clamp(x) functions in formula (7), the present disclosure proposes to use a direct gradient estimation method to calculate ,Right now:

[0146] (20)

[0147] Among them, the maximum and minimum values ​​of the quantized integers The definition of l The membrane potential firing threshold of the layer The maximum and minimum values ​​of the quantized integers in the first and second examples of the gradient The definition is consistent.

[0148] It should be understood that The gradient of about The gradient and about The sum of the gradients of The gradient of about The gradient and about The sum of the gradients of The gradient of about The gradient and about The sum of the gradients of .

[0149] In addition, it can also be based on the first loss in the total loss of the object classification model , update the weight parameters of the object classification model. The specific method of this part is relatively common and will not be described in detail.

[0150] Figure 4 A flow chart illustrating a method for monitoring data width parameters and updating data quantization parameters according to an exemplary embodiment of the present disclosure is shown.

[0151] As an exemplary embodiment, step S401 may be performed during the execution of step S101 .

[0152] Reference Figure 4 In step S401, for each layer of the object classification model, if it is monitored that the integer value corresponding to the data bit width parameter to be used for the current batch of training samples is inconsistent with the integer value corresponding to the data bit width parameter used for the previous batch of training samples, then the data quantization parameter that can minimize the data quantization error is calculated and used for the current batch of training samples.

[0153] Specifically, the multi-bit pulse neural network will use each batch of training samples to update the parameters once during the training process, and in the process of forward propagation of the model, it is necessary to 、 and Mapping to integers by rounding 、 and (i.e., the integer value corresponding to the data width parameter). Therefore, when the data width updated based on the previous batch of training samples undergoes an integer mutation, there will be a data width in the forward process of the current batch of training samples. (like and ) and quantization step size (like and ). Specifically, when After mutation, the original Will fail, so you need to use the mutation right To update; when After mutation, the original Will fail, so you need to use the mutation right to update.

[0154] To solve the above problems, the present invention designs a method for the integer form of the data bit width parameter (i.e. and ) of the monitor (i.e., Figure 5 The data width monitor in the training process of each batch of training samples (batch) tracks the integer value of the data width parameter of each layer of the model. When it detects that the value has changed from the training of the previous batch of training samples, the monitor triggers the data quantization parameter and This ensures that the distribution of the model's pulse activation values ​​and the distribution of the quantized model weight parameters during the training of the current batch of training samples remain consistent with those during the training of the previous batch of training samples, alleviating the mismatch between the bit width and the quantization step size caused by the change in the parameter bit width.

[0155] As an exemplary embodiment, step S401 may include:

[0156] Step S1: The data width monitor first records the initial data width , and then observe the forward propagation process of each batch of training samples. When it is found that the current batch of training samples will be used (i.e., based on the updated training samples of the previous batch ) and the one used for the previous batch of training samples If different, it will be updated Record and trigger the quantization step update mechanism at the same time.

[0157] Step S2: The data width monitor also records the quantized data The effective maximum value and the effective minimum The initial values ​​of these two are "negative infinity" and "positive infinity" respectively. When the quantization step update mechanism is triggered, the maximum value of the quantized data of the current batch of training samples will be read first. and minimum value .for In terms of quantified data for ,for Then .

[0158] Step S3: According to and , we can calculate a relatively accurate quantization step size:

[0159] (twenty one)

[0160] In order to reduce the quantization error more accurately, the grid search method can be used to and Make further confirmation.

[0161] Step S4: First calculate the range of quantized data of the current batch of training samples Then, at the kth iteration of the grid search method, Reset to , where K is the total number of iterations of the grid search method. Reset to But if the quantized data are all in the non-negative domain, Keep it at 0.

[0162] Step S5: According to formula (21), calculate the quantization step size in the current grid search iteration Then follow , quantize the quantized data and get , and calculate the quantization error Among them, the mean(x) function represents the element-by-element average, and pow represents the power size.

[0163] Step S6: The one with the smallest quantization error and As the output of the grid search algorithm, the effective maximum and minimum values ​​are then updated:

[0164] (twenty two)

[0165] Among them, the functions min(a,b) and max(a,b) represent the minimum and maximum values ​​of a and b respectively.

[0166] Step S7: Based on the updated effective maximum and effective minimum values, calculate the correct Overwrite the previous value.

[0167] Furthermore, as an exemplary embodiment, to reduce the significant time consumption associated with the grid search method during model training and to ensure the priority of quantization step size optimization through gradient optimization, the bit width parameter monitoring and quantization step size update mechanism proposed in this disclosure can be limited to phases of model training where data bit width changes significantly. For example, when the difference between the average data bit width and a preset target data bit width falls below a certain threshold, the bit width parameter monitoring and quantization step size update mechanism is automatically disabled.

[0168] According to exemplary embodiments of the present disclosure, it is proposed to parameterize variables related to various quantization step sizes of a multi-bit spiking neural network and optimize learning through gradients. Furthermore, for integer bit-width variables related to model size and computational complexity, not only are these parameters parameterized, but their gradients are also optimized through gradient scaling and direct gradient estimation to alleviate their optimization difficulties.

[0169] Furthermore, it is proposed to construct a data bit width monitor during the training process of the parameterized multi-bit pulse model. The monitor compares the integer values ​​of the data bit width between the previous and next batches of training samples. When the data bit width undergoes an integer mutation, the mismatch between the quantization step size and the data bit width increases sharply. If the optimization is performed solely through gradient, the effect will be very poor. Therefore, when the monitor detects an integer mutation in the data bit width, it will adaptively recalculate and use the quantization step size that minimizes the current quantization error, thereby avoiding the increase in the error in the calculation results of the current batch of training samples caused by the integer mutation in the data bit width.

[0170] Furthermore, once the model's data width parameters are learnable, a specific supervisory signal is proposed to ensure that the learned data width remains within a specific range. Specifically, the present disclosure calculates the difference between the average data width and the preset target data width and then calculates the second norm to obtain an additional loss signal for the data width.

[0171] Figure 6 A flowchart illustrating an object classification method according to an exemplary embodiment of the present disclosure is shown.

[0172] Reference Figure 6 , in step S601, obtain samples to be classified.

[0173] In step S602, the sample to be classified is input into the object classification model to obtain a classification result, wherein the object classification model is trained by executing the training method described in the above exemplary embodiment.

[0174] Figure 7 A structural block diagram of a training device for an object classification model according to an exemplary embodiment of the present disclosure is shown. The object classification model is a spiking neural network model.

[0175] Reference Figure 7According to an exemplary embodiment of the present disclosure, the training device of the object classification model includes: a first prediction unit 701, a loss calculation unit 702, and a parameter updating unit 703.

[0176] Specifically, the first prediction unit 701 is configured to input the current batch of training samples into the object classification model to obtain prediction results of the current batch of training samples.

[0177] The loss calculation unit 702 is configured to calculate the total loss of the object classification model based on the prediction results of the current batch of training samples and the predetermined training labels.

[0178] The parameter updating unit 703 is configured to update the model parameters of the object classification model based on the total loss.

[0179] In addition to the weight parameters, the model parameters also include data quantization parameters related to the data quantization step size and data bit width parameters related to the model size and calculation amount.

[0180] As an exemplary embodiment, the data quantization parameter may include but is not limited to at least one of the following items: a membrane potential emission threshold and a weight parameter quantization step size.

[0181] As an exemplary embodiment, the data bit width parameter may include but is not limited to at least one of the following items: pulse bit width, weight parameter bit width, and pulse sequence length.

[0182] As an exemplary embodiment, the loss calculation unit 702 can be configured to: calculate a first loss based on the prediction results of the current batch of training samples and the training labels; calculate a second loss based on the average bit width of each weight parameter and a preset target weight bit width; calculate a third loss based on the average pulse sequence length of each feature element and a preset target pulse sequence length; calculate a fourth loss based on the average pulse bit width of each feature element and a preset target pulse bit width; calculate the total loss of the object classification model based on the first loss, the second loss, the third loss and the fourth loss.

[0183] As an exemplary embodiment, the parameter update unit 703 can be configured to: calculate a first gradient scaling coefficient based on the pulse bit width of the current layer, and update the pulse bit width of the current layer based on the total loss and the first gradient scaling coefficient; calculate a second gradient scaling coefficient based on the weight parameter bit width of the current layer, and update the weight parameter bit width of the current layer based on the total loss and the second gradient scaling coefficient; calculate a third gradient scaling coefficient based on the pulse bit width of the current layer, and update the membrane potential emission threshold of the current layer based on the total loss and the third gradient scaling coefficient; calculate a fourth gradient scaling coefficient based on the weight parameter bit width of the current layer, and update the weight parameter quantization step of the current layer based on the total loss and the fourth gradient scaling coefficient; and update the pulse sequence length of the current layer based on the total loss.

[0184] As an exemplary embodiment, the parameter updating unit 703 may be configured to: calculate the gradient of the first loss item with respect to the pulse bit width of the current layer based on the first loss item in the total loss and the first gradient scaling coefficient, wherein the first loss item is the first loss; calculate the gradient of the fourth loss item in the total loss with respect to the pulse bit width of the current layer, wherein the fourth loss item is the product of the fourth loss and its corresponding penalty coefficient; take the sum of the gradient of the first loss item with respect to the pulse bit width of the current layer and the gradient of the fourth loss item with respect to the pulse bit width of the current layer as the gradient of the pulse bit width of the current layer; and update the pulse bit width of the current layer based on the gradient of the pulse bit width of the current layer.

[0185] As an exemplary embodiment, the parameter update unit 703 can be configured to: calculate the gradient of the first loss term with respect to the weight parameter bit width of the current layer based on the first loss term in the total loss and the second gradient scaling coefficient, wherein the first loss term is the first loss; calculate the gradient of the second loss term in the total loss with respect to the weight parameter bit width of the current layer, wherein the second loss term is the product of the second loss and its corresponding penalty coefficient; take the sum of the gradient of the first loss term with respect to the weight parameter bit width of the current layer and the gradient of the second loss term with respect to the weight parameter bit width of the current layer as the gradient of the weight parameter bit width of the current layer; and update the weight parameter bit width of the current layer based on the gradient of the weight parameter bit width of the current layer.

[0186] As an exemplary embodiment, the parameter update unit 703 can be configured to: calculate the gradient of the first loss term with respect to the membrane potential emission threshold of the current layer based on the first loss term in the total loss and the third gradient scaling coefficient, wherein the first loss term is the first loss; use the gradient of the first loss term with respect to the membrane potential emission threshold of the current layer as the gradient of the membrane potential emission threshold of the current layer; and update the membrane potential emission threshold of the current layer based on the gradient of the membrane potential emission threshold of the current layer.

[0187] As an exemplary embodiment, the parameter update unit 703 can be configured to: calculate the gradient of the first loss term with respect to the weight parameter quantization step of the current layer based on the first loss term in the total loss and the fourth gradient scaling coefficient, wherein the first loss term is the first loss; use the gradient of the first loss term with respect to the weight parameter quantization step of the current layer as the gradient of the weight parameter quantization step of the current layer; and update the weight parameter quantization step of the current layer based on the gradient of the weight parameter quantization step of the current layer.

[0188] As an exemplary embodiment, the parameter updating unit 703 may be configured to: calculate the gradient of the first loss term in the total loss with respect to the pulse sequence length of the current layer, wherein the first loss term is the first loss; calculate the gradient of the third loss term in the total loss with respect to the pulse sequence length of the current layer, wherein the third loss term is the product of the third loss and its corresponding penalty coefficient; take the sum of the gradient of the first loss term with respect to the pulse sequence length of the current layer and the gradient of the third loss term with respect to the pulse sequence length of the current layer as the gradient of the pulse sequence length of the current layer; and update the pulse sequence length of the current layer based on the gradient of the pulse sequence length of the current layer.

[0189] As an exemplary embodiment, the parameter updating unit 703 can also be configured as follows: for each layer of the object classification model, if it is monitored that the integer value corresponding to the data bit width parameter to be used for the current batch of training samples is inconsistent with the integer value corresponding to the data bit width parameter used for the previous batch of training samples, then the data quantization parameter that can minimize the data quantization error is calculated and used for the current batch of training samples.

[0190] Figure 8 A structural block diagram of an object classification device according to an exemplary embodiment of the present disclosure is shown.

[0191] Reference Figure 8 , an object classification device according to an exemplary embodiment of the present disclosure: a sample acquisition unit 801 and a second prediction unit 802 .

[0192] In particular, the sample obtaining unit 801 is configured to obtain a sample to be classified.

[0193] The second prediction unit 802 is configured to input the sample to be classified into an object classification model to obtain a classification result. The object classification model is trained by performing the training method described in the above example embodiments.

[0194] It should be understood that the specific processes performed by the training apparatus and the object classification apparatus according to the example embodiments of the present disclosure have been described in detail above with reference to the above example embodiments, and details related thereto will not be repeated here. Figures 1 to 6

[0195] It should be understood that each unit and module in the training apparatus and the object classification apparatus according to the example embodiments of the present disclosure can be implemented by hardware components and / or software components.

[0196] Figure 9 A structural block diagram of a computer device according to an example embodiment of the present disclosure is shown.

[0197] Referring to Figure 9 , the computer device includes at least one memory 901 and at least one processor 902, and the at least one memory 901 stores a set of computer executable instructions, and when the set of computer executable instructions is executed by the at least one processor 902, the training method of the object classification model and / or the object classification method as described in the above example embodiments are performed.

[0198] As an example, the computer device can be a PC computer, a tablet device, a personal digital assistant, a smart phone, or other devices capable of executing the above instructions. Here, the computer device is not necessarily a single computer device, but can also be a collection of any devices or circuits capable of executing the above instructions (or instruction sets) alone or jointly. The computer device can also be part of an integrated control system or a system manager, or can be configured as a portable computer device that interfaces with local or remote (e.g., via wireless transmission) devices.

[0199] In the computer device, the processor 902 can include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a special-purpose processor system, a microcontroller, or a microprocessor. As an example and not limitation, the processor 902 can also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0200] ​The processor 902 can execute instructions or codes stored in the memory 901, wherein the memory 901 can also store data. Instructions and data can also be sent and received over the network via the network interface device, wherein the network interface device can use any known transmission protocol.

[0201] The memory 901 may be integrated with the processor 902, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. Furthermore, the memory 901 may comprise a separate device, such as an external disk drive, a storage array, or any other storage device usable by a database system. The memory 901 and the processor 902 may be operatively coupled or may communicate with each other, for example, via an I / O port, a network connection, or the like, such that the processor 902 can access files stored in the memory.

[0202] In addition, the computer device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the computer device may be connected to each other via a bus and / or a network.

[0203] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute the object classification model training method and / or object classification method as described in the above exemplary embodiments. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as a multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device configured to store a computer program and any associated data, data files and data structures in a non-transitory manner and provide the computer program and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be executed in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0204] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, and instructions in the computer program product may be executed by at least one processor to complete the object classification model training method and / or object classification method as described in the above exemplary embodiments.

[0205] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0206] It should be understood that the present disclosure is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training an object classification model executed by a computer device, characterized in that: The object classification model is a pulse neural network model, and the object classification model is used to classify images or videos. The input of the object classification model is: image data or video data, and the output of the object classification model is: image classification results or video classification results; Wherein, the training method includes: Inputting the current batch of training samples into the object classification model to obtain prediction results of the current batch of training samples; Calculating the total loss of the object classification model based on the prediction results of the current batch of training samples and the predetermined training labels; Based on the total loss, updating model parameters of the object classification model; The model parameters include: weight parameters, data quantization parameters related to data quantization step length, and data bit width parameters related to model size and computational complexity; the data quantization parameters include: membrane potential emission threshold and weight parameter quantization step length; the data bit width parameters include: pulse bit width, weight parameter bit width, and pulse sequence length; The step of calculating the total loss of the object classification model based on the prediction results of the current batch of training samples and the predetermined training labels includes: Calculating a first loss based on the prediction results of the current batch of training samples and the training labels; Calculate the second loss based on the average bit width of each weight parameter and the preset target weight bit width; calculating a third loss based on an average pulse sequence length of each characteristic element and a preset target pulse sequence length; Calculating a fourth loss based on an average pulse bit width of each characteristic element and a preset target pulse bit width; Calculating a total loss of the object classification model based on the first loss, the second loss, the third loss, and the fourth loss; The step of updating the model parameters of the object classification model based on the total loss includes: Calculating a first gradient scaling factor based on a pulse bit width of a current layer, and updating the pulse bit width of the current layer based on the total loss and the first gradient scaling factor; Calculating a second gradient scaling factor based on a weight parameter bit width of a current layer, and updating the weight parameter bit width of the current layer based on the total loss and the second gradient scaling factor; Calculating a third gradient scaling coefficient based on the pulse bit width of the current layer, and updating the membrane potential emission threshold of the current layer based on the total loss and the third gradient scaling coefficient; Calculating a fourth gradient scaling coefficient based on a weight parameter bit width of a current layer, and updating a quantization step size of the weight parameter of the current layer based on the total loss and the fourth gradient scaling coefficient; Based on the total loss, the pulse train length of the current layer is updated.

2. The training method according to claim 1, characterized in that The step of updating the pulse bit width of the current layer based on the total loss and the first gradient scaling factor includes: Calculating, based on a first loss term in the total loss and the first gradient scaling coefficient, a gradient of the first loss term with respect to the pulse bit width of the current layer, wherein the first loss term is the first loss; Calculating a gradient of a fourth loss term in the total loss with respect to a pulse bit width of the current layer, wherein the fourth loss term is a product of the fourth loss and a corresponding penalty coefficient; Taking the sum of the gradient of the first loss term with respect to the pulse bit width of the current layer and the gradient of the fourth loss term with respect to the pulse bit width of the current layer as the gradient of the pulse bit width of the current layer; Based on the gradient of the pulse bit width of the current layer, the pulse bit width of the current layer is updated.

3. The training method according to claim 1, characterized in that The step of updating the weight parameter bit width of the current layer based on the total loss and the second gradient scaling coefficient includes: Calculate, based on a first loss term in the total loss and the second gradient scaling coefficient, a gradient of the first loss term with respect to a weight parameter bit width of the current layer, wherein the first loss term is the first loss; Calculate the gradient of a second loss term in the total loss with respect to the bit width of a weight parameter of the current layer, wherein the second loss term is the product of the second loss and its corresponding penalty coefficient; Taking the sum of the gradient of the first loss term with respect to the weight parameter bit width of the current layer and the gradient of the second loss term with respect to the weight parameter bit width of the current layer as the gradient of the weight parameter bit width of the current layer; Based on the gradient of the weight parameter bit width of the current layer, the weight parameter bit width of the current layer is updated.

4. The training method according to claim 1, characterized in that The step of updating the membrane potential emission threshold of the current layer based on the total loss and the third gradient scaling coefficient includes: Calculating, based on a first loss term in the total loss and the third gradient scaling coefficient, a gradient of the first loss term with respect to the membrane potential emission threshold of the current layer, wherein the first loss term is the first loss; Taking the gradient of the first loss term with respect to the membrane potential emission threshold of the current layer as the gradient of the membrane potential emission threshold of the current layer; Based on the gradient of the membrane potential emission threshold of the current layer, the membrane potential emission threshold of the current layer is updated.

5. The training method according to claim 1, wherein: The step of updating the quantization step size of the weight parameter of the current layer based on the total loss and the fourth gradient scaling coefficient includes: Calculate, based on a first loss term in the total loss and the fourth gradient scaling coefficient, a gradient of the first loss term with respect to a quantization step of a weight parameter of the current layer, wherein the first loss term is the first loss; Taking the gradient of the first loss term with respect to the quantization step of the weight parameter of the current layer as the gradient of the quantization step of the weight parameter of the current layer; Based on the gradient of the quantization step of the weight parameter of the current layer, the quantization step of the weight parameter of the current layer is updated.

6. The training method according to claim 1, characterized in that The step of updating the pulse sequence length of the current layer based on the total loss includes: Calculating a gradient of a first loss term in the total loss with respect to a pulse sequence length of the current layer, wherein the first loss term is the first loss; Calculating a gradient of a third loss term in the total loss with respect to a pulse sequence length of the current layer, wherein the third loss term is a product of the third loss and a corresponding penalty coefficient; Taking the sum of the gradient of the first loss term with respect to the pulse sequence length of the current layer and the gradient of the third loss term with respect to the pulse sequence length of the current layer as the gradient of the pulse sequence length of the current layer; The pulse sequence length of the current layer is updated based on the gradient of the pulse sequence length of the current layer.

7. The training method according to claim 1, characterized in that In the step of inputting the current batch of training samples into the object classification model to obtain the prediction results of the current batch of training samples, For each layer of the object classification model, if it is monitored that the integer value corresponding to the data bit width parameter to be used for the current batch of training samples is inconsistent with the integer value corresponding to the data bit width parameter used for the previous batch of training samples, then the data quantization parameter that can minimize the data quantization error is calculated and used for the current batch of training samples.

8. A method for object classification performed by a computer device, characterized in that: include: Obtain samples to be classified; Inputting the sample to be classified into the object classification model to obtain a classification result; Wherein, the object classification model is trained by executing the training method according to any one of claims 1 to 7; The object classification model is used to classify images or videos. The input of the object classification model is image data or video data, and the output of the object classification model is image classification results or video classification results.

9. A training device for an object classification model, characterized in that: The object classification model is a pulse neural network model, and the object classification model is used to classify images or videos. The input of the object classification model is: image data or video data, and the output of the object classification model is: image classification results or video classification results; Wherein, the training device comprises: A first prediction unit is configured to input a current batch of training samples into the object classification model to obtain a prediction result of the current batch of training samples; a loss calculation unit configured to calculate a total loss of the object classification model based on the prediction results of the current batch of training samples and predetermined training labels; a parameter updating unit configured to update model parameters of the object classification model based on the total loss; The model parameters include: weight parameters, data quantization parameters related to data quantization step length, and data bit width parameters related to model size and computational complexity; the data quantization parameters include: membrane potential emission threshold and weight parameter quantization step length; the data bit width parameters include: pulse bit width, weight parameter bit width, and pulse sequence length; Wherein, the loss calculation unit is configured as follows: Calculating a first loss based on the prediction results of the current batch of training samples and the training labels; Calculate the second loss based on the average bit width of each weight parameter and the preset target weight bit width; calculating a third loss based on an average pulse sequence length of each characteristic element and a preset target pulse sequence length; Calculating a fourth loss based on an average pulse bit width of each characteristic element and a preset target pulse bit width; Calculating a total loss of the object classification model based on the first loss, the second loss, the third loss, and the fourth loss; Wherein, the parameter updating unit is configured as follows: Calculating a first gradient scaling factor based on a pulse bit width of a current layer, and updating the pulse bit width of the current layer based on the total loss and the first gradient scaling factor; Calculating a second gradient scaling factor based on a weight parameter bit width of a current layer, and updating the weight parameter bit width of the current layer based on the total loss and the second gradient scaling factor; Calculating a third gradient scaling coefficient based on the pulse bit width of the current layer, and updating the membrane potential emission threshold of the current layer based on the total loss and the third gradient scaling coefficient; Calculating a fourth gradient scaling coefficient based on a weight parameter bit width of a current layer, and updating a quantization step size of the weight parameter of the current layer based on the total loss and the fourth gradient scaling coefficient; Based on the total loss, the pulse train length of the current layer is updated.

10. The training device according to claim 9, characterized in that The parameter updating unit is configured to: Calculating, based on a first loss term in the total loss and the first gradient scaling coefficient, a gradient of the first loss term with respect to the pulse bit width of the current layer, wherein the first loss term is the first loss; Calculating a gradient of a fourth loss term in the total loss with respect to a pulse bit width of the current layer, wherein the fourth loss term is a product of the fourth loss and a corresponding penalty coefficient; Taking the sum of the gradient of the first loss term with respect to the pulse bit width of the current layer and the gradient of the fourth loss term with respect to the pulse bit width of the current layer as the gradient of the pulse bit width of the current layer; Based on the gradient of the pulse bit width of the current layer, the pulse bit width of the current layer is updated.

11. The training device according to claim 9, characterized in that The parameter updating unit is configured to: Calculate, based on a first loss term in the total loss and the second gradient scaling coefficient, a gradient of the first loss term with respect to a weight parameter bit width of the current layer, wherein the first loss term is the first loss; Calculate the gradient of a second loss term in the total loss with respect to the bit width of a weight parameter of the current layer, wherein the second loss term is the product of the second loss and its corresponding penalty coefficient; Taking the sum of the gradient of the first loss term with respect to the weight parameter bit width of the current layer and the gradient of the second loss term with respect to the weight parameter bit width of the current layer as the gradient of the weight parameter bit width of the current layer; Based on the gradient of the weight parameter bit width of the current layer, the weight parameter bit width of the current layer is updated.

12. The training device according to claim 9, characterized in that The parameter updating unit is configured to: Calculating, based on a first loss term in the total loss and the third gradient scaling coefficient, a gradient of the first loss term with respect to the membrane potential emission threshold of the current layer, wherein the first loss term is the first loss; Taking the gradient of the first loss term with respect to the membrane potential emission threshold of the current layer as the gradient of the membrane potential emission threshold of the current layer; Based on the gradient of the membrane potential emission threshold of the current layer, the membrane potential emission threshold of the current layer is updated.

13. The training device according to claim 9, characterized in that The parameter updating unit is configured to: Calculate, based on a first loss term in the total loss and the fourth gradient scaling coefficient, a gradient of the first loss term with respect to a quantization step of a weight parameter of the current layer, wherein the first loss term is the first loss; Taking the gradient of the first loss term with respect to the quantization step of the weight parameter of the current layer as the gradient of the quantization step of the weight parameter of the current layer; Based on the gradient of the quantization step of the weight parameter of the current layer, the quantization step of the weight parameter of the current layer is updated.

14. The training device according to claim 9, characterized in that The parameter updating unit is configured to: Calculating a gradient of a first loss term in the total loss with respect to a pulse sequence length of the current layer, wherein the first loss term is the first loss; Calculating a gradient of a third loss term in the total loss with respect to a pulse sequence length of the current layer, wherein the third loss term is a product of the third loss and a corresponding penalty coefficient; Taking the sum of the gradient of the first loss term with respect to the pulse sequence length of the current layer and the gradient of the third loss term with respect to the pulse sequence length of the current layer as the gradient of the pulse sequence length of the current layer; The pulse sequence length of the current layer is updated based on the gradient of the pulse sequence length of the current layer.

15. The training device according to claim 9, characterized in that The parameter updating unit is further configured to: For each layer of the object classification model, if it is monitored that the integer value corresponding to the data bit width parameter to be used for the current batch of training samples is inconsistent with the integer value corresponding to the data bit width parameter used for the previous batch of training samples, then the data quantization parameter that can minimize the data quantization error is calculated and used for the current batch of training samples.

16. An object classification device, characterized in that: include: A sample acquisition unit is configured to acquire samples to be classified; A second prediction unit is configured to input the sample to be classified into an object classification model to obtain a classification result; Wherein, the object classification model is trained by executing the training method according to any one of claims 1 to 7; The object classification model is used to classify images or videos. The input of the object classification model is image data or video data, and the output of the object classification model is image classification results or video classification results.

17. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by at least one processor, the at least one processor is enabled to execute the object classification model training method according to any one of claims 1 to 7 and / or the object classification method according to claim 8.

18. A computer device, characterized in that: The computer device comprises: at least one processor; at least one memory storing computer-executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the object classification model training method according to any one of claims 1 to 7 and / or the object classification method according to claim 8.

19. A computer program product comprising computer executable instructions, characterized in that When the computer-executable instructions are executed by at least one processor, the training method of the object classification model according to any one of claims 1 to 7 and / or the object classification method according to claim 8 are implemented.

Images