Learning device, learning method, image processing device, image processing method, and program

Abstract

To provide a learning device, a method, a program, and an image processing device, which achieve joint optimization of weight and quantization scale and significantly improve the trade-off between MAC and accuracy of a sparsely connected ΣΔ network.SOLUTION: A learning device for a neural network model consisting of multiple feature map layers, includes: an input unit configured to accept, as training data, input of time-series image data and correct answer data; and a calculation unit configured to learn a weight parameter and a quantization scale of the neural network model. The calculation unit obtains, for each corresponding pixel of a feature map obtained from image data at two times, a quantization scale range in which pixel values at the two times are quantized to the same value, and sets a center value of the quantization scale range as an optimal value of the quantization scale. Further, the calculation unit uses a loss function that includes an error between an output of the neural network model and the correct answer data, and a sum of differences between the quantization scale and the optimal value of each pixel, to learn the weight parameter and the quantization scale.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a technique for training a neural network model used in an image processing device. [Background technology]

[0002] Convolutional neural networks used in image recognition and other applications perform processing by repeatedly convolving input image data with a learned weight vector. In a convolutional neural network, the convolution operation is performed on the entire input data. For example, when processing video data, the convolution operation is performed on the entire image even if there is no change in part of the input image or if there is only a small change. However, performing calculations even when there is no change can be considered wasteful processing. In particular, there is a lot of wasteful processing with data that has a high frame rate, and as a result, a high output rate cannot be achieved.

[0003] In response to this, a neural network has been proposed that processes only the difference between time series data and the previous data, allowing for efficient calculations when there is little change (Non-Patent Document 1). This neural network uses the convolution operation w*x t the previously calculated value w*x t-1 and the convolution operation of the differential information w*(x t -x t-1 ) is a ΣΔ neuron model rewritten in

[0004] In a conventional dense neural network, a convolution operation must be performed every time an input is received, but in the ΣΔ neuron model, the calculation for a new input is w*Δx, so if Δx contains 0, the operation for that input can be skipped.

[0005] Neural networks are often used in edge devices such as autonomous vehicles and mobile phones to process spatiotemporal data such as video. In neural networks, processing data independently for each frame requires a massive amount of computation. However, it is known that data that is close in time is highly correlated, and the differences between each pixel in consecutive frames are very sparse. Approaches for efficiently processing spatiotemporal data are being investigated from both the perspectives of algorithms and processors.

[0006] Non-Patent Document 2 proposes a ΣΔ network for efficient video processing. It reformulates a quantized neural network into an equivalent quantized recurrent neural network. This network recursively updates states and outputs using sparse time differences. The sparse update mechanism reduces the number of multiply-accumulate (MAC) operations. This model has proven effective for various tasks, including classification, object detection, and pose estimation. It has also been shown to be applicable to asynchronous event data. ΣΔ networks can perform highly efficient inference, especially when mapped to processors that can exploit dynamic sparsity. These devices are becoming commonplace due to their efficiency and advances in CMOS technology. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Amirhossein Habibian et al. “Skip-Convolutions for Efficient Video Processing” CVPR2021 [Non-patent document 2] Peter O'Connor, Max Welling "Sigma delta quantized networks" arXiv preprint arXiv:1611.02024, 2016 Summary of the Invention [Problem to be solved by the invention]

[0008] The challenge here is how to train a ΣΔ network. Existing methods first train the ΣΔ network without quantizing it, and then optimize the quantization scale for testing after quantization. To optimize the quantization scale, a greedy algorithm or layer-by-layer stochastic gradient descent algorithm is then used.

[0009] Increasing the quantization scale can reduce MAC, but this comes at a trade-off with accuracy. Although joint optimization of weights and quantization scale by end-to-end (E2E) learning seems promising, existing methods for learning the quantization scale of quantized neural networks, such as least square quantization (LSQ), cannot learn the quantization scale of ΣΔ networks to reduce MAC.

[0010] In view of the above background, an object of the present invention is to significantly improve the trade-off between MAC and accuracy in sparsely connected ΣΔ networks by realizing simultaneous optimization of weights and quantization scales. [Means for solving the problem]

[0011] The learning device of the present invention is a learning device for a neural network model consisting of a plurality of layers of feature maps, and includes an input unit that receives input of time-series image data and correct answer data as teacher data, a calculation unit that learns weight parameters and quantization scales of the neural network model, and a storage unit that stores the weight parameters and quantization scales obtained by calculation in the calculation unit. The calculation unit determines, for each corresponding pixel in a feature map obtained from image data at two times, a range of quantization scales in which pixel values ​​at the two times are quantized to the same value, and sets the center value of the range of quantization scales as an optimal value s of the quantization scale. optThe error between the output of the neural network model and the correct data, and the optimal value s of the quantization scale and each pixel are opt The weight parameters and quantization scale are learned using a loss function that includes the sum of the differences between

[0012] In the learning device of the present invention, the calculation unit selects the smallest quantization scale range from among a plurality of ranges, and sets the center value of the range as the optimum quantization scale value s opt It may also be possible to use the following.

[0013] In the learning device of the present invention, the calculation unit calculates a quantization scale and an optimal value s opt The difference may be calculated using the l0 norm.

[0014] In the learning device of the present invention, the quantization scale may be common to each layer or each channel.

[0015] In the learning device of the present invention, the calculation unit calculates the quantization scale and the optimal value s opt The weight parameters and quantization scale may be learned using a loss function obtained by multiplying the difference between

[0016] The image processing device of the present invention includes a memory unit that stores weight parameters and quantization scales of a neural network model trained by the learning device described above, an input unit that accepts input of time-series image data, and a calculation unit including a plurality of event-based processors that perform calculations when the data is input, wherein the calculation unit is configured to calculate a difference between the time-series image data and a frame of a previous time, quantize the difference using the quantization scale read from the memory unit, and calculate a feature map for the current time based on the feature map for the previous time and a value obtained by multiplying the quantized difference by the weight parameters read from the memory unit.

[0017] Moreover, an image processing device of the present invention includes a memory unit that stores weight parameters, a quantization scale, and a connection map of a neural network model trained by the above-described learning device; an input unit that accepts input of time-series image data; and a calculation unit including a plurality of event-based processors that perform calculations when the data is input, wherein the calculation unit multiplies the connection mask and weight parameters read from the memory unit to generate a pruned neural network model, calculates a difference from the previous time for the time-series image data, quantizes the difference using the quantization scale read from the memory unit, and calculates a feature map for the current time based on the feature map for the previous time and a value obtained by multiplying the quantized difference by the weight parameter.

[0018] The learning method of the present invention is a learning method for a neural network model consisting of a plurality of layers of feature maps, and includes the steps of receiving input of time-series image data and correct answer data as training data, applying the time-series image data to a neural network model to be learned to perform inference and obtain output data, and during the inference process, determining a range of quantization scales in which pixel values ​​at two times are quantized to the same value for each corresponding pixel in a feature map obtained from image data at two times, and setting the center value of the range of the quantization scale as an optimal value s of the quantization scale. opt and determining the error between the output data of the neural network model and the correct data, the quantization scale and the optimal value s of each pixel as opt and a step of learning the weight parameters and the quantization scale using a loss function including the sum of the differences between

[0019] The program of the present invention is a program for training a neural network model consisting of a plurality of layers of feature maps, and includes the steps of: receiving input of time-series image data and correct answer data as training data in a computer; applying the time-series image data to a neural network model to be trained to perform inference and obtain output data; determining, in the inference process, a range of quantization scales in which pixel values ​​at two times are quantized to the same value for each corresponding pixel in a feature map obtained from image data at two times; and setting the center value of the range of the quantization scale as an optimal value s of the quantization scale. opt and determining the error between the output data of the neural network model and the correct data, the quantization scale and the optimal value s of each pixel as opt and a step of learning the weight parameters and the quantization scale using a loss function including the sum of the differences between the [Brief explanation of the drawings]

[0020] [Figure 1] (a) A diagram showing the correspondence between the dense neuron state and the neuron state using the difference from the previous time. (b) A diagram showing the quantized formula of the formula shown in Figure 1(a). [Figure 2] FIG. 10 is a diagram showing changes in the values ​​to which the feature amount xt at time t and the feature amount xt+1 at time t+1 are quantized when the quantization scale is increased. [Figure 3] This figure shows the range in which the feature xt and the feature xt+1 are in the same step when the quantization scale s is increased from s=sLB. [Figure 4] FIG. 1 is a diagram illustrating a configuration of a learning device according to an embodiment. [Figure 5] 10 is a flowchart illustrating processing performed by the learning device according to the embodiment. [Figure 6] 1 is a diagram illustrating a configuration of an image processing device according to an embodiment. [Figure 7] FIG. 1 is a diagram showing the configuration of an SoC (System-on-a-chip) on which the neural network of the present embodiment is implemented. [Figure 8] 1A is a diagram showing a detailed configuration of a neuron core, and FIG. 1B is a diagram showing a conceptual diagram of the processing executed by the neuron core. DETAILED DESCRIPTION OF THE INVENTION

[0021] An image processing device and a neural network model generating device used in the image processing device according to an embodiment of the present invention will be described below with reference to the drawings. (First embodiment) [overview] Figure 1(a) shows the correspondence between the state of a dense neuron and the state of a neuron using the difference from the previous time. The dense neuron shown on the left side of Figure 1(a) performs a convolution operation w* on an input I to generate an internal state X t This is a neuron that calculates (Internal State). The input I to the neuron is the output of the previous layer neuron, which is the activation of the previous layer neuron. σ denotes the activation function.

[0022] The time-difference neuron shown on the right side of Figure 1(a) has an internal state X at time t. t The internal state X of the previous time t-1 t-1 The difference w*ΔI t More specifically, the difference ΔI between the input at time t-1 and the input at time t t The value obtained by performing the convolution operation on the internal state X at time t-1 t-1 By adding to the internal state X at time t t In Figure 1, ΔI t Although the subscript "(l)" indicating the layer is added to the right of the above, it is omitted in the specification.

[0023] In a temporal difference neuron, if there is no difference between the input I at time t-1 and time t (i.e., ΔI t =0), then the internal state X at time t t is the internal state X at time t-1 t-1When an event-based processor is used, no calculation is required if there is no difference input. Therefore, the difference ΔI t There is a demand to make it as sparse as possible.

[0024] Figure 1(b) is a quantized version of the equation shown in Figure 1(a). The equation for calculating the internal state X is the same as in Figure 1(a), but the difference ΔI t The equation for finding the input I at time t-1 is quantized. t-1 and input I at time t t (=σ(X t )) are each quantized to a discrete value, so the difference will often be 0. The larger the quantization scale s, the higher the probability that the difference will be 0, making it possible to reduce the amount of calculations. If we only consider the quantization scale s, it would be sufficient to increase the quantization scale s, but increasing the quantization scale s means that more data will be lost due to quantization. In light of the above background, the subject of the present invention is how to learn the quantization scale.

[0025] If the feature quantity at time t-1 and the feature quantity at time t are quantized to the same value, the difference ΔI t becomes 0. In the present invention, learning is performed so that more pixels are quantized to the same value.

[0026] Figure 2 shows the feature x at time t when the quantization scale is increased. t and the feature x at time t+1 t+1 1 is a diagram showing the change in the quantized value of the feature quantity x at time t. t and the feature x at time t+1 t+1 are quantized to the same value, the quantization scale s is t+1 -x t It is necessary that |x t+1 -x t |=s LB In Figure 2, the quantization scale s=s LB The graph shows how the quantization scale is gradually increased starting from .

[0027] In the example shown in Figure 2, s = s LB When t and feature x t+1 Since the values ​​of x and s are different due to quantization, the difference between them will not be 0. As the quantization scale s is increased, the feature x t and feature x t+1 will become the same value after quantization (will be on the same step). If the quantization scale s is further increased, the feature x t and feature x t+1 The quantized value of is then taken at different steps, then at the same step, and so on, until finally, the feature x t and feature x t+1 The quantized values ​​are all 0 and are on the same step. However, if the quantized data is 0, it means that there is no feature, which is meaningless.

[0028] Figure 3 shows the quantization scale s as s = s LB When increasing from t and feature x t+1 This figure shows the range in which the feature x t and feature x t+1 This means that the feature x t and feature x t+1 There are multiple ranges found where quantizes to the same value.

[0029] Feature x t and feature x t+1 When the quantization scale is at its lower limit, the feature x t and feature x t+1 are on the same step, but the smaller feature is at the edge of that step. Conversely, when the quantization scale is at the upper limit of the range, the feature x t and feature x t+1Although the feature quantities x and x2 are on the same step, the larger feature quantity is located at the edge of that step. When the quantization scale is at the upper or lower limit of the range in which the quantization scale is on the same step, even a small change causes the quantized data to move to a different step, resulting in an unstable state. When the quantization scale is at the center value of the range in which the quantized data is on the same step, the quantized data is stable on the same step. The smaller the quantization scale, the higher the proportion of the feature quantities before quantization that are retained. Therefore, in this embodiment, the feature quantity x t and feature x t+1 The smallest range (in Figure 3, s) of the range in which the quantized data of LB The center value of this range is s opt Let quantization scale s be s opt Learn to get closer to this.

[0030] By the way, feature x t and feature x t+1 Depending on the value of x, there may be only a range where the quantized data is 0 in order to place the quantized data in the same step. For example, t =0.8, feature x t+1 =2.2, where the difference between two feature quantities (2.2 - 0.8 = 1.4) is greater than the smaller feature quantity (0.8). In this case, the range in which the quantized data is set to 0 has a lower limit but no upper limit, so the center of the range cannot be determined. In this case, the lower limit is used as the reference value, and s opt The lower limit is the feature x t , feature x t+1 The larger value of s is doubled. opt It may also be possible to use the following.

[0031] In addition, the feature x t and feature x t+1 is the value for each pixel in the feature map, so s opt is calculated for each pixel. In contrast, the quantization scale s is a parameter common to each layer or channel. The quantization scale s is calculated for all pixels in the feature map.opt The regular term L of the quantization scale s is calculated so that the difference between s In the following equation, s opt To indicate that is calculated for each pixel, an index i is added to the right.

number

[0032] In this embodiment, the regular term L used in the loss function s Let quantization scale s and optimal value s opt The L2 norm of the difference is used, but it is also possible to use the L0 norm or L1 norm of the difference.

[0033] (Learning device) 4 is a diagram showing the configuration of a learning device 10 according to an embodiment. Learning device 10 includes an input unit 11 that receives input of training data, a calculation unit 12 that uses the training data to learn a neural network model and a quantization scale, and a storage unit 13 that stores the neural network model and the quantization scale obtained by training.

[0034] The neural network model generated by the learning device 10 is a neural network model for executing a predetermined task, such as detecting vehicles, pedestrians, or signs from time-series image data. The neural network model consists of multiple layers, and a feature map is generated in each layer.

[0035] The neural network model of this embodiment has neurons for time difference, i.e., neurons that do not need to perform state update calculations when there is no difference between the input of each layer and the current time.

[0036] The input unit 11 receives a large number of time-series image data and task-correct answer data as training data. The calculation unit 12 uses the time-series image data and task-correct answer data to train a neural network model and a quantization scale. The specific processing performed by the calculation unit 12 will be described in the description of the operation of the learning device 10.

[0037] 5 is a flowchart showing the operation of learning device 10 according to the embodiment. Learning device 10 receives input of a large number of time-series image data and task-correct data as training data (S10). If the task is to detect a vehicle, the correct data is data that identifies the area of ​​the vehicle shown in the time-series data, and if the task is to detect a pedestrian, the correct data is data that identifies the area of ​​the pedestrian.

[0038] The learning device 10 applies the time-series image data to the neural network model to be learned, performs inference processing, and outputs result data for the task (S11). The learning device 10 inputs the time-series image data one frame at a time in chronological order, and obtains result data from each frame. As described above, the neural network model is a time-difference neural network model, and performs convolution on the difference data between the previous time and the current time, and adds it to the state at the previous time to calculate the state at the current time. The quantization scale s used here is also a learning target, and the quantization scale s uses an initial value or the value used up to that point.

[0039] The learning device 10 determines the optimal value s of the quantization scale based on the feature amount of each pixel of the feature map obtained in the inference process. opt The optimum value s is calculated (S12). opt The method for calculating s is as explained using FIG. 3. That is, the learning device 10 calculates the range of quantization scales in which the pixel values ​​at the two consecutive times are quantized to the same value for each corresponding pixel in the feature map obtained from image data at two consecutive times, and calculates the smallest quantization scale (s LB The center value of the range (close to s opt It is decided that:

[0040] Specifically, the learning device 10 determines the optimal value s of the quantization scale. opt is determined as follows: In the following formula, [ ] indicates rounding off.

number

[0041] The learning device 10 calculates the error L between the output data by inference and the correct answer data. task , and the quantization scale s and the optimal value s for each pixel opt The sum of the errors L s The weight parameters and quantization scale s of the neural network model are learned using the following loss function, which includes (S13).

number

[0042] The following equation shows the derivative of the regularization term of the quantization scale s used during learning. Here, x t Here is the differential equation with respect to x t-1 The same is true for .

number

number

[0043] The learning device 10 determines whether to end learning (S14). The determination of whether to end learning is based on criteria such as whether processing has been performed on all time-series images or whether the update width of the learning parameters and quantization scale has become equal to or less than a predetermined threshold.

[0044] If it is determined not to end the learning, the process returns to the inference process (S11) to perform further learning. If it is determined to end the learning, the learning device 10 stores the neural network model and quantization scale data obtained by the learning in the storage unit 13 (S15).

[0045] Although the learning device 10 of this embodiment has been described, a program that realizes the processing of the calculation unit 12 of the above embodiment is also included in the scope of the present invention.

[0046] (Image processing device) 6 is a diagram showing the configuration of an image processing device 20 that performs image processing using the neural network model and quantization scale generated by the above-described learning device 10. The neural network model and quantization scale generated by the learning device 10 are stored in a storage unit 24.

[0047] The image processing device 20 includes an input unit 21 that inputs time-series image data to be processed, a calculation unit 22 that processes the time-series image data using a model including a plurality of neurons, and an output unit 23 that outputs the calculation results by the calculation unit 22. The time-series image data is video data or image data captured with continuous shooting. A specific application example of the image processing device 20 is the processing of video data captured by a vehicle, and the image processing device 20 performs object detection processing to detect vehicles, people, signs, etc. by image processing.

[0048] The calculation unit 22 is a data-driven calculation device that performs calculation processing when data is input, and a graph processor or a data flow processor is used. Note that the calculation device is not limited to these, and an FPGA or a CPU can also be used, but it is preferable that the calculation device is a device that can perform large-scale parallel processing.

[0049] Fig. 7 is a diagram showing an example of the configuration of an SoC (System-on-a-chip) on which the neural network of this embodiment is implemented. As shown in Fig. 7, the SoC is configured by arranging neuron cores that constitute each layer of the neural network.

[0050] Fig. 8(a) shows the detailed configuration of each neuron core, and Fig. 8(b) shows the conceptual diagram of the process executed by the neuron core in Fig. 8(a). Each neuron core receives an event from the previous layer, updates the membrane potential by multiplying the received event by a weight stored in SRAM, and fires when the difference from the membrane potential at the time of the previous firing exceeds a threshold.

[0051] As shown in Figure 8(a), the SRAM storing the weights is located within the neuron core, so all processing is performed within the processor without accessing external memory, which is advantageous from the viewpoint of power consumption.

[0052] The learning device 10 of the first embodiment optimizes the neural network model and the quantization scale using a loss function that minimizes the weight parameters and quantization scale of the neural network model, and can generate a neural network model that can make the difference from the previous time sparse using the optimal quantization scale. Therefore, by performing image processing using the neural network model and quantization scale generated by the learning device 10, image processing can be performed with a reduced computational load.

[0053] (Second embodiment) The learning device of the second embodiment simultaneously learns the weight parameters and quantization scales of a neural network model that has a connection mask. First, the connection mask will be explained. The connection mask represents a parameter used for pruning the neural network, and takes on two values: 0 or 1. By taking the Hadamard product of the connection mask and the weight parameters, when the mask value is 0, the product with the weight parameters becomes 0, so the connection disappears (they are pruned). When the mask value is 1, the weight parameters remain at their original values.

[0054] The following loss function is used to train a neural network model including masked connections:

number

[0055] The mask m has a value of 0 when the weight parameter w is less than γ and a value of 1 when it is greater than or equal to γ. S In this case, the quantization scale s and the optimal value s opt The Hadamard product of the error and the mask m is taken. Note that the quantization scale s and the optimal value s opt The function Ξ applied to the mask m is the quantization scale s and the optimal value s so that the Hadamard product with the mask m can be calculated. opt This function unfolds the difference of . Unfolding means returning the folded state for the convolution operation to a matrix that can be used for matrix multiplication. The unfolding operation is also known as "im2col" (image to column).

[0056] As shown in the above equation, the quantization scale s and the optimal value s opt The reason why we apply mask m to the error between s and s is that it does not matter what the quantization scale s is for branches whose weight is set to 0 by mask m. In other words, in learning the quantization scale s, the optimal value s for the part where the weight becomes 0 and the connection is lost is opt In order to avoid the influence of the , the product is taken with the combined mask.

[0057] In the case of a neural network model that includes mask connections, the optimal value s opt The weight parameter w and the quantization scale s can be optimized using

[0058] In this embodiment, an example has been given in which the threshold value γ that determines the mask value is fixed and the weight parameter w and quantization scale s are optimized, but it is also possible to introduce an independent parameter that determines whether or not to mask, and optimize the weight parameter w, quantization scale s, and mask parameter simultaneously.

[0059] Although the learning device of the present invention has been described in detail above using an embodiment, the present invention is not limited to the above embodiment. In this embodiment, an example of object detection processing has been described, but the task of the neural network model generated by the learning device is not limited to object detection processing. For example, the learning device and image processing device of the present invention can also be applied to image classification processing and pose estimation processing of objects appearing in images.

[0060] In this embodiment, an example of image processing of video captured by an in-vehicle camera has been given, but it can also be used to process event data captured by an event camera. An event camera processes a pixel only when the luminance of that pixel changes by more than a certain threshold value at that time t i and pixel coordinate x i , polarity ρ i Event e containing (-1 or +1) i ={t i ,x i ,ρ i An event camera is a camera that outputs data in the format:}. By accumulating events that occurred within a certain period of time in the past (for example, the past one second), the camera can process the images in the same way as normal video. [Explanation of symbols]

[0061] 10 Learning Device 11 Input section 12 Arithmetic section 13 Storage section 20 Image processing device 21 Input section 22 Arithmetic section 23 Output section 24 Memory section

Claims

1. A learning device for a neural network model consisting of a plurality of feature map layers, an input unit that accepts input of time-series image data and correct answer data as training data; a calculation unit that learns weight parameters and quantization scales of the neural network model; a storage unit for storing weighting parameters and quantization scales obtained by learning in the calculation unit; Equipped with The calculation unit For each corresponding pixel in the feature map obtained from the image data at the two times, a range of quantization scales is calculated in which the pixel values ​​at the two times are quantized to the same value; The center value of the range of the quantization scale is set as the optimum value s of the quantization scale. opt year, The error between the output of the neural network model and the correct data, and the optimum value s of each pixel and the quantization scale opt A learning device that learns weight parameters and quantization scales using a loss function that includes the sum of differences between

2. The calculation unit selects the smallest quantization scale range from among a plurality of ranges, and sets the center value of the range as the optimum quantization scale value s opt 2. The learning device according to claim 1, wherein:

3. The calculation unit calculates a quantization scale and an optimal value s in the loss function. opt The difference between ０ The learning device according to claim 1, wherein the calculation is performed by a norm.

4. The learning device according to claim 1 , wherein the quantization scale is common to each layer or each channel.

5. The calculation unit calculates the quantization scale and the optimum value s opt 2. The learning device according to claim 1, wherein the weight parameters and the quantization scale are learned using a loss function obtained by multiplying the difference between

6. a storage unit that stores weight parameters and quantization scales of a neural network model trained by the learning device according to any one of claims 1 to 4; an input unit that accepts input of time-series image data; a calculation unit including a plurality of event-based processors that perform calculations when data is input; Equipped with the calculation unit calculates a difference between the time-series image data and a previous frame, and quantizes the difference using the quantization scale read from the storage unit; The image processing device calculates a feature map for the current time based on a value obtained by multiplying the quantized difference by a weighting parameter read from the storage unit and the feature map for the previous time.

7. a storage unit that stores weight parameters, quantization scales, and connection maps of a neural network model trained by the learning device according to claim 5; an input unit that accepts input of time-series image data; a calculation unit including a plurality of event-based processors that perform calculations when data is input; Equipped with the calculation unit multiplies the connection mask read from the storage unit by a weight parameter to generate a pruned neural network model; calculating a difference between the time-series image data and a previous time, and quantizing the difference using the quantization scale read from the storage unit; an image processing device that calculates a feature map for a current time based on a value obtained by multiplying the quantized difference by the weight parameter and the feature map for a previous time;

8. A method for training a neural network model consisting of multiple layers of feature maps, comprising: A step of receiving input of time-series image data and correct answer data as training data; applying the time-series image data to a neural network model to be trained to perform inference and obtain output data; In the process of the inference, for each pixel corresponding to the feature map obtained from the image data at two times, a range of quantization scales in which the pixel values ​​at the two times are quantized to the same value is calculated, and the center value of the range of the quantization scales is set as the optimal value s of the quantization scale. opt and determining The error between the output data of the neural network model and the correct data, and the optimum value s of each pixel and the quantization scale opt A step of learning weight parameters and quantization scales using a loss function including a sum of differences between A learning method that includes:

9. A program for training a neural network model consisting of multiple layers of feature maps, the program comprising: A step of receiving input of time-series image data and correct answer data as training data; applying the time-series image data to a neural network model to be trained to perform inference and obtain output data; In the process of the inference, for each pixel corresponding to the feature map obtained from the image data at two times, a range of quantization scales in which the pixel values ​​at the two times are quantized to the same value is calculated, and the center value of the range of the quantization scales is set as the optimal value s of the quantization scale. opt and determining The error between the output data of the neural network model and the correct data, and the optimum value s of each pixel and the quantization scale opt A step of learning weight parameters and quantization scales using a loss function including a sum of differences between A program that executes the following.

Images