Information processing apparatus, information processing method, and program

Abstract

To solve the problem in which a quantization error is accumulated in each layer of a neural network and is finally increased to be output.SOLUTION: An information processing apparatus comprises: acquisition means that acquires second quantization information which includes information associated with quantization of a second network in the information processing apparatus that generates a deep layer learning model having a first network and the second network connected to the first network; determination means that generates first quantization information which is information associated with quantization of the first network on the basis of the second quantization information; and first quantization means in which first quantization based on the first quantization information is applied to the first network.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, neural networks have been applied to a variety of applications, including image processing, speech processing, and language processing. Since modern neural networks have many layers and require a large amount of calculations, high-speed computers are used during training. However, for inference after training, computing resources are often limited, so more efficient calculation methods are required.

[0003] A method of performing calculations during inference is known in which the data is quantized to lower precision values ​​than those used during learning. By quantizing parameters or output tensors, the system can be deployed and operated even on devices with limited computing resources, such as edge devices and embedded devices.

[0004] In order to obtain a quantized result model with a high execution speed, it is important to perform quantization based on information about the computer architecture of the device to which it is deployed. The technology described in Non-Patent Document 1 determines the number of bits after quantization to speed up calculations based on information about the computing cores and memory usage of the edge device to which it is deployed, and also optimizes the calculation schedule. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] 8-bit Inference with TensorRT Szymon Migacz, NVIDIA,May 8, 2017, [online], [Retrieved August 14, 2023], Internet, <URL: https: / / on-demand.gputechconf.com / gtc / 2017 / presentation / s7310-8-bit-inference-with-tensorrt.pdf> [Non-Patent Document 2] Matias Tassano and 2 others, “FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation”, [online], April 29, 2020, [Retrieved August 14, 2023], Internet,<URL:https: / / arxiv.org / pdf / 1907.01361.pdf> [Non-Patent Document 3] Markus Nagel and 5 others, “A White Paper on Neural Network Quantization”, [online], June 15, 2021, [Retrieved August 14, 2023], Internet,<URL: https: / / arxiv.org / pdf / 2106.08295.pdf> Summary of the Invention [Problem to be solved by the invention]

[0006] In the above-mentioned method, quantization error occurs as a result of reducing the calculation accuracy by reducing the number of bits during the quantization process of each layer. This quantization error accumulates in each layer of the neural network and eventually becomes large when it is output. [Means for solving the problem]

[0007] In order to solve this problem, for example, an information processing device of the present invention has the following configuration. An information processing device that generates a deep learning model having a first network and a second network connected to the first network, obtaining means for obtaining second quantization information including information regarding quantization of the second network; A determination means for generating first quantization information, which is information regarding quantization of the first network, based on the second quantization information; a first quantization means for applying a first quantization based on the first quantization information to the first network; The present invention is characterized by comprising: Effect of the Invention

[0008] According to the present invention, first quantization information regarding the first quantization of the first network is determined based on second quantization information regarding the second quantization of the second network, so that the quantizations can be made to correspond to each other, and the accumulation of quantization errors associated with an increase in layers, for example, can be suppressed. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an information processing system. [Diagram 2] FIG. 2 is a diagram showing an example of a hardware configuration of an apparatus in the information processing system. [Diagram 3] FIG. 1 is a functional block diagram of an information processing system according to a first embodiment. [Figure 4] 4 is a flowchart showing a process flow in the information processing system according to the first embodiment. [Diagram 5] FIG. 1 is a diagram for explaining the layer structure of a neural network that performs noise removal processing. [Figure 6] FIG. 1 is a diagram illustrating the serial application of two quantization techniques. [Figure 7] 5A to 5C are diagrams illustrating specific examples of second quantization information. [Figure 8] A diagram explaining quantization of a tensor histogram. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] <CNNについて> First, a description will be given of a Convolutional Neural Network (CNN) that is used in information processing technology that applies deep learning and that appears in each of the following embodiments. CNN is a technology that repeats convolution of a filter generated by training with data such as image data, followed by nonlinear calculation. The filter is also called a local receptive field. Image data obtained by convolving the filter with the image data and then performing nonlinear calculation is called a feature map. Learning is performed using training data (training images or data sets) consisting of a pair of input image data (also called student image data) and output image data (also called teacher image data). Simply put, learning is the generation of filter values ​​that can convert input image data to corresponding output image data with high accuracy from training data.

[0012] When image data has RGB color channels or when a feature map is composed of multiple pieces of image data, the filter used for convolution has multiple channels according to the image data or feature map. In other words, the convolution filter is expressed as a four-dimensional array that includes the number of channels in addition to the vertical and horizontal sizes and the number of images. The process of nonlinear calculation after convolving the filter with the image data (or feature map) is expressed in units of layers. For example, the feature map and filter are expressed as the nth layer feature map and the nth layer filter. For example, a CNN that repeats the convolution of the filter and nonlinear calculation three times has a three-layer network structure. Such nonlinear calculation processing can be formulated as the following equation (1).

[0013]

number

[0014] In the above formula (1), W n is the nth layer filter, b n is the bias of the nth layer, f is a nonlinear operator, and X n is the nth layer feature map, and * is the convolution operator. The (l) on the right indicates that it is the lth filter or feature map. The filters and biases are generated by learning, which will be described later, and are collectively called "network parameters." For example, a sigmoid function or ReLU (Rectified Linear Unit) is used as a nonlinear operation. The formula for ReLU is given by the following formula (2).

[0015]

number

[0016] As shown in the above formula (2), negative elements of the input vector X become zero, and positive elements remain unchanged.

[0017] As networks using CNN, ResNet in the field of image recognition and its application RED-Net in the field of noise removal are well-known. In both cases, by stacking multiple layers of CNN and performing convolution of filters multiple times, high-precision processing is achieved. For example, ResNet features a network structure with a path that shortcuts the convolutional layer, enabling the realization of a 152-layer deep network and achieving high-precision recognition approaching the human recognition rate. The reason why processing is made more precise by multiple-layer CNN is simply that by repeating non-linear operations multiple times, a non-linear relationship between input and output can be expressed.

[0018] <CNN Learning> Next, the learning of CNN will be explained. The learning of CNN is generally performed by minimizing the objective function expressed by the following formula (3) for the training data consisting of a set of input image (student image) data for training and corresponding output image (teacher image) data.

[0019]

Equation

[0020] In the above formula (3), L is a loss function that measures the error between the correct answer and its estimation. Y i is the i-th output image data, X iis the i-th input image data. F is a function that collectively represents the calculations of formula (1) performed in each layer of CNN. θ is the network parameters (filter and bias). ||Z||2 is the L2 norm, or more simply, the square root of the sum of the squares of the elements of vector Z. n is the total number of training data used for learning. Since the total number of training data is generally large, in the Stochastic Gradient Descent (SGD) method, a portion of the training image data is randomly selected and used for learning. This reduces the computational load in learning using a large amount of training image data. There are various methods known for minimizing (optimizing) the objective function, such as the momentum method, AdaGrad method, AdaDelta method, and Adam method. The Adam method is given by the following formula (4).

[0021]

number

[0022] In the above formula (4), θ i t is the i-th network parameter at the t-th iteration, and g is θ i t is the gradient of the loss function L with respect to m, v are moment vectors, α is the base learning rate, β1, β2 are hyperparameters, and ε is a small constant. Note that there are no guidelines for selecting an optimization method in learning, so any method can basically be used, but it is known that there are differences in convergence between methods, resulting in differences in learning time.

[0023] First Embodiment In this embodiment, a method for reducing artifacts caused by the above-mentioned quantization error will be described by taking as an example a case of high image quality processing that performs noise removal as processing using a neural network. Note that the application of the present technology is not limited to high image quality processing of noise removal, and may be applied to neural networks that perform other image processing, audio processing, and language processing.

[0024] <Example of information processing system configuration> Fig. 1 is a diagram showing an example of the configuration of an information processing system according to this embodiment. The information processing system shown in Fig. 1 includes a cloud server 200 that generates learning data and estimates image quality degradation and learns restoration (hereinafter, degradation restoration learning), and an edge device 100 that performs degradation restoration (hereinafter, degradation restoration inference). The cloud server 200 and the edge device 100 are connected via the Internet. The cloud server 200 and the edge device 100 are, for example, computers.

[0025] <Edge device hardware configuration> FIG. 2 is a diagram showing an example of a hardware configuration of an apparatus of an information processing system. The edge device 100 of this embodiment is an image processing apparatus that performs degradation restoration inference by applying learned network parameters provided from a cloud server 200 to image data input from an imaging apparatus 10. A user applies image quality improvement processing to image data using an image processing application installed in the edge device 100. The edge device 100 has a CPU 101, a RAM 102, a ROM 103, a mass storage device 104, a general-purpose interface (I / F) 105, and a network I / F 106. The CPU 101, the RAM 102, the ROM 103, the mass storage device 104, the general-purpose interface (I / F) 105, and the network I / F 106 are mutually connected by a system bus 107. The edge device 100 is connected to the imaging apparatus 10, the input device 20, the external storage device 30, and the display device 40 via the general-purpose I / F 105.

[0026] The CPU 101 executes programs stored in the ROM 103 using the RAM 102 as a work memory, and performs overall control of each component of the edge device 100 via a system bus 107. The mass storage device 104 is, for example, a hard disk drive (HDD) or a solid state drive (SSD), and stores various programs handled by the edge device 100 and data such as parameters required for executing the programs. The CPU 101 writes data to the mass storage device 104 and reads data stored in the mass storage device 104 via the system bus 107. The general-purpose I / F 105 is, for example, a serial bus interface such as USB, IEEE1394, or HDMI (registered trademark).

[0027] The edge device 100 acquires data from the external storage device 30 (for example, various storage media such as a memory card, a CF card, an SD card, a USB memory, etc.) via the general-purpose I / F 105. The edge device 100 accepts user instructions from an input device 20 such as a mouse or a keyboard via the general-purpose I / F 105. The edge device 100 outputs image data processed by the CPU 101 to a display device 40 (for example, various image display devices such as a liquid crystal display, etc.) via the general-purpose I / F 105. The edge device 100 acquires data of a captured image (RAW image) to be subjected to high image quality processing for noise removal from the imaging device 10 via the general-purpose I / F 105. The network I / F 106 is an interface for connecting to the Internet. The edge device 100 accesses the cloud server 200 by an installed web browser and acquires network parameters for degradation restoration inference.

[0028] <Cloud server hardware configuration> The cloud server 200 of this embodiment is a server device that provides cloud services on the Internet. More specifically, the cloud server 200 generates learning data and performs degradation restoration learning, and provides the network parameters and network structure of the learned deep learning model, which is the learning result, in response to a request from the edge device 100. The cloud server 200 has a CPU 201, a ROM 202, a RAM 203, a mass storage device 204, and a network I / F 205. The CPU 201, the ROM 202, the RAM 203, the mass storage device 204, and the network I / F 205 are connected to each other by a system bus 206. The CPU 201 reads out a control program stored in the ROM 202 or the mass storage device 204 and executes various processes, thereby controlling the operation of the entire device. The RAM 203 is used as a temporary storage area such as a main memory and a work area of ​​the CPU 201. The mass storage device 204 is a large-capacity secondary storage device such as an HDD or SSD that stores image data and various programs. The network I / F 205 is an interface for connecting to the Internet. The cloud server 200 provides the above-mentioned network parameters in response to a request from the web browser of the edge device 100 via the network I / F 205.

[0029] Although there are other components of the edge device 100 and the cloud server 200 than those described above, they are not the main focus of this embodiment, and therefore will not be described. In this embodiment, the cloud server 200 generates learning data and performs degradation restoration learning. The edge device 100 downloads a trained deep learning model, which is the learning result, and performs degradation restoration inference on the RAW image data. The above system configuration is an example, and is not limited to this. For example, the functions of the cloud server 200 may be subdivided, and the generation of learning data and the degradation restoration learning may be performed by separate devices. Furthermore, the imaging device 10, which has both the functions of the edge device 100 and the cloud server 200, may be configured to perform all of the generation of learning data, the degradation restoration learning, and the degradation restoration inference.

[0030] <Issues and approaches to existing quantization methods> The issues that arise when a neural network performing noise removal processing as represented by Non-Patent Document 2 is quantized using the quantization technique as represented by Non-Patent Document 1, and approaches to address the issues will be described in detail with reference to Figs. 5 and 6.

[0031] FIG. 5 shows the layer structure of a neural network that performs noise removal processing, as disclosed in Non-Patent Document 2. Noise removal is a process that removes noise that is superimposed when an image is captured. The purpose of noise removal is to obtain a high-quality image 510 in which noise has been removed from an input degraded image 501 that contains noise. The method of Non-Patent Document 2 uses two neural networks, a first network 502 in Step 1 and a second network 507 in Step 2, to reduce noise in stages and achieve high noise removal performance. In FIG. 5, for simplicity, one network is assumed to have three layers, but in reality, more than this number of layers may be held. Each of layers 1 to 6 executes a convolution process 512, an activation process 513, and a quantization process 514, and outputs the results of the processing. In the following description, when it is necessary to distinguish between the processing of either network 502 or 507, the processing of the first network 502, etc. is described with the addition of "first", such as first quantization processing, and the processing of the second network 507, etc. is described with the addition of "second", such as second quantization processing. The first network 502 and the second network 507 are connected. The connection here includes a direct connection and an indirect connection via an adder or the like. The number of networks to be connected may be changed as appropriate. The anteroposterior relationship of the connected networks may be changed as appropriate. Input / output data of each layer is expressed in a three-dimensional tensor data format of length x width x channel. Note that the quantization process 514 in the figure is inserted when applying quantization to the architecture of Non-Patent Document 2.

[0032] Next, the data flow of the above-mentioned noise removal process will be described. Tensor 0, which is a degraded image 501 containing noise, is input to the first network 502, and each process is executed on the degraded image 501 from layer 1 to layer 3, and tensor 3, which is a noise component (degraded component) of the degraded image, is output. An addition process 505 is performed in which residual map 503, which is the obtained tensor 3, is added to the degraded image 501, and image 506, which is the image quality improvement result of Step 1, is output as tensor 4. In image 506, which is the image quality improvement result at this point in time, the noise in the degraded image 501 has not been sufficiently removed. In order to remove the noise remaining in image 506, tensor 4, which is the image quality improvement result of Step 1, is input to the second network 507 of Step 2, and each process is executed on tensor 4 from layer 4 to layer 6. As a result, residual map 508, which is tensor 7, which is the processing result of Step 2, is output. In an addition process 509, the residual map 508 is added to the image quality improvement resultant image 506 of Step 1, and an image quality improvement resultant image 510, which is a tensor 8, is output. It is desirable to improve the calculation accuracy of the residual maps 503, 508 by proceeding with learning so that the loss calculations 504, 512 calculated based on the respective residual maps 503, 508 are reduced.

[0033] However, when the weight parameters of the convolution process 512 and the values ​​of each tensor are quantized using the method of Non-Patent Document 1, quantization errors occur in each layer, and errors accumulate throughout the entire network. As a result, when looking at a partial image 511 (highlighted) that is an enlarged view of a flat portion of the high-quality image 510, it can be seen that residual noise, which is a variation in pixel values, and false patterns in which the residual noise is connected vertically and horizontally to appear as a pattern have occurred in an area that should be flat.

[0034] The reason for this quantization error is that the values ​​of the weight parameters and tensors in the deep learning model that are input to the quantization method are continuous values ​​that span the entire real number range. In this case, rounding errors inevitably occur and accumulate when quantization is applied.

[0035] Therefore, the technology disclosed in this application suppresses the occurrence of new quantization errors by applying the above-mentioned quantization to a deep learning model generated so that the values ​​of each weight parameter and tensor have discrete values ​​in a range that is a value range that is significantly restricted after quantization. A deep learning model having this restricted range is obtained by a quantization method with a powerful function of reducing quantization errors as disclosed in Non-Patent Document 3. The method of Non-Patent Document 3 outputs a deep learning model with suppressed quantization errors by performing learning to reduce the generated quantization errors by error backpropagation.

[0036] FIG. 6 is a diagram for explaining the serial application of two quantization methods of the embodiment. Based on the above, in this embodiment, as shown in FIG. 6, quantization information (called first quantization information) capable of reducing quantization error as represented by Non-Patent Document 3 is applied to a first network 502 of a deep learning model, and the first network 502 is quantized (called first quantization). After that, in this embodiment, quantization information (called second quantization information) capable of optimizing for a specific edge device as shown in Non-Patent Document 1 is applied to a second network 507 of a deep learning model to which the first quantization by the first quantization information is applied, and the second network 507 is quantized (called second quantization). As a result, this embodiment generates a deep learning model to which quantization by both quantization information is applied. Furthermore, this embodiment determines the first quantization information based on the second quantization information. As a result, this embodiment controls the first quantization of the first network 502 based on the second quantization information such as the number of bits, type, and precision when performing the second quantization, and reduces the quantization error generated in the second network 507.

[0037] <Processing flow of the entire system> The functional blocks and processing flow of the entire information processing system in this embodiment will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is a functional block diagram of the information processing system. Fig. 4 is a flowchart showing the processing flow in the information processing system.

[0038] As shown in Fig. 3(a), cloud server 200 has model acquisition unit 301, second quantization information acquisition unit 302, first quantization information determination unit 303, first quantization unit 304, and second quantization unit 308. As shown in Fig. 3(b), first quantization unit 304 has learning data acquisition unit 305, forward propagation unit 306, and weight parameter update unit 307. Cloud server 200 realizes the functions of model acquisition unit 301, second quantization information acquisition unit 302, first quantization information determination unit 303, first quantization unit 304, and second quantization unit 308 by CPU 201 loading a program stored in ROM 202 or mass storage device 204 into RAM 203.

[0039] The edge device 100 includes an inference unit 309 and a performance evaluation unit 310. The edge device 100 realizes the functions of the inference unit 309 and the performance evaluation unit 310 by the CPU 101 loading a program stored in the ROM 103 or the mass storage device 104, etc., into the RAM 102.

[0040] The configuration shown in Fig. 3 can be modified or changed as appropriate. For example, one functional unit may be divided into multiple functional units, or two or more functional units may be integrated into one functional unit. The configuration shown in Fig. 3 may be realized by two or more devices. In this case, each device is connected via a circuit, wired or wireless network, and can realize each process described below by performing cooperative operations through data communication with each other.

[0041] In step S401, the model acquisition unit 301 acquires information on a trained deep learning model to which the quantization information is to be applied from the mass storage device 104 of the edge device 100. Note that the model acquisition unit 301 may read information on the trained deep learning model from another device such as the ROM 202, the mass storage device 204, the external storage device 30 of the edge device 100, and the network I / F 106.

[0042] In step S402, the second quantization information acquisition unit 302 acquires second quantization information, which is information when quantizing the second network 507, from the model acquisition unit 301 or the performance evaluation unit 310. Specifically, in the first step S402, the second quantization information acquisition unit 302 acquires the second quantization information from the model acquisition unit 301, and from the second time onwards, the second quantization information acquisition unit 302 may acquire updated second quantization information that has been performance evaluated from the performance evaluation unit 310. The second quantization information includes four types of information, namely, the number of bits, type, performance evaluation value, and architecture information of the tensor of the second network 507 including the second quantization process. FIG. 7 is a diagram for explaining a specific example of the second quantization information. The second quantization information is held as an electronic file as in the description 701 in FIG. 7. Each description of the description 701 will be explained below.

[0043] The number of bits indicates the number of bits of each tensor of the deep learning model output by each layer of the second network 507. For example, the technique of Non-Patent Document 1 selects an optimal number of bits from 8 bits, 16 bits, and 32 bits for each tensor and performs quantization. In this case, as in description 701, the item of the number of bits describes options for the number of bits to be used as an array.

[0044] The type is the type of the numerical value of each tensor of the second network 507 quantized by the second quantization information. The type in the description 701 is an example of using a signed integer type. Note that the type may be an unsigned integer type, a signed floating point type, or the like.

[0045] The performance evaluation value is a value indicating the relationship between the reduction rate of the value range of a tensor and the performance in the second network 507 to which the second quantization of the second quantization information is applied. When the quantization process is performed on a tensor, the dynamic range of the variable is reduced, and the value range of the tensor that can be expressed is narrowed. FIG. 8 is a diagram explaining the quantization of a histogram of a tensor. For example, consider the case where the distribution of the values ​​of a certain tensor is as shown in histogram 801 in FIG. 8(a), and the tensor of histogram 801 has a minimum value 802 and a maximum value 803. When the quantization process is applied to the tensor of histogram 801, the value range is restricted as shown by number line 804 in FIG. 8(b) and number line 805 in FIG. 8(c), and the possible values ​​become discrete values. As shown in FIG. 8(b), the wider the value range, the better the expressiveness at the ends, but the values ​​become sparse in the center, and the quantization error increases, so there is a trade-off relationship. Therefore, it is necessary to find a reduction rate of the value range that maximizes performance. Therefore, the second quantization is performed in advance using the second quantization information, and the performance for each reduction rate of the value range at that time is recorded as a performance evaluation value in the description 701. Here, the performance indicates the smallness of the distance between the output image with high image quality and the correct image. Note that other performance evaluation methods may be used.

[0046] As shown in Fig. 7, the architecture information includes at least one of the connection relationship of layers of the second network used in the second quantization based on the second quantization information and the output destination of the output tensor. For each layer, a weight parameter, IDs of the input tensor and the output tensor, and the number of bits after quantization are described as the architecture information. Note that the method of describing the architecture information is not limited to the example of Fig. 7, and other information related to the architecture may be described.

[0047] Second quantization information acquisition unit 302 outputs the above-mentioned second quantization information to first quantization information determination unit 303 and second quantization unit 308. Note that the second quantization information may include information other than the above-mentioned four types of information.

[0048] In step S403, first quantization information determination unit 303 determines, based on the second quantization information, first quantization information that is information for quantizing (first quantization) first network 502. The first quantization information includes three types of information: the number of bits of the weight parameters of each layer of the first network, the number of bits of input / output tensors input and output at each layer of the first network, a quantization formula, and a value range.

[0049] The first quantization information determination unit 303 determines the number of bits of at least one of the weight parameters and the input / output tensor values ​​of the first network 502 so as to match the minimum number of bits of the weight parameters and the input / output tensor values ​​indicated by the second quantization information. For example, in the example of FIG. 7, the number of bits is determined to be 8 bits. This parameter is passed to the quantization process 514, and the value is discretized into at most 2 to the power of 8 kinds of decimal values ​​during quantization. In this way, no more information is lost during the second quantization process of the second network 507 than during the first quantization process of the first network 502, and the occurrence of new quantization errors can be avoided.

[0050] However, when the type is specified as a floating-point type in the second quantization information, the first quantization information determination unit 303 sets the number of bits of the first quantization information to at least 16 bits in order to ensure the expressiveness of the mantissa. In this way, the first quantization information determination unit 303 determines the number of bits of the tensor of the first quantization of the first network 502 included in the first quantization information, taking into account the type of the tensor used in the second quantization, thereby suppressing the increase in quantization error due to insufficient expressiveness of the mantissa. Note that the method of determining the number of bits is not limited to the above, and the first quantization information determination unit 303 may determine the number of bits using another method, such as increasing or decreasing the number of bits from the above value depending on the amount of calculation resources, or changing the number of bits taking into account the magnitude of the quantization error of each layer.

[0051] The quantization formula is determined as follows:

[0052]

number

[0053] x is the value before quantization. y is the value after quantization. f clip The minimum value of α min And the maximum value is α max This is an operation to clip so that the range is reduced by quantization. [] is the Gaussian symbol, and realizes the rounding of values ​​by quantization. c is the quantization unit that represents the step size of the value after quantization.

[0054] In the example of the architecture information in FIG. 7, the layer 1 receives the tensor 0, which is a degraded image, as an input. When the input degraded image has pixel values ​​of 0 to 255, in order to store it in a signed integer type, it is necessary to secure a 9-bit storage area of ​​-255 to +255. On the other hand, since the amount of information of the output tensor 1 is 8 bits, it is necessary to reduce the amount of information after the convolution operation, and the first quantization information determination unit 303 selects the rounding formula in the upper part of Equation (5). In contrast, in the layer 2, since both the input tensor and the convolution filter coefficients have an amount of information of 8 bits, there is little need to reduce the number of bits on the output side. Therefore, the first quantization information determination unit 303 selects the formula in the lower part of Equation (5) in which the rounding is omitted. In this way, the first quantization information determination unit 303 reads out the calculation accuracy in the second quantization of the second network 507 from the architecture information and determines the quantization formula for the first quantization of the first network 502 based on this, which reduces unnecessary processing and speeds up the quantization processing. The quantization formula is not limited to the method of determination of formula (5), and may be determined based on other rules.

[0055] The first quantization information determination unit 303 determines the range [α min ,α max ] is set to a value that maximizes the performance evaluation value included in the second quantization information. In the example of FIG. 7, the range reduction rate that maximizes the evaluation value is 2%. Therefore, when the distribution of the values ​​of each tensor when various degraded images are input to the neural network is as shown in histogram 801, the first quantization information determination unit 303 sets the top 2% point 808 of the histogram as α max Let αmin =-α max Based on this range, the first quantization information determination unit 303 sets the quantization unit c=α max / 128. In this embodiment, an example is shown in which the relationship between the range and the accuracy evaluation value is described in the second quantization information, but the first quantization information determination unit 303 may measure and describe the relationship between the information such as the number of bits and the quantization formula and the accuracy evaluation value in advance, and may adopt the number of bits and the quantization formula that maximizes the accuracy evaluation value. In this way, the first quantization information determination unit 303 determines the first quantization information including the number of bits, the quantization formula, and the range of the tensor of the first network 502 so as to improve the evaluation value, thereby improving the performance of the second network 507 including the second quantization process.

[0056] However, for the tensor which is the image outputted outside the first network 502, since it is used for calculation with the degraded image and for calculating the image quality improvement result image, the first quantization information determination unit 303 needs to set the value range so as to match the quantization unit of the external calculation. In the example of Fig. 7, since the layer 3 generates the tensor 3 which is added to the tensor 0, it can be seen from the architecture information that the first network 502 outputs the degraded image to the outside.

[0057] When the possible pixel values ​​of the degraded image are 0, 1, 2, ..., 255, the residual map tensor 3 must match the quantization unit of the degraded image and take values ​​of -128, -127, ..., 126, 127. This is because if the quantization density becomes coarser than this, the first network 502 will not be able to remove noise of pixel values ​​at one scale. Also, if the quantization density becomes finer than this, the value range will be narrowed and the first network 502 will not be able to remove noise with large displacement. This is because noise that cannot be removed will cause quantization errors such as residual noise and false patterns. If tensor 0 is a normalized value, i.e., if the possible values ​​are 0, 1 / 255, 2 / 255, ..., 254 / 255, and 1, the first quantization information determination unit 303 similarly scales tensor 3 so that it can take the values ​​-128 / 255, -127 / 255, ..., 126 / 255, and 127 / 255. In other words, the first quantization information determination unit 303 determines the value range of tensor 3 output from the first network 502 based on the architecture information.

[0058] In this way, the first quantization information determination unit 303 can suppress the occurrence of correction residues for inputs due to quantization errors by determining the value range so that the quantization units are approximately the same in the first quantization of the first network 502 and the second quantization of the second network 507. In addition, the first quantization information determination unit 303 can determine the value range based on information on whether the tensor is output from the network, thereby making it possible to determine the value range according to the above-mentioned use of the tensor, and thus improving the performance of the second quantization process of the second network 507.

[0059] The first quantization information determination unit 303 outputs the first quantization information determined as above to the first quantization unit 304. Note that, although the number of bits of each weight parameter, the number of bits of each tensor, the quantization formula, and the range are set as the first quantization information in the above, the first quantization information determination unit 303 may include items other than these in the first quantization information.

[0060] In step S404, the first quantization unit 304 applies first quantization to the first network 502 of the deep learning model based on the first quantization information. The first quantization unit 304 performs quantization learning, which absorbs quantization error by error backpropagation, as described in Non-Patent Document 3, on the deep learning model to which the first quantization based on the first quantization information has been applied. The quantization learning procedure is as shown in FIG. 4(b) and the following steps S405 to 407.

[0061] In step S405, the learning data acquiring unit 305 acquires learning data, which is a sequence of pairs of student data as a degraded image with noise superimposed thereon and teacher data as a noise-free teacher image, from a medium such as the mass storage device 104 or the external storage device 30. Note that the learning data acquiring unit 305 may acquire only the teacher image and generate a degraded image by superimposing noise on the teacher image.

[0062] In step S406, the forward propagation unit 306 executes a forward propagation process and outputs the image quality improvement result image, which is an output tensor, as an inference result. Specifically, the forward propagation unit 306 inputs the degraded image acquired from the learning data acquisition unit 305 to a deep learning model, and calculates an image quality improvement result image in which noise has been removed from the degraded image. In the convolution process 512, the forward propagation unit 306 executes an operation using a convolution filter with weight parameters quantized based on the first quantization information. In the quantization process 514, the forward propagation unit 306 also executes a quantization process to which the first quantization information is applied.

[0063] In step S407, the weighting parameter updating unit 307 calculates the error between the obtained image quality improvement result image and the teacher image to obtain a loss value L. The weighting parameter updating unit 307 reduces the quantization error by backpropagating the loss value and updating the weighting parameters according to Equation (4). At this time, since the update amount of the weighting parameters is a continuous value, the weighting parameter updating unit 307 temporarily cancels the quantization of the discretized weighting parameters before updating them.

[0064] The first quantization unit 304 executes the process of steps S405 to 407 until the error is within a certain range or a given number of repetitions is reached, and obtains a deep learning model in which the first quantized weight parameters are updated. In this way, the weight parameter update unit 307 quantizes while updating the weight parameters so as to reduce the error with the teacher data, thereby suppressing the quantization error and obtaining a quantized deep learning model. Note that, although the quantization method of Non-Patent Document 3 is applied in steps S405 to 407, other quantization methods may be applied.

[0065] The first quantization unit 304 outputs the deep learning model in which the first network 502 has been first quantized and the weight parameters have been updated to the second quantization unit 308. This makes the weight parameters and architecture common to the first network 502 and the second network 507, and the second quantized second network 507 can inherit the small quantization error achieved by the first quantization.

[0066] In step S408, the second quantization unit 308 applies second quantization to the deep learning model that has been first quantized by the first quantization information, to the second network 507 based on the second quantization information. The second quantization is a quantization method in which optimization is performed to increase the execution speed in the edge device to which the model is deployed, as typified by Non-Patent Document 1. Specifically, the second quantization unit 308 inputs a plurality of calibration data to measure the distribution of each layer, and determines the number of bits of each tensor so as to ensure both the invariance of the distribution before and after quantization and the avoidance of slowdown on the edge device. As a result, the second quantization unit 308 performs second quantization on the parameters of the second network 507 based on the second quantization information. As a result, the second quantization unit 308 transmits the deep learning model that has been first quantized and second quantized to the edge device 100.

[0067] In step S409, the inference unit 309 of the edge device 100 inputs the degraded image to the deep learning model that has been second quantized based on the second quantization information, and calculates a high-image-quality resultant image.

[0068] In step S410, the performance evaluation unit 310 evaluates the image quality of the image quality improvement resultant image using evaluation indices such as the error from the teacher image, PSNR (Peak Signal To Noise Ratio), and SSIM (Structural Similarity).

[0069] In step S411, if the performance evaluation unit 310 determines that the evaluation value does not meet the given threshold value or that re-learning is necessary, the cloud server 200 updates the performance evaluation value of the description 701 based on the evaluation result, updates the second quantization information, and executes the process again from step S402. On the other hand, if the performance evaluation unit 310 determines that the evaluation value meets the given threshold value or that re-learning is not necessary, the flow ends. The determination in step S411 may be made by the user. In this case, the cloud server 200 may obtain an input from the user and decide whether to execute the process from step S402 onwards again or to end the flow.

[0070] In this embodiment, an example of noise removal as image quality improvement is shown, but this embodiment can be applied to other image quality improvement methods such as super-resolution and blur removal. In this embodiment, an example of quantizing a deep learning model for image quality improvement is described, but the deep learning model to be quantized is not limited to this. The object of quantization can also be applied to other deep learning models for image processing such as a recognition model or an image generation model, a deep learning model for voice processing, a deep learning model for language processing, and other deep learning models for performing other processing.

[0071] As described above, according to this embodiment, the first quantization information is determined based on the second quantization information, so that the quantization of the first network 502 and the second network 507 can be made to correspond to each other. As a result, this embodiment can determine the quantization information and quantize both networks so that the quantization error in the first quantization of the first network 502 is not accumulated due to the quantization error in the second quantization of the second network 507. As a result, this embodiment can reduce the accumulation of quantization error accompanying an increase in the number of networks and layers. Therefore, for example, in the noise removal process of an image, artifacts such as false patterns and residual noise in the output image can be suppressed.

[0072] In addition, in this embodiment, the first quantization information is determined based on the second quantization information of the second network 507 that has been second quantized to optimize it for the edge device 100 to which it is deployed, so that the accumulation of quantization errors can be reduced without making any algorithm changes that may impair the optimization performance of the second network 507.

[0073] In this embodiment, the second quantization information includes any of information regarding the weight parameters of the second network 507, information such as the number of bits regarding the output tensor, and architecture information of the second network 507, so that the first quantization information can be appropriately determined in accordance with the target of quantization, the content of quantization, etc.

[0074] In this embodiment, first quantization information including either information regarding the weight parameters of the first network 502 or a quantization formula is determined, so that information such as the number of bits regarding the weight parameters of the first network 502 and a quantization formula that can reduce the accumulation of quantization errors can be determined based on the second quantization information.

[0075] In this embodiment, a deep learning model including a first network 502 to which first quantization based on first quantization information has been applied is applied to a second network 507, and second quantization based on second quantization information is applied to the second network 507. As a result, in this embodiment, quantization is applied to two networks using quantization information corresponding to each other, so that accumulation of quantization errors can be reliably reduced.

[0076] In this embodiment, an image quality improvement result image, which is an input tensor based on a degraded image, which is an output tensor output from a first network 502, is input to a second network 507. In this manner, in this embodiment, the first network 502 and the second network 507 are quantized with corresponding quantization information, and tensors are input and output between the networks 502 and 507 connected in series with each other, so that accumulation of quantization errors can be reduced.

[0077] In this embodiment, the first quantized first network 502 acquires learning data, outputs an inference result by forward propagation processing, and updates the weight parameters based on the inference result. As a result, in this embodiment, learning is performed by the first network 502 with less accumulated quantization error, so that the weight parameters can be updated to those with even smaller errors.

[0078] In this embodiment, since the second quantization information includes a performance evaluation value, the first quantization information can be determined so as to increase the performance evaluation value of the second network 507. As a result, in this embodiment, the performance of the second network 507 can be further improved.

[0079] In this embodiment, the number of bits of at least one of the weight parameters and the value of the output tensor of the first quantization information is determined so as to match the minimum value of the number of bits indicated by the second quantization information. This prevents the loss of information during the quantization process of the second network 507 from occurring more than during the quantization process of the first network 502, and makes it possible to avoid the occurrence of new quantization errors.

[0080] In this embodiment, the occurrence of correction residues for inputs due to quantization errors can be suppressed by determining the value range so that the quantization units of the first quantization of the first network 502 and the second quantization of the second network 507 are approximately the same.

[0081] In this embodiment, based on the architecture information, it is determined whether or not the tensor output from the first network 502 is to be output to the outside, and the value range of the tensor is determined. This makes it possible to determine the value range according to the use of the output tensor of the first network 502, and improves the performance of the second network 507.

[0082] In this embodiment, the number of bits of the output tensor of the first network 502 is determined based on the type of the output tensor of the second network 507, thereby making it possible to suppress the increase in quantization error due to insufficient expressiveness of the mantissa.

[0083] <Second embodiment> In the first embodiment, a mode of reducing quantization error using a second quantization process with high optimization performance for a specific edge device is described. In the present embodiment, a lightweight and high-speed second quantization with limited options for the number of calibration data and the number of bits of variables after quantization is targeted. Then, an example is described in which the total processing time of the first and second quantizations is reduced compared to the single quantization method represented by Non-Patent Document 3 by determining the first quantization information based on the second quantization information without changing the processing of the second quantization method.

[0084] Note that a description of the contents common to the first embodiment, such as the basic configuration of the information processing system, will be omitted, and the following description will focus on the differences.

[0085] In step S402, the second quantization information acquisition unit 302 acquires second quantization information including information on the number of calibration data and the number of bits after quantization. The second quantization information acquisition unit 302 outputs the second quantization information to the first quantization information determination unit 303 and the second quantization unit 308.

[0086] In step S403, the first quantization information determination unit 303 determines the first quantization information based on the second quantization information. In consideration of the fact that the second quantization is performed in the next stage of the first quantization, the first quantization information determination unit 303 determines the first quantization information so as to minimize the number of iterations of quantization learning and to end the quantization process in a short execution time. When the given number of iterations in the first embodiment is n, the ratio of the number of calibration data in the second embodiment to the first embodiment is r, and the given speed-up coefficient is k (=100), the first quantization information determination unit 303 determines the number of iterations to be n / rk.

[0087] In steps S404 to S408, the first quantization unit 304 and the second quantization unit 308 perform first quantization and second quantization based on the obtained quantization information, and obtain a deep learning model of the quantization result.

[0088] In step S409, the inference unit 309 of the edge device 100 inputs the degraded image to the deep learning model resulting from the second quantization, and calculates a high-image-quality resultant image.

[0089] According to this embodiment, the time required for the quantization process can be reduced by dividing the process into two parts: the lightweight first quantization process is responsible for generating a deep learning model capable of reducing quantization errors, and the lightweight second quantization process is responsible for the actual calculation precision reduction process. In other words, the total processing time required for the first quantization process and the second quantization process can be reduced compared to the single quantization method typified by Non-Patent Document 3.

[0090] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0091] The disclosure of this specification includes the following information processing device, information processing method, and program. (Item 1) An information processing device that generates a deep learning model having a first network and a second network connected to the first network, obtaining means for obtaining second quantization information including information regarding quantization of the second network; A determination means for generating first quantization information, which is information regarding quantization of the first network, based on the second quantization information; a first quantization means for applying a first quantization based on the first quantization information to the first network; An information processing device comprising: (Item 2) The acquiring means acquires the second quantization information including at least one of information on weight parameters of the second network, information on an output tensor, and second architecture information including a connection relationship of layers of the second network and an output destination of the output tensor. 2. The information processing device according to item 1, (Item 3) The determining means determines the first quantization information including at least one of information regarding a weight parameter of the first network and a quantization formula. 3. The information processing device according to item 1 or 2. (Item 4) a second quantization means for applying second quantization based on the second quantization information to the second network of the deep learning model including the first network to which the first quantization has been applied; 4. The information processing device according to any one of items 1 to 3, (Item 5) The determining means determines the first quantization information including at least one of a number of bits, a quantization formula, and a value range of an input / output tensor of each layer of the first network. 5. The information processing device according to any one of items 1 to 4, (Item 6) The acquisition means acquires the second quantization information including at least one of the number of bits of input / output tensors of a layer of the second network, the type of the input / output tensors, an output tensor inferred using weight parameters obtained by the second quantization, and second architecture information including a connection relationship of the layers of the second network and an output destination of the output tensor. 5. The information processing device according to item 4, (Item 7) The second network is input with an input tensor based on an output tensor output based on at least one of the first quantized weight parameters and first architecture information including a connection relationship of layers of the first network and an output destination of an output tensor. 5. The information processing device according to item 4, (Item 8) The first quantization means A learning data acquisition means for acquiring learning data consisting of student data and teacher data; a forward propagation means for performing a forward propagation process using the student data as an input and outputting an inference result; and a weight parameter update means for updating a weight parameter so as to reduce an error between the inference result and the teaching data. 8. The information processing device according to any one of items 1 to 7, (Item 9) The acquiring means acquires the second quantization information including an evaluation value representing accuracy of the output of the second network. 9. The information processing device according to any one of items 1 to 8, (Item 10) The first quantization means causes the number of bits of either the weight tensor or the output tensor of the first network that has been first quantized to coincide with the minimum value of the number of bits indicated by the second quantization information. 10. The information processing device according to any one of items 1 to 9, (Item 11) The determining means determines the first quantization information including first architecture information including a connection relationship of layers of the first network and an output destination of an output tensor; The first quantization means determines a quantization formula for the first network based on the first quantization information. 11. The information processing device according to any one of items 1 to 10, (Item 12) The first architecture information includes a computation precision of the output tensor. 12. The information processing device according to item 11, (Item 13) The first quantization means determines a range of values ​​of an output tensor output from a layer of the first network based on the first architecture information. 12. The information processing device according to item 11, (Item 14) The first quantization means determines a range of values ​​to be quantized so that a quantization unit indicating a step width of values ​​of an output tensor output from a layer of the first network and a layer of the second network is approximately the same. 14. The information processing device according to item 13, (Item 15) The determining means determines the first quantization information including the first architecture information including information on whether the output tensor of the first network is output to the outside. 14. The information processing device according to item 13, (Item 16) The determining means determines the first quantization information including at least one of a number of bits of an output tensor of the first network, a quantization formula, and a value range so as to improve the evaluation value. 10. The information processing device according to item 9, (Item 17) The determining means determines the first quantization information including the number of bits of the output tensor of the first network based on a type of the output tensor of the second network indicated by the second quantization information. 17. The information processing device according to any one of items 1 to 16, (Item 18) The deep learning model acquires image data and learns from it. 18. The information processing device according to any one of items 1 to 17, (Item 19) The first network reduces a quantization error by the first quantization information; The second network is optimized for a device that executes the deep learning model using the second quantized information. 19. The information processing device according to any one of items 1 to 18, (Item 20) 1. An information processing method for generating a deep learning model having a first network and a second network connected to the first network, obtaining second quantization information including information regarding quantization of the second network; A determination step of generating first quantization information, which is information regarding quantization of the first network, based on the second quantization information; a first quantization step of applying a first quantization based on the first quantization information to the first network; An information processing method comprising: (Item 21) A program for causing a computer to function as each of the means of the information processing device according to any one of items 1 to 19.

[0092] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0093] 100...edge device, 200...cloud server, 502...first network, 507...second network, 301...model acquisition unit, 302...second quantization information acquisition unit, 303...first quantization information determination unit, 304...first quantization unit, 308...second quantization unit.

Claims

1. An information processing device that generates a deep learning model having a first network and a second network connected to the first network, obtaining means for obtaining second quantization information including information regarding quantization of the second network; A determination means for generating first quantization information, which is information regarding quantization of the first network, based on the second quantization information; a first quantization means for applying a first quantization based on the first quantization information to the first network; An information processing device comprising:

2. The acquiring means acquires the second quantization information including at least one of information on weight parameters of the second network, information on an output tensor, and second architecture information including a connection relationship of layers of the second network and an output destination of the output tensor.

2. The information processing apparatus according to claim 1,

3. The determining means determines the first quantization information including at least one of information regarding a weight parameter of the first network and a quantization formula.

2. The information processing apparatus according to claim 1,

4. The deep learning model further includes a second quantization means for applying second quantization based on the second quantization information to the second network of the deep learning model including the first network to which the first quantization has been applied.

2. The information processing apparatus according to claim 1,

5. The determining means determines the first quantization information including at least one of a number of bits, a quantization formula, and a value range of an input / output tensor of each layer of the first network.

2. The information processing apparatus according to claim 1,

6. The acquisition means acquires the second quantization information including at least one of the number of bits of input / output tensors of a layer of the second network, the type of the input / output tensor, an output tensor inferred using weight parameters obtained by the second quantization, and second architecture information including a connection relationship of the layers of the second network and an output destination of the output tensor.

5. The information processing apparatus according to claim 4.

7. The second network is input with an input tensor based on an output tensor output based on at least one of the first quantized weight parameters and first architecture information including a connection relationship of layers of the first network and an output destination of an output tensor.

5. The information processing apparatus according to claim 4.

8. The first quantization means A learning data acquisition means for acquiring learning data consisting of student data and teacher data; a forward propagation means for performing a forward propagation process using the student data as an input and outputting an inference result; and a weight parameter update means for updating a weight parameter so as to reduce an error between the inference result and the teaching data.

2. The information processing apparatus according to claim 1,

9. The acquiring means acquires the second quantization information including an evaluation value representing accuracy of the output of the second network.

2. The information processing apparatus according to claim 1,

10. The first quantization means causes the number of bits of either the weight tensor or the output tensor of the first network that has been first quantized to coincide with the minimum value of the number of bits indicated by the second quantization information.

2. The information processing apparatus according to claim 1,

11. The determining means determines the first quantization information including first architecture information including a connection relationship of layers of the first network and an output destination of an output tensor; The first quantization means determines a quantization formula for the first network based on the first quantization information.

2. The information processing apparatus according to claim 1,

12. The first architecture information includes a computation precision of the output tensor.

12. The information processing apparatus according to claim 11 .

13. The first quantization means determines a range of values ​​of an output tensor output from a layer of the first network based on the first architecture information.

12. The information processing apparatus according to claim 11 .

14. The first quantization means determines a range of values ​​to be quantized so that a quantization unit indicating a step width of values ​​of an output tensor output from a layer of the first network and a layer of the second network is approximately the same.

14. The information processing apparatus according to claim 13,

15. The determining means determines the first quantization information including the first architecture information including information on whether the output tensor of the first network is output to the outside.

14. The information processing apparatus according to claim 13,

16. The determining means determines the first quantization information including at least one of a number of bits of an output tensor of the first network, a quantization formula, and a value range so as to improve the evaluation value.

10. The information processing apparatus according to claim 9,

17. The determining means determines the first quantization information including the number of bits of the output tensor of the first network based on a type of the output tensor of the second network indicated by the second quantization information.

2. The information processing apparatus according to claim 1,

18. The deep learning model acquires image data and learns from it.

2. The information processing apparatus according to claim 1,

19. The first network reduces a quantization error by the first quantization information; The second network is optimized for a device that executes the deep learning model using the second quantized information.

2. The information processing apparatus according to claim 1,

20. 1. An information processing method for generating a deep learning model having a first network and a second network connected to the first network, obtaining second quantization information including information regarding quantization of the second network; A determination step of generating first quantization information, which is information regarding quantization of the first network, based on the second quantization information; a first quantization step of applying a first quantization based on the first quantization information to the first network; An information processing method comprising:

21. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 19.

Images