Learning method and learning device

Abstract

To aim at gaining both of lightening of a machine learning model and inference precision.SOLUTION: A learning method permits learning of a neural network having an extraction section for extracting a feature amount from an input image, and a detection section for detecting the class and coordinates of an object photographed in the input image on the basis of the extracted feature amount, and includes a first step of allowing the extraction section to learn a weighting parameter and a quantization parameter as learning objects by low-bit operation of 8 bits or less, and a second step of allowing the neural network to learn a weighting parameter and a quantization parameter as learning objects by low-bit operation of 8 bits or less.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a learning method and a learning device. [Background technology]

[0002] In recent years, a convolutional neural network (CNN) has become known as a model used in image recognition processing and the like (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6896306 Summary of the Invention [Problem to be solved by the invention]

[0004] As a method for reducing the computational load of such convolutional neural networks, quantization, which reduces the number of bits in input data or intermediate data, is known. However, there is concern that the downside of reducing the computation load by performing quantization is that the inference accuracy may decrease. In other words, it is not easy to achieve both a reduction in computational load and high computational accuracy.

[0005] Therefore, an object of the present invention is to provide a learning method and a learning device that can achieve both lightweight machine learning models and high inference accuracy. [Means for solving the problem]

[0006] [1] In order to solve the above problems, one aspect of the present invention is a learning method for training a neural network having an extraction unit that extracts features from an input image and a detection unit that detects the class and coordinates of an object captured in the input image based on the extracted features, the learning method including a first step of training the extraction unit using low-bit operations of 8 bits or less to learn weight parameters and quantization parameters, and a second step of training the neural network using low-bit operations of 8 bits or less to learn weight parameters and quantization parameters.

[0007] [2] Furthermore, one aspect of the present invention is the learning method described in [1] above, wherein the first step is performed using a learning detection unit that is different from the detection unit used during inference.

[0008] [3] Furthermore, one aspect of the present invention is the learning method described in [2] above, wherein the extraction unit includes a plurality of convolution operation layers and quantization operation layers connected in series, and when the first step in the learning stage is executed, output information of the final stage of the plurality of convolution operation layers and quantization operation layers connected in series is input to the learning detection unit, and when the second step in the learning stage is executed and in the inference stage, output information of each of the convolution operation layers and quantization operation layers is input to the detection unit.

[0009] [4] Furthermore, one aspect of the present invention is a learning method according to any one of [1] to [3] above, wherein the first step and the second step are performed based on training data in which an image is at least associated with a class of an object depicted in the image, and the amount of training data used in the first step is 100 times or more the amount of training data used in the second step.

[0010] [5] Moreover, one aspect of the present invention is a learning method according to any one of [1] to [4] above, wherein the first step trains a system to solve a classification problem for determining a class of an object depicted in the input image, and the second step trains a system to solve an object detection problem for determining the class and coordinates of an object depicted in the input image.

[0011] [6] Moreover, one aspect of the present invention is a learning device that includes a learning unit that trains a neural network having an extraction unit that extracts features from an input image and a detection unit that detects the class and coordinates of an object in the input image based on the extracted features, in which, as a first step, the learning unit trains the extraction unit using low-bit operations of 8 bits or less to learn weight parameters and quantization parameters as learning objects, and after the first step is performed, as a second step, the learning unit trains the neural network using low-bit operations of 8 bits or less to learn weight parameters and quantization parameters as learning objects. Effect of the Invention

[0012] According to the present invention, it is possible to achieve both a lightweight machine learning model and high inference accuracy. [Brief description of the drawings]

[0013] [Figure 1] FIG. 2 is a diagram illustrating an example of a functional configuration of a learning device according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a functional configuration of a calculation unit according to the embodiment. [Diagram 3] FIG. 1 is a diagram illustrating an example of a configuration of a convolutional neural network according to an embodiment. [Figure 4] 1 is a flowchart illustrating an example of a learning method according to an embodiment. [Diagram 5] FIG. 2 is a diagram for explaining a first step of the learning method according to the embodiment. [Figure 6] FIG. 11 is a diagram for explaining a second step of the learning method according to the embodiment. [Figure 7]2 is a block diagram showing an example of an internal configuration of a learning device according to an embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] [Embodiment] Hereinafter, the learning method and learning device according to the aspects of the present invention will be described in detail with reference to the accompanying drawings, showing preferred embodiments. Note that the aspects of the present invention are not limited to these embodiments, and include various modifications or improvements. In other words, the components described below include those that a person skilled in the art can easily imagine, and those that are substantially the same, and the components described below can be appropriately combined. In addition, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the present invention. In addition, in the following drawings, the scale and number of each structure may be different from the scale and number of the actual structure in order to make each structure easier to understand.

[0015] 1 is a diagram showing an example of the functional configuration of a learning device according to an embodiment. With reference to the figure, an example of the functional configuration of the learning device 10 will be described. The learning device 10 generates and trains a convolutional neural network. In the following description, a person who generates and trains a convolutional neural network using the learning device 10 may be referred to as a user.

[0016] The learning device 10 includes a learning control unit 100, an operation receiving unit 300, a presentation unit 400, a data input unit 500, a storage unit 600, a calculation unit 700, and a data output unit 800. Each of these functional units is realized, for example, by using an electronic circuit. Each functional unit may include a storage means such as a semiconductor memory or a magnetic hard disk device, as necessary. Each function may be realized by a computer having a CPU (Central Processing Unit) and software. All or a part of each functional unit may be realized by hardware (e.g., a circuit unit; circuitry) such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field-Programmable Gate Array). All or a part of each functional unit may be realized by a combination of software and hardware.

[0017] In addition, all or part of the functions of the learning device 10 may be realized using an external accelerator such as a CPU, GPU, or hardware provided in an external device such as a cloud server. All or part of the functions of the learning device 10 may be configured to improve the calculation speed by using, for example, a GPU or dedicated hardware with high calculation performance on a cloud server. In addition, the learning device 10 may be composed of multiple devices (computers), and each functional block may be distributed among the multiple devices.

[0018] The operation receiving unit 300 receives input operations from a user. The operation receiving unit 300 receives the input operations in response to the user operating an input device such as a keyboard, a mouse, a touch panel, or a microphone (not shown). The operation receiving unit 300 outputs information related to the operation acquired from the user to the learning control unit 100 as operation information OPI.

[0019] The presentation unit 400 presents information to the user. The presentation unit 400 presents information to the user based on the presentation information PRI output from the learning control unit 100. The presentation unit 400 presents information visually to the user by outputting information to a display device (not shown), such as a liquid crystal display or an organic EL (Electroluminescence) display. The presentation unit 400 may also present information audibly to the user by outputting information to an audio output device (not shown), such as a speaker.

[0020] The presentation unit 400 may present a GUI (Graphical User Interface) image generated by the calculation unit 700, a console screen for receiving commands, etc. Furthermore, when the calculation unit 700 requires information input from a user, the presentation unit 400 may present to the user a message urging the user to input information by operating an input device (not shown). Furthermore, the presentation unit 400 may present a GUI image, etc., required for the user to input information.

[0021] The learning control unit 100 performs overall control regarding the generation and learning of the convolutional neural network. The learning control unit 100 acquires operation information OPI from the operation reception unit 300, and outputs presentation information PRI to the presentation unit 400. The learning control unit 100 also exchanges control signals CI among the data input unit 500, the storage unit 600, the calculation unit 700, and the data output unit 800, thereby performing overall control regarding the generation and learning of the convolutional neural network.

[0022] The data input unit 500 receives input of hardware information HW, network information NW, etc., necessary for generating a neural network circuit. The neural network circuit is a circuit in which a convolutional neural network generated or learned by the learning device 10 can be incorporated into an IoT (Internet of Things) device or the like. The hardware information HW, network information NW, etc., input to the data input unit 500 are input as data described in a predetermined data format, for example. The input hardware information HW, network information NW, etc., are stored in the storage unit 600. The hardware information HW, network information NW, etc., may be input or changed by a user via the operation reception unit 300. The data input unit 500 outputs the hardware information HW, network information NW, etc., to the storage unit 600 as input information II.

[0023] The storage unit 600 includes, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, a read only memory (ROM), etc. The storage unit 600 stores hardware information HW, network information NW, etc. input by the data input unit 500. Information to be stored in the storage unit 600 is acquired from the data input unit 500 as input information II. The storage unit 600 may also store information obtained as a result of the learning device 10 performing learning.

[0024] The calculation unit 700 performs a predetermined calculation to generate and learn a convolutional neural network. Information used in the calculation is stored in the storage unit 600 and acquired as stored information MI. The calculation unit 700 generates a functional model of the convolutional neural network. The calculation unit 700 may also generate a neural network circuit capable of incorporating the functional model of the convolutional neural network into an IoT device or the like. Furthermore, the calculation unit 700 may generate software that operates the neural network circuit.

[0025] The data output unit 800 acquires information obtained as a result of the calculation by the calculation unit 700 as calculation result information RI, and outputs the acquired information to a user, a predetermined storage device, etc. The information obtained as a result of the calculation by the calculation unit 700 may be learned parameters related to a functional model of the convolutional neural network, information related to a neural network circuit, software, etc.

[0026] FIG. 2 is a diagram showing an example of the functional configuration of the calculation unit according to the embodiment. With reference to the figure, an example of the functional configuration of the calculation unit 700 will be described. First, information used by the calculation unit 700 for calculation will be described. Information used by the calculation unit 700 for calculation is stored in the storage unit 600. The storage unit 600 stores network information NW1, inference network information NW2, a learning data set DS, and learned parameters PM. The learning data set DS and the inference network information NW2 are input data input to the learning device 10. The learned parameters PM are output data output by the learning device 10.

[0027] The network information NW1 is information about a convolutional neural network used for learning. The network information NW1 includes, for example, information that defines the function of the convolutional neural network. Specifically, the network information NW1 may include the network configuration of the convolutional neural network, input data information, output data information, quantization information, and the like. The input data information may include the input data type, such as an image or a voice, and the input data size, and the like. In addition, the network information NW1 may be information about the convolutional neural network that is the target of learning by the learning device 10, and is not limited to information generated by the learning device 10. In other words, the network information NW1 may be information generated by a device other than the learning device 10.

[0028] The inference network information NW2 is information about the inference calculation executed by the neural network circuit. The inference network information NW2 includes, for example, information that defines the function of the inference calculation of the neural network that can be executed by the neural network circuit. Specifically, the inference network information NW2 may include the circuit configuration of the neural network circuit, the function of the arithmetic unit, the data bit width, etc. In addition, in contrast to the inference network information NW2, the network information NW1 can also be said to be learning network information for learning.

[0029] The learning dataset DS includes teacher data used for learning. The learning dataset DS also broadly includes learning data used for learning and test data used for inference testing.

[0030] The trained parameters PM are parameters obtained as a result of training. The trained parameters PM include various parameters that are trained by the training device 10. The trained parameters PM may be, for example, weight parameters, quantization parameters, batch normalization processing, scale processing, and the like.

[0031] The calculation unit 700 includes a learning unit 710, an inference unit 720, a functional model generation unit 730, and a software generation unit 740. Each of these functional units is realized, for example, by using an electronic circuit. Each functional unit may include a storage means such as a semiconductor memory or a magnetic hard disk device, as necessary. Each function may be realized by a computer having a CPU and software. All or a part of each functional unit may be realized by hardware (e.g., a circuit unit; circuitry) such as an ASIC, a PLD, or an FPGA. All or a part of each functional unit may be realized by a combination of software and hardware.

[0032] The learning unit 710 performs learning of the convolutional neural network using the network information NW1, the inference network information NW2, and the learning data set DS stored in the storage unit 600. As a result of the learning performed by the learning unit 710, a learned parameter PM is generated.

[0033] The inference unit 720 performs inference using the inference network information NW2 stored in the storage unit 600. Data used for inference is input by the user via, for example, the operation reception unit 300. In addition, a data set used when inference is performed as an inference test may be stored in the storage unit 600 as a learning data set DS.

[0034] The functional model generating unit 730 generates (configures) a functional model of a convolutional neural network based on a request from a user. The functional model generating unit 730 outputs network information NW1, which is information on the functional model of the convolutional neural network.

[0035] The software generation unit 740 generates software SW for operating the neural network circuit based on the network information NW1 and the inference network information NW2. The generated software SW may include software for transferring the learned parameters PM to the neural network circuit as necessary.

[0036] The data output unit 800 outputs the network information NW1, the learned parameters PM, and the software SW output by the calculation unit 700.

[0037] 3 is a diagram showing an example of the configuration of a convolutional neural network according to an embodiment. With reference to the diagram, an example of a convolutional neural network 200 that is the learning target of the learning device 10 will be described.

[0038] The convolutional neural network 200 is a multi-layer network including a convolutional layer 210 that performs a convolutional operation, a quantization operation layer 220 that performs a quantization operation, and an output layer 230. In at least a part of the convolutional neural network 200, the convolutional layer 210 and the quantization operation layer 220 are alternately connected. The convolutional neural network 200 is a model that is widely used in image recognition and video recognition. The convolutional neural network 200 may further include a layer having other functions, such as a fully connected layer.

[0039] The convolution layer 210 performs a convolution operation on the input data a using a weight w. The convolution layer 210 performs a multiply-and-accumulate operation with the input data a and the weight w as input. The input data a (also called activation data or feature map) to the convolution layer 210 is multidimensional data such as image data. In this embodiment, the input data a may be a three-dimensional tensor consisting of elements (x, y, c). The convolution layer 210 performs a convolution operation on the low-bit input data a. In this embodiment, the elements of the input data a are 2-bit unsigned integers (0, 1, 2, 3). The elements of the input data a may be, for example, 4-bit or 8-bit integers including sign information.

[0040] In addition, when the input data input to the convolutional neural network 200 has a format different from that of the input data a to the convolutional layer 210, such as a 32-bit floating-point type, the convolutional neural network 200 may further have an input layer that performs type conversion and quantization before the convolutional layer 210.

[0041] The weights w (also called filters or kernels) of the convolution layer 210 are multidimensional data having elements that are learnable parameters. In this embodiment, the weights w may be, for example, four-dimensional tensors. The weights w may also have, for example, d three-dimensional tensors. The convolution layer 210 performs a convolution operation using low-bit weights w. In this embodiment, the elements of the weights w are 1-bit signed integers (0,1), with a value of "0" representing +1 and a value of "1" representing -1. The convolution layer 210 outputs output data f obtained as a result of the convolution operation to the quantization operation layer 220.

[0042] The quantization operation layer 220 performs a quantization operation and the like on the output of the convolution operation output by the convolution layer 210. The quantization operation layer 220 includes a pooling layer 221, a batch normalization layer 222, an activation function layer 223, and a quantization layer 224.

[0043] The pooling layer 221 performs calculations such as average pooling and MAX pooling on the output data f of the convolution calculation output by the convolution layer 210. The pooling layer 221 compresses the output data f of the convolution layer 210 as a result of the calculation.

[0044] The batch normalization layer 222 performs calculations on the output data of the quantization calculation layer 220 and the output data of the pooling layer 221, thereby normalizing the data distribution.

[0045] The activation function layer 223 performs calculations on the outputs of the quantization calculation layer 220, the pooling layer 221, and the batch normalization layer 222 using an activation function such as a ReLU (Rectified Linear Unit) function.

[0046] The quantization layer 224 performs a quantization operation on the outputs of the pooling layer 221 and the activation function layer 223 based on the quantization parameter.

[0047] The output layer 230 outputs the result of the convolutional neural network 200 using an identity function, a softmax function, etc. The layer preceding the output layer 230 may be the convolutional layer 210 or the quantization operation layer 220.

[0048] In the convolutional neural network 200, the quantized output data of the quantization layer 224 is input to the convolutional layer 210, so the load of the convolutional calculation in the convolutional layer 210 is smaller than that of other convolutional neural networks that do not perform quantization.

[0049] Next, a learning method of the learning device 10 according to this embodiment will be described with reference to FIGS.

[0050] First, a brief explanation of the learning method using conventional technology will be given. Conventionally, most neural networks for image processing consist of a module called a backbone that obtains a feature map from an input image, and a module designed for each task from the obtained feature map. For example, in the case of an object detection model, the backbone is connected to modules with structures specific to object detection, called necks and heads.

[0051] Among conventional neural networks having such a configuration, there are lightweight neural networks in which weights, input data, intermediate data, etc. are quantized to, for example, 8 bits or less. Conventionally, PTQ (Post Training Quantization) and QAT (Quantization Aware Training) are known as methods for creating such lightweight neural networks. PTQ is a method for quantizing float parameters that have been learned without quantization. QAT is a method for simulating quantization during learning to perform optimization.

[0052] Currently, parameters themselves that have been trained in a state of sufficient numerical precision such as Float using a large-scale data set and a GPU (Graphics Processing Unit) with sufficient resources have already been made public. Examples of pre-trained models that have already been made public include Resnet and the like. Therefore, according to PTQ, since the backbone can be quantized by reusing such already existing parameters, re-learning is not necessary. However, according to PTQ, when the trained parameters are reduced in bit size, errors associated with quantization occur. Here, according to QAT, since learning is performed with low bits, there is no need to perform re-quantization after learning, and the accuracy degradation due to the reduction in bit size as in PTQ can be suppressed. However, according to PTQ, there is a problem that re-learning is required. However, there is a problem that the accuracy is significantly reduced due to errors associated with quantization when reducing the bit size of parameters by PTQ. On the other hand, by learning a model reduced in bit size using QAT, the accuracy degradation due to the reduction in bit size can be suppressed.

[0053] According to this embodiment, a large-scale data set and a GPU with sufficient resources are used to perform learning at low bits without using parameters trained in a state of sufficient numerical precision such as Float. According to this embodiment, first, learning of the backbone is performed at low bits by QAT by solving a classification task, and then learning of the entire neural network including the backbone pre-trained by QAT as well as initialized necks and heads is performed, thereby achieving both weight reduction of the neural network and inference accuracy. In this embodiment, the low bits may be 12 bits or less, and preferably 8 bits or less. In this embodiment, the low bits are more preferably extremely low bits such as 2 bits or 1 bit. Furthermore, it is preferable that the learning according to this embodiment is performed with a bit number that is a power of 2 (for example, 8 bits, 4 bits, or 2 bits). This is because learning is performed with a bit number that is a power of 2, and the internal circuit, such as the bus width of the memory when reading out data, is designed with a bit number that is a power of 2. Therefore, learning with a bit number that is a power of 2 makes it possible to efficiently transfer data.

[0054] The learning method according to this embodiment is a method for training a neural network that includes a backbone that extracts features from an input image, and a neck and a head that detect the class and coordinates of an object captured in the input image based on the extracted features. In the following description, the backbone may be referred to as an extraction unit, and the configuration that includes the neck and head may be referred to as a detection unit.

[0055] FIG. 4 is a flowchart showing an example of a learning method according to the embodiment. With reference to the same figure, an example of the learning method according to the embodiment will be described. The learning method according to the embodiment can also be used for a network that uses a backbone for extracting feature amounts such as semantic segmentation for identifying the region of a subject, pose estimation for detecting the posture of an object reflected in an image, action recognition for recognizing the action of an object reflected in an image, and depth estimation for detecting depth from two-dimensional data. In addition, the learning method according to the embodiment can be said to be an example of the processing by the learning unit 710 described above.

[0056] (Step S11) First, the extraction unit quantized by QAT is trained. The model trained at this time includes a trainable layer for matching the output of the classification task in addition to the extraction unit. This process is also referred to as the first step.

[0057] (Step S12) Next, an extraction unit for the parameters learned in step 11 and a subsequent detection unit are prepared, and learning is performed using QAT. This process is also referred to as the second step. In the second step, learning is performed using a dataset for a task other than classification. The dataset for a task other than classification may be, for example, object detection, semantic segmentation, etc.

[0058] 5 is a diagram for explaining the first step of the learning method according to the embodiment. With reference to the figure, an example of the learning method in the first step will be described. In the first step of the learning stage, an extraction unit 20 to be learned and a detection unit for learning 31 are used as the configuration of the convolutional neural network.

[0059] The extraction unit 20 is the backbone. An input image (Input) is input to the extraction unit 20. The input image input to the extraction unit 20 may be a patch image obtained by cutting out an image for each predetermined number of pixels. The extraction unit 20 includes an input layer, a convolution operation layer, a quantization layer, an output layer, and the like, as described with reference to FIG. 3. A plurality of convolution operation layers and quantization layers may be provided. The extraction unit 20 may also be said to include a plurality of convolution operation layers and quantization operation layers connected in series. In the illustrated example, the extraction unit 20 includes a first layer 21 to a fourth layer 24. Output information of the final stage of a plurality of convolution operation layers and quantization operation layers connected in series (for example, it may be output information of an output layer, and in the illustrated example, it is output information of the fourth layer 24) is input to the learning detection unit 31.

[0060] The learning detection unit 31 includes at least a classification output layer 313. The learning detection unit 31 is different from the detection unit used during inference. The learning detection unit 31 is preferably lighter than the detection unit used during inference. The classification output layer 313 performs a series of differentiable processes required to make the output of the detection unit correspond to the final output of the classification problem. More specifically, the learning detection unit 31 may not use a fully connected layer, but may apply a convolution layer with the number of output channels equal to the number of classes to the output of the detection unit, and then apply an average pooling layer to the output as the final output.

[0061] 6 is a diagram for explaining the second step of the learning method according to the embodiment. With reference to the same figure, an example of the learning method in the second step will be described. In the second step of the learning stage, an extraction unit 20 to be learned and a detection unit 30 are used as the configuration of the convolutional neural network. Specifically, the detection unit 30 includes a neck 40 and a head 50.

[0062] The extraction unit 20 is the backbone used in the first step. An input image (Input) is input to the extraction unit 20. The input image input to the extraction unit 20 may be a patch image cut out from an image for each predetermined number of pixels. Output information of each layer (e.g., a convolution operation layer and a quantization operation layer) included in the extraction unit 20 is input to the detection unit 30. Specifically, in the example shown in the figure, output information of the second layer 22 to the fourth layer 24 is input to the detection unit 30.

[0063] The neck 40 includes a first layer 41 to a third layer 43. Each layer of the neck 40 includes, for example, a plurality of convolution operation layers and an upsampling layer. The neck 40 acquires information from the extraction unit 20 and performs upsampling based on the acquired information. The neck 40 outputs the result of the upsampling to the head 50.

[0064] The head 50 includes a first layer 51 to a third layer 53. The head 50 estimates the class and coordinates (bounding box) based on the output result from the neck 40. In this embodiment, a case where a configuration of FPN (Feature Pyramid Networks) is adopted will be described, but the network structure according to this embodiment is not limited to this example.

[0065] Next, the difference between the first step and the second step in the learning stage will be described. In both the first step and the second step, learning is performed based on teacher data in which at least an image and a class of an object captured in the image are associated with each other. That is, the type of teacher data is common to the first step and the second step. However, the number of teacher data used in learning is different between the first step and the second step. Specifically, in the first step, large-scale learning is performed, for example, learning is performed based on about several million images. On the other hand, in the second step, small-scale (or medium-scale) learning is performed, for example, learning is performed based on about tens of thousands of images, which is different from the first step. That is, it can be said that the number of teacher data used in the first step is 100 times or more the number of teacher data used in the second step.

[0066] In addition, the first step and the second step have different problems to be solved in learning. Specifically, in the first step, the system learns to solve a classification problem for determining the class of an object captured in an input image. Meanwhile, in the second step, the system learns to solve an object detection problem for determining the class and coordinates of an object captured in an input image. That is, in the first step, the system determines the class of the object, and does not learn the location where the object exists, such as a bounding box. Meanwhile, in the second step, the system learns not only the class of the object captured in the input image, but also the location where the object exists, such as a bounding box. In other words, even if the same image is included as the input image, the first step and the second step have different training data because the target of learning is different.

[0067] The configuration in the second step of the learning stage shown in the figure may also be used in the inference process.

[0068] FIG. 7 is a block diagram showing an example of the internal configuration of the learning device according to this embodiment. At least some of the functions of the learning device 10 can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions included in a program read from the RAM 902 or the like. The central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations according to each instruction. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. The input / output port 903 is a port through which the central processing unit 901 exchanges data with an external input / output device or the like. The input / output devices 904 and 905 are input / output devices. The input / output devices 904 and 905 exchange data with the central processing unit 901 via the input / output port 903. The bus 906 is a common communication path used inside the computer. For example, the central processing unit 901 reads and writes data from the RAM 902 via the bus 906. For example, the central processing unit 901 accesses the input / output port via the bus 906. In addition, all or part of each functional unit provided in the learning device 10 may be realized using hardware (e.g., a circuit unit; circuitry) such as an ASIC, a PLD, or an FPGA. In addition, all or part of each functional unit provided in the learning device 10 may be realized by a combination of software and hardware.

[0069] [Summary of this embodiment] According to the embodiment described above, the learning method according to this embodiment is a learning method for training a neural network including an extraction unit 20 that extracts a feature amount from an input image and a detection unit 30 that detects the class and coordinates of an object captured in the input image based on the extracted feature amount. In addition, the learning method according to this embodiment has a first step, whereby the extraction unit 20 is trained (i.e., the backbone is trained by QAT) with weight parameters and quantization parameters as learning objects by low-bit operations of 8 bits or less, and has a second step, whereby the neural network is trained (i.e., the entire neural network is trained by QAT) with weight parameters and quantization parameters as learning objects by low-bit operations of 8 bits or less. According to this embodiment, since learning is performed by low-bit operations, it is not necessary to reduce the bits of a trained model that has been trained in advance. That is, according to this embodiment, there is no decrease in inference accuracy when quantization is performed. Therefore, according to this embodiment, both weight reduction and calculation accuracy can be achieved.

[0070] Moreover, according to the learning method of the embodiment described above, in the first step, learning is performed using the detection unit for learning 31 that is different from the detection unit 30 used during inference. The detection unit for learning 31 includes, for example, a fully connected layer 311 and an output layer 312, and is lighter than the detection unit 30 that includes a plurality of convolution operation layers and quantization operation layers. That is, according to this embodiment, learning is performed using the detection unit for learning 31 that is lighter than the detection unit 30 used during inference. Therefore, according to this embodiment, it is possible to achieve both weight reduction and calculation accuracy.

[0071] According to the learning method of the embodiment described above, the extraction unit 20 includes a plurality of convolution operation layers and quantization operation layers connected in series. When the first step is executed in the learning stage, as shown in FIG. 5, output information of the final stage of the plurality of convolution operation layers and quantization operation layers connected in series is input to the learning detection unit 31. When the second step is executed in the learning stage and in the inference stage, output information of each of the convolution operation layers and quantization operation layers is input to the detection unit 30 as shown in FIG. 6. That is, according to this embodiment, when the first step is executed in the learning stage, learning is performed using the learning detection unit 31, which is lighter than the detection unit 30 used in the inference, and when the second step is executed, learning is performed using the detection unit 30 used in the inference. Therefore, according to this embodiment, it is possible to achieve both weight reduction and calculation accuracy.

[0072] According to the learning method of the above-described embodiment, the first step and the second step are performed based on teacher data in which at least an image and a class of an object captured in the image are associated with each other. The number of images in the teacher data used in the first step is 100 times or more the number of teacher data used in the second step. Specifically, the number of teacher data used in the first step may be about several million, and the number of teacher data used in the second step may be about several hundred to tens of thousands. It can be said that the first step is large-scale learning, and the second step is small-scale to medium-scale learning. According to this embodiment, the first step ensures calculation accuracy by performing large-scale learning with low bits, and the second step performs small-scale to medium-scale learning with low bits to reduce the weight. Therefore, according to this embodiment, it is possible to achieve both weight reduction and calculation accuracy.

[0073] Moreover, according to the learning method of the embodiment described above, the first step is to learn to solve a classification problem for determining the class of an object captured in an input image, and the second step is to learn to solve an object detection problem for determining the class and coordinates of an object captured in an input image. That is, according to this embodiment, by not determining the coordinates in the first step, the network is composed of only an extraction unit and a lightweight classification output layer, and the learning time can be shortened.

[0074] In addition, all or part of the functions of each unit of the learning device according to the above-mentioned embodiment may be realized by recording a program for realizing these functions on a computer-readable recording medium, reading the program recorded on the recording medium into a computer system, and executing it. Note that the "computer system" referred to here includes hardware such as the OS and peripheral devices.

[0075] In addition, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage units such as hard disks built into computer systems. Furthermore, the term "computer-readable recording medium" may also include those that dynamically hold a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and those that hold a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in such cases. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0076] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the spirit of the present invention. [Explanation of symbols]

[0077] 10...learning device, 100...learning control unit, 200...convolutional neural network, 300...operation reception unit, 400...presentation unit, 500...data input unit, 600...storage unit, 700...calculation unit, 800...data output unit, 710...learning unit, 720...inference unit, 730...functional model generation unit, 740...software generation unit, NW1...network information, NW2...inference network information, DS...learning data set, PM...learned parameters, SW...software, 20...extraction unit, 30...detection unit

Claims

1. A learning method for training a neural network including an extraction unit that extracts a feature amount from an input image, and a detection unit that detects a class and coordinates of an object captured in the input image based on the extracted feature amount, comprising: a first step of training the extraction unit using a low-bit operation of 8 bits or less, with weight parameters and quantization parameters as training objects; a second step of training the neural network using low-bit operations of 8 bits or less, with weight parameters and quantization parameters as training objects; A learning method that has

2. In the first step, learning is performed using a learning detection unit different from the detection unit used during inference. The learning method according to claim 1 .

3. The extraction unit includes a plurality of convolution operation layers and quantization operation layers connected in series; When the first step in the learning stage is executed, output information of a final stage of a plurality of convolution operation layers and quantization operation layers connected in series is input to the learning detection unit; When the second step is executed in the learning stage and in the inference stage, output information of each of the convolution operation layers and the quantization operation layers is input to the detection unit. The learning method according to claim 2 .

4. In the first step and the second step, learning is performed based on teacher data in which an image is associated with at least a class of an object depicted in the image; The number of pieces of training data used in the first step is 100 times or more the number of pieces of training data used in the second step. A learning method according to any one of claims 1 to 3.

5. The first step is to train the system to solve a classification problem for determining a class of an object captured in the input image; The second step is to train the system to solve an object detection problem that determines the class and coordinates of an object captured in the input image. A learning method according to any one of claims 1 to 3.

6. a learning unit that trains a neural network including an extraction unit that extracts a feature from an input image and a detection unit that detects a class and coordinates of an object captured in the input image based on the extracted feature; The learning unit, as a first step, causes the extraction unit to learn by using a weight parameter and a quantization parameter as learning objects through low-bit arithmetic of 8 bits or less; After the first step is performed, in the second step, the neural network is trained using low-bit arithmetic of 8 bits or less, with weight parameters and quantization parameters as training objects. Learning device.

Images