Learning device, learning method, and program

The adaptive control of regularization in deep learning using weighting and rescaling functions addresses uniform regularization issues, enhancing learning efficiency and generalization by balancing weight contributions and preventing overfitting.

JP7586335B2Active Publication Date: 2024-11-19NEC CORP
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023549281
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-09-27
Publication Date
2024-11-19
Estimated Expiration
2041-09-27

AI Technical Summary

Technical Problem

Conventional regularization methods in deep learning apply uniform strength across all training data, leading to weaker regularization for easy data and stronger regularization for difficult data, resulting in overfitting and reduced learning efficiency.

Method used

Adaptive control of regularization strength by using a weighting function that increases rapidly for over- or under-confident class scores and a rescaling function that increases slowly, calculated in mini-batch units to adjust weight contributions and overall regularization.

Benefits of technology

Enhances learning efficiency by dynamically adjusting regularization based on data difficulty, preventing overfitting and improving generalization performance through balanced weight normalization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007586335000006
    Figure 0007586335000006
  • Figure 0007586335000007
    Figure 0007586335000007
  • Figure 0007586335000008
    Figure 0007586335000008
Patent Text Reader

Abstract

In this learning device, an estimation means performs inference for training data using an inference model, and outputs a class score. A weight calculation means calculates a weight from the output class score by using a weight function that increases more rapidly than a linear increase when the class score is too large or too small. A weight sum calculation means calculates a sum of weights for a mini-batch including a predetermined number of pieces of training data. A regularization term calculation means calculates a regularization term by applying, to the sum, a rescaling function defined to be a monotonously increasing function that increases more slowly than a linear increase. An optimization means optimizes the inference model by using a total loss including the regularization term.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a method for training a machine learning model. [Background technology]

[0002] It is known that regularization is performed to suppress overfitting when training a large-scale machine learning model such as deep learning. For example, Patent Document 1 discloses a method for updating weight parameters of a neural network using a cost function in which a regularization term is added to an error function. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-43596 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional methods apply regularization uniformly to all training data, which can result in weak regularization for training data that is easy to predict, leading to overfitting, or strong regularization for training data that is difficult to predict, resulting in reduced learning efficiency.

[0005] One objective of the present disclosure is to adaptively control the strength of regularization in deep learning depending on training data. [Means for solving the problem]

[0006] In one aspect of the present disclosure, a learning device includes: an inference means for performing inference on the training data using the inference model and outputting a class score; a weight calculation means for calculating a weight from the output class score using a weight function that increases more rapidly than linearly when the class score is too large or too small; a weight summation calculation means for calculating the sum of the weights over a mini-batch containing a predetermined number of training data; a regularization term calculation means for calculating a regularization term by applying a rescaling function, which is a monotonically increasing function that increases more slowly than linearly, to the sum; An optimization means for optimizing the inference model using a total loss including the regularization term; Equipped with.

[0007] In another aspect of the present disclosure, a method of learning comprises: Use the inference model to make inferences on the training data and output class scores. Calculating weights from the output class scores using a weighting function that increases more rapidly than linearly when the class scores are too high or too low; Calculate the sum of the weights over a mini-batch containing a predetermined number of training data; applying a rescaling function to the sum to calculate a regularization term; The total loss, including the regularization term, is used to optimize the inference model.

[0008] In yet another aspect of the present disclosure, program teeth, Use the inference model to make inferences on the training data and output class scores. Calculating weights from the output class scores using a weighting function that increases more rapidly than linearly when the class scores are too high or too low; Calculate the sum of the weights over a mini-batch containing a predetermined number of training data; applying a rescaling function to the sum to calculate a regularization term; A process of optimizing the inference model using a total loss including the regularization term is performed by a computer. 。 Effect of the Invention

[0009] According to the present disclosure, in deep learning, it becomes possible to adaptively control the strength of regularization depending on the training data. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a hardware configuration of a learning device according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing the functional configuration of the learning device of the first embodiment. [Diagram 3] Examples of weighting and rescaling functions are given below. [Figure 4] 4 is a flowchart of a learning process performed by the learning device of the first embodiment. [Diagram 5] FIG. 11 is a block diagram showing the functional configuration of a learning device according to a second embodiment. [Figure 6] 10 is a flowchart of a learning process performed by a learning device according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings. First Embodiment [Learning device] (Hardware configuration) 1 is a block diagram showing the hardware configuration of a learning device 100 according to a first embodiment. As shown in the figure, the learning device 100 includes an interface (I / F) 11, a processor 12, a memory 13, a recording medium 14, and a database (DB) 15.

[0012] The interface 11 inputs and outputs data to and from an external device. Specifically, a training data set used for learning is input to the learning device 100 through the interface 11.

[0013] The processor 12 is a computer such as a CPU (Central Processing Unit) and controls the entire learning device 100 by executing a program prepared in advance. The processor 12 may be a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). The processor 12 executes a learning process, which will be described later.

[0014] The memory 13 is composed of a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 13 is also used as a working memory while the processor 12 is executing various processes.

[0015] The recording medium 14 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or a semiconductor memory, and is configured to be detachable from the learning device 100. The recording medium 14 records various programs executed by the processor 12. When the learning device 100 executes various processes, the programs recorded on the recording medium 14 are loaded into the memory 13 and executed by the processor 12. The DB 15 stores a training dataset input via the I / F 11 as necessary.

[0016] (Functional configuration) 2 is a block diagram showing the functional configuration of the learning device 100 according to the first embodiment. The learning device 100 includes an inference unit 21, a loss function calculation unit 22, a summation calculation unit 23, a weighting function calculation unit 24, a weighted summation calculation unit 25, a rescaling function calculation unit 26, and a parameter update unit 27.

[0017] A training data set is input to the learning device 100. The training data set includes a plurality of training data x i And for each training data x i The correct class y corresponding to i The training data x i is input to the inference unit 21, and the correct class y i is input to the loss function calculation unit 22.

[0018] The inference unit 21 performs inference using a deep learning model that is the subject of learning by the learning device 100. Specifically, the inference unit 21 includes a neural network that constitutes the deep learning model that is the subject of learning. The inference unit 21 performs inference using input training data x i We perform inference on v and obtain the class score v → i In detail, the inference unit 21 outputs the training data x i Then, we classify the classes and obtain the class score v → i In this specification, for convenience, the vector "→" is written as a superscript to the right of "v". → i is input to the loss function calculation unit 22 and the weight function calculation unit 24.

[0019] The loss function calculation unit 22 calculates the class score v by using a loss function prepared in advance. → i Loss on l cls,i Specifically, the loss function calculation unit 22 calculates the loss function x i Class score for v → i and the training data x i The correct class for y i Using the above, the loss l is calculated as shown in equation (1). cls,i Calculate the calculated loss l cls,i is input to the sum calculation unit 23.

[0020]

number

[0021] On the other hand, the weight function calculation unit 24 calculates the class score v generated by the inference unit 21. → i Based on the training data x i Specifically, the weight function calculation unit 24 calculates the weight for the training data xi The class score v is the inference result for → i Then, the weight w, which is a single real value, is calculated by the following formula (2). i Determine.

[0022]

number

[0023] As a weighting function, the class score v → i A function is chosen that grows rapidly when the confidence scores of each class in v are over or underestimated. "Rapidly" means faster than linearly. The condition for the weight function to grow rapidly is that the class score v → i This is necessary to emphasize over- or under-confidence scores in v. That is, by using a fast-growing function to calculate the weights, → i If contains over- or under-valued confidence score values, those over- or under-valued values ​​are emphasized and weighted by w i is a larger value. Thus, the selection of the weight function determines the contribution of the weight of each training data to the gradient of the regularization term described later. Note that the weight function calculation unit 24 does not simply calculate the class score v → i The confidence scores of each class in are input to the weight function, and the output weights are i The value of w is not a normalized value. i to the weight sum calculation unit 25.

[0024] The weight sum calculation unit 25 calculates the weight w i A mini-batch is a set of a predetermined number (e.g., N) of training data. Specifically, the weight sum calculation unit 25 calculates the sum of N training data x i The N weights w i Calculate the sum S of

[0025]

number

[0026] The weight sum calculation unit 25 outputs the calculated sum S to the rescaling function calculation unit .

[0027] The rescaling function calculation unit 26 calculates the rescaling function based on the input sum S to obtain the normalization term L reg Specifically, the rescaling function calculation unit 26 generates a normalization term L reg Generate.

[0028]

number

[0029] In formula (4), "g(S)" is a rescaling function. A slowly increasing monotonically increasing function is selected as the rescaling function g(S). Note that this slowly increasing monotonically increasing function is different from the mathematical "slowly increasing function."

[0030] Here, "gradual" means slower than linear. The condition that the rescaling function g(S) is gradual is necessary to prevent a rapidly growing weight function from increasing the gradient of the regularization term, which results in unstable learning. In other words, the weights w i If we use it as it is, the regularization may be too strong, so we use the rescaling function g(S) to set the weight w i In this respect, the rescaling function g(S) adjusts the overall scale of the weights w i It can be considered that the rescaling function calculation unit 26 normalizes the normalization term L obtained in this way and adjusts the strength of the overall regularization. reg is output to the sum calculation unit 23.

[0031] The summation calculation unit 23 calculates the loss l input from the loss function calculation unit 22. cls,i and the normalization term L input from the rescaling function calculation unit 26. reg Specifically, the sum calculation unit 23 calculates the loss l by the following formula (5): cls,i and the regularization term L reg The total loss L is calculated by adding the sum of the i training data and dividing the sum by the number of training data N contained in the mini-batch.

[0032]

number

[0033] Then, the sum calculation unit 23 outputs the obtained total loss L to the parameter update unit 27.

[0034] The parameter update unit 27 optimizes the inference unit 21 based on the input total loss L. Specifically, the parameter update unit 27 updates the parameters of the neural network constituting the inference unit 21 based on the total loss L. In this manner, the deep learning model constituting the inference unit 21 is trained.

[0035] As described above, according to the learning device 100 of the first embodiment, the contribution of each training data to the regularization term can be adaptively determined by calculating the regularization term in units of mini-batches. In addition, the learning device 100 can prevent overfitting by strengthening regularization for simple training data and by weakening regularization for difficult training data by emphasizing excessive or underestimated inference results output by the inference unit 21 using a weight function, thereby improving the efficiency of learning. Furthermore, the learning device 100 can normalize the weight partially emphasized using the weight function by adjusting the overall scale of the weight using a rescale function, thereby adjusting the strength of the overall regularization. As a result, it is possible to adaptively determine the strength of regularization according to the training data and obtain higher generalization performance, i.e., classification accuracy.

[0036] In the above configuration, the inference unit 21 is an example of an inference means, the loss function calculation unit 22 is an example of a loss calculation means, the weight function calculation unit 24 is an example of a weight calculation means, the weight sum calculation unit 25 is an example of a weight calculation means, the rescaling function calculation unit 26 is an example of a regularization term calculation means, and the parameter update unit 27 is an example of an optimization means.

[0037] (Example of a function) Figure 3 shows examples of weighting and rescaling functions. In the first example, the weighting function is → i The confidence score for each class in v ic The rescaling function is a function that calculates the square of the sum S output by the weight summation calculation unit 25.

[0038] In the second example, the weight function is → i The confidence score for each class in v ic The rescaling function is a function that calculates the logarithm of the sum S output by the weight sum calculation unit 25.

[0039] In the third example, the weight function is → i Positive and negative confidence scores for each class in v ic The rescaling function is a function that calculates the logarithm of the sum S output by the weight sum calculation unit 25.

[0040] (Learning process) 4 is a flowchart of the learning process by the learning device 100. This process is realized by the processor 12 shown in FIG. 1 executing a program prepared in advance and operating as each element shown in FIG.

[0041] First, the inference unit 21 calculates the training data x i(Step S11). The inference unit 21 performs inference on the class score v → i to the loss function calculation unit 22 and the weight function calculation unit 24. The loss function calculation unit 22 outputs the class score v → i Based on this, loss l is calculated using equation (1). cls,i and outputs it to the sum calculation unit 23 (step S12).

[0042] Next, the weighting function calculation unit 24 calculates the class score v → i Based on this, the weight w i Then, the weight sum calculation unit 25 calculates the weight w for each mini-batch using the formula (3) and outputs the weight sum calculation unit 25 (step S13). i The rescaling function calculation unit 26 then calculates the sum S of the input sum S and outputs it to the rescaling function calculation unit 26 (step S14). Next, the rescaling function calculation unit 26 calculates the normalization term L from the input sum S using the rescaling function. reg and outputs the result to the sum calculation unit 23 (step S15). Note that the processing of step S12 and steps S13 to S15 may be performed in the reverse order, or may be performed in parallel.

[0043] Next, the summation calculation unit 23 calculates the loss l input from the loss function calculation unit 22. cls,i and the normalization term L input from the rescaling function calculation unit 26. reg Based on this, the total sum of losses (total loss L) is calculated using equation (5) and output to parameter update unit 27 (step S16). Next, parameter update unit 27 updates the parameters of the neural network constituting inference unit 21 based on the total sum of losses (total loss L) (step S17).

[0044] Next, it is determined whether or not a learning termination condition has been met (step S18). The termination condition may be, for example, that all training data has been used, or that the accuracy of the inference unit 21 has reached a predetermined accuracy. If the termination condition has not been met (step S18: No), the process returns to step S11, and the processes of steps S11 to S17 are performed using the next training data. On the other hand, if the termination condition has been met (step S18: Yes), the learning process ends.

[0045] <Second embodiment> 5 is a block diagram showing the functional configuration of the learning device of the second embodiment. The learning device 200 includes an inference means 201, a weight calculation means 202, a weight sum calculation means 203, a regularization term calculation means 204, and an optimization means 205.

[0046] FIG. 6 is a flowchart of the learning process by the learning device 200 of the second embodiment. First, the inference means 201 performs inference on the training data and outputs a class score (step S21). Next, the weight calculation means 202 calculates weights from the class scores output by the inference means 201 using a weight function that increases more rapidly than linearly when the class score is over or underestimated (step S22). Next, the weight sum calculation means 203 calculates the sum of weights over a mini-batch including a predetermined number of training data (step S23). Next, the regularization term calculation means 204 applies a rescaling function, which is a monotonically increasing function that increases more slowly than linearly, to the sum to calculate a regularization term (step S24). Then, the optimization means 205 optimizes the inference means using a loss including the regularization term (step S25).

[0047] According to the learning device 200 of the second embodiment, in deep learning, it becomes possible to adaptively control the strength of regularization according to the training data.

[0048] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes.

[0049] (Appendix 1) an inference means for performing inference on the training data using the inference model and outputting a class score; a weight calculation means for calculating a weight from the output class score using a weight function that increases more rapidly than linearly when the class score is too large or too small; a weight summation calculation means for calculating the sum of the weights over a mini-batch containing a predetermined number of training data; a regularization term calculation means for calculating a regularization term by applying a rescaling function, which is a monotonically increasing function that increases more slowly than linearly, to the sum; An optimization means for optimizing the inference model using a total loss including the regularization term; A learning device comprising:

[0050] (Appendix 2) 2. The learning device according to claim 1, wherein the regularization term calculation means increases a value of the regularization term when the class score is high, and decreases a value of the regularization term when the class score is low.

[0051] (Appendix 3) a loss calculation means for calculating a loss based on the class score and a correct class corresponding to the training data; 3. The learning device according to claim 1, wherein the total loss is a sum of the loss and the regularization term.

[0052] (Appendix 4) The class scores include a confidence score for each class for one training data set; The weighting function is a function that sums the squares of the confidence scores of the classes across all classes; The learning device according to any one of claims 1 to 3, wherein the rescaling function is a function that calculates the square root of the sum.

[0053] (Appendix 5) The class scores include a confidence score for each class for one training data set; The weighting function is a function that sums the natural logarithm of the squared confidence scores of the classes across all classes; The learning device according to any one of claims 1 to 3, wherein the rescaling function is a function that calculates the logarithm of the sum.

[0054] (Appendix 6) The class scores include a confidence score for each class for one training data set; The weighting function is a function that sums the natural logarithm of the confidence scores of each of the classes across all the classes; The learning device according to any one of claims 1 to 3, wherein the rescaling function is a function that calculates the logarithm of the sum.

[0055] (Appendix 7) Use the inference model to make inferences on the training data and output class scores. Calculating weights from the output class scores using a weighting function that increases more rapidly than linearly when the class scores are too high or too low; Calculate the sum of the weights over a mini-batch containing a predetermined number of training data; applying a rescaling function to the sum to calculate a regularization term; A learning method that optimizes the inference model using a total loss including the regularization term.

[0056] (Appendix 8) Use the inference model to make inferences on the training data and output class scores. Calculating weights from the output class scores using a weighting function that increases more rapidly than linearly when the class scores are too high or too low; Calculate the sum of the weights over a mini-batch containing a predetermined number of training data; applying a rescaling function to the sum to calculate a regularization term; A recording medium recording a program that causes a computer to execute a process of optimizing the inference model using a total loss including the regularization term.

[0057] Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above-mentioned embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. [Explanation of symbols]

[0058] 12 processors 21 Reasoning part 22 Loss function calculation section 23 Sum Calculation Section 24 Weighting function calculation unit 25 Weight sum calculation unit 26 Rescale function calculation section 27 Parameter Update Section 100, 200 Learning Device

Claims

1. an inference means for performing inference on the training data using the inference model and outputting a class score; a weight calculation means for calculating a weight from the output class score using a weight function that increases more rapidly than linearly when the class score is too large or too small; a weight summation calculation means for calculating the sum of the weights over a mini-batch containing a predetermined number of training data; a regularization term calculation means for calculating a regularization term by applying a rescaling function, which is a monotonically increasing function that increases more slowly than linearly, to the sum; An optimization means for optimizing the inference model using a total loss including the regularization term; A learning device comprising:

2. 2. The learning device according to claim 1, wherein the regularization term calculation means increases a value of the regularization term when the class score is high, and decreases a value of the regularization term when the class score is low.

3. a loss calculation means for calculating a loss based on the class score and a correct class corresponding to the training data; The learning device according to claim 1 , wherein the total loss is a sum of the loss and the regularization term.

4. The class scores include a confidence score for each class for one training data set; The weighting function is a function that sums the squares of the confidence scores of the classes across all classes; The learning device according to claim 1 , wherein the rescaling function is a function that calculates a square root of the sum.

5. The class scores include a confidence score for each class for one training data set; The weighting function is a function that sums the natural logarithm of the squared confidence scores of the classes across all classes; The learning device according to claim 1 , wherein the rescaling function is a function for calculating a logarithm of the sum.

6. The class scores include a confidence score for each class for one training data set; The weighting function is a function that sums the natural logarithm of the confidence scores of each of the classes across all the classes; The learning device according to claim 1 , wherein the rescaling function is a function for calculating a logarithm of the sum.

7. A computer comprising: Use the inference model to make inferences on the training data and output class scores. Calculating weights from the output class scores using a weighting function that increases more rapidly than linearly when the class scores are too high or too low; Calculate the sum of the weights over a mini-batch containing a predetermined number of training data; applying a rescaling function to the sum to calculate a regularization term; A learning method that optimizes the inference model using a total loss including the regularization term.

8. Use the inference model to make inferences on the training data and output class scores. Calculating weights from the output class scores using a weighting function that increases more rapidly than linearly when the class scores are too high or too low; Calculate the sum of the weights over a mini-batch containing a predetermined number of training data; applying a rescaling function to the sum to calculate a regularization term; A program that causes a computer to execute a process of optimizing the inference model using a total loss including the regularization term.

Citation Information

Patent Citations

  • Quantization parameter optimization method and quantization parameter optimization device

    JP2021043596A

  • Binocular image depth estimation method, device, equipment, program, and medium

    JP2021502626A

  • Control method, information processing device, and control program

    WO2021144943A1