Iterative Convergence Calculation Device, Iterative Convergence Calculation Method, and Program

Abstract

The iterative convergence calculation device (1) includes an iterative calculation unit (11) that performs iterative convergence calculations based on a set compression ratio, and an iterative control unit (12) that performs an update to decrease the compression ratio, sequentially sets the compression ratio decreased by the update in the iterative calculation unit 11, and repeatedly causes the iterative calculation unit (11) to perform iterative convergence calculations.

Description

Technical Field

[0001] The present disclosure relates to an iterative convergence calculation device, an iterative convergence calculation method, and a program.

Background Art

[0002] Iterative convergence calculations are widely used in fields such as data analysis and optimization. For example, iterative convergence calculations are used in the learning process of machine learning models. In image data analysis, iterative convergence calculations are used in many tasks of image processing such as image reconstruction or filtering. Also, in non-linear optimization problems, an optimal solution is searched for using iterative convergence calculations. In control system design or optimal control, iterative convergence calculations are required in the calculation of state feedback of a controlled object or an optimal control method.

[0003] For example, Patent Document 1 describes a tensor decomposition process for tensor-decomposing a convolutional filter included in a neural network (hereinafter referred to as NN) for low-rank approximation, and calculates, as the rank to be used in tensor decomposition, the rank corresponding to an evaluation value less than a predetermined threshold among evaluation values representing the degree of low-rank approximation. By the low-rank approximation of NN, the number of weighted parameters required to represent NN is compressed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the conventional iterative convergence calculation, there is a trade-off relationship between the calculation speed until the calculation converges and the accuracy of the calculation result. That is, if one is prioritized, the other will be sacrificed. For example, in the NN optimization method described in Patent Document 1, iterative convergence calculation is used in the tensor decomposition process for low-rank approximation. Although the learning speed is improved by compressing the number of weighted parameters by low-rank approximation, the prediction accuracy of the model may decrease due to the loss of information amount associated with the compression of the number of parameters.

[0006] The present disclosure aims to solve the above problems and obtain an iterative convergence calculation device capable of performing iterative convergence calculation that achieves both an improvement in calculation speed and suppression of a decrease in the accuracy of calculation results.

Means for Solving the Problems

[0007] The iterative convergence calculation device according to the present disclosure includes an iterative calculation unit that performs iterative convergence calculation based on a set compression rate, and an iterative control unit that performs an update to decrease the compression rate, sequentially sets the decreased compression rate by the update to the iterative calculation unit, and causes the iterative calculation unit to repeatedly perform iterative convergence calculation. The iterative calculation unit performs learning of a neural network compressed at the set compression rate, and the iterative control unit sequentially sets the initial value of the compression rate and the decreased compression rate by the update to the iterative calculation unit, and causes the iterative calculation unit to repeatedly perform learning of the neural network. When the initial value of the compression rate and the decreased compression rate by the update are sequentially set by the iterative control unit 、 Compresses the number of learning parameters of the neural network by tensor decomposition of the fully connected layer or convolutional layer in the neural network Calculate a neural network compressed at a compression rate by the processing to be performed to do.

Effects of the Invention

[0008] According to the present disclosure, an update to decrease the compression rate is performed, the decreased compression rate by the update is sequentially set, and iterative convergence calculation based on the set compression rate is repeatedly performed. Thereby, the iterative convergence calculation device according to the present disclosure can perform iterative convergence calculation that achieves both an improvement in calculation speed and suppression of a decrease in the accuracy of calculation results.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Modes for Carrying Out the Invention

[0010] Embodiment 1. The iterative convergence calculation device according to Embodiment 1 repeatedly performs iterative convergence calculation while reducing the compression rate in a specific problem. Iterative convergence calculation is a method of repeatedly performing calculations on a given problem and approximating the solution until the convergence condition is satisfied. For example, an initial value of the solution is set, an iterative step, which is an operation for updating the value of the solution, is performed, and the iterative step is repeatedly performed until the convergence condition is satisfied. Iterative convergence calculation is widely used in various fields such as analysis of non-linear equations, solution methods for optimization problems, numerical integration, and learning algorithms in machine learning.

[0011] (Application examples of iterative convergence calculation) Iterative convergence calculation is widely used in various fields such as image AI for image classification or recognition, generative AI, physical simulation, and speech recognition. For the task of image classification or recognition in image AI, for example, a machine learning model of NN is used, and iterative convergence calculation is used for the learning of this machine learning model. Specifically, for example, the weights and biases of the model are adjusted by a method called backpropagation. This process is iterative convergence calculation, and it is repeatedly performed until the model is learned to minimize the error between the prediction of the training data and the actual label.

[0012] In generative AI, iterative convergence calculation is used for the learning of the generative model. For example, in an adversarial generative network (GAN), the generator and the discriminator learn while competing with each other. The generator generates fake data from input noise, and the discriminator determines whether it is actual data. These models perform iterative convergence calculation alternately, and learning is performed so that the generated data approaches the real thing.

[0013] In physical simulation, numerical calculations are performed iteratively. For example, finite element analysis (FEA) or computational fluid dynamics (CFD) sets elements that divide the analysis region into multiple parts and numerically analyzes the behavior of each element. In these analyses, iterative convergence calculations are performed until the convergence conditions are met. For example, the interaction between elements or the update of boundary conditions are carried out, and the calculations are repeated until the simulation results converge.

[0014] In speech recognition, conversion from speech data to text is performed. In speech recognition using an NN, a network for converting speech signals to text is learned. This learning process uses iterative convergence calculations and the network is optimized to capture the characteristics of the speech data.

[0015] (Compression ratio) The compression ratio is an index indicating the degree to which the number of objects to be compressed has decreased, and can be expressed as the ratio of the number after compression to the original number. For example, in image AI, a large amount of computing resources are required for the training or inference of large-scale NN models. Therefore, by using the compression ratio to reduce the number of model parameters, the computational cost can be reduced. Similarly, in generative AI, by using the compression ratio to reduce the number of parameters of the generative model, the efficiency of training and inference is improved. In physical simulation, complex analysis models are represented by large-scale meshes or particles. By using the compression ratio to adjust the size of the meshes or particles, the computational cost is reduced and efficient analysis is possible. Also in speech recognition, since large-scale models may be used, the computational cost can be reduced by using the compression ratio to reduce the number of model parameters.

[0016] (Overview of the iterative convergence calculation device) FIG. 1 is a block diagram showing the configuration of the iterative convergence calculation device 1 according to Embodiment 1. In FIG. 1, the iterative convergence calculation device 1 operates according to each procedure of initial value setting of the compression ratio, update of the compression ratio, iterative convergence calculation, convergence determination, and repetition. The initial value of the compression ratio is determined according to the characteristics of the problem to be processed or the required accuracy, but is set to a value higher than at least 1-fold. Here, "compression ratio of 1-fold" represents that no compression is performed. That is, "a value higher than 1-fold" means a value indicating that at least some compression is performed.

[0017] The iterative convergence calculation device 1 performs an update to decrease the compression ratio used in the repeatedly performed iterative convergence calculation. By this update process, the repeatedly performed iterative convergence calculation can be efficiently performed, and since the amount of compressed information gradually increases, the calculation accuracy also improves. The update process of the compression ratio may vary depending on the problem or algorithm to be processed. For example, in the compression of an NN, the compression ratio of the number of model parameters used in the iterative convergence operation for converging the number of model parameters is updated. For example, in mesh structure analysis, the compression ratio of mesh elements used in the iterative convergence operation for merging and simplifying a plurality of mesh elements is updated. Note that these are merely examples.

[0018] The iterative convergence calculation device 1 performs an iterative convergence calculation using the updated compression ratio. By the iterative convergence calculation, a converged solution is obtained based on the updated compression ratio. The iterative convergence calculation is considered to converge when a certain convergence condition is reached. For example, it is determined to converge when the change in the solution becomes less than a certain threshold. The process of updating to decrease the compression ratio and the iterative convergence calculation is repeated until a certain end condition is achieved. Thereby, the iterative convergence calculation device can perform an iterative convergence calculation that achieves both an improvement in calculation speed and suppression of a decrease in the accuracy of the calculation result.

[0019] As shown in FIG. 1, the iterative convergence calculation device 1 is realized by a computer connected to the storage unit 2. The storage unit 2 stores a data set used for iterative convergence calculation. For example, for image AI, a data set of images is used. In generative AI, a data set such as handwritten numbers is used. In physical simulation, three-dimensional grid data or the like is used. In speech recognition, a data set of speech or the like is used. Note that the storage unit 2 is a storage device connected to the computer that functions as the iterative convergence calculation device 1. Further, the storage unit 2 may be built into the iterative convergence calculation device 1, or may be a storage unit provided in an external device that can be communicatively connected via a wired or wireless network by a communication unit provided in the computer.

[0020] The iterative convergence calculation device 1 includes an iterative calculation unit 11 and an iterative control unit 12. For example, a program that constitutes an information processing application for realizing the functions of the iterative calculation unit 11 and the iterative control unit 12 is stored in a memory provided in the computer. By executing the information processing application read by the processor provided in the computer from the memory, the functions of the iterative calculation unit 11 and the iterative control unit 12 are realized.

[0021] The iterative calculation unit 11 performs an iterative convergence calculation based on the set compression rate. For example, the iterative calculation unit 11 performs appropriate processing for reducing the weights, biases, or other parameters of the model based on the set compression rate. For example, there are processes such as zero reduction, low-rank approximation, or quantization. Thereafter, the iterative calculation unit 11 starts a normal iterative calculation process using the model compressed in this way. The iterative calculation unit 11 updates the parameters of the model, for example, and repeatedly performs calculations until the convergence condition is satisfied.

[0022] The iterative control unit 12 causes the iterative calculation unit 11 to repeatedly perform iterative convergence calculations while reducing the compression rate used in the iterative convergence calculations. For example, the iterative control unit 12 performs an update to reduce the compression rate, sequentially sets the reduced compression rate to the iterative calculation unit 11, and causes the iterative calculation unit 11 to repeatedly perform iterative convergence calculations. For example, when causing the iterative calculation unit 11 to repeatedly perform iterative convergence calculations, the iterative control unit 12 updates the compression rate to a value lower than the previously set value, and based on the updated compression rate, instructs the iterative calculation unit 11 to perform iterative convergence calculations. When the iterative calculation unit 11 satisfies the convergence condition, the iterative control unit 12 determines that the calculation is complete and causes the iterative calculation unit 11 to output the final solution. When the convergence condition is not satisfied, the iterative control unit 12 further performs an update to reduce the compression rate and continues the iterative convergence calculations by the iterative calculation unit 11.

[0023] Also, the iterative control unit 12 may adjust the compression rate update method and schedule in order to optimize the convergence performance or calculation efficiency. Thereby, the efficiency of the iterative convergence calculations is improved, and the calculation resources can be effectively utilized. Further, the iterative control unit 12 manages the compression rate set for the iterative calculation unit 11 during the iterative convergence calculations by the iterative calculation unit 11. In this way, by repeatedly performing a reduction in the compression rate and iterative convergence calculations, the calculation efficiency is improved, and analysis or learning of large-scale problems becomes possible.

[0024] Next, the operation of the iterative convergence calculation device 1 will be described. FIG. 2A is a flowchart showing the iterative convergence calculation method according to Embodiment 1. The iterative control unit 12 sets an initial value of the compression rate to the iterative calculation unit 11 (step ST1). For example, the iterative control unit 12 determines a reduction amount for reducing the compression rate for each update and a final compression rate estimated to obtain a model with a target accuracy based on the constraints of the calculation resources and the requirements regarding the performance of the model. The iterative control unit 12 determines the initial value of the compression rate so as to fall within the target calculation speed based on the determined compression rate and the reduction amount of the compression rate for each update. The iterative control unit 12 sets the determined initial value of the compression rate to the iterative calculation unit 11.

[0025] The iterative calculation unit 11 performs an iterative convergence calculation based on the compression rate set by the iteration control unit 12 (step ST2). For example, the iterative calculation unit 11 updates the model parameters using the compressed model and iteratively performs calculations until the objective function or the convergence condition is satisfied. At this time, the iteration control unit 12 determines whether or not the end condition of the iterative convergence calculation by the iterative calculation unit 11 has been reached (step ST3). For example, the end condition is a condition that indicates whether or not the change amount or the residual itself of the residual obtained from the objective function reaches a target value. The end condition may also be when a predetermined number of iterations has ended.

[0026] When the end condition is reached (step ST3; YES), the series of processes in FIG. 2A is ended. On the other hand, when the end condition has not been reached (step ST3; NO), the iteration control unit 12 performs an update to decrease the compression rate (step ST4). For example, when the amount of decrease in the compression rate is set in the iteration control unit 12, the iteration control unit 12 sets in the iterative calculation unit 11 a compression rate decreased by the amount of decrease from the compression rate after the previous update. Thereafter, returning to step ST2, the iterative calculation unit 11 performs an iterative convergence calculation using the updated compression rate. By executing the above method, the iterative convergence calculation device 1 can perform an iterative convergence calculation that achieves both an improvement in calculation speed and suppression of a decrease in the accuracy of the calculation result.

[0027] In the above description, an example is shown in which the iteration control unit 12 explicitly performs the process of setting the initial value of the compression rate and updating the compression rate. However, for example, it is also possible to store in the storage unit 2 the initial value of the compression rate and the update amount determined in advance, and have the iterative calculation unit 11 read out those data and perform the process of setting the initial value of the compression rate and updating the compression rate. In that case, the iteration control unit 12 may only perform the confirmation of the end condition and the control of the repetition. The same applies to other embodiments.

[0028] Next, in order to compare with the iterative convergence calculation method according to Embodiment 1, a conventional iterative convergence calculation method will be described. FIG. 2B is a flowchart showing a conventional iterative convergence calculation method. First, a pre-compression model (non-compressed model) is created as an initial model (step ST100). Next, the parameters of the model are updated, and calculations are repeatedly performed until the objective function or convergence condition is satisfied (step ST101). At this time, it is determined whether or not the iterative convergence calculation has reached the end condition (step ST102). If the end condition is reached (step ST102; YES), the series of processes in FIG. 2B is terminated. On the other hand, if the end condition has not been reached (step ST102; NO), an update for improving the compression rate is performed (step ST103). A compression rate improved by a predetermined amount from the compression rate after the previous update is calculated, and the process returns to step ST101, where iterative convergence calculation using the updated compression rate is performed.

[0029] In the conventional iterative convergence calculation method, an initial model that is not compressed is calculated, and an update for improving the compression rate is performed. Since the compression rate increases each time an update is performed, the number of parameters of the model gradually decreases accordingly. That is, the amount of information of the compressed model is less than that of the initial model. Since the initial model has the largest number of parameters and the largest amount of information in this way, the learning speed of the initial model is slower than that of the compressed model. In contrast, the iterative convergence calculation method according to Embodiment 1 performs iterative convergence calculation of model learning while performing an update for decreasing the compression rate starting from a model compressed at the initial value of the compression rate. Therefore, with the iterative convergence calculation method according to Embodiment 1, it is possible to perform iterative convergence calculation that achieves both an improvement in calculation speed and suppression of a decrease in the accuracy of the calculation result.

[0030] Next, the hardware configuration for realizing the functions of the iterative convergence calculation device 1 will be described. FIG. 3 is a block diagram showing an example of a hardware configuration for realizing the functions of the iterative convergence calculation device 1. Each function of the iterative calculation unit 11 and the iterative control unit 12 included in the iterative convergence calculation device 1 is realized by a processing circuit. That is, the iterative convergence calculation device 1 includes a processing circuit for executing the processing from step ST1 to step ST4 shown in FIG. 2A. The processing circuit may be a CPU (Central Processing Unit) that executes a program stored in a memory.

[0031] For example, when the iterative convergence calculation device 1 can be wirelessly connected to an external device, the iterative calculation unit 11 acquires data received from the external device via the input interface 100 by a communication unit (not shown in FIG. 1) included in the iterative convergence calculation device 1. Also, when the iterative convergence calculation device 1 is connected to an external storage device by a wired data line, the iterative calculation unit 11 acquires the data stored in the storage device via the input interface 100. In this case, the iterative convergence calculation device 1 may not include the communication unit. Also, the iterative control unit 12 acquires control data received from the external device via the input interface 100 by the communication unit. The iterative control unit 12 controls the iterative convergence calculation by the iterative calculation unit 11 according to the acquired control data.

[0032] The iterative calculation unit 11 outputs the calculation result of the iterative convergence calculation to an external device via the output interface 101. For example, the display information indicating the iterative convergence calculation result is output to a display device (not shown in FIG. 1) via the output interface 101. Also, the iterative calculation unit 11 may transmit the iterative convergence calculation result to an external device by controlling the communication unit via the output interface 101.

[0033] Each function of the iterative calculation unit 11 and the iterative control unit 12 included in the iterative convergence calculation device 1 is realized by software, firmware, or a combination of software and firmware. Note that the software or firmware is described as a program and stored in the memory 103.

[0034] The processor 102 realizes the functions of the iterative calculation unit 11 and the iterative control unit 12 included in the iterative convergence calculation device 1 by reading and executing the program stored in the memory 103. For example, the iterative convergence calculation device 1 includes a memory 103 for storing a program that, when executed by the processor 102, results in the execution of the processes from step ST1 to step ST4 shown in FIG. 2A. These programs cause the computer to execute the procedures or methods of the processes performed by the iterative calculation unit 11 and the iterative control unit 12. The memory 103 may be a computer-readable storage medium storing a program for causing the computer to function as the iterative calculation unit 11 and the iterative control unit 12.

[0035] The memory 103 includes, for example, non-volatile or volatile semiconductor memories such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically-EPROM) (registered trademark), magnetic disks, flexible disks, optical disks, compact disks, mini disks, DVDs, and the like.

[0036] Part of the functions of the iterative calculation unit 11 and the iterative control unit 12 included in the iterative convergence calculation device 1 may be realized by dedicated hardware, and the other part may be realized by software or firmware. For example, the function of the iterative calculation unit 11 may be realized by a processing circuit that is dedicated hardware, and the function of the iterative control unit 12 may be realized by the processor 102 reading and executing the program stored in the memory 103. Thus, the processing circuit can realize the above functions by hardware, software, firmware, or a combination thereof.

[0037] As described above, the iterative convergence calculation device 1 according to Embodiment 1 includes an iterative calculation unit 11 that performs iterative convergence calculation based on a set compression rate, and an iterative control unit 12 that performs an update to decrease the compression rate, sequentially sets the decreased compression rate to the iterative calculation unit 11, and causes the iterative calculation unit 11 to repeatedly perform iterative convergence calculation. Thereby, the iterative convergence calculation device 1 can perform iterative convergence calculation that achieves both an improvement in calculation speed and suppression of a decrease in the accuracy of calculation results.

[0038] The iterative convergence calculation method according to Embodiment 1 includes steps ST1 and ST2 in which the iterative calculation unit 11 performs iterative convergence calculation based on a set compression rate, and steps ST3 and ST4 in which the iterative control unit 12 performs an update to decrease the compression rate, sequentially sets the decreased compression rate to the iterative calculation unit, and causes the iterative calculation unit 11 to repeatedly perform iterative convergence calculation. By executing the above method, the iterative convergence calculation device 1 can perform iterative convergence calculation that achieves both an improvement in calculation speed and suppression of a decrease in the accuracy of calculation results.

[0039] A computer that executes the program according to Embodiment 1 functions as an iterative calculation unit 11 that performs iterative convergence calculation based on a set compression rate, and an iterative control unit 12 that performs an update to decrease the compression rate, sequentially sets the decreased compression rate to the iterative calculation unit 11, and causes the iterative calculation unit 11 to repeatedly perform iterative convergence calculation. A computer that executes the program according to Embodiment 1 can perform iterative convergence calculation that achieves both an improvement in calculation speed and suppression of a decrease in the accuracy of calculation results.

[0040] Embodiment 2. In Embodiment 1, an iterative convergence calculation device that can be used in various fields was shown. In this Embodiment 2, an iterative convergence calculation device for NN learning will be described.

[0041] FIG. 4 is a block diagram showing the configuration of the iterative convergence calculation device 1A according to Embodiment 2. In FIG. 4, the iterative convergence calculation device 1A operates according to each procedure of initial value setting of the compression rate, update of the compression rate, iterative convergence calculation related to NN learning, convergence determination, and repetition. The initial value of the compression rate is determined according to the characteristics of the problem to be processed or the required accuracy. As shown in FIG. 4, the iterative convergence calculation device 1A is realized by a computer connected to the storage unit 2A.

[0042] The storage unit 2A stores a data set used for NN learning. The data set includes, for example, an image data set, a text data set, an audio data set, and a time-series data set. The image data set is used for NN learning for image recognition or image generation. The text data set is used for NN learning for natural language processing. The audio data set is used for NN learning for audio recognition or audio generation. The time-series data set is used for NN learning for time-series prediction or sequence generation. Note that the storage unit 2A is a storage device connected to a computer that functions as the iterative convergence calculation device 1A. Also, the storage unit 2A may be built into the iterative convergence calculation device 1A, or may be a storage unit provided in an external device that can be communicatively connected via a wired or wireless network by a communication unit included in the computer.

[0043] (Overview of Iterative Convergence Calculation Device) The iterative convergence calculation device 1A includes an iterative learning unit 11A and a learning control unit 12A. For example, a program that constitutes an information processing application for realizing the functions of the iterative learning unit 11A and the learning control unit 12A is stored in a memory included in the computer. By executing the information processing application read from the memory by a processor included in the computer, the functions of the iterative learning unit 11A and the learning control unit 12A are realized.

[0044] The iterative learning unit 11A is an iterative calculation unit that performs learning of the NN compressed at the set compression rate. For example, the iterative learning unit 11A performs appropriate processing for reducing the weights, biases, or other parameters of the NN model based on the set compression rate. For example, the iterative learning unit 11A updates the model parameters and iteratively performs calculations related to learning until the convergence condition is satisfied.

[0045] The learning control unit 12A is an iterative control unit that sequentially sets the compression rate decreased by the update to the iterative learning unit 11A and causes the iterative learning unit 11A to repeatedly perform learning of the NN. For example, the learning control unit 12A instructs the iterative learning unit 11A to perform learning of the NN based on the updated compression rate. When the iterative learning unit 11A satisfies the convergence condition, the learning control unit 12A determines the end of the learning of the NN by the iterative learning unit 11A and causes the iterative learning unit 11A to output the final solution. If the convergence condition is not satisfied, the learning control unit 12A further performs an update to decrease the compression rate and continues the learning of the NN by the iterative learning unit 11A.

[0046] Also, the learning control unit 12A may adjust the compression rate update method and schedule in order to optimize the convergence performance or calculation efficiency. Thereby, the efficiency of the iterative convergence calculation is improved and the calculation resources can be effectively utilized. Further, the learning control unit 12A manages the compression rate set for the iterative learning unit 11A during the learning of the NN by the iterative learning unit 11A. Thereby, by repeatedly performing the decrease in the compression rate and the iterative convergence calculation, the calculation efficiency is improved and analysis or learning of large-scale problems becomes possible.

[0047] Next, the iterative convergence calculation method according to Embodiment 2 will be described. FIG. 5 is a flowchart showing the iterative convergence calculation method according to Embodiment 2, and shows a series of operations by the iterative convergence calculation apparatus 1A. First, the learning control unit 12A sets an initial value of the compression rate for the iterative learning unit 11A (step ST1A). For example, the learning control unit 12A determines a reduction amount for reducing the compression rate for each update and a final compression rate at which it is estimated that a model with a target accuracy can be obtained, based on constraints on computational resources and requirements regarding the performance of the model. The learning control unit 12A determines an initial value of the compression rate so as to fall within the target calculation speed, based on the determined compression rate and the reduction amount of the compression rate for each update. The learning control unit 12A sets the determined initial value of the compression rate for the iterative learning unit 11A.

[0048] FIG. 6 is a diagram showing the NN before compression. The NN shown in FIG. 6 is a convolutional NN (hereinafter referred to as CNN). For example, the iterative learning unit 11A performs learning of the CNN compressed at the compression rate set by the learning control unit 12A. In the CNN, the convolutional layer performs a convolution operation on the input data of the CNN. In the convolutional layer, kernels, which are a plurality of filters, execute a convolution operation to detect different features. For example, in a CNN that extracts features of an image, features such as edges, textures, and patterns are extracted. The ReLU (Rectified Linear Unit), which is an activation function, is a non-linear function that outputs 0 when the input is less than 0 and outputs the original value otherwise.

[0049] Max pooling downsamples the output of the convolutional layer by taking out the maximum value from each region of the convolutional layer. In the flattening layer, multi-dimensional data is converted into one dimension. Thereby, the output from the pooling layer is converted into a form suitable for the fully connected layer. In the fully connected layers (1) and (2), all the perceptrons in the previous layer are connected to all the perceptrons in the next layer. Thereby, the model performs final classification or output. The Softmax function, which is an activation function, converts the output of the network into a probability distribution. It normalizes the values of the perceptrons in the output layer to obtain the probability of belonging to each class.

[0050] The iterative learning unit 11A compresses the CNN shown in FIG. 6 at the compression rate set by the learning control unit 12A. For example, the iterative learning unit 11A calculates a compressed model as an initial learning network by means of a tensor network, low-rank approximation, or pruning method.

[0051] (Initial model generation by tensor network) FIG. 7 is a diagram showing an overview of a tensor network and shows the fully connected layers of the CNN in FIG. 6. In FIG. 7, the perceptron 20 is connected to the perceptrons 21-1 and 21-2. With one perceptron 20 as the input, binary values are output from the perceptrons 21-1 and 21-2. The subscript i is the subscript corresponding to the weight in the connection between the perceptron 20 and the perceptron 21-1, and the subscript j is the subscript corresponding to the weight in the connection between the perceptron 20 and the perceptron 21-2. The subscript k is the subscript corresponding to the connection between the perceptrons 21-1 and 21-2. The subscript m corresponds to the output of the perceptron 21-1, and the subscript n corresponds to the output of the perceptron 21-2.

[0052] The iterative learning unit 11A performs tensor decomposition on the fully connected layer or convolutional layer of the CNN to compress the number of parameters of the CNN. For example, in the fully connected layer (1) connecting 5408 dimensions and 1024 dimensions in the CNN shown in FIG. 6, the operation is performed by a matrix operation W having 5408×1024 elements. The iterative learning unit 11A assumes, for example, the tensor network shown in FIG. 7 for the fully connected layer (1), and ikm V jkn performs tensor decomposition into the product of a third-order tensor of the form, and sets the value of k regarding the connection between the perceptron 20 and the perceptron 21-2 to be sufficiently small to compress the total number of parameters. Thereby, the iterative learning unit 11A can generate a compressed model by setting the value of k according to the set compression rate.

[0053] (Initial model generation by low-rank approximation) The iterative learning unit 11A may generate an initial model by compressing the model through low-rank approximation. Low-rank approximation is a method of approximating high-dimensional data such as matrices or tensors with lower-rank approximate matrices or tensors. FIG. 8 is a diagram showing an overview of low-rank approximation. The matrix operation W in the fully connected layer or the convolutional layer is subjected to singular value decomposition as shown in the following formula (1). In the following formula (1), W is an m×n matrix. U is an m×m orthogonal matrix, Σ is an m×n diagonal matrix, and V is an n×n orthogonal matrix. T indicates the transpose matrix. Thereafter, the iterative learning unit 11A sets a threshold for the singular values and sets the singular value components below the threshold to 0, so that, as shown in FIG. 8, the matrices U, Σ, and V become matrices with partially remaining elements and are low-rank approximated. In this way, the iterative learning unit 11A can generate a compressed model by performing low-rank approximation according to the set compression rate. W = UΣV T (1)

[0054] (Initial model generation by pruning method) The iterative learning unit 11A may generate an initial model by compressing the model through a pruning method. The pruning method is a method of compressing the model by deleting unimportant connections or parameters of the model. FIG. 9 is a diagram showing an overview of compression by the pruning method. In FIG. 9, k perceptrons 22-k are each connected to k perceptrons 23-k in the next layer. The iterative learning unit 11A deletes connections or parameters with low importance in the model by the pruning method. For example, the iterative learning unit 11A selects only k weights w1 indicated by thick lines among the weights of the perceptrons shown in FIG. 9 and sets 0 to the other weights w2 indicated by broken lines to compress the calculation of the model. Thereby, the iterative learning unit 11A can generate a compressed model by the pruning method according to the set compression rate.

[0055] Subsequently, the iterative learning unit 11A repeatedly performs learning of the compressed NN model based on the compression rate set by the learning control unit 12A (step ST2A). In the learning of the NN, the weights or biases are adjusted to minimize the error between the output of the model for the training data and the correct label. In algorithms such as backpropagation, learning is performed by repeatedly calculating the gradient and updating the weights. At this time, the learning control unit 12A determines whether the end condition of the NN learning has been reached (step ST3A). For example, the learning control unit 12A may determine the end condition of the NN learning by the first determination method or the second determination method shown below.

[0056] (First determination method) The iterative learning unit 11A continues the NN learning when the residual from the target variable or the change amount of the residual in the iterative calculation is greater than the threshold, and ends the learning if it is below the threshold. For example, the iterative learning unit 11A updates the parameters of the NN using the training data set to improve the predicted value for the target variable. The iterative learning unit 11A calculates the residual between the target variable and the predicted value of the network every time the model is trained. The residual represents the difference between the target variable and the output of the network. The iterative learning unit 11A checks whether the change amount of the residual is greater than the set threshold. If the change amount of the residual is greater than the threshold, the learning continues. On the other hand, when the change amount of the residual becomes less than or equal to the threshold, the learning is ended. Thereby, the iterative convergence calculation device 1A can end the NN learning at an appropriate timing.

[0057] (Second determination method) Also, the learning control unit 12A may continue the NN learning when the number of times the NN learning is repeated is less than the threshold number of times, and end the learning when it reaches the threshold number of times or more. For example, the number of iterations to complete the learning of the NN is called the "number of epochs". The number of epochs depends on the nature of the problem targeted by the model, the size of the dataset, or the complexity of the model, etc. Thereby, the iterative convergence calculation device 1A can end the learning of the NN at an appropriate timing.

[0058] When the end condition is reached (step ST3A; YES), the series of processes in FIG. 5 is ended. On the other hand, when the end condition has not been reached (step ST3A; NO), the learning control unit 12A performs an update to decrease the compression rate (step ST4A). After this, it returns to step ST2A, and the iterative learning unit 11A performs the learning of the NN using the updated compression rate.

[0059] (Specific example of the method for decreasing the compression rate) The learning control unit 12A may decrease the compression rate according to the progress of the learning of the NN. For example, the learning control unit 12A monitors the progress of learning during the training of the model. This includes the number of learning epochs, the value of the loss function, accuracy, or other performance metrics. The learning control unit 12A uses a high compression rate in the initial stage of the learning of the NN to efficiently reduce the size of the model. Then, as the learning of the NN progresses, the learning control unit 12A decreases the compression rate to the compression rate required for the model parameters to exhibit optimal performance. Here, after adjusting the compression rate, the learning control unit 12A evaluates the performance of the model, and if necessary based on this evaluation result, decreases the compression rate. By repeating this procedure, the learning control unit 12A can update to the compression rate corresponding to the optimal model performance.

[0060] Hereinafter, the effects obtained by executing the above method using a known dataset will be described. FIG. 10 is a diagram showing a configuration example of an NN. In FIG. 10, the NN has an input layer of 784 dimensions, two hidden layers of 4096 dimensions, and an output layer of 10 dimensions, and the perceptrons of these adjacent layers are respectively connected. In the hidden layer on the input side, operations are performed by matrix operation W1, and in the hidden layer on the output side, operations are performed by matrix operation W2.

[0061] FIG. 11 is a graph showing the transition of the correct answer rate over time. In FIG. 11, the horizontal axis represents time (learning time), and the vertical axis represents the correct answer rate of the training data by the model of the learning result at each time point. The results in FIG. 11 are obtained by performing learning of an NN having two hidden layers with 4096 perceptrons shown in FIG. 10 using a known MNIST data set.

[0062] The learning results shown in FIG. 11 are the results obtained by performing low-rank approximation for matrix operations W1 and W2 from 4096 to a rank of k0 = 20. The compression rate in this low-rank approximation is 4096 / 20 = 205 times. Also, the result indicated by symbol A is the result of learning with only low-rank approximation. The result indicated by symbol B is the result of applying the method of the present disclosure and learning by shifting from low-rank approximation (compression rate: 205 times) to full rank (no compression, i.e., compression rate: 1 time). The result indicated by symbol C is the result of learning with only full rank.

[0063] When only low-rank approximation is performed, as shown by symbol A, the learning speed is about 3 times faster than when only full rank is used, and the accuracy degradation at this time is about 5%. On the other hand, when the iterative convergence calculation device 1A performs learning by gradually decreasing the compression rate from the initial value of 205 times and shifting from low-rank approximation to full rank, as shown by symbol B, the learning speed is about 1.5 times faster than when only full rank is used, and the accuracy degradation at this time is about 0% to 1%. Thus, according to the method of the present disclosure, in which a value exceeding 1-fold (in this example, 205-fold) is set as the initial compression ratio and then the compression ratio is gradually decreased to perform iterative convergence calculation, the learning speed can be increased compared to the case of convergence with full rank only, and the accuracy degradation can be suppressed compared to the case of convergence with low rank only. In other words, according to the method of the present disclosure, it is possible to achieve both high-speed convergence calculation and suppression of accuracy degradation of the calculation result. Note that if the initial value of the compression ratio is set to about 10% of the original size for learning purposes of the MNIST dataset as described above, it is considered that an effect of achieving both high-speed convergence calculation and suppression of accuracy degradation of the calculation result can be obtained approximately. In addition, since the optimal network size varies depending on the learning data, the above is merely an example, and an initial value suitable for the network size may be set. Theoretically, it is considered that the effect of the method of the present disclosure can be obtained if a value higher than 1-fold (i.e., other than no compression) is set as the initial value of the compression ratio.

[0064] As described above, in the iterative convergence calculation device 1A according to Embodiment 2, the iterative learning unit 11A performs learning of the NN compressed at the set compression ratio. The learning control unit 12A sequentially sets the compression ratio decreased by the update to the iterative learning unit 11A and repeatedly causes the iterative learning unit 11A to perform learning of the NN. Thereby, the iterative convergence calculation device 1A can perform iterative convergence calculation that achieves both improvement in the learning speed of the NN and suppression of degradation in the accuracy of the calculation result model.

[0065] In the iterative convergence calculation device 1A according to Embodiment 2, when the initial value of the compression ratio and the compression ratio decreased by the update are sequentially set by the learning control unit 12A, the iterative learning unit 11A calculates the NN compressed at the set compression ratio. Thereby, the iterative convergence calculation device 1A can perform iterative convergence calculation that achieves both improvement in the learning speed of the NN and suppression of degradation in the accuracy of the calculation result model.

[0066] In the iterative convergence calculation device 1A according to Embodiment 2, the iterative learning unit 11A compresses the number of learning parameters of the NN by tensor-decomposing the fully connected layer or the convolutional layer in the NN. As a result, the iterative convergence calculation device 1A can generate a compressed model.

[0067] In the iterative convergence calculation device 1A according to Embodiment 2, the iterative learning unit 11A compresses the number of learning parameters of the NN by performing singular value decomposition on the matrix operation in the fully connected layer or the convolutional layer of the NN. As a result, the iterative convergence calculation device 1A can generate a compressed model.

[0068] In the iterative convergence calculation device 1A according to Embodiment 2, the iterative learning unit 11A selects a number of weights determined according to the compression rate among the weights at the nodes of the NN, and sets the other weights to 0. As a result, the iterative convergence calculation device 1A can generate a compressed model.

[0069] In the iterative convergence calculation device 1A according to Embodiment 2, the iterative learning unit 11A continues the learning of the NN when the residual from the target variable or the change amount of the residual in the iterative calculation is greater than the threshold value, and ends the learning if it is below the threshold value. As a result, the iterative convergence calculation device 1A can end the learning of the NN at an appropriate timing.

[0070] In the iterative convergence calculation device 1A according to Embodiment 2, the learning control unit 12A continues the learning of the NN when the number of times the learning of the NN is repeated is less than the threshold number of times, and ends the learning when it reaches the threshold number of times or more. As a result, the iterative convergence calculation device 1A can end the learning of the NN at an appropriate timing.

[0071] In the iterative convergence calculation device 1A according to Embodiment 2, the learning control unit 12A decreases the compression rate according to the progress of the learning of the NN. As a result, the iterative convergence calculation device 1A can end the learning of the NN at an appropriate timing.

[0072] Embodiment 3. In Embodiment 1, an iterative convergence calculation device that can be used in various fields was shown. In this Embodiment 3, an iterative convergence calculation device for performing mesh structure analysis will be described. Mesh structure analysis divides a complex object or structure into a finite number of elements and approximates physical quantities within each element. Finite element analysis, including mesh structure analysis, needs to solve the boundary conditions or force balance between elements in order to simulate mechanical behavior. Iterative convergence calculations are used to obtain this solution.

[0073] FIG. 12 is a block diagram showing the configuration of an iterative convergence calculation device 1B according to Embodiment 3. In FIG. 12, the iterative convergence calculation device 1B operates according to each procedure of initial value setting of mesh coarseness, update of mesh coarseness, iterative convergence calculation related to meshing of space, convergence determination, and repetition. Here, the mesh coarseness is an index indicating the degree to which the number of mesh elements has decreased, and can be represented by the ratio of the number of mesh elements to the original number. The mesh coarseness is one form of the compression rate described so far. Also, the initial value of the mesh coarseness is determined according to the characteristics of the problem to be processed or the required accuracy. As shown in FIG. 12, the iterative convergence calculation device 1B is realized by a computer connected to a storage unit 2B.

[0074] The storage unit 2B stores a data set used for mesh structure analysis. The data set includes, for example, geometry data, mesh data, and boundary condition data. The geometry data is data representing the shape or size of an object or structure that is the target of mesh structure analysis. The mesh data is data representing a mesh generated from the geometry data, and represents the mesh by elements such as triangles or quadrilaterals, for example. The boundary condition data is data indicating the boundary conditions of a system or object. The boundary condition data includes fixed vertices or surfaces, and regions to which external forces or loads are applied. These boundary conditions are used when simulating the behavior of an object. Note that the storage unit 2B is a storage device connected to a computer that functions as the iterative convergence calculation device 1B. Also, the storage unit 2B may be built into the iterative convergence calculation device 1B, or may be a storage unit provided in an external device that can be communicatively connected via a wired or wireless network by a communication unit included in the computer.

[0075] (Overview of the iterative convergence calculation device) The iterative convergence calculation device 1B includes a mesh calculation unit 11B and an iterative control unit 12B. For example, in the memory included in the computer, a program that constitutes an information processing application for realizing the functions of the mesh calculation unit 11B and the iterative control unit 12B is stored. By executing the information processing application read from the memory by the processor included in the computer, the functions of the mesh calculation unit 11B and the iterative control unit 12B are realized.

[0076] The mesh calculation unit 11B is an iterative calculation unit that performs meshing of a space using a set compression ratio. For example, the mesh calculation unit 11B performs each process of generating an initial mesh, meshing the space, and executing iterative calculations based on the set compression ratio. For example, the mesh calculation unit 11B meshes the space using an initial value of mesh coarseness, and repeatedly performs meshing of the space and calculation of mesh coarseness until the mesh coarseness is reduced to reach its target value.

[0077] The iterative control unit 12B is an iterative control unit that sequentially sets the mesh coarseness reduced by the update to the mesh calculation unit 11B and causes the mesh calculation unit 11B to repeatedly perform the meshing of the space. For example, the iterative control unit 12B instructs the mesh calculation unit 11B to perform the meshing of the space based on the updated mesh coarseness. When the mesh calculation unit 11B satisfies the convergence condition, the iterative control unit 12B determines the end of the meshing of the space by the mesh calculation unit 11B and causes the mesh calculation unit 11B to output the final solution. If the convergence condition is not satisfied, the iterative control unit 12B further performs an update to reduce the mesh coarseness and continues the meshing of the space by the mesh calculation unit 11B.

[0078] Also, the iterative control unit 12B may adjust the update method and schedule of the mesh coarseness in order to optimize the convergence performance or calculation efficiency. As a result, the efficiency of the iterative convergence calculation is improved, and the calculation resources can be effectively utilized. Further, the iterative control unit 12B manages the mesh coarseness set for the mesh calculation unit 11B in the process of the meshing of the space by the mesh calculation unit 11B. As a result, by repeatedly performing the reduction of the mesh coarseness and the iterative convergence calculation, the calculation efficiency is improved, and the analysis or learning of large-scale problems becomes possible.

[0079] Next, the iterative convergence calculation method according to Embodiment 3 will be described. FIG. 13 is a flowchart showing the iterative convergence calculation method according to Embodiment 3, and shows a series of operations by the iterative convergence calculation apparatus 1B. First, the iterative control unit 12B sets an initial value of the compression rate to the mesh calculation unit 11B (step ST1B). For example, the iterative control unit 12B determines the reduction amount for reducing the mesh coarseness for each update and the final mesh coarseness estimated to obtain an analysis result of the target accuracy based on the constraints of the calculation resources and the requirements regarding the performance of the model. The iterative control unit 12B determines the initial value of the mesh coarseness so as to fall within the target calculation speed based on the determined mesh coarseness and the reduction amount of the mesh coarseness for each update. The iterative control unit 12B sets the determined initial value of the mesh coarseness to the mesh calculation unit 11B.

[0080] The mesh calculation unit 11B performs an iterative convergence calculation of meshing based on the mesh coarseness set from the iterative control unit 12B as mesh structure analysis (step ST2B). For example, the mesh calculation unit 11B updates the previously set mesh coarseness and repeatedly performs the calculations necessary for meshing with the updated coarseness. Here, the mesh calculation unit 11B may set the meshes coarsely at equal intervals in the target space, or may set them coarsely focusing only on the principal components according to the values of the mesh elements in the target space.

[0081] FIG. 14 is a diagram showing an overview of the meshing process. The left diagram in FIG. 14 is a space meshed according to the previously set mesh coarseness and has coarse meshes. The mesh calculation unit 11B is set with the mesh coarseness decreased each time it is updated. The mesh calculation unit 11B calculates an interpolation value of the mesh according to the mesh coarseness decreased each time it is updated, and inserts mesh elements corresponding to the interpolation value into the previously set mesh, so that the target space becomes fine meshes as shown in the right diagram in FIG. 14. As a method for calculating the interpolation value, for example, linear interpolation or Catmull-Rom interpolation is used.

[0082] The iterative control unit 12B determines whether or not the end condition of the iterative convergence calculation by the mesh calculation unit 11B has been reached (step ST3B). For example, the iteration end condition is a condition indicating whether or not the final mesh coarseness at which it is estimated that the analysis result of the target accuracy obtained when calculating the initial value of the mesh coarseness has been reached.

[0083] When the iteration end condition is reached (step ST3B; YES), the series of processes in FIG. 13 is terminated. On the other hand, if the iteration end condition has not been reached (step ST3B; NO), the iteration control unit 12B performs an update to reduce the mesh coarseness (step ST4B). For example, when the amount of reduction in mesh coarseness is set in the iteration control unit 12B, the iteration control unit 12B sets the mesh coarseness reduced by the amount of reduction from the mesh coarseness after the previous update in the mesh calculation unit 11B. After that, returning to step ST2B, the mesh calculation unit 11B performs an iterative convergence calculation for meshing using the updated mesh coarseness. By executing the above method, the iterative convergence calculation device 1B can perform an iterative convergence calculation that achieves both an improvement in the calculation speed of spatial meshing and suppression of a decrease in the accuracy of mesh structure analysis of the calculation results.

[0084] In addition, in FIG. 14, mesh elements are set in a three-dimensional space, but Embodiment 3 is not limited to this. For example, a four-dimensional mesh obtained by adding a dimension in the time direction to the three dimensions of the target space may be handled. Furthermore, the iterative convergence calculation device 1B according to Embodiment 3 may be applied to mesh analysis (such as quantum state calculation) in a high-dimensional space of five or more dimensions.

[0085] As described above, in the iterative convergence calculation device 1B according to Embodiment 3, the mesh calculation unit 11B performs a meshing process for meshing the target space with a coarseness according to the set compression ratio. The iteration control unit 12B sequentially sets the compression ratio reduced by the update in the mesh calculation unit 11B and causes the mesh calculation unit 11B to repeatedly perform the meshing process. Thereby, the iterative convergence calculation device 1B can perform an iterative convergence calculation that achieves both an improvement in the calculation speed of spatial meshing and suppression of a decrease in the accuracy of mesh structure analysis of the calculation results.

[0086] In addition, combinations of each embodiment or deformations of any constituent elements of each embodiment or omissions of any constituent elements in each embodiment are possible.

Industrial Applicability

[0087] The iterative convergence calculation device according to the present disclosure can be used, for example, in data analysis and optimization.

Explanation of Signs

[0088] 1, 1A, 1B iterative convergence calculation device, 2, 2A, 2B storage unit, 11 iterative calculation unit, 11A iterative learning unit, 11B mesh calculation unit, 12 iterative control unit, 12A learning control unit, 12B iterative control unit, 20, 21-1, 21-2, 22-k, 23-k perceptron, 100 input interface, 101 output interface, 102 processor, 103 memory.

Claims

1. An iterative calculation unit that performs iterative convergence calculations based on the set compression ratio, An iterative control unit that performs an update to reduce the compression ratio, sequentially sets the reduced compression ratio to the iterative calculation unit, and causes the iterative calculation unit to repeatedly perform iterative convergence calculations. Equipped with, The iterative calculation unit performs training on the neural network compressed with the set compression ratio. The iterative control unit sequentially sets the compression ratio, which has been reduced by the update, to the iterative calculation unit, and causes the iterative calculation unit to repeatedly perform the learning of the neural network. The iterative calculation unit, upon sequentially setting the initial value of the compression ratio and the compression ratio reduced by the iterative control unit, calculates the neural network compressed with the set compression ratio and compresses the number of learning parameters of the neural network by tensor decomposition of the fully connected layers or convolutional layers in the neural network. An iterative convergence computing device characterized by the following features.

2. An iterative calculation unit that performs iterative convergence calculations based on a set compression ratio, An iterative control unit that performs an update to reduce the compression ratio, sequentially sets the reduced compression ratio to the iterative calculation unit, and causes the iterative calculation unit to repeatedly perform iterative convergence calculations. Equipped with, The iterative calculation unit performs training on the neural network compressed with the set compression ratio. The iterative control unit sequentially sets the compression ratio, which has been reduced by the update, to the iterative calculation unit, and causes the iterative calculation unit to repeatedly perform the learning of the neural network. The iterative calculation unit, upon sequentially setting the initial value of the compression ratio and the compression ratio reduced by the iterative control unit, calculates the neural network compressed with the set compression ratio, and compresses the number of learning parameters of the neural network by singular value decomposition of the matrix operations in the fully connected or convolutional layers of the neural network. An iterative convergence computing device characterized by the following features.

3. The iterative calculation unit selects a number of weights from the nodes of the neural network according to the compression ratio, and sets the remaining weights to 0. The iterative convergence computing device according to claim 1 or 2.

4. The iteration control unit continues training the neural network if the residual from the target variable or the change in the residual in the iterative convergence calculation is greater than a threshold, and terminates training if it is less than or equal to the threshold. The iterative convergence computing device according to claim 1 or 2.

5. The iteration control unit continues the neural network's learning process if the number of times the neural network has been trained is less than a threshold number of times, and terminates the learning process when the number of times exceeds the threshold number of times. The iterative convergence computing device according to claim 1 or 2.

6. The iteration control unit reduces the compression ratio according to the progress of the neural network's learning. The iterative convergence computing device according to claim 1 or 2.

7. The iterative calculation unit performs a meshing process that meshes the target space with a coarseness corresponding to the set compression ratio. The iterative control unit sequentially sets the compression ratio, which has been reduced by the update, to the iterative calculation unit, and causes the iterative calculation unit to repeatedly perform the meshing process. The iterative convergence computing device according to claim 1 or 2.

8. An iterative convergence calculation method using an iterative convergence calculation device, The iterative calculation unit performs an iterative convergence calculation based on the set compression ratio, The iterative control unit performs an update that reduces the compression ratio, sequentially sets the reduced compression ratio to the iterative calculation unit, and causes the iterative calculation unit to repeatedly perform iterative convergence calculations. The iterative calculation unit performs training on the neural network compressed with the set compression ratio. The iterative control unit sequentially sets the compression ratio, which has been reduced by the update, to the iterative calculation unit, and causes the iterative calculation unit to repeatedly perform the learning of the neural network. When the iterative calculation unit receives the initial value of the compression ratio and the compression ratio reduced by the iterative control unit in sequence, it calculates the neural network compressed with the set compression ratio and compresses the number of learning parameters of the neural network by tensor decomposition of the fully connected layers or convolutional layers in the neural network. A method for iterative convergence calculation characterized by the following:

9. An iterative convergence calculation method using an iterative convergence calculation device, The iterative calculation unit performs an iterative convergence calculation based on the set compression ratio, The iterative control unit performs an update that reduces the compression ratio, sequentially sets the reduced compression ratio to the iterative calculation unit, and causes the iterative calculation unit to repeatedly perform iterative convergence calculations. The iterative calculation unit performs training on the neural network compressed with the set compression ratio. The iterative control unit sequentially sets the compression ratio, which has been reduced by the update, to the iterative calculation unit, and causes the iterative calculation unit to repeatedly perform the learning of the neural network. When the iterative calculation unit receives the initial value of the compression ratio and the compression ratio reduced by the iterative control unit in sequence, it calculates the neural network compressed with the set compression ratio, and compresses the number of learning parameters of the neural network by singular value decomposition of the matrix operations in the fully connected layer or convolutional layer of the neural network. A method for iterative convergence calculation characterized by the following:

10. Computers, An iterative calculation unit that performs iterative convergence calculations based on the set compression ratio, and An iterative control unit performs an update that reduces the compression ratio, sequentially sets the reduced compression ratio to the iterative calculation unit, and causes the iterative calculation unit to repeatedly perform iterative convergence calculations. To make it function as, The iterative calculation unit performs training on the neural network compressed with the set compression ratio. The iterative control unit sequentially sets the compression ratio, which has been reduced by the update, to the iterative calculation unit, and causes the iterative calculation unit to repeatedly perform the learning of the neural network. When the iterative calculation unit receives the initial value of the compression ratio and the compression ratio reduced by the iterative control unit in sequence, it calculates the neural network compressed with the set compression ratio and compresses the number of learning parameters of the neural network by tensor decomposition of the fully connected layers or convolutional layers in the neural network. A program characterized by the following features.

11. A computer, An iterative calculation unit that performs iterative convergence calculations based on the set compression ratio, and An iterative control unit performs an update that reduces the compression ratio, sequentially sets the reduced compression ratio to the iterative calculation unit, and causes the iterative calculation unit to repeatedly perform iterative convergence calculations. To make it function as, The iterative calculation unit performs training on the neural network compressed with the set compression ratio. The iterative control unit sequentially sets the compression ratio, which has been reduced by the update, to the iterative calculation unit, and causes the iterative calculation unit to repeatedly perform the learning of the neural network. When the iterative calculation unit receives the initial value of the compression ratio and the compression ratio reduced by the iterative control unit in sequence, it calculates the neural network compressed with the set compression ratio, and compresses the number of learning parameters of the neural network by singular value decomposition of the matrix operations in the fully connected layer or convolutional layer of the neural network. A program characterized by the following features.

Images