Iterative convergence calculation device, iterative convergence calculation method, and program

Abstract

This iterative convergence calculation device (1) is provided with: an iterative calculation unit (11) that performs iterative convergence calculation on the basis of a set compression rate; and an iterative control unit (12) that performs update to reduce the compression rate, sequentially sets the compression rate reduced by the update to the iterative calculation unit 11, and repeatedly performs iterative convergence calculation by the iterative calculation unit (11).

Description

Iterative convergence calculation device, iterative convergence calculation method and program

[0001] The present disclosure relates to an iterative convergence calculation device, an iterative convergence calculation method, and a program.

[0002] Iterative convergence calculations are widely used in fields such as data analysis and optimization. For example, it is used in the training process of machine learning models. In image data analysis, iterative convergence calculations are used in many image processing tasks, such as image reconstruction or filtering. Iterative convergence calculations are also used to find optimal solutions in nonlinear optimization problems. In control system design or optimal control, iterative convergence calculations are required for state feedback of the controlled object or for calculating the optimal control method.

[0003] For example, Patent Document 1 describes a method for optimizing a neural network (NN) by performing a tensor decomposition process to tensor-decompose a convolution filter of the NN for low-rank approximation, and calculating a rank corresponding to an evaluation quantity that is less than a predetermined threshold among evaluation quantities that represent the degree of low-rank approximation as the rank to be used in the tensor decomposition. The low-rank approximation of the NN reduces the number of weighting parameters required to represent the NN.

[0004] International Publication No. 2019 / 146189

[0005] However, in conventional iterative convergence calculations, there is a trade-off between the calculation speed until the calculation converges and the accuracy of the calculation results, and prioritizing one has the disadvantage that the other is sacrificed. For example, in the NN optimization method described in Patent Document 1, iterative convergence calculations are used in tensor decomposition processing for low-rank approximation, and although the learning speed is improved by compressing the number of weighting parameters using low-rank approximation, there is a possibility that the prediction accuracy of the model will decrease due to a loss of information associated with compressing the number of parameters.

[0006] The present disclosure is intended to solve the above-mentioned problems, and aims to provide an iterative convergence calculation device that can perform iterative convergence calculations that achieve both improved calculation speed and suppression of deterioration in accuracy of calculation results.

[0007] The iterative convergence calculation device according to the present disclosure includes an iterative calculation unit that performs iterative convergence calculation based on a set compression ratio, and an iterative control unit that performs an update to reduce the compression ratio, sequentially sets the reduced compression ratios in the iterative calculation unit as a result of the update, and repeatedly performs iterative convergence calculation by the iterative calculation unit.

[0008] According to the present disclosure, an update is performed to reduce the compression ratio, the reduced compression ratios are sequentially set, and iterative convergence calculations based on the set compression ratios are repeatedly performed. This allows the iterative convergence calculation device according to the present disclosure to perform iterative convergence calculations that achieve both improved calculation speed and suppressed deterioration in the accuracy of the calculation results.

[0009] FIG. 2A is a block diagram showing the configuration of an iterative convergence calculation device according to embodiment 1. FIG. 2A and FIG. 2B are flowcharts showing an iterative convergence calculation process. FIG. 2B is a block diagram showing a hardware configuration for realizing the functions of the iterative convergence calculation device according to embodiment 1. FIG. 2C is a block diagram showing the configuration of an iterative convergence calculation device according to embodiment 2. FIG. 2D is a flowchart showing an iterative convergence calculation method according to embodiment 2. FIG. 2E is a diagram showing a neural network before compression. FIG. 2F is a diagram showing an outline of a tensor network. FIG. 2G is a diagram showing an outline of low-rank approximation. FIG. 2H is a diagram showing an outline of compression by pruning. FIG. 2H is a diagram showing an example configuration of a neural network. FIG. 2H is a graph showing the transition of the accuracy rate over time. FIG. 2I is a block diagram showing the configuration of an iterative convergence calculation device according to embodiment 3. FIG. 2I is a flowchart showing an iterative convergence calculation method according to embodiment 3. FIG. 2I is a diagram showing an outline of meshing processing.

[0010] Embodiment 1. An iterative convergence calculation device according to embodiment 1 repeatedly performs iterative convergence calculations on a specific problem while decreasing the compression ratio. Iterative convergence calculations are a method of repeatedly performing calculations on a given problem and approximating a solution until a convergence condition is satisfied. For example, an initial value of the solution is set, and an iterative step, which is a calculation for updating the value of the solution, is performed, and the iterative step is repeated until the convergence condition is satisfied. Iterative convergence calculations are widely used in various fields, such as analysis of nonlinear equations, solution of optimization problems, numerical integration, and learning algorithms for machine learning.

[0011] (Application Example of Iterative Convergence Calculation) Iterative convergence calculation is widely used in various fields, such as image AI that performs image classification or recognition, generative AI, physical simulation, and voice recognition. For example, a neural network (NN) machine learning model is used for the image classification or recognition task in image AI, and iterative convergence calculation is used to train this machine learning model. For example, specifically, the weights and biases of the model are adjusted by a technique called backpropagation. This process is iterative convergence calculation, and is repeated until the model is trained to minimize the error between the prediction of training data and the actual label.

[0012] In generative AI, iterative convergence calculations are used to train generative models. For example, a generative adversarial network (GAN) is trained by a generator and a classifier competing with each other. The generator generates fake data from input noise, and the classifier determines whether it is real data. These models alternate through iterative convergence calculations, training the generated data to be closer to the real thing.

[0013] In physical simulations, numerical calculations are performed iteratively. For example, finite element analysis (FEA) or computational fluid dynamics (CFD) divide the analysis domain into multiple elements, and numerically analyze the behavior of each element. In these analyses, iterative convergence calculations are performed until the convergence condition is met. For example, the interactions between elements or boundary conditions are updated, and calculations are repeated until the simulation results converge.

[0014] Speech recognition involves converting speech data into text. In speech recognition using neural networks, a network is trained to convert speech signals into text. This training process uses iterative convergence calculations to optimize the network to capture the features of the speech data.

[0015] (Compression Ratio) The compression ratio is an index indicating the degree to which the number of compressed objects has been reduced, and can be expressed as the ratio of the number of compressed objects to the original number. For example, in image AI, the learning or inference of a large-scale NN model requires a large amount of computational resources. Therefore, by using the compression ratio to reduce the number of model parameters, computational costs can be reduced. Similarly, in generative AI, by using the compression ratio to reduce the number of parameters of a generative model, the efficiency of learning and inference can be improved. In physical simulations, complex analytical models are represented by large meshes or particles. By adjusting the size of the mesh or particles using the compression ratio, computational costs can be reduced and efficient analysis is possible. In speech recognition, large-scale models may also be used, and by using the compression ratio to reduce the number of model parameters, computational costs can be reduced.

[0016] (Outline of Iterative Convergence Calculation Device) FIG. 1 is a block diagram showing the configuration of an iterative convergence calculation device 1 according to a first embodiment. In FIG. 1, the iterative convergence calculation device 1 operates through the procedures of setting an initial value of the compression ratio, updating the compression ratio, iterative convergence calculation, convergence determination, and repetition. The initial value of the compression ratio is determined according to the characteristics of the problem to be processed or the required accuracy, but is set to a value at least higher than 1. Here, a "compression ratio of 1" indicates that no compression is performed. In other words, a "value higher than 1" indicates that at least some compression is performed.

[0017] The iterative convergence calculation device 1 performs an update to reduce the compression ratio used in the repeated iterative convergence calculation. This update process allows the repeated iterative convergence calculation to be performed efficiently, and the amount of compressed information is gradually increased, thereby improving calculation accuracy. The compression ratio update process may differ depending on the problem or algorithm being processed. For example, in NN compression, the compression ratio of the number of model parameters used in the iterative convergence calculation to converge the number of model parameters is updated. For example, in mesh structure analysis, the compression ratio of mesh elements used in the iterative convergence calculation to merge and simplify multiple mesh elements is updated. Note that these are merely examples.

[0018] The iterative convergence calculation device 1 performs iterative convergence calculation using the updated compression ratio. A converged solution is obtained based on the updated compression ratio through the iterative convergence calculation. The iterative convergence calculation is deemed to have converged when a certain convergence condition is reached. For example, convergence is determined when the change in the solution is equal to or less than a certain threshold. The process of updating to reduce the compression ratio and the iterative convergence calculation is repeated until a certain termination condition is achieved. This enables the iterative convergence calculation device to perform iterative convergence calculation that achieves both improved calculation speed and suppressed deterioration in the accuracy of the calculation results.

[0019] As shown in FIG. 1, the iterative convergence calculation device 1 is realized by a computer connected to a storage unit 2. The storage unit 2 stores data sets used for the iterative convergence calculation. For example, an image data set is used for image AI. A data set of handwritten numbers or the like is used for generative AI. Three-dimensional grid data or the like is used for physical simulation. A voice data set or the like is used for voice recognition. The storage unit 2 is a storage device connected to the computer functioning as the iterative convergence calculation device 1. The storage unit 2 may be built into the iterative convergence calculation device 1, or may be a storage unit included in an external device that can be connected for communication via a wired or wireless network by a communication unit included in the computer.

[0020] The iterative convergence calculation apparatus 1 includes an iterative calculation unit 11 and an iterative control unit 12. For example, a memory included in the computer stores a program constituting an information processing application for realizing each function of the iterative calculation unit 11 and the iterative control unit 12. A processor included in the computer reads the information processing application from the memory and executes it, thereby realizing each function of the iterative calculation unit 11 and the iterative control unit 12.

[0021] The iterative calculation unit 11 performs iterative convergence calculation based on the set compression ratio. For example, the iterative calculation unit 11 performs appropriate processing to reduce the weights, biases, or other parameters of the model based on the set compression ratio. For example, this processing may include zero reduction, low-rank approximation, or quantization. Then, the iterative calculation unit 11 starts a normal iterative calculation process using the model compressed in this way. For example, the iterative calculation unit 11 updates the model parameters and performs iterative calculation until a convergence condition is satisfied.

[0022] The iterative control unit 12 causes the iterative calculation unit 11 to repeatedly perform iterative convergence calculations while decreasing the compression ratio used in the iterative convergence calculations. For example, the iterative control unit 12 performs an update to decrease the compression ratio, sequentially sets the updated decreased compression ratios in the iterative calculation unit 11, and causes the iterative calculation unit 11 to repeatedly perform iterative convergence calculations. For example, when causing the iterative calculation unit 11 to repeat the iterative convergence calculations, the iterative control unit 12 updates the compression ratio to a value lower than the previously set value and instructs the iterative calculation unit 11 to perform iterative convergence calculations based on the updated compression ratio. When the iterative calculation unit 11 satisfies the convergence condition, the iterative control unit 12 determines the end of the calculations and causes the iterative calculation unit 11 to output a final solution. When the convergence condition is not satisfied, the iterative control unit 12 performs an update to further decrease the compression ratio and causes the iterative calculation unit 11 to continue the iterative convergence calculations.

[0023] The iterative control unit 12 may also adjust the compression rate update method and schedule to optimize convergence performance or calculation efficiency. This improves the efficiency of the iterative convergence calculation and enables effective use of calculation resources. Furthermore, the iterative control unit 12 manages the compression rate set in the iterative calculation unit 11 during the iterative convergence calculation by the iterative calculation unit 11. In this way, by repeatedly performing the reduction in the compression rate and the iterative convergence calculation, calculation efficiency is improved and analysis or learning of large-scale problems becomes possible.

[0024] Next, the operation of the iterative convergence calculation apparatus 1 will be described. FIG. 2A is a flowchart showing an iterative convergence calculation method according to the first embodiment. The iterative control unit 12 sets an initial value of the compression ratio to the iterative calculation unit 11 (step ST1). For example, the iterative control unit 12 determines the amount by which the compression ratio is reduced for each update and the final compression ratio estimated to obtain a model with the target accuracy based on the constraints of calculation resources and requirements related to model performance. The iterative control unit 12 determines the initial value of the compression ratio based on the determined compression ratio and the amount by which the compression ratio is reduced for each update so as to fall within the target calculation speed. The iterative control unit 12 sets the determined initial value of the compression ratio to the iterative calculation unit 11.

[0025] The iterative calculation unit 11 performs an iterative convergence calculation based on the compression ratio set by the iterative control unit 12 (step ST2). For example, the iterative calculation unit 11 updates the model parameters using the compressed model and performs iterative calculations until the objective function or the convergence condition is satisfied. At this time, the iterative control unit 12 determines whether an end condition for the iterative convergence calculation performed by the iterative calculation unit 11 has been reached (step ST3). For example, the end condition is a condition indicating whether a target value, which is the amount of change in the residual obtained from the objective function or the residual itself, has been reached. Alternatively, the end condition may be the point in time when a predetermined number of iterations has been completed.

[0026] If the termination condition is met (step ST3; YES), the series of processes in FIG. 2A ends. On the other hand, if the termination condition is not met (step ST3; NO), the iterative control unit 12 performs an update to reduce the compression ratio (step ST4). For example, if a reduction amount of the compression ratio is set in the iterative control unit 12, the iterative control unit 12 sets a compression ratio that is reduced by the reduction amount from the compression ratio after the previous update in the iterative calculation unit 11. Thereafter, the process returns to step ST2, and the iterative calculation unit 11 performs an iterative convergence calculation using the updated compression ratio. By executing the above method, the iterative convergence calculation device 1 can perform an iterative convergence calculation that achieves both improved calculation speed and suppressed deterioration in the accuracy of the calculation results.

[0027] In the above description, an example is shown in which the iterative control unit 12 explicitly sets the initial value of the compression ratio and updates the compression ratio. However, it is also possible, for example, to store predetermined initial values ​​and update amounts of the compression ratio in the storage unit 2, and have the iterative calculation unit 11 read out the data to set the initial value of the compression ratio and update the compression ratio. In this case, the iterative control unit 12 may only check the termination condition and control the repetition. This is also true for other embodiments.

[0028] Next, a conventional iterative convergence calculation method will be described for comparison with the iterative convergence calculation method according to the first embodiment. FIG. 2B is a flowchart showing the conventional iterative convergence calculation method. First, a pre-compression model (uncompressed model) is created as an initial model (step ST100). Next, the model parameters are updated, and calculations are repeated until the objective function or convergence condition is satisfied (step ST101). At this time, it is determined whether the iterative convergence calculation has reached a termination condition (step ST102). If the termination condition has been reached (step ST102; YES), the series of processes shown in FIG. 2B is terminated. On the other hand, if the termination condition has not been reached (step ST102; NO), an update is performed to improve the compression ratio (step ST103). A compression ratio improved by a predetermined amount from the compression ratio after the previous update is calculated, and the process returns to step ST101 to perform the iterative convergence calculation using the updated compression ratio.

[0029] In conventional iterative convergence calculation methods, an uncompressed initial model is calculated, and updates are performed to improve the compression rate. As the compression rate increases with each update, the number of parameters in the model gradually decreases. That is, the amount of information in the compressed model is smaller than that in the initial model. Because the initial model has the largest number of parameters and the largest amount of information, the learning speed of the initial model is slower than that of the compressed model. In contrast, the iterative convergence calculation method according to the first embodiment starts with a model compressed at the initial compression rate, and performs iterative convergence calculation to learn the model while performing updates that reduce the compression rate. Therefore, the iterative convergence calculation method according to the first embodiment can perform iterative convergence calculation that achieves both improved calculation speed and suppressed degradation in the accuracy of the calculation results.

[0030] Next, a hardware configuration for realizing the functions of the iterative convergence calculation apparatus 1 will be described. FIG. 3 is a block diagram showing an example of a hardware configuration for realizing the functions of the iterative convergence calculation apparatus 1. The functions of the iterative calculation unit 11 and the iterative control unit 12 included in the iterative convergence calculation apparatus 1 are realized by a processing circuit. That is, the iterative convergence calculation apparatus 1 includes a processing circuit for executing the processes of steps ST1 to ST4 shown in FIG. 2A. The processing circuit may be a CPU (Central Processing Unit) that executes a program stored in a memory.

[0031] For example, if the iterative convergence calculation apparatus 1 can be wirelessly connected to an external device, the iterative calculation unit 11 acquires data received from the external device via the input interface 100 by a communication unit (not shown in FIG. 1 ) included in the iterative convergence calculation apparatus 1. Also, if the iterative convergence calculation apparatus 1 is connected to an external storage device via a wired data line, the iterative calculation unit 11 acquires data stored in the storage device via the input interface 100. In this case, the iterative convergence calculation apparatus 1 does not need to be equipped with the communication unit. Also, the iterative control unit 12 acquires control data received from the external device via the input interface 100 by the communication unit. The iterative control unit 12 controls the iterative convergence calculation by the iterative calculation unit 11 in accordance with the acquired control data.

[0032] The iterative calculation unit 11 outputs the calculation results of the iterative convergence calculation to an external device via the output interface 101. For example, display information showing the iterative convergence calculation results is output to a display device (not shown in Fig. 1) via the output interface 101. The iterative calculation unit 11 may also transmit the iterative convergence calculation results to the external device by controlling the communication unit via the output interface 101.

[0033] The functions of the iterative calculation unit 11 and the iterative control unit 12 included in the iterative convergence calculation apparatus 1 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 103.

[0034] The processor 102 reads and executes programs stored in the memory 103 to implement the functions of the iterative calculation unit 11 and the iterative control unit 12 included in the iterative convergence calculation apparatus 1. For example, the iterative convergence calculation apparatus 1 includes a memory 103 for storing programs that, when executed by the processor 102, result in the processing of steps ST1 to ST4 shown in FIG. 2A being executed. These programs cause a computer to execute the procedures or methods of the processing performed by the iterative calculation unit 11 and the iterative control unit 12. The memory 103 may be a computer-readable storage medium that stores programs for causing a computer to function as the iterative calculation unit 11 and the iterative control unit 12.

[0035] The memory 103 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically-EPROM) (registered trademark), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD.

[0036] Some of the functions of the iterative calculation unit 11 and the iterative control unit 12 included in the iterative convergence calculation apparatus 1 may be realized by dedicated hardware, and the other functions may be realized by software or firmware. For example, the functions of the iterative calculation unit 11 may be realized by a processing circuit that is dedicated hardware, and the functions of the iterative control unit 12 may be realized by the processor 102 reading and executing a program stored in the memory 103. In this way, the processing circuit can realize the above functions by hardware, software, firmware, or a combination of these.

[0037] As described above, the iterative convergence calculation apparatus 1 according to the first embodiment includes the iterative calculation unit 11 that performs iterative convergence calculation based on the set compression ratio, and the iterative control unit 12 that performs an update to reduce the compression ratio, sequentially sets the compression ratios reduced by the update in the iterative calculation unit 11, and repeatedly performs iterative convergence calculation by the iterative calculation unit 11. This enables the iterative convergence calculation apparatus 1 to perform iterative convergence calculation that achieves both an improvement in calculation speed and suppression of a decrease in the accuracy of the calculation results.

[0038] The iterative convergence calculation method according to the first embodiment includes steps ST1 and ST2 in which the iterative calculation unit 11 performs iterative convergence calculation based on the set compression ratio, and steps ST3 and ST4 in which the iterative control unit 12 performs an update to reduce the compression ratio, sequentially sets the updated reduced compression ratio in the iterative calculation unit, and causes the iterative convergence calculation unit 11 to repeatedly perform iterative convergence calculation. By executing the above method, the iterative convergence calculation device 1 can perform iterative convergence calculation that achieves both an improvement in calculation speed and suppression of a decrease in the accuracy of the calculation results.

[0039] The computer that executes the program according to the first embodiment functions as an iterative calculation unit 11 that performs iterative convergence calculation based on a set compression ratio, and an iterative control unit 12 that performs an update to reduce the compression ratio, sequentially sets the compression ratio reduced by the update in the iterative calculation unit 11, and causes the iterative convergence calculation to be repeated by the iterative calculation unit 11. The computer that executes the program according to the first embodiment can perform iterative convergence calculation that achieves both an improvement in calculation speed and suppression of a decrease in the accuracy of the calculation results.

[0040] Second Embodiment In the first embodiment, an iterative convergence calculation device that can be used in various fields was shown, but in the second embodiment, an iterative convergence calculation device that performs learning of a neural network will be described.

[0041] FIG. 4 is a block diagram showing the configuration of an iterative convergence calculation device 1A according to a second embodiment. In FIG. 4, the iterative convergence calculation device 1A operates through the procedures of setting an initial value of the compression ratio, updating the compression ratio, iterative convergence calculation related to NN learning, convergence determination, and repetition. The initial value of the compression ratio is determined according to the characteristics of the problem to be processed or the required accuracy. As shown in FIG. 4, the iterative convergence calculation device 1A is realized by a computer connected to a storage unit 2A.

[0042] The storage unit 2A stores data sets used for training the neural network. Examples of data sets include an image data set, a text data set, a speech data set, and a time-series data set. The image data set is used for training a neural network for image recognition or image generation. The text data set is used for training a neural network for natural language processing. The speech data set is used for training a neural network for speech recognition or speech generation. The time-series data set is used for training a neural network for time-series prediction or sequence generation. The storage unit 2A is a storage device connected to a computer functioning as the iterative convergence calculation device 1A. The storage unit 2A may be built into the iterative convergence calculation device 1A, or may be a storage unit included in an external device that can be connected to the computer via a communication unit provided in the computer via a wired or wireless network.

[0043] (Outline of the Iterative Convergence Calculation Device) The iterative convergence calculation device 1A includes an iterative learning unit 11A and a learning control unit 12A. For example, a memory included in the computer stores a program constituting an information processing application for realizing each function of the iterative learning unit 11A and the learning control unit 12A. A processor included in the computer executes the information processing application read from the memory, thereby realizing each function of the iterative learning unit 11A and the learning control unit 12A.

[0044] The iterative learning unit 11A is an iterative calculation unit that performs learning of the neural network compressed at a set compression rate. For example, the iterative learning unit 11A performs appropriate processing to reduce the weights, biases, or other parameters of the neural network model based on the set compression rate. For example, the iterative learning unit 11A updates the model parameters and repeatedly performs learning-related calculations until a convergence condition is satisfied.

[0045] The learning control unit 12A is an iterative control unit that sequentially sets the compression rate, which has been reduced by the update, in the iterative learning unit 11A and causes the iterative learning unit 11A to repeatedly learn the NN. For example, the learning control unit 12A instructs the iterative learning unit 11A to learn the NN based on the updated compression rate. When the iterative learning unit 11A satisfies the convergence condition, the learning control unit 12A determines that the NN learning by the iterative learning unit 11A has ended and causes the iterative learning unit 11A to output a final solution. If the convergence condition is not satisfied, the learning control unit 12A performs an update that further reduces the compression rate and causes the iterative learning unit 11A to continue learning the NN.

[0046] The learning control unit 12A may also adjust the compression rate update method and schedule to optimize convergence performance or calculation efficiency. This improves the efficiency of the iterative convergence calculation and enables effective use of calculation resources. Furthermore, the learning control unit 12A manages the compression rate set in the iterative learning unit 11A during the NN learning process by the iterative learning unit 11A. This improves calculation efficiency by repeatedly reducing the compression rate and performing iterative convergence calculation, making it possible to analyze or learn large-scale problems.

[0047] Next, an iterative convergence calculation method according to the second embodiment will be described. FIG. 5 is a flowchart illustrating the iterative convergence calculation method according to the second embodiment, showing a series of operations performed by the iterative convergence calculation device 1A. First, the learning control unit 12A sets an initial value of the compression rate in the iterative learning unit 11A (step ST1A). For example, based on the constraints of calculation resources and requirements related to model performance, the learning control unit 12A determines the amount by which the compression rate is reduced for each update and the final compression rate estimated to obtain a model with the target accuracy. Based on the determined compression rate and the amount by which the compression rate is reduced for each update, the learning control unit 12A determines the initial value of the compression rate so that the calculation speed falls within the target range. The learning control unit 12A sets the determined initial value of the compression rate in the iterative learning unit 11A.

[0048] FIG. 6 is a diagram illustrating a NN before compression. The NN illustrated in FIG. 6 is a convolutional NN (hereinafter referred to as CNN). For example, the iterative learning unit 11A performs learning on a CNN compressed at a compression rate set by the learning control unit 12A. In a CNN, the convolutional layer performs convolution operations on input data to the CNN. In the convolutional layer, multiple filter kernels perform convolution operations to detect different features. For example, a CNN that extracts image features extracts features such as edges, textures, and patterns. The activation function, ReLU (Rectified Linear Unit), is a nonlinear function that outputs 0 if the input is less than 0 and outputs the same value otherwise.

[0049] Max pooling downsamples the output of the convolutional layer by taking the maximum value from each region of the convolutional layer. The flattening layer converts multidimensional data into one dimension, converting the output from the pooling layer into a format suitable for the fully connected layer. In the fully connected layers (1) and (2), all perceptrons in the previous layer are connected to all perceptrons in the next layer. This allows the model to perform the final classification or output. The Softmax function, an activation function, converts the network output into a probability distribution. It normalizes the values ​​of the perceptrons in the output layer to represent the probability of belonging to each class.

[0050] The iterative learning unit 11A compresses the CNN shown in Fig. 6 at a compression rate set by the learning control unit 12A. For example, the iterative learning unit 11A calculates a compressed model as an initial learning network using a tensor network, low-rank approximation, or pruning method.

[0051] (Generation of Initial Models Using Tensor Networks) FIG. 7 is a diagram showing an overview of a tensor network, illustrating the fully connected layer of the CNN in FIG. 6. In FIG. 7, the perceptron 20 is connected to the perceptron 21-1 and the perceptron 21-2. One perceptron 20 receives input, and the perceptrons 21-1 and 21-2 output two values. The subscript i corresponds to the weight in the connection between the perceptron 20 and the perceptron 21-1, and the subscript j corresponds to the weight in the connection between the perceptron 20 and the perceptron 21-2. The subscript k corresponds to the connection between the perceptron 21-1 and the perceptron 21-2. The subscript m corresponds to the output of the perceptron 21-1, and the subscript n corresponds to the output of the perceptron 21-2.

[0052] The iterative learning unit 11A tensor-decomposes the fully connected layer or convolutional layer of the CNN to compress the number of parameters of the CNN. For example, in the fully connected layer (1) of the CNN shown in FIG. 6, which connects 5408 dimensions and 1024 dimensions, calculations are performed using a matrix operation W having 5408 × 1024 elements. For example, the iterative learning unit 11A assumes the tensor network shown in FIG. 7 for the fully connected layer (1), and ｉｋｍ V ｊｋｎ The total number of parameters is compressed by setting the value of k relating to the connection between perceptron 20 and perceptron 21-2 to a sufficiently small value. This allows the iterative learning unit 11A to generate a compressed model by setting the value of k according to the set compression rate.

[0053] (Generation of Initial Model by Low-Rank Approximation) The iterative learning unit 11A may generate an initial model by compressing the model using low-rank approximation. Low-rank approximation is a technique for approximating high-dimensional data such as matrices or tensors with a lower-rank approximation matrix or tensor. FIG. 8 is a diagram illustrating an overview of low-rank approximation. A matrix operation W in a fully connected layer or convolutional layer is subjected to singular value decomposition as shown in the following formula (1). In the following formula (1), W is an m×n matrix. U is an m×m orthogonal matrix, Σ is an m×n diagonal matrix, and V is an n×n orthogonal matrix. T represents a transposed matrix. After this, the iterative learning unit 11A sets a threshold for the singular values ​​and sets singular value components below the threshold to 0, thereby performing low-rank approximation on the matrices U, Σ, and V, as shown in FIG. 8, by which matrices U, Σ, and V become matrices with some elements remaining. In this way, the iterative learning unit 11A can generate a compressed model by performing low-rank approximation according to the set compression rate. W = UΣV Ｔ (1)

[0054] (Generation of Initial Model by Pruning) The iterative learning unit 11A may generate an initial model by compressing the model using pruning. Pruning is a method for compressing a model by removing unimportant connections or parameters from the model. FIG. 9 is a diagram illustrating an overview of compression using pruning. In FIG. 9, k perceptrons 22-k are each connected to k perceptrons 23-k in the next layer. The iterative learning unit 11A uses pruning to remove connections or parameters with low importance in the model. For example, the iterative learning unit 11A compresses the calculation of the model by selecting weights w1 for only k perceptrons, indicated by bold lines, from the weights of the perceptrons shown in FIG. 9, and setting the remaining weights w2, indicated by dashed lines, to 0. This allows the iterative learning unit 11A to generate a compressed model using pruning in accordance with the set compression rate.

[0055] Next, the iterative learning unit 11A iteratively learns the compressed NN model based on the compression rate set by the learning control unit 12A (step ST2A). In NN learning, weights or biases are adjusted to minimize the error between the model output for the training data and the correct label. In algorithms such as backpropagation, learning is performed by iteratively calculating gradients and updating weights. At this time, the learning control unit 12A determines whether the NN learning termination condition has been reached (step ST3A). For example, the learning control unit 12A may determine the NN learning termination condition using the first or second determination method described below.

[0056] (First Determination Method) The iterative learning unit 11A continues learning of the NN when the residual from the response variable or the change in the residual in the iterative calculation is greater than a threshold, and terminates learning when it is equal to or less than the threshold. For example, the iterative learning unit 11A uses a training dataset to update the parameters of the NN to improve the predicted value for the response variable. The iterative learning unit 11A calculates the residual between the response variable and the predicted value of the network each time the model is trained. The residual represents the difference between the response variable and the network output. The iterative learning unit 11A checks whether the change in the residual is greater than a set threshold. If the change in the residual is greater than the threshold, learning continues. On the other hand, if the change in the residual is equal to or less than the threshold, learning terminates. This allows the iterative convergence calculation device 1A to terminate learning of the NN at an appropriate time.

[0057] (Second Determination Method) The learning control unit 12A may continue NN learning if the number of iterations of NN learning is less than a threshold number, and may terminate learning when the number of iterations reaches or exceeds the threshold number. For example, the number of iterations required to terminate NN learning is called the "epoch number." The epoch number depends on the nature of the problem targeted by the model, the size of the data set, the complexity of the model, and the like. This allows the iterative convergence calculation device 1A to terminate NN learning at an appropriate time.

[0058] If the termination condition is met (step ST3A; YES), the series of processes in Fig. 5 is terminated. On the other hand, if the termination condition is not met (step ST3A; NO), the learning control unit 12A performs an update to lower the compression rate (step ST4A). After this, the process returns to step ST2A, and the iterative learning unit 11A performs NN learning using the updated compression rate.

[0059] (Specific Example of a Method for Reducing Compression Rate) The learning control unit 12A may reduce the compression rate according to the progress of NN learning. For example, the learning control unit 12A monitors the progress of learning during model training. This includes the number of learning epochs, the value of the loss function, accuracy, or other performance metrics. The learning control unit 12A uses a high compression rate in the early stages of NN learning to efficiently reduce the model size. Then, as NN learning progresses, the learning control unit 12A reduces the compression rate to a level required for the model parameters to achieve optimal performance. Here, after adjusting the compression rate, the learning control unit 12A evaluates the model performance and, if necessary, reduces the compression rate based on the evaluation results. By repeating this procedure, the learning control unit 12A can update the compression rate to a level corresponding to optimal model performance.

[0060] The effects obtained by executing the above method using a known data set will be described below. Fig. 10 is a diagram showing an example of the configuration of a neural network. In Fig. 10, the neural network has a 784-dimensional input layer, two 4096-dimensional hidden layers, and a 10-dimensional output layer, with perceptrons in adjacent layers connected to each other. In the input-side hidden layer, calculations are performed using matrix operation W1, and in the output-side hidden layer, calculations are performed using matrix operation W2.

[0061] Figure 11 is a graph showing the change in accuracy rate over time. In Figure 11, the horizontal axis represents time (learning time), and the vertical axis represents the accuracy rate for training data using the model of the learning results at each point in time. The results in Figure 11 were obtained by training a neural network (NN) with two hidden layers, each with 4096 perceptrons, as shown in Figure 10, using the known MNIST dataset.

[0062] The learning results shown in FIG. 11 show that for matrix operations W1 and W2, the rank is k ０ = 20. The compression rate in the low-rank approximation is 4096 / 20 = 205 times. The result indicated by symbol A is the result of learning using only low-rank approximation. The result indicated by symbol B is the result of applying the method of the present disclosure, and is the result of learning after transitioning from low-rank approximation (compression rate: 205 times) to full-rank (no compression, i.e., compression rate: 1 time). The result indicated by symbol C is the result of learning using only full-rank.

[0063] When only low-rank approximation was performed, the learning speed was approximately three times faster than when only full-rank approximation was performed, as indicated by symbol A, with accuracy degradation of approximately 5%. On the other hand, when the iterative convergence calculation device 1A gradually reduced the compression rate from the initial value of 205x and performed learning to transition from low-rank approximation to full-rank approximation, the learning speed was approximately 1.5 times faster than when only full-rank approximation was performed, with accuracy degradation of approximately 0% to 1%, as indicated by symbol B. Thus, the method disclosed herein, which sets an initial compression rate greater than 1x (205x in this example) and then gradually reduces the compression rate to perform iterative convergence calculations, can speed up the learning speed compared to when convergence was performed using only full-rank approximation, while suppressing accuracy degradation compared to when convergence was performed using only low-rank approximation. In other words, the method disclosed herein can achieve both faster convergence calculations and suppressed accuracy degradation in the calculation results. In addition, if the initial value of the compression rate is set to about 10% of the original size when the MNIST dataset is used for learning, it is believed that it will be possible to achieve both an increase in the speed of the convergence calculation and suppression of deterioration in the accuracy of the calculation results. Furthermore, since the optimal network size varies depending on the learning data, the above is merely an example, and it is sufficient to set an initial value appropriate for the network size. Theoretically, it is believed that the effects of the method disclosed herein can be obtained by setting the initial compression rate to a value higher than 1 (i.e., other than no compression).

[0064] As described above, in the iterative convergence calculation device 1A according to the second embodiment, the iterative learning unit 11A performs learning of a neural network compressed at a set compression rate. The learning control unit 12A sequentially sets the compression rate, which has been reduced by updating, in the iterative learning unit 11A, and causes the iterative learning unit 11A to repeatedly perform learning of the neural network. This enables the iterative convergence calculation device 1A to perform iterative convergence calculations that simultaneously improve the learning speed of the neural network and suppress deterioration in the accuracy of the model in the calculation results.

[0065] In the iterative convergence calculation device 1A according to the second embodiment, when the learning control unit 12A sequentially sets the initial value of the compression rate and the compression rate reduced by updating, the iterative learning unit 11A calculates a neural network compressed at the set compression rate. This enables the iterative convergence calculation device 1A to perform an iterative convergence calculation that achieves both an improvement in the learning speed of the neural network and suppression of a decrease in the accuracy of the model of the calculation results.

[0066] In the iterative convergence calculation device 1A according to the second embodiment, the iterative learning unit 11A reduces the number of learning parameters of the neural network by tensor decomposing the fully connected layer or the convolutional layer in the neural network. This allows the iterative convergence calculation device 1A to generate a reduced model.

[0067] In the iterative convergence calculation apparatus 1A according to the second embodiment, the iterative learning unit 11A reduces the number of learning parameters of the neural network by performing singular value decomposition on the matrix operations in the fully connected layer or convolutional layer of the neural network. This allows the iterative convergence calculation apparatus 1A to generate a reduced model.

[0068] In the iterative convergence calculation device 1A according to the second embodiment, the iterative learning unit 11A selects a number of weights determined according to the compression rate from among the weights at the nodes of the neural network, and sets the other weights to 0. This allows the iterative convergence calculation device 1A to generate a compressed model.

[0069] In the iterative convergence calculation device 1A according to the second embodiment, the iterative learning unit 11A continues learning of the neural network when the residual from the objective variable or the amount of change in the residual in the iterative calculation is greater than a threshold, and terminates learning when the residual is equal to or less than the threshold. This allows the iterative convergence calculation device 1A to terminate learning of the neural network at an appropriate timing.

[0070] In the iterative convergence calculation device 1A according to the second embodiment, the learning control unit 12A continues the learning of the NN when the number of iterations of the NN learning is less than a threshold number, and terminates the learning when the number of iterations reaches or exceeds the threshold number. This allows the iterative convergence calculation device 1A to terminate the learning of the NN at an appropriate timing.

[0071] In the iterative convergence calculation apparatus 1A according to the second embodiment, the learning control unit 12A reduces the compression rate in accordance with the progress of the neural network learning, thereby enabling the iterative convergence calculation apparatus 1A to terminate the neural network learning at an appropriate timing.

[0072] Embodiment 3. In the first embodiment, an iterative convergence calculation device that can be used in various fields was described, but in this third embodiment, an iterative convergence calculation device that performs mesh structure analysis will be described. Mesh structure analysis divides a complex object or structure into a finite number of elements and approximates the physical quantities within each element. Finite element analysis, including mesh structure analysis, needs to solve the boundary conditions or force balance between elements in order to simulate mechanical behavior. Iterative convergence calculation is used to obtain this solution.

[0073] FIG. 12 is a block diagram showing the configuration of an iterative convergence calculation apparatus 1B according to a third embodiment. In FIG. 12, the iterative convergence calculation apparatus 1B operates through the following steps: initial mesh roughness setting, mesh roughness update, iterative convergence calculation for meshing the space, convergence determination, and repetition. Here, mesh roughness is an index indicating the degree to which the number of mesh elements has been reduced and can be expressed as the ratio of the number of mesh elements to the original number. Mesh roughness is one form of the compression ratio described above. The initial mesh roughness value is determined according to the characteristics of the problem being processed or the required accuracy. As shown in FIG. 12, the iterative convergence calculation apparatus 1B is realized by a computer connected to a storage unit 2B.

[0074] The storage unit 2B stores data sets used in the mesh structure analysis. Examples of the data sets include geometry data, mesh data, and boundary condition data. The geometry data represents the shape or size of an object or structure to be analyzed using mesh structure analysis. The mesh data represents a mesh generated from the geometry data, and expresses the mesh using elements such as triangles or quadrangles. The boundary condition data represents the boundary conditions of a system or object. The boundary condition data includes fixed vertices or surfaces, and regions to which external forces or loads are applied. These boundary conditions are used to simulate the behavior of the object. The storage unit 2B is a storage device connected to a computer functioning as the iterative convergence calculation device 1B. The storage unit 2B may be built into the iterative convergence calculation device 1B, or may be a storage device included in an external device that can be connected to the computer via a communication unit provided in the computer via a wired or wireless network.

[0075] (Outline of Iterative Convergence Calculation Device) The iterative convergence calculation device 1B includes a mesh calculation unit 11B and an iterative control unit 12B. For example, a program constituting an information processing application for realizing each function of the mesh calculation unit 11B and the iterative control unit 12B is stored in a memory included in the computer. A processor included in the computer executes the information processing application read from the memory, thereby realizing each function of the mesh calculation unit 11B and the iterative control unit 12B.

[0076] The mesh calculation unit 11B is an iterative calculation unit that meshes the space using a set compression ratio. For example, the mesh calculation unit 11B performs processes such as generating an initial mesh, meshing the space, and performing iterative calculations based on the set compression ratio. For example, the mesh calculation unit 11B meshes the space using an initial value of mesh coarseness, and iteratively meshes the space and calculates the mesh coarseness by reducing the mesh coarseness until it reaches the target value.

[0077] The iterative control unit 12B is an iterative control unit that sequentially sets the mesh coarseness reduced by the update to the mesh calculation unit 11B and causes the mesh calculation unit 11B to repeatedly mesh the space. For example, the iterative control unit 12B instructs the mesh calculation unit 11B to mesh the space based on the updated mesh coarseness. When the mesh calculation unit 11B satisfies the convergence condition, the iterative control unit 12B determines that the mesh calculation unit 11B has finished meshing the space and causes the mesh calculation unit 11B to output a final solution. When the convergence condition is not satisfied, the iterative control unit 12B performs an update that further reduces the mesh coarseness and causes the mesh calculation unit 11B to continue meshing the space.

[0078] The iterative control unit 12B may also adjust the mesh coarseness update method and schedule to optimize convergence performance or calculation efficiency. This improves the efficiency of the iterative convergence calculation and enables effective use of calculation resources. Furthermore, the iterative control unit 12B manages the mesh coarseness set in the mesh calculation unit 11B during the process of meshing the space by the mesh calculation unit 11B. This improves calculation efficiency by repeatedly performing reduction in mesh coarseness and iterative convergence calculation, making it possible to analyze or learn large-scale problems.

[0079] Next, an iterative convergence calculation method according to the third embodiment will be described. FIG. 13 is a flowchart illustrating the iterative convergence calculation method according to the third embodiment, showing a series of operations performed by the iterative convergence calculation apparatus 1B. First, the iterative control unit 12B sets an initial value of the compression rate to the mesh calculation unit 11B (step ST1B). For example, based on the constraints of calculation resources and requirements related to model performance, the iterative control unit 12B determines the amount by which the mesh coarseness is reduced at each update and the final mesh coarseness estimated to obtain an analysis result with the target accuracy. Based on the determined mesh coarseness and the amount of reduction in mesh coarseness at each update, the iterative control unit 12B determines an initial value of the mesh coarseness so as to fall within the target calculation speed. The iterative control unit 12B sets the determined initial value of the mesh coarseness to the mesh calculation unit 11B.

[0080] The mesh calculation unit 11B performs iterative convergence calculations for meshing based on the mesh roughness set by the iterative control unit 12B as the mesh structure analysis (step ST2B). For example, the mesh calculation unit 11B updates the mesh roughness set previously and iteratively performs calculations necessary for meshing at the updated roughness. Here, the mesh calculation unit 11B may set the mesh roughly at equal intervals in the target space, or may set the mesh roughly by focusing only on the main component according to the values ​​of the mesh elements in the target space.

[0081] FIG. 14 is a diagram illustrating an overview of the meshing process. The diagram on the left in FIG. 14 shows a space that has been meshed according to the mesh coarseness previously set, resulting in a coarse mesh. A mesh coarseness that is reduced with each update is set in the mesh calculation unit 11B. The mesh calculation unit 11B calculates an interpolated mesh value according to the mesh coarseness reduced with each update, and inserts mesh elements corresponding to the interpolated value into the previously set mesh, thereby forming a fine mesh for the target space, as shown in the diagram on the right in FIG. 14. Examples of methods that can be used to calculate the interpolated value include linear interpolation and Catmull-Rom interpolation.

[0082] The iterative control unit 12B determines whether or not a termination condition for the iterative convergence calculation by the mesh calculation unit 11B is reached (step ST3B). For example, the iterative termination condition is a condition indicating whether or not a final mesh roughness, which is determined when the initial value of the mesh roughness is calculated and is estimated to provide an analysis result with a target accuracy, is reached.

[0083] If the iteration termination condition is met (step ST3B; YES), the series of processes in FIG. 13 is terminated. On the other hand, if the iteration termination condition is not met (step ST3B; NO), the iterative control unit 12B performs an update to reduce the mesh roughness (step ST4B). For example, if a mesh roughness reduction amount is set in the iterative control unit 12B, the iterative control unit 12B sets, in the mesh calculation unit 11B, a mesh roughness that is reduced by the reduction amount from the mesh roughness after the previous update. After this, the process returns to step ST2B, and the mesh calculation unit 11B performs an iterative convergence calculation of meshing using the updated mesh roughness. By executing the above method, the iterative convergence calculation device 1B can perform an iterative convergence calculation that simultaneously improves the calculation speed of spatial meshing and suppresses a decrease in the accuracy of the mesh structure analysis of the calculation results.

[0084] 14, mesh elements are set in a three-dimensional space, but the third embodiment is not limited to this. For example, a four-dimensional mesh may be handled in which a time dimension is added to the three dimensions of the target space. Furthermore, the iterative convergence calculation apparatus 1B according to the third embodiment may be applied to mesh analysis of a high-dimensional space of five or more dimensions (such as quantum state calculation).

[0085] As described above, in the iterative convergence calculation apparatus 1B according to the third embodiment, the mesh calculation unit 11B performs a meshing process to mesh the target space with a coarseness according to the set compression rate. The iterative control unit 12B sequentially sets the compression rate, which has been reduced by the update, in the mesh calculation unit 11B, and causes the mesh calculation unit 11B to repeatedly perform the meshing process. This enables the iterative convergence calculation apparatus 1B to perform an iterative convergence calculation that achieves both an improvement in the calculation speed for meshing the space and suppression of a decrease in the accuracy of the mesh structure analysis of the calculation results.

[0086] It is possible to combine the embodiments, modify any of the components of the embodiments, or omit any of the components of the embodiments.

[0087] The iterative convergence calculation device according to the present disclosure can be used for, for example, data analysis and optimization.

[0088] 1, 1A, 1B Iterative convergence calculation device, 2, 2A, 2B Memory unit, 11 Iterative calculation unit, 11A Iterative learning unit, 11B Mesh calculation unit, 12 Iterative control unit, 12A Learning control unit, 12B Iterative control unit, 20, 21-1, 21-2, 22-k, 23-k Perceptron, 100 Input interface, 101 Output interface, 102 Processor, 103 Memory.

Claims

1. An iterative convergence calculation device comprising: an iterative calculation unit that performs iterative convergence calculation based on a set compression ratio; and an iterative control unit that performs an update to reduce the compression ratio, sequentially sets the reduced compression ratios in the iterative calculation unit as a result of the update, and causes the iterative convergence calculation unit to repeatedly perform iterative convergence calculation.

2. The iterative convergence calculation device according to claim 1, characterized in that the iterative calculation unit performs training of the neural network compressed at the set compression rate, and the iterative control unit sequentially sets the compression rate reduced by updating in the iterative calculation unit, causing the iterative calculation unit to repeatedly train the neural network.

3. The iterative convergence calculation device according to claim 2, characterized in that, when the initial value of the compression ratio and the compression ratio reduced by updating are sequentially set by the iterative control unit, the iterative calculation unit calculates the neural network compressed at the set compression ratio.

4. The iterative convergence calculation device according to claim 3, wherein the iterative calculation unit compresses the number of learning parameters of the neural network by tensor decomposing a fully connected layer or a convolutional layer in the neural network.

5. The iterative convergence calculation device according to claim 3, characterized in that the iterative calculation unit reduces the number of learning parameters of the neural network by singular value decomposition of matrix operations in a fully connected layer or a convolutional layer of the neural network.

6. The iterative convergence calculation device according to claim 3, characterized in that the iterative calculation unit selects a number of weights determined according to the compression ratio from among the weights in the nodes of the neural network, and sets the other weights to 0.

7. An iterative convergence calculation device according to any one of claims 3 to 6, characterized in that the iterative control unit continues learning of the neural network when the residual from the target variable or the amount of change in the residual in the iterative convergence calculation is greater than a threshold, and terminates learning when it is equal to or less than the threshold.

8. An iterative convergence calculation device according to any one of claims 3 to 6, characterized in that the iterative control unit continues learning of the neural network when the number of times learning of the neural network has been repeated is less than a threshold number of times, and terminates learning when the number of times is equal to or greater than the threshold number of times.

9. An iterative convergence calculation device according to any one of claims 3 to 8, characterized in that the iterative control unit reduces the compression rate according to the progress of learning of the neural network.

10. The iterative convergence calculation device according to claim 1, characterized in that the iterative calculation unit performs a meshing process to mesh the target space at a coarseness according to the set compression ratio, and the iterative control unit sequentially sets the compression ratio lowered by updating in the iterative calculation unit, and causes the iterative calculation unit to repeatedly perform the meshing process.

11. An iterative convergence calculation method using an iterative convergence calculation device, comprising: a step in which an iterative calculation unit performs iterative convergence calculation based on a set compression ratio; and a step in which an iterative control unit performs an update to reduce the compression ratio, sequentially sets the reduced compression ratios in the iterative calculation unit as a result of the update, and repeatedly performs iterative convergence calculation by the iterative calculation unit.

12. A program for causing a computer to function as an iterative calculation unit that performs iterative convergence calculation based on a set compression ratio, and an iterative control unit that performs an update to reduce the compression ratio, sequentially sets the reduced compression ratio in the iterative calculation unit, and repeatedly performs iterative convergence calculation by the iterative calculation unit.

Images