Information processing program, information processing method, and information processing device

Abstract

To deal with issues that there are unnecessary calculations generated since training needs to be continued until failures of mechanical learning such as the fact that a loss value does not drop or that a value is a NaN (Not a Number) become obvious in deep learning, it is difficult to determine how much training has normally done even if the mechanical learning is to be returned to the check point, and there is no mechanism to return to an 8-bit data type after the learning has moved to an FP32 operation in an existing technique, and therefore calculations of the FP32 has to be thereafter continued.SOLUTION: An information processing program causes a computer to execute processing of: detecting a sign of a learning failure during an operation of a lower bit than an operation of a predetermined bit in deep learning; performing rollback at an operation part where the sign is detected to perform re-calculation by an operation of the predetermined bit; determining whether returning to an operation of the lower bit from an operation of the predetermined bit is to be approved; and shifting to the operation of the lower bit when the returning is possible.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an information processing program, an information processing method, and an information processing device. [Background technology]

[0002] In recent years, technology to optimize calculation accuracy has been attracting attention as a technique for speeding up machine learning. Machine learning calculations generally use 32-bit floating-point numbers, but there are many cases where fewer bits are sufficient for solving problems using machine learning. By calculating with fewer bits, benefits such as faster calculations, improved power performance, and reduced memory resources can be obtained.

[0003] Currently, many companies are working on the development and application of technologies to optimize the accuracy of calculations. For example, a technology is known that performs calculations with fewer bits by using 8-bit or 16-bit floating point numbers on a GPU (Graphics Processing Unit). Another known technology is one that performs inference using 8-bit integers by using a TPU (Tensor Processing Unit), a processor specialized for tensor calculations.

[0004] In this context, a processor has been proposed that performs calculations with optimal precision by varying the fixed point according to the stage of machine learning. This processor is designed specifically for deep learning, a type of machine learning, and optimizes calculation precision by taking advantage of the property that "the fluctuation between iterations of the values ​​to be calculated becomes smaller" and the property that "the distribution of values ​​held by tensors becomes narrower" as deep learning training progresses. Note that one iteration is one repetition in machine learning.

[0005] In the first half of the learning (hereinafter sometimes referred to as "pre-learning"), when the numerical fluctuations between iterations and the variance of the values ​​of tensors are large, training is performed using conventional 32-bit floating point, and in the middle to latter half (hereinafter sometimes referred to as "main learning"), when the variance becomes smaller, training is performed using 8-bit or 16-bit fixed-point arithmetic, where the decimal point position is variable. Note that 32-bit floating-point arithmetic includes, for example, FP32 arithmetic, and 8-bit or 16-bit fixed-point arithmetic includes, for example, DL-INT (Deep Learning Integer) arithmetic and QINT (Quantized Integer) arithmetic.

[0006] In fixed-point number arithmetic, the decimal point position of a numerical value can be adjusted as necessary to allow data to be expressed in 8 or 16 bits. In the case of DL-INT, the decimal point position is determined by a value (hereinafter sometimes referred to as the "Q value") determined from the distribution of decimal points of each element of a tensor (hereinafter sometimes referred to as "statistical information"). For example, during calculations, values ​​are read from the numerical data expressed in 8 bits and the Q value, and calculations are performed.

[0007] When calculating the Q value, the value "0" is calculated as a special value, although this is implementation-dependent. This is because in deep learning calculations, "0" is sometimes used as a tensor mask or for padding image regions, and if "0" were included in the Q value calculation as a normal value, the numerical data would not be fully represented.

[0008] In addition, 8-bit and 16-bit calculations have the disadvantage that they are prone to failure in machine learning. Therefore, as a means of avoiding learning failures, there is a method of periodically saving the state during training as a checkpoint, for example once every 5,000 iterations, and recalculating from the checkpoint when learning fails. In this way, in general, in deep learning, training proceeds while periodically creating such checkpoints. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] JP 7-84975 A [Patent Document 2] JP 2019-125014 A [Patent Document 3] JP 2020-161031 A [Patent Document 4] U.S. Patent No. 5,845,051 [Patent Document 5] International Publication No. 2018 / 139266 Summary of the Invention [Problem to be solved by the invention]

[0010] However, in conventional methods, training must be continued until the failure of machine learning becomes apparent, such as the loss value not decreasing or the value becoming NaN (Not a Number), resulting in a lot of wasted calculations. In addition, even if you return to a checkpoint, it is difficult to determine to what extent training was performed correctly. Furthermore, in conventional methods, for example, once the data has been switched to FP32 arithmetic, there is no mechanism to return to an 8-bit data type, so calculations must continue in FP32 arithmetic thereafter.

[0011] One aspect is to improve the efficiency of calculations in deep learning. [Means for solving the problem]

[0012] In the first proposal, in deep learning, a computer is caused to detect signs of learning failure during an operation of a lower bit than an operation of a specified bit, roll back to the operation point where the signs were detected and recalculate using the operation of the specified bit, determine whether it is possible to return from the operation of the specified bit to the operation of the lower bit, and if return is possible, execute a process to transition to the operation of the lower bit. Effect of the Invention

[0013] One aspect is to improve the efficiency of calculations in deep learning. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an example of calculation by the information processing device 100 according to the first embodiment. [Diagram 2] FIG. 2 is a flowchart of a usage scene of a user according to the first embodiment. [Diagram 3] FIG. 3 is a block diagram of a functional configuration of the information processing device 100 according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of an anomaly detection method according to the first embodiment. [Diagram 5] FIG. 5 is a diagram illustrating another example of the anomaly detection method according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a recovery process after the abnormality avoidance process according to the first embodiment. [Figure 7] FIG. 7 is a diagram of an example of functional blocks in the return process according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a data flow during normal operation according to the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of the Q value according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a process flow (1) of the main process according to the first embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of a process flow (2) of the main process according to the first embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of a processing flow of the sub-process according to the first embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, the information processing program, the information processing method, and the information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to these embodiments. In addition, each embodiment can be appropriately combined within a range that does not cause a contradiction. EXAMPLES

[0016] [Description of Information Processing Device 100] The information processing device 100 according to the first embodiment is an example of a computer that executes machine learning model construction using a machine learning framework that provides functions such as deep learning. In deep learning, in the pre-learning in the first half of the learning, training is performed using 32-bit floating-point arithmetic because of the large variation in values ​​between iterations and the large variance of values ​​of tensors, and in the main learning in the second half of the learning, training is performed using 8-bit or 16-bit fixed-point arithmetic in which the decimal point position is variable, thereby optimizing the arithmetic processing.

[0017] Fig. 1 is a diagram illustrating an example of a calculation by the information processing device 100 according to the first embodiment. As illustrated in Fig. 1, when the information processing device 10 receives an input of a problem that a user wants to solve, the information processing device 10 defines a calculation model for deriving a solution and performs learning of the calculation model to derive the solution to the problem.

[0018] The example of a computational model shown in Figure 1 shows a structure in which the graph structure changes depending on whether iteration = i or iteration = i+1. In general, statistical information is managed in association with each node that can be identified by "computation graph structure information," and when performing a calculation at each node of the computation graph, training is performed using the statistical information of the output of the corresponding calculation in the previous iteration instead of the decimal point position of the current iteration. Note that a computation graph is constructed by combining calculations, and in deep learning it is constructed by Add, Mul, Sum, Matrix Multiply, Convolution, etc.

[0019] Next, a flow when a user trains a computational model using the information processing device 100 will be described. Fig. 2 is a flowchart showing a flow of a usage scene of a user according to the first embodiment. As shown in Fig. 2, when a user inputs information about a problem to be solved to the information processing device 100 (S101), the information processing device 100 uses a general machine learning algorithm to determine a computation graph and a computation path from the input information (S102).

[0020] Next, the information processing device 100 executes arithmetic processing in each node (S103 to S104). Specifically, the information processing device 100 executes arithmetic processing set in the node using information of the input tensor, generates a hash value, and outputs the arithmetic result and the hash value as an output tensor to the next node.

[0021] After that, when the learning is completed and the structure of the computational model is determined, the information processing device 100 converts it into a solution form that the user can understand (S105), and outputs the solution to the problem that the user wants to solve to a display, a memory unit, a user terminal, etc. (S106).

[0022] [Functional configuration of information processing device 100] 3 is a block diagram illustrating a functional configuration of the information processing device 100 according to the first embodiment. As illustrated in FIG. 3, the information processing device 100 includes a control unit 110, a calculation unit 120, a deep learning dedicated calculation unit 130, and a database 140.

[0023] The control unit 110 is a processing unit that controls the entire information processing device 100, and is, for example, a processor. The control unit 110 has a Q value calculation function, an abnormality detection function, a rollback function, a recovery function, etc. Each function is an example of an electronic circuit that the processor has or an example of a process that the processor executes.

[0024] The control unit 110 detects a sign of learning failure during a calculation of a lower bit than FP32 calculation in deep learning. The control unit 110 also rolls back to the calculation point where the sign of learning failure was detected, and instructs the calculation unit 120 and the deep learning dedicated calculation unit 130 to perform recalculation with FP32 calculation. The control unit 110 also determines whether or not it is possible to return from FP32 calculation to a low-bit calculation such as DL-INT calculation or QINT calculation, and if it is possible to return, instructs the calculation unit 120 and the deep learning dedicated calculation unit 130 to transition to the low-bit calculation.

[0025] The process of detecting a sign of learning failure includes a process of detecting the sign when the Q value difference of the input tensor is outside the allowable range or when the fluctuation range of the Q value of the output tensor is equal to or greater than a predetermined threshold. The process of detecting a sign of learning failure also includes a process of determining whether or not the sampling values ​​for calculating the Q value are all 0, and when the sampling values ​​are all 0, detecting the sign based on past Q values. The process of detecting a sign of learning failure also includes a process of detecting the sign when the number of elements that have overflowed or underflowed with respect to the number of elements to be sampled is equal to or greater than a predetermined threshold.

[0026] The calculation unit 120 is a processing unit that executes pre-processing of calculations for machine learning, memory control related thereto, etc. For example, in the case of pre-learning, the calculation unit 120 outputs the calculation type and calculation parameters of the calculation node to the deep learning dedicated calculator 130 and requests a 32-bit floating point calculation.

[0027] In addition, for example, in the case of main learning, the calculation unit 120 specifies the decimal point position of the calculation according to the statistical information. After that, the calculation unit 120 outputs the calculation type of the calculation node, the calculation parameters, and the decimal point position to the deep learning dedicated calculator 130, and requests a fixed point calculation such as a DL-INT calculation.

[0028] The deep learning dedicated computing unit 130 is an accelerator that handles various calculations of the calculation node, and more specifically, refers to a GPU (Graphics Processing Unit) or a TPU (Tensor Processing Unit). If the information processing device 100 does not have an accelerator, the host CPU (Central Processing Unit) handles the calculations. In this case, the information processing device 100 does not need to have the deep learning dedicated computing unit 130.

[0029] Furthermore, when pre-learning is instructed, the deep learning dedicated arithmetic unit 130 executes 32-bit floating point arithmetic using the notified operation type and operation parameters of the calculation node, and outputs the operation result. When main learning is instructed, the deep learning dedicated arithmetic unit 130 executes fixed point arithmetic such as DL-INT arithmetic using the notified decimal point position, using the notified operation type and operation parameters of the calculation node, and outputs the operation result. Note that the deep learning dedicated arithmetic unit 130 may be, for example, a processing unit executed by a deep learning dedicated AI (Artificial Intelligence) processor (DLU) or the like, and may be an example of an electronic circuit possessed by the DLU or an example of a process executed by the DLU.

[0030] The database 140 is an example of a storage device that stores various data and programs executed by the control unit 110, and is, for example, a memory or a hard disk.

[0031] [Function details of the information processing device 100] Next, the details of the functions of the information processing device 100 according to the first embodiment will be described with reference to FIG. 4-12. In the description of the first embodiment, DL-INT operation is used as an example of low-bit operation in deep learning, but QINT operation may be used. In addition, the operation device can be applied to both a CPU and an accelerator, but the description will be given assuming that the operation device is a CPU.

[0032] In Example 1, which aims to improve the efficiency of calculations in deep learning, deep learning is realized by the following three steps, which are roughly divided: (1) detection of a sign of learning failure in DL-INT calculation, (2) rolling back to the point where the abnormality was detected and recalculating with FP32 calculation, and (3) transitioning to DL-INT calculation when it is possible to return from FP32 calculation to DL-INT calculation.

[0033] In the deep learning according to the first embodiment, machine learning by DL-INT calculation is divided into two processes, a main process and a sub-process, and proceeds. This is because the sub-process performs the abnormality detection process and the Q value calculation of DL-INT calculation, and the main process concentrates on the calculation. In the information processing device 100, the above three steps are processed asynchronously by these two processes. The processing flow of the main process is shown in Figs. 10 and 11, and the processing flow of the sub-process is shown in Fig. 12, but each process will be described later.

[0034] Next, a method for detecting a sign of learning failure during low-bit calculation in deep learning will be described in more detail with reference to Figures 4 and 5. There are four patterns of signs of learning failure during low-bit calculation in deep learning, and in the first embodiment, anomalies in each pattern can be detected using the following four types of parameters. The four types of parameters are (1) the Q value difference of the input tensor, (2) the fluctuation range of the Q value of the output tensor, (3) the sampling value for calculating the Q value, and (4) the number of elements that overflow or underflow with respect to the number of elements to be sampled.

[0035] [Anomaly detection method based on Q value difference of input tensors] The Q value varies depending on the tensor. For example, operations that receive multiple inputs, such as Add, Sub, and Sum, may have different Q values ​​for the input tensors. The Q value represents the decimal point position, and the allowable difference in input Q values ​​varies in fixed-point number operations. Therefore, a condition for an allowable input Q value is defined for each operation, and the information processing device 100 checks whether the input Q value satisfies the condition before starting the operation. Note that the condition may be set to a generic value as a default value, and may be changed later by the user.

[0036] Here, an anomaly detection based on the Q value difference of the input tensor will be described with a concrete example. For example, if the allowable range of the Q value difference of Add is set to 2, in the calculation: Z = X + Y, (Q X , Q Y If (Q )=(2, 1), the Q value difference is within the allowable range, and the information processing device 100 determines that it is operating normally. X , Q Y When Q value difference (Q value difference) is greater than the allowable range, the information processing device 100 detects an abnormality.

[0037] Similarly, if the tolerance for the Q value difference of the convolution is set to 2, then in the calculation: Y = Conv(X, Weight), (Q X , Q w When (Q )=(3, 4), the Q value difference is within the allowable range, so the information processing device 100 determines that it is operating normally. X , Q w If Q value difference (Q value)=(1, 4), the Q value difference exceeds the allowable range, and the information processing device 100 detects an abnormality.

[0038] Next, the process when an abnormality is detected with respect to the Q-value difference of the input tensor will be described. FIG. 4 is a diagram showing an example of the abnormality detection method according to the first embodiment. The example of FIG. 4 shows a computational graph including operations A-D. For example, as shown in FIG. 4, when the abnormality detection function detects an abnormality in operations A or B by the DL-INT operation (NG route), the information processing apparatus 100 inserts a conversion operation (Cast) to the FP32 operation and executes operations C and D by the FP32 operation. On the other hand, when the abnormality detection function does not detect an abnormality (OK route), the information processing apparatus 100 continuously executes operations C and D by the DL-INT operation.

[0039] The determination of whether to execute by the DL-INT operation or shift to the FP32 operation and execute by such an abnormality detection function is made by checking the data type of the input tensor of each operation. The determination corresponds to, for example, the determination of "whether FP32 is included in the data type of the input Tensor" in the main process shown in FIG. 10.

[0040] Note that not only the operation corresponding to the detected abnormality is made FP32, but also the subsequent operations are made FP32 because if there is an operation where an abnormality has occurred once, there is a high possibility that the subsequent operations will also cause an abnormality. Therefore, by setting that when the data type of the input tensor is FP32, the calculation is performed by the FP32 operation, the information processing apparatus 100 can propagate the occurrence of an abnormality in the previous operation to the subsequent operations.

[0041] [Abnormality Detection Method Based on the Variation Range of the Q-Value of the Output Tensor] Next, the abnormality detection method based on the variation range of the Q-value of the output tensor will be described. In 8-bit learning, it is assumed that the variation between iterations of the numerical values to be calculated is small. Therefore, when the variation range of the Q-value is large between iterations, the assumption is broken. For abnormality detection, the formula: |Q - Q avg | < T is used, and the information processing apparatus 100 determines that an abnormality has been detected when the formula is not satisfied. Here, Q in the formula is the Q-value of the output tensor, Q avgrepresents the average value of the Q values ​​of the output tensor of the past N times, and T represents the threshold. Note that the past number of times N and the threshold value T may be set to general-purpose values ​​as default values ​​and may be changed later by the user.

[0042] Next, a process when an abnormality is detected in the fluctuation range of the Q value of the output tensor will be described. FIG. 5 is a diagram showing another example of the abnormality detection method according to the first embodiment. In the example of FIG. 5, calculations A, B, and C by DL-INT calculation are executed in order. For example, the information processing device 100 checks the DL-INT abnormality flag before executing calculation B, and determines whether an abnormality has occurred in the calculation before calculation B. This determination corresponds to, for example, the determination of "Is the DL-INT abnormality flag of the abnormality detection function set?" in the main process shown in FIG. 11.

[0043] When the information processing device 100 detects an abnormality based on the DL-INT abnormality flag, the information processing device 100 rolls back to the calculation point where the abnormality occurred based on the calculation ID where the abnormality occurred, and performs recalculation using FP32 calculation.

[0044] 5, the information processing device 100 records a unique ID (operation ID) of an operation in which an abnormality has occurred when the information processing device 100 detects an abnormality in the database 140. Then, when the information processing device 100 confirms that the DL-INT abnormality flag is set, it determines a rollback destination based on the operation ID.

[0045] After the rollback, the information processing device 100 resumes the processing by FP32 arithmetic. The resumption of the processing corresponds to, for example, the Yes route of "Is the processing started by rollback?" in the main process shown in FIG. 11. Similarly to the anomaly detection method based on the Q value difference of the input tensor, the information processing device 100 executes the subsequent calculation after the rollback by FP32 arithmetic. Note that the determination of whether to roll back and execute by FP32 arithmetic also corresponds to, for example, the determination of "Does the data type of the input Tensor include FP32?" in the main process shown in FIG. 10, similar to the anomaly detection method based on the Q value difference of the input tensor.

[0046] Moreover, if no abnormality is detected by the DL-INT abnormality flag, the information processing device 100 continues to execute the operation B in the DL-INT operation. After execution of the operation B, as shown in FIG. 5, the information processing device 100 calculates a new Q value from the obtained numerical data and the used Q value, and inputs a JOB that performs CHECK to the sub-process. Here, the used Q value is, for example, the Q value of the previous intention. Moreover, inputting to the sub-process corresponds to, for example, "input Q value calculation process to sub-process" in the main process shown in FIG. 11. Note that the main process by the information processing device 100 executes the operation B without waiting for the completion of the processing in the sub-process.

[0047] Meanwhile, the sub-process by the information processing device 100 processes a JOB input from the main process. The sub-process starts processing after the calculation of calculation B is completed and output numerical data is obtained. In the sub-process, the information processing device 100 receives the calculation result and the calculation ID from the main process. Next, the information processing device 100 samples the output numerical data of the calculation result, and randomly extracts elements to be used in the Q-value calculation. This sampling corresponds to, for example, "sampling a value to be used in the Q-value calculation from the numerical data" in the sub-process shown in FIG. 12.

[0048] Next, the information processing device 100 performs a check process on the obtained numerical data, the Q value. This check process corresponds to the sub-process "CHECK on the Q value change amount" shown in FIG. 12. avg is calculated from the Q values ​​registered in the database 140 up to now. If the information processing device 100 detects an abnormality, it sets a DL-INT abnormality flag and records the calculation ID. If a JOB has already been submitted to the sub-process, that JOB is cleared since it has become unnecessary due to rollback. On the other hand, if no abnormality is detected, the information processing device 100 registers the Q value in the database 140. This registration process corresponds to the sub-process "Register Q value in database" shown in FIG. 12.

[0049] [Anomaly detection method using sampled values ​​to calculate Q value] Next, an anomaly detection method using sampled values ​​for calculating a Q value will be described. The information processing device 100 uses output numerical data to calculate the Q value, but since it is actually expensive to calculate from all values, the Q value is calculated from sampled numerical data.

[0050] At this time, as described above, the value "0" is treated as a special value and is not included in the calculation of the Q value. Therefore, when the sampled numerical values ​​are all "0" (All 0), the Q value is also a special value. Note that when a special Q value is used, the actual value cannot be obtained from 8-bit numerical data. Also, at this time, the information processing device 100 distinguishes whether All 0 occurred by chance due to sampling or whether All 0 occurred inevitably due to calculation. If All 0 occurs inevitably due to calculation, a special Q value can be used.

[0051] Here, a specific example will be given to explain an example where All 0 occurs. For example, as an example where All 0 inevitably occurs, as shown in the following formula (1), when calculating an inner product, if one side is a 0 vector, the output will also be 0.

[0052]

number

[0053] As an example of All 0 occurring by chance, the matrix operation of the input tensor and weights may result in 0, as shown in the following equation (2).

[0054]

number

[0055] In order to distinguish between these two cases in which All 0 occurs, when All 0 occurs, the information processing device 100 calculates the Q value from the past Q value.avg Calculate Q avg Also check that Q is a special value. avg If the DL-INT value is not a special value, the information processing device 100 sets a DL-INT abnormality flag as an abnormal case.

[0056] The process when an anomaly is detected in the sampling value for calculating the Q value is the same as the process when an anomaly is detected in the fluctuation range of the Q value of the output tensor described above. Note that the CHECK process by the anomaly detection method using the sampling value for calculating the Q value corresponds to, for example, the "CHECK for ALL "0"" subprocess shown in FIG.

[0057] [Method of detecting anomalies based on the number of overflowed or underflowed elements] Next, we will explain the method of detecting anomalies based on the number of overflowed or underflowed elements. In DL-INT calculations, when the number of bits in the integer part is n and the number of bits in the decimal part is m, the expressible range is -(2 n-1 )~2 n-1 -2 -m (n+m+1=8). If the output obtained by the calculation exceeds this range, an overflow or underflow occurs. If the number of overflowed or underflowed elements is small, there is no problem in continuing learning, but if the ratio of overflowed or underflowed elements becomes high, learning will fail. Therefore, the information processing device 100 checks whether the ratio of overflowed or underflowed elements exceeds a certain value.

[0058] Here, a specific example will be given to explain how to check whether the ratio of the number of overflowed or underflowed elements exceeds a certain value. For example, if the number of elements to be sampled is 256, the number of overflowed or underflowed elements is n, and the threshold is 5% or less, the information processing device 100 detects an abnormality when n / 256 is 5% or more, and determines normal operation when it is less than 5%. If the information processing device 100 detects an abnormality, it sets a DL-INT abnormality flag.

[0059] The process when an anomaly is detected in the number of overflowed or underflowed elements is the same as the process when an anomaly is detected in the fluctuation range of the Q value of the output tensor described above. Note that the CHECK process using the anomaly detection method based on the number of overflowed or underflowed elements corresponds to the subprocess "CHECK for overflow / underflow" shown in Figure 12, for example.

[0060] [Return process] Next, a description will be given of a return process after an abnormality is detected. If an abnormality is detected and the process continues to shift to FP32 calculation, the processing performance will remain degraded. Therefore, when the abnormality is resolved and a return to DL-INT calculation is possible, the information processing device 100 will shift from FP32 calculation to DL-INT calculation to recover performance.

[0061] Fig. 6 is a diagram illustrating an example of a recovery process after an abnormality avoidance process according to the embodiment 1. Fig. 6 illustrates a calculation status indicating a calculation state, a transition thereof, a transition condition to each calculation status, and a data type of a calculation when each calculation status is in a state.

[0062] Fig. 7 is a diagram illustrating an example of a functional block in the recovery process according to the embodiment 1. As illustrated in Fig. 7, in each calculation such as the DL-INT calculation or the FP32 calculation, the information processing device 100 holds an internal state, which is a calculation status, and a state counter, which is a counter of each state.

[0063] In the DL-INT Enable state, the information processing device 100 checks the data type of the previous iteration, and if the data type of the operation is FP32, changes the state to DL-INT Disable. Also, if the input tensor contains FP32, the information processing device 100 changes the state to DL-INT Disable. In addition, in the DL-INT Disable state, the information processing device 100 checks the return condition and attempts to return to DL-INT operation. The return condition is, for example, (1) that training has been repeated K times in the DL-INT Disable state, or (2) that the retry rate for the past L times is equal to or less than a predetermined threshold value X%. The retry rate may be the abnormality occurrence rate.

[0064] The values ​​L, K, and X may be default values ​​that are general-purpose and can be changed later by the user. The retry rate can be calculated, for example, using the formula: Retry rate = DL-INT Retry count in the past L times / DL-INT Enable count in the past L times x 100. The information processing device 100 can record the DL-INT Enable count, DL-INT Disable count, and DL-INT Retry count in the past L times in a status counter, and obtain each value from the status counter.

[0065] Then, when the above-mentioned restoration conditions (1) and (2) are satisfied, the information processing device 100 changes the state from DL-INT Disable to DL-INT Enable. Note that this state update corresponds to, for example, "update of state counter" of the main process shown in FIG. 10.

[0066] Note that reversion condition (1) is effective, for example, when the cause of the learning failure is insufficient pre-learning. By repeating training further in the DL-INT Disable state, the properties of the numerical values ​​to be calculated being small between iterations and the properties of the distribution of the tensor values ​​being narrow are satisfied, and it is expected that training will proceed normally after reversion. Furthermore, reversion condition (2) is effective when the calculation is potentially unstable. Depending on the configuration of the computation graph, there may be calculations that are unstable when converted to DL-INT, so for calculations with a high probability of occurrence of abnormalities, it is acceptable to leave them converted to FP32 without reverting.

[0067] FIG. 8 is a diagram showing an example of a data flow during normal operation in the DL-INT Enable state. FIG. 9 is a diagram showing an example of a Q value according to the first embodiment. FIG. 8 is an example when arithmetic ID=2, and arithmetic-2 has two inputs and one output. Q[i][t] in the database 140 is a Q value in arithmetic ID=i and iteration=t. In the example shown in FIG. 9, Q[2][t-1] is the Q value referred to in FIG. 8, and Q[2][t] is the updated Q value. Each numerical data is a multidimensional matrix.

[0068] [effect] As described above, in deep learning, the information processing program detects signs of learning failure during lower bit operations than a specified bit operation, such as FP32 operations, rolls back to the operation point where the signs of learning failure were detected and recalculates with the specified bit operation, determines whether it is possible to return from the specified bit operation to the lower bit operation, and if it is possible to return to the low bit operation, causes the information processing device 100 to execute processing to transition to the low bit operation.

[0069] As a result, the information processing device 100 can detect signs of learning failure as an anomaly and minimize the processing required for recalculation by rolling back before the failure of machine learning becomes apparent, such as when the loss value does not decrease or the value becomes NaN. In addition, the information processing device 100 can achieve high-speed calculation during deep learning, improved power performance, and memory resource saving by returning from a predetermined bit operation to a low bit operation. In this way, the information processing device 100 can achieve the efficiency of calculation in deep learning.

[0070] In addition, the process of transitioning to a low-bit calculation includes a process of transitioning to a DL-INT calculation or a QINT calculation.

[0071] This enables the information processing device 100 to improve performance during deep learning.

[0072] The process of detecting a sign of learning failure also includes a process of detecting a sign when the Q value difference of the input tensor is outside the allowable range.

[0073] This enables the information processing device 100 to detect signs of learning failure as abnormal values, thereby improving the efficiency of calculations in deep learning.

[0074] The process of detecting a sign of learning failure also includes a process of detecting the sign when the fluctuation range of the Q value of the output tensor is equal to or greater than a predetermined threshold.

[0075] This enables the information processing device 100 to detect signs of learning failure as abnormal values, thereby improving the efficiency of calculations in deep learning.

[0076] In addition, the process of detecting a sign of learning failure includes a process of determining whether or not all sampling values ​​for calculating the Q value are 0, and if all sampling values ​​are 0, detecting a sign based on past Q values.

[0077] This enables the information processing device 100 to detect signs of learning failure as abnormal values, thereby improving the efficiency of calculations in deep learning.

[0078] The process of detecting a sign of learning failure also includes a process of detecting the sign when the number of elements that have overflowed or underflowed with respect to the number of elements to be sampled is equal to or greater than a predetermined threshold value.

[0079] This enables the information processing device 100 to detect signs of learning failure as abnormal values, thereby improving the efficiency of calculations in deep learning.

[0080] The process of determining whether or not return is possible includes a process of repeating training with a predetermined bit operation a first predetermined number of times, and determining that return to low-bit operation is possible if the abnormality occurrence rate is equal to or less than a second predetermined number of times.

[0081] This enables the information processing device 100 to improve performance in deep learning while normally performing training.

[0082] [system] The information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. In addition, the specific examples, distributions, values, etc. described in the embodiments are merely examples and can be changed as desired.

[0083] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure. In other words, all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0084] Furthermore, each processing function performed by each device may be realized, in whole or in part, by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware using wired logic.

[0085] [Hardware] The hardware configuration of the above-mentioned information processing device 100 will be described. Fig. 13 is a diagram showing an example of the hardware configuration. As shown in Fig. 13, the information processing device 100 has a communication unit 100a, a HDD (Hard Disk Drive) 100b, a memory 100c, and a processor 100d. Moreover, each unit shown in Fig. 13 is connected to each other via a bus or the like.

[0086] The communication unit 100a is a network interface card or the like, and communicates with other servers. The HDD 100b stores programs and databases that operate the functions shown in FIG.

[0087] The processor 100d reads out a program that executes the same processes as the respective processing units shown in FIG. 3 from the HDD 100b or the like and loads it in the memory 100c, thereby operating a process that executes each function described in FIG. 3. For example, this process executes a function similar to that of each processing unit possessed by the information processing device 100. Specifically, for example, the processor 100d reads out a program having a function similar to that of each processing unit shown in FIG. 3 from the HDD 100b or the like. Then, the processor 100d executes a process that executes a process similar to that of the processing by each processing unit shown in FIG. 3.

[0088] In this way, the information processing device 100 operates as an information processing device that executes each process by reading and executing a program. The information processing device 100 can also realize functions similar to those of the above-mentioned embodiment by reading the program from a recording medium using a medium reading device and executing the read program. Note that the program in this other embodiment is not limited to being executed by the information processing device 100. For example, the present invention can be similarly applied to a case where another computer or server executes a program, or a case where these execute a program in cooperation with each other.

[0089] This program can be distributed via a network such as the Internet. This program can also be recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), or a digital versatile disk (DVD), and can be executed by being read out from the recording medium by a computer.

[0090] In addition, the following supplementary notes are disclosed regarding the embodiments including the above-mentioned Example 1.

[0091] (Appendix 1) In deep learning, detecting signs of learning failure during operations of lower bits than the operations of a given bit, Rolling back to the calculation point where the symptom was detected and recalculating the predetermined bit; determining whether or not it is possible to return from the operation of the predetermined bit to the operation of the low bit; If it is possible to return to the low-bit operation, the low-bit operation is performed. An information processing program that causes a computer to execute a process.

[0092] (Supplementary Note 2) The information processing program according to Supplementary Note 1, wherein the process of transitioning to a low-bit operation includes a process of transitioning to a DL-INT operation or a QINT operation.

[0093] (Supplementary Note 3) The information processing program according to Supplementary Note 1 or 2, characterized in that the process of detecting the symptom includes a process of detecting the symptom when a Q value difference of an input tensor is outside a tolerance range.

[0094] (Appendix 4) The information processing program according to appendix 1 or 2, characterized in that the process of detecting the sign includes a process of detecting the sign when a fluctuation range of the Q value of the output tensor is equal to or greater than a predetermined threshold.

[0095] (Appendix 5) The process of detecting the symptom is It is determined whether or not all the sampling values ​​for calculating the Q value are 0; If the sampling values ​​are all 0, the symptom is detected based on the past Q value. 3. The information processing program according to claim 1 or 2, comprising:

[0096] (Appendix 6) The information processing program according to appendix 1 or 2, characterized in that the process of detecting the symptom includes a process of detecting the symptom when the number of elements that have overflowed or underflowed with respect to the number of elements to be sampled is equal to or greater than a predetermined threshold.

[0097] (Appendix 7) The process for determining whether or not the recovery is possible is as follows: The training is repeated a first predetermined number of times with respect to the operation of the predetermined bit, and when the abnormality occurrence rate is equal to or less than a second predetermined number of times, it is determined that a return to the operation of the low bit is possible. 7. The information processing program according to claim 1, further comprising:

[0098] (Appendix 8) In deep learning, detecting signs of learning failure during operations of lower bits than the operations of a given bit, Rolling back to the calculation point where the symptom was detected and recalculating the predetermined bit; determining whether or not it is possible to return from the operation of the predetermined bit to the operation of the low bit; If it is possible to return to the low-bit operation, the low-bit operation is performed. An information processing method characterized in that the processing is executed by a computer.

[0099] (Supplementary Note 9) The information processing method according to Supplementary Note 8, wherein the process of transitioning to a low-bit operation includes a process of transitioning to a DL-INT operation or a QINT operation.

[0100] (Supplementary Note 10) The information processing method according to Supplementary Note 8 or 9, characterized in that the process of detecting the symptom includes a process of detecting the symptom when a Q value difference of an input tensor is outside a tolerable range.

[0101] (Supplementary Note 11) The information processing method according to Supplementary Note 8 or 9, characterized in that the process of detecting the sign includes a process of detecting the sign when the fluctuation range of the Q value of the output tensor is equal to or greater than a predetermined threshold.

[0102] (Appendix 12) The process of detecting the symptom comprises: It is determined whether or not all the sampling values ​​for calculating the Q value are 0; If the sampling values ​​are all 0, the symptom is detected based on the past Q value. 10. The information processing method according to claim 8 or 9, further comprising the steps of:

[0103] (Appendix 13) The information processing method according to appendix 8 or 9, characterized in that the process of detecting the symptom includes a process of detecting the symptom when the number of elements that have overflowed or underflowed with respect to the number of elements to be sampled is equal to or greater than a predetermined threshold.

[0104] (Appendix 14) The process for determining whether or not the recovery is possible is as follows: The training is repeated a first predetermined number of times with respect to the operation of the predetermined bit, and when the abnormality occurrence rate is equal to or less than a second predetermined number of times, it is determined that a return to the operation of the low bit is possible. 14. The information processing method according to any one of claims 8 to 13, further comprising the step of:

[0105] (Appendix 15) In deep learning, detecting signs of learning failure during an operation of a lower bit than an operation of a predetermined bit, Rolling back to the calculation point where the symptom was detected and recalculating the predetermined bit; determining whether or not it is possible to return from the operation of the predetermined bit to the operation of the low bit; If it is possible to return to the low-bit operation, the low-bit operation is performed. An information processing device comprising a control unit for executing processing.

[0106] (Supplementary Note 16) The information processing device according to Supplementary Note 15, wherein the process of transitioning to a low-bit operation includes a process of transitioning to a DL-INT operation or a QINT operation.

[0107] (Supplementary Note 17) The information processing device according to Supplementary Note 15 or 16, wherein the process of detecting the symptom includes a process of detecting the symptom when a Q value difference of an input tensor is outside a tolerance range.

[0108] (Supplementary Note 18) The information processing device according to Supplementary Note 15 or 16, characterized in that the process of detecting the symptom includes a process of detecting the symptom when a fluctuation range of the Q value of the output tensor is equal to or greater than a predetermined threshold.

[0109] (Appendix 19) The process of detecting the symptom comprises: It is determined whether or not all the sampling values ​​for calculating the Q value are 0; If the sampling values ​​are all 0, the symptom is detected based on the past Q value. 17. The information processing device according to claim 15 or 16, further comprising a process.

[0110] (Addendum 20) The information processing device according to Addendum 15 or 16, characterized in that the process of detecting the symptom includes a process of detecting the symptom when the number of elements that have overflowed or underflowed with respect to the number of elements to be sampled is equal to or greater than a predetermined threshold.

[0111] (Appendix 21) The process for determining whether or not the return is possible is as follows: The training is repeated a first predetermined number of times with respect to the operation of the predetermined bit, and when the abnormality occurrence rate is equal to or less than a second predetermined number of times, it is determined that a return to the operation of the low bit is possible. 21. The information processing device according to any one of claims 15 to 20, further comprising a process.

[0112] (Appendix 22) A processor; a memory operatively connected to said processor; An information processing device comprising: In deep learning, a sign of learning failure during an operation of a lower bit than an operation of a predetermined bit is detected; Rolling back to the calculation point where the symptom was detected and recalculating the predetermined bit; determining whether or not it is possible to return from the operation of the predetermined bit to the operation of the low bit; If it is possible to return to the low-bit operation, the low-bit operation is performed. 2. An information processing device that executes processing. EXAMPLES

[0113] Although the embodiments of the present invention have been described above, the present invention may be embodied in various different forms other than the above-described embodiments. [Explanation of symbols]

[0114] 100 Information processing device 100a Communications Department 100b HDD 100c Memory 100d Processor 110 Control section 120 Arithmetic section 130 Deep learning dedicated computing device 140 Databases

Claims

1. Using deep learning including pre-learning using a floating point of a predetermined number of bits and main learning using a floating point of a low bit number having a number of bits less than the predetermined number of bits after the completion of the pre-learning, when main learning is performed on a computation graph having a plurality of nodes each corresponding to an operation, detecting a sign of learning failure during the operation of the low bit number, When the symptom is detected in a first node among the plurality of nodes, a rollback is performed to the first node, and recalculation is performed using the operation of the predetermined bit, and the operation of the predetermined bit is executed in each node after the first node; determining whether or not to return from the operation of the predetermined bit to the operation of the lower bit every time an operation of the predetermined bit is executed in each node after the first node; When a second node capable of returning to the low-bit operation is detected, the low-bit operation is executed in each node after the second node. The process is executed by a computer, The process of detecting a sign of learning failure and the process of determining whether or not the learning can be restored are executed in a main process of the deep learning, The process of executing the calculation is performed in a sub-process of the deep learning, The information processing program, wherein the main process and the sub-process are executed asynchronously.

2. 2. The information processing program according to claim 1, wherein the process of executing a low-bit operation includes a process of executing a DL-INT (Deep Learning Integer) operation or a QINT (Quantized Integer) operation.

3. 3. The information processing program according to claim 1, wherein the process of detecting the symptom includes a process of detecting the symptom when a Q value difference of an input tensor is outside a tolerance range.

4. 3. The information processing program according to claim 1, wherein the process of detecting the symptom includes a process of detecting the symptom when a fluctuation range of a Q value of an output tensor is equal to or greater than a predetermined threshold.

5. The process of detecting the symptom includes: determining whether or not all sampling values ​​for calculating the Q value are 0; If the sampling values ​​are all 0, the symptom is detected based on the past Q value.

3. The information processing program according to claim 1, further comprising a process for:

6. 3. The information processing program according to claim 1, wherein the process of detecting the symptom includes a process of detecting the symptom when the number of elements that have overflowed or underflowed with respect to the number of elements to be sampled is equal to or greater than a predetermined threshold value.

7. The process of determining whether or not the return is possible is The training is repeated a first predetermined number of times with the operation of the predetermined bit, and when the abnormality occurrence rate is equal to or less than a second predetermined number of times, it is determined that a return to the operation of the low bit is possible.

7. The information processing program according to claim 1, further comprising a process for:

8. Using deep learning including pre-learning using a floating point of a predetermined bit and main learning using a floating point of a low bit having a bit number less than the predetermined bit after the completion of the pre-learning, when main learning is performed on a computation graph having a plurality of nodes each corresponding to an operation, detecting signs of learning failure during the operation of the low bit, When the symptom is detected in a first node among the plurality of nodes, a rollback is performed to the first node, and recalculation is performed using the operation of the predetermined bit, and the operation of the predetermined bit is executed in each node after the first node; determining whether or not to return from the operation of the predetermined bit to the operation of the lower bit every time an operation of the predetermined bit is executed in each node after the first node; When a second node capable of returning to the low-bit operation is detected, the low-bit operation is executed in each node after the second node. The computer executes the process, The process of detecting a sign of learning failure and the process of determining whether or not the learning can be restored are executed in a main process of the deep learning, The process of executing the calculation is performed in a sub-process of the deep learning, The information processing method according to claim 1, wherein the main process and the sub-process are executed asynchronously.

9. Using deep learning including pre-learning using a floating point of a predetermined bit and main learning using a floating point of a low bit having a bit number less than the predetermined bit after the completion of the pre-learning, when main learning is performed on a computation graph having a plurality of nodes each corresponding to an operation, detecting signs of learning failure during the operation of the low bit, When the symptom is detected in a first node among the plurality of nodes, a rollback is performed to the first node, and recalculation is performed using the operation of the predetermined bit, and the operation of the predetermined bit is executed in each node after the first node; determining whether or not to return from the operation of the predetermined bit to the operation of the lower bit every time an operation of the predetermined bit is executed in each node after the first node; When a second node capable of returning to the low-bit operation is detected, the low-bit operation is executed in each node after the second node. A control unit that executes processing, The process of detecting a sign of learning failure and the process of determining whether or not the learning can be restored are executed in a main process of the deep learning, The process of executing the calculation is performed in a sub-process of the deep learning, 2. An information processing apparatus, comprising: a main process and a sub-process that are executed asynchronously.

Images