Inference pre-processing device and inference pre-processing method

Abstract

Provided is an inference pre-processing device comprising: a hardware model construction unit that refers to an inference hardware configuration and converts an input inference model into a trial model capable of processing equivalent to the inference hardware; an inference trial unit that uses the trial model and inference target sample data to repeat inference simulations while changing a fixed-point position; an accuracy measurement unit that uses correct answer data of the sample data to evaluate the inference simulation results of the inference trial unit; and a fixed-point position determination unit that determines an initial fixed-point position for dynamic fixed-point position control of the inference hardware, on the basis of the result of the evaluation by the accuracy measurement unit.

Description

Inference preprocessing device and inference preprocessing method

[0001] The disclosed technology relates to an inference preprocessing device and an inference preprocessing method.

[0002] Deep learning is a machine learning technique that uses neural networks that replicate the mechanisms of human neurons, and is applied to fields such as video processing and natural language processing. In the field of video processing, many methods have been proposed that use deep learning models called convolutional neural networks (CNNs) to perform image recognition, which determines the class of an object in an image, object detection, which determines the position and class of an object, and semantic segmentation, which infers the class of an object for each pixel. In addition, in the field of natural language processing, models equipped with attention mechanisms, such as transformers, are used for tasks such as machine translation or summarization. Such deep learning models have achieved performance that exceeds that of conventional machine learning models, and there is a movement to utilize deep learning in various fields, such as medicine and industry.

[0003] Factors contributing to the improved performance of deep learning models include improvements in computer computing power and advances in cloud technology. The use of GPUs (Graphical Processing Units) for large-scale parallel computing and the emergence of cloud services equipped with numerous GPUs have made it easier to train large-scale deep learning models. When using a trained model to perform inference processing on data acquired by devices such as in-vehicle cameras and smartphones, there are two methods: performing inference on the cloud or performing inference on the device that acquired the data. The former method impairs real-time performance due to network delays, and transmitting data to the cloud via the Internet can pose security risks and privacy violations. Therefore, in recent years, attention has been focused on edge AI, which performs inference processing on the device that acquired the data.

[0004] While edge AI enables real-time data processing while protecting privacy, it has faced challenges such as difficulty in securing power and computational resources. In particular, when performing inference processing on mobile devices such as drones and smartphones, weight and other limitations make it impossible to install devices with high power consumption, such as GPUs. Therefore, by installing specialized hardware for inference processing, known as inference hardware, on the device, power consumption is reduced while ensuring the computational resources required for edge AI. Furthermore, to reduce circuit size and power consumption, calculations typically performed using 32-bit floating point on GPUs are often quantized to 16-bit or 8-bit fixed point or integers on inference hardware. When performing calculations using fixed point, parameters or intermediate outputs are quantized to a specified number of bits using the fixed point position specific to the inference hardware. Furthermore, for inference hardware with adjustable fixed point positions, a method has been proposed in which the fixed point position to be used when calculating the next input data is determined based on the inference results for a given input data.

[0005] Fabrizio Indirli, Ahmet Erdem, and Cristina Silvano, “A Tile-based Fused-layer CNN Accelerator for FPGAs”, IEEE International Conference on Electronics Circuits and Systems, 2020

[0006] Fixed-point position control has been performed dynamically on inference hardware. In this case, the initial fixed-point position can be set arbitrarily, but if the initial value is not set appropriately, the convergence of the fixed-point position during dynamic control can be slow, and the inference accuracy can also be reduced.

[0007] The disclosed technology has been made in consideration of the above points, and aims to provide an inference preprocessing device and an inference preprocessing method that accelerate the convergence of dynamic adjustment of fixed-point positions on inference hardware.

[0008] A first aspect of the present disclosure is an inference pre-processing device comprising: a hardware model construction unit that converts an input inference model into a trial model that can perform processing equivalent to that of the inference hardware by referring to the configuration of inference hardware; an inference trial unit that repeats inference simulations while changing the fixed-point position using the trial model and sample data to be inferred; an accuracy measurement unit that evaluates the results of the inference simulation by the inference trial unit using ground truth data of the sample data; and a fixed-point position determination unit that determines an initial fixed-point position for dynamic fixed-point position control of the inference hardware based on the results of the evaluation by the accuracy measurement unit.

[0009] A second aspect of the present disclosure is an inference preprocessing method, in which a processor converts an input inference model into a trial model that can perform processing equivalent to that of the inference hardware by referring to the configuration of inference hardware, repeats an inference simulation using the trial model and sample data to be inferred while changing the fixed-point position, evaluates the results of the inference simulation using ground truth data of the sample data, and performs processing to determine an initial fixed-point position for dynamic fixed-point position control of the inference hardware based on the results of the evaluation.

[0010] According to the disclosed technology, it is possible to provide an inference preprocessing device and an inference preprocessing method that accelerate the convergence of dynamic adjustment of fixed-point positions on inference hardware.

[0011] FIG. 1 is a diagram illustrating an example of the configuration of inference hardware. FIG. 2 is a flowchart illustrating inference processing executed by the inference hardware illustrated in FIG. 1. FIG. 3 is a block diagram illustrating a hardware configuration of an inference pre-processing device according to an embodiment of the disclosed technology. FIG. 4 is a block diagram illustrating an example of the functional configuration of the inference pre-processing device. FIG. 5 is a flowchart illustrating the flow of inference pre-processing by the inference pre-processing device. FIG. 6 is a block diagram illustrating an example of the functional configuration of the inference pre-processing device. FIG. 7 is a diagram illustrating an example of blocks that affect each other's calculation results. FIG. 8 is a diagram illustrating an example of blocks that affect each other's calculation results. FIG. 9 is a block diagram illustrating an example of a model optimization. FIG. 10 is a flowchart illustrating the flow of inference pre-processing by the inference pre-processing device.

[0012] First, a description will be given of techniques related to embodiments of the disclosed technology. FIG. 1 is a diagram illustrating an example configuration of inference hardware. The inference preprocessing device of the inference hardware illustrated in FIG. 1 is hardware capable of performing dynamic fixed-point position control, which adjusts the fixed-point position by feeding back past inference results. The inference hardware 1000 illustrated in FIG. 1 includes an inference processing unit 1010, a counter unit 1020, and a decimal point position determination unit 1030. The inference processing unit 1010 receives an inference model and data to be inferred as input, and performs arithmetic processing required for inference. The counter unit 1020 includes two types of counters, an upper limit saturation counter and a lower limit saturation counter, which respectively count the number of times the absolute value of the output feature value of a certain layer exceeds the maximum value of the range at the current fixed-point position and the number of times it falls below the minimum step. The minimum step is the value when only the least significant bit is set to 1. The decimal point position determination unit 1030 determines the fixed point position to be used when inferring the next input data from the count value of the upper limit saturation counter of the counter unit 1020, the count value of the lower limit saturation counter, and the input data, and reflects this in the inference processing unit 1010.

[0013] 2 is a flowchart showing the inference processing executed by the inference hardware 1000 shown in FIG. In step S1001, the inference processing unit 1010 quantizes parameters. Specifically, prior to inputting video, the inference processing unit 1010 converts the kernel and bias of the convolution operation into 8-bit fixed-point numbers so that the entire range of values ​​can be represented.

[0014] Following step S1001, in step S1002, the inference processing unit 1010 performs a convolutional layer operation on the input to that layer, and temporarily stores the operation result as a 16-bit decimal number.

[0015] Following step S1002, in step S1003, the inference processing unit 1010 determines for each output pixel whether the 16-bit calculation result can be expressed in the range when converted to 8 bits at the current fixed-point position. More specifically, the inference processing unit 1010 determines whether the value of the calculation result is equal to or less than the minimum step of the current fixed-point position. If the value of the calculation result is equal to or less than the minimum step of the current fixed-point position (step S1003; Yes), the inference processing unit 1010 increments the lower limit saturation counter of the counter unit 1020 by 1 to indicate lower limit saturation in step S1004. On the other hand, if the value of the calculation result exceeds the minimum step of the current fixed-point position (step S1003; No), the inference processing unit 1010 determines in step S1005 whether the value of the calculation result is equal to or greater than the maximum value represented by the current fixed-point position. If the value of the calculation result is equal to or greater than the maximum value represented by the current fixed-point position (step S1005; Yes), the inference processing unit 1010 increments the upper limit saturation counter of the counter unit 1020 by 1 to indicate upper limit saturation in step S1006. On the other hand, if the value of the calculation result is not equal to or greater than the maximum value represented by the current fixed-point position (step S1005; No), the inference processing unit 1010 skips the processing of step S1006.

[0016] Next, in step S1007, the inference processing unit 1010 determines whether all pixels of the input image have been checked. If all pixels of the input image have not been checked (step S1007; No), the inference processing unit 1010 returns to the process of step S1003. If all pixels of the input image have been checked (step S1007; Yes), the decimal point position determination unit 1030 determines in step S1008 whether the lower limit saturation count value is equal to or greater than the threshold T ｌｏｗｅｒ It is determined whether the lower limit saturation count value is equal to or greater than the threshold T ｌｏｗｅｒ If the lower limit saturation count value is equal to or greater than the threshold T ｌｏｗｅｒ If the upper limit saturation count value is not equal to or greater than the threshold T ｕｐｐｅｒ It is determined whether the upper limit saturation count value is equal to or greater than the threshold T ｕｐｐｅｒ If the upper limit saturation count value is equal to or greater than the threshold T ｕｐｐｅｒ If not (step S1010; No), the decimal point position determining unit 1030 skips the process of step S1011.

[0017] Next, in step S1012, the inference processing unit 1010 determines whether input to the layer is complete. If input to the layer is not complete (step S1012; No), the inference processing unit 1010 returns to the processing of step S1002. On the other hand, if input to the layer is complete (step S1012; Yes), the inference processing unit 1010 ends the inference processing for the input data.

[0018] The inference hardware 1000 can search for the optimal fixed-point position by repeatedly performing the series of processes shown in FIG. 2 for each layer each time an input is received from the initial fixed-point position.

[0019] However, when fixed-point position control is performed dynamically in this manner using inference hardware 1000, although the initial fixed-point position can be set arbitrarily, if the initial value is not set appropriately, the convergence of the fixed-point position during dynamic control may be slow, and the inference accuracy may also be reduced.

[0020] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same reference numerals are used to designate identical or equivalent components and parts in each drawing. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0021] First Embodiment FIG. 3 is a block diagram showing the hardware configuration of an inference preprocessing device 10. As shown in FIG.

[0022] 3, the inference preprocessing device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0023] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads programs from the ROM 12 or the storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the programs stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores an inference preprocessing program that accelerates the convergence of dynamic adjustment of fixed-point positions on the inference hardware.

[0024] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured by a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and various data.

[0025] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to input various types of information.

[0026] The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may be a touch panel type and function as the input unit 15.

[0027] The communication interface 17 is an interface for communicating with other devices, and the communication may use, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark).

[0028] Next, the functional configuration of the inference preprocessing device 10 will be described.

[0029] FIG. 4 is a block diagram showing an example of the functional configuration of the inference preprocessing device 10.

[0030] 4, the inference preprocessing device 10 has, as its functional components, a hardware model construction unit 101, an inference trial unit 102, an accuracy measurement unit 103, and a fixed-point position determination unit 104. Each functional component is realized by the CPU 11 reading out an inference preprocessing program stored in the ROM 12 or the storage 14, expanding it in the RAM 13, and executing it.

[0031] The hardware model construction unit 101 refers to the configuration of the inference hardware 1000 and creates, from the input inference model, a trial model that simulates the inference processing on the inference hardware 1000. In other words, the hardware model construction unit 101 refers to the configuration of the inference hardware 1000 and converts the input inference model into a trial model that can perform processing equivalent to that of the inference hardware 1000.

[0032] The hardware model construction unit 101 quantizes the kernel and bias of the inference model according to the number of operation bits of the inference hardware 1000. Here, any method for quantizing the kernel and bias may be used, but an example is a method of finding the fixed-point position n after quantization of the parameter params using the following formula (1). When this method is used, a fixed-point position is obtained whose range after quantization includes the maximum value of the parameter.

[0033]

[0034] The hardware model construction unit 101 inserts a mechanism for quantizing input data and output data of each layer. The construction of a trial model and the insertion of a quantization mechanism can be performed by any method, but one example is a method in which software modules equivalent to hardware that performs calculations and quantization that can be processed by the inference hardware 1000 are prepared, and the trial model is constructed by combining the modules while analyzing the structure of the inference model. Using this method, the inference preprocessing device 10 can be applied to the input of any inference model.

[0035] Inference trial unit 102 repeats inference simulation while changing the fixed-point position using the trial model created by hardware model construction unit 101 and sample data of the inference target. Inference trial unit 102 quantizes the input sample data of the inference target before inputting it to the trial model, and attempts inference processing in inference hardware 1000 using the quantized kernel and bias. Inference trial unit 102 keeps the fixed-point position of each intermediate layer constant during the inference processing trial, repeatedly attempts inference for possible fixed-point positions, and obtains inference trial results when inference is performed with each combination of fixed-point positions. Inference trial unit 102 then passes the output of the final layer of the trial model to accuracy measurement unit 103 as the inference trial result.

[0036] The accuracy measurement unit 103 uses the correct answer data of the sample data to be inferred to evaluate the results of the inference simulation by the inference trial unit 102. The accuracy measurement unit 103 switches the evaluation index depending on the task to be solved by the deep learning model, such as mean average precision (mAP) when evaluating an object detection model, or mean interaction over union (mIoU) when evaluating semantic segmentation.

[0037] The fixed-point position determination unit 104 searches for an optimal fixed-point position and determines an initial fixed-point position for dynamic fixed-point position control based on the evaluation result of the accuracy measurement unit 103. In other words, the fixed-point position determination unit 104 passes the fixed-point position with the highest inference accuracy to the inference hardware 1000 as the initial fixed-point position for dynamic fixed-point position control.

[0038] If correct data cannot be obtained, for example, in the case of an object detection model, the accuracy measurement unit 103 measures the number of objects detected when a certain fixed-point position is used, and the fixed-point position determination unit 104 may pass the fixed-point position that can detect the most objects among multiple decimal point positions to the inference hardware 1000 as the initial fixed-point position for dynamic fixed-point position control.

[0039] Next, the operation of the inference preprocessing device 10 will be described.

[0040] 5 is a flowchart showing the flow of inference preprocessing by the inference preprocessing device 10. The CPU 11 reads out an inference preprocessing program from the ROM 12 or the storage 14, loads it into the RAM 13, and executes it, thereby performing inference preprocessing.

[0041] In step S101 , the CPU 11 , as the hardware model construction unit 101 , quantizes the kernel and bias of the inference model in accordance with the number of calculation bits of the inference hardware 1000 .

[0042] Following step S101, in step S102, the CPU 11 inserts, as the hardware model construction unit 101, a quantization module for the input data and the output data of each layer.

[0043] Following step S102, in step S103, the CPU 11, as the inference trial unit 102, quantizes the input sample data to be inferred when inputting it into the trial model, and uses the quantized kernel and bias to simulate the inference processing in the inference hardware 1000 at the specified decimal point position.

[0044] Following step S103, in step S104, the CPU 11 functions as the accuracy measurement unit 103 to evaluate the result of the inference simulation in step S103 using the correct answer data of the sample data to be inferred.

[0045] Following step S104, in step S105, the CPU 11, functioning as the accuracy measurement unit 103, determines whether simulations have been performed at all decimal point positions. If the result of the determination in step S105 indicates that simulations have not been performed at all decimal point positions (step S105; No), the CPU 11 changes the decimal point position and returns to the processing in step S103. On the other hand, if the result of the determination in step S105 indicates that simulations have been performed at all decimal point positions (step S105; Yes), in step S106, the CPU 11, functioning as the fixed-point position determination unit 104, searches for an optimal fixed-point position based on the evaluation results and determines the initial fixed-point position for dynamic fixed-point position control. In other words, the CPU 11, functioning as the fixed-point position determination unit 104, passes the fixed-point position with the highest inference accuracy to the inference hardware 1000 as the initial fixed-point position for dynamic fixed-point position control.

[0046] By performing this processing, the CPU 11 can start dynamic adjustment from a fixed-point position that is close to the optimum, making it possible to converge dynamic fixed-point position control with fewer inputs than before, thereby achieving more accurate inference on the inference hardware 1000.

[0047] Second Embodiment FIG. 6 is a block diagram showing an example of the functional configuration of an inference preprocessing device 10. As shown in FIG.

[0048] 6, the inference pre-processing device 10 has, as its functional configuration, a hardware model construction unit 101, an inference trial unit 102, an accuracy measurement unit 103, a fixed-point position determination unit 104, and a processing allocation unit 105. Each functional configuration is realized by the CPU 11 reading out an inference pre-processing program stored in the ROM 12 or storage 14, expanding it in the RAM 13, and executing it. Here, the processing allocation unit 105, which is added to the configuration of the inference pre-processing device 10 shown in FIG. 4, will be described.

[0049] The processing allocation unit 105 allocates the computational processing of the inference model to each processor by referring to the configuration information of the inference hardware 1000, and outputs a model to be executed by the inference hardware 1000. The processing allocation unit 105 also passes the allocation information for each processor to the hardware model construction unit 101. The hardware model construction unit 101 constructs a trial model based on the allocation information for each processor.

[0050] 7 is a flowchart showing the flow of inference preprocessing by the inference preprocessing device 10. The CPU 11 reads out an inference preprocessing program from the ROM 12 or the storage 14, loads it into the RAM 13, and executes it, thereby performing inference preprocessing.

[0051] The type and number of processors on the inference hardware 1000 are arbitrary, but here we assume that the inference hardware 1000 is equipped with two types of processors: processor A, which performs dynamic fixed-point position control and processes convolution operations using fixed points, and processor B, which does not perform dynamic fixed-point position control and can process general-purpose operations.

[0052] In step S201, the CPU 11, functioning as the processing allocation unit 105, determines whether each operation of the inference model can be processed by processor A. If the result of the determination in step S201 is that the operation of the inference model can be processed by processor A (step S201; Yes), in step S202, the CPU 11, functioning as the processing allocation unit 105, allocates the operation of the inference model to processor A. Following step S202, in step S203, the CPU 11, functioning as the hardware model construction unit 101, quantizes the kernel and bias of the inference model in accordance with the number of operation bits of the inference hardware 1000. Following step S203, in step S204, the CPU 11, functioning as the hardware model construction unit 101, inserts quantization modules for the input data and the output data of each layer.

[0053] If the result of the determination in step S201 is that the calculation of the inference model cannot be processed by processor A (step S201; No), in step S205, the CPU 11, as the processing allocation unit 105, allocates the calculation of the inference model to processor B. Following step S205, in step S206, the CPU 11, as the processing allocation unit 105, starts conversion of decimal precision and adds connection processing necessary for exchanging data with processor A.

[0054] Once it has been determined whether all calculations of the inference model can be processed by processor A, in step S207, the CPU 11, as the inference trial unit 102, quantizes the input sample data to be inferred when inputting it into the trial model, and uses the quantized kernel and bias to simulate the inference processing in the inference hardware 1000 at the specified decimal point position.

[0055] Following step S207, in step S208, the CPU 11 functions as the accuracy measurement unit 103 to evaluate the result of the inference simulation in step S207 using the correct answer data of the sample data to be inferred.

[0056] Following step S208, in step S209, the CPU 11, functioning as the accuracy measurement unit 103, determines whether simulations have been performed at all decimal point positions. If the result of the determination in step S209 indicates that simulations have not been performed at all decimal point positions (step S209; No), the CPU 11 changes the decimal point position and returns to the processing of step S207. On the other hand, if the result of the determination in step S209 indicates that simulations have been performed at all decimal point positions (step S209; Yes), the CPU 11, functioning as the fixed-point position determination unit 104, searches for an optimal fixed-point position based on the evaluation results and determines the initial fixed-point position for dynamic fixed-point position control. In other words, the CPU 11, the fixed-point position determination unit 104, passes the fixed-point position with the highest inference accuracy to the inference hardware 1000 as the initial fixed-point position for dynamic fixed-point position control.

[0057] By providing the processing allocation unit 105 in this manner, the inference preprocessing device 10 can determine the initial fixed-point position of the dynamic fixed-point position control even in a heterogeneous environment in which multiple processors are mixed on a single inference hardware 1000, making it applicable to a variety of inference hardware 1000.

[0058] Third Embodiment FIG. 8 is a block diagram showing an example of the functional configuration of an inference preprocessing device 10. As shown in FIG.

[0059] 8, the inference pre-processing device 10 has, as its functional configuration, a hardware model construction unit 101, an inference trial unit 102, an accuracy measurement unit 103, a fixed-point position determination unit 104, and an adjustment interval determination unit 106. Each functional configuration is realized by the CPU 11 reading out an inference pre-processing program stored in the ROM 12 or storage 14, expanding it in the RAM 13, and executing it. Here, we will explain the adjustment interval determination unit 106, which is added to the configuration of the inference pre-processing device 10 shown in FIG. 4.

[0060] The adjustment interval determination unit 106 executes a process of searching for an interval where dynamic fixed-point position control is to be implemented with priority. The adjustment interval determination unit 106 identifies a processing block that is likely to cause upper-limit saturation or lower-limit saturation from the inference trial result obtained by the inference trial unit 102 and the initial fixed-point position obtained by the fixed-point position determination unit 104, and notifies the decimal point position determination unit 1030 of the inference hardware 1000 of the identified processing block as a dynamic adjustment interval. In other words, the interval where dynamic fixed-point position control is to be implemented with priority refers to an interval that is likely to cause upper-limit saturation or lower-limit saturation. Here, a processing block refers to a group of multiple consecutive arithmetic operations, such as a convolution operation.

[0061] In dynamic fixed-point position control, if the fixed-point position of only the layer where upper or lower limit saturation occurs is changed, even if the saturation in that layer is resolved, saturation may occur in subsequent layers, resulting in only a shift in the location of saturation. Figures 9A and 9B are diagrams showing examples of blocks that affect each other's calculation results. For example, the layers included in the Residual Block shown in Figure 9A are strongly interconnected, and changes in the output of one layer significantly contribute to the occurrence of saturation in the other layer. Furthermore, as shown in Figure 9B, the output of the Residual Block further influences subsequent layers via the Route layer. Therefore, blocks with such strong interconnections are defined as dynamic control sections, and a list of dynamic control sections within the inference model is passed to the decimal point position determination unit 1030 of the inference hardware 1000.

[0062] 8A and 8B, a method may be used in which a connection pattern of layers that are closely connected to each other is defined in advance, and layer sections corresponding to matching processing blocks are extracted as dynamic control sections by a method such as pattern matching. Alternatively, the method of extracting dynamic control sections by the adjustment section determination unit 106 may be a method in which the number of occurrences of upper limit saturation and lower limit saturation for the intermediate output of each layer in the inference trial unit 102 is counted, and sections where saturation is likely to occur when the fixed-point position is changed are extracted.

[0063] By providing the adjustment interval determination unit 106 in this manner, the inference pre-processing device 10 can clarify the intervals that require priority adjustment during dynamic control, and can more efficiently search for the optimal fixed-point position than when the adjustment interval determination unit 106 is not provided.

[0064] Fourth Embodiment FIG. 10 is a block diagram showing an example of the functional configuration of an inference preprocessing device 10. As shown in FIG.

[0065] 10, the inference pre-processing device 10 has, as its functional configuration, a hardware model construction unit 101, an inference trial unit 102, an accuracy measurement unit 103, a fixed-point position determination unit 104, a process allocation unit 105, an adjustment interval determination unit 106, and a process optimization unit 107. Each functional configuration is realized by the CPU 11 reading out an inference pre-processing program stored in the ROM 12 or storage 14, expanding it in the RAM 13, and executing it. Here, the process optimization unit 107, which is added to the configuration of the inference pre-processing device 10 shown in FIGS. 4, 6, and 8, will be described.

[0066] The process optimization unit 107 executes a process of searching for sections that can be integrated based on the dynamic adjustment sections determined by the adjustment section determination unit 106. Specifically, the process optimization unit 107 executes an optimization process that is performed when inference is executed on the inference hardware 1000, based on the dynamic adjustment sections determined by the adjustment section determination unit 106, and then generates an execution model in which the dynamic control sections of the fixed-point positions are integrated. When executing the optimization process, the process optimization unit 107 may further use allocation information that is a result of allocation by the process allocation unit 105.

[0067] In order to improve computational efficiency during inference processing by the inference hardware 1000, model optimization, including the integration of operations within the model, is often performed in advance outside the inference hardware 1000. Figures 11A to 11C are diagrams showing examples of model optimization. For example, as shown in Figure 11A, optimization is commonly performed to integrate a convolutional layer (Conv) with surrounding primitive operations such as padding (Pad), batch normalization (BN), and activation function processing (ReLU). Furthermore, an optimization called layer integration has also been proposed, in which a convolutional layer as shown in Figure 11A and surrounding operations are combined into a Conv block, and multiple Conv blocks are integrated to perform computations as shown in Figures 11B and 11C (Non-Patent Document 1).

[0068] If the inference hardware has a processor capable of performing both such optimization and dynamic fixed-point position control, the processing optimization unit 107 constructs an execution model so that the dynamic adjustment section and the integrated optimization section of the calculation match as closely as possible, thereby enabling the inference pre-processing device 10 to achieve more efficient calculation processing than if the processing optimization unit 107 is not provided.

[0069] 12 is a flowchart showing the flow of inference preprocessing by the inference preprocessing device 10. The CPU 11 reads out an inference preprocessing program from the ROM 12 or the storage 14, loads it into the RAM 13, and executes it, thereby performing inference preprocessing.

[0070] In step S301, the CPU 11, as the process optimization unit 107, searches for any section that can be layer-merged among the dynamic adjustment sections output from the adjustment section determination unit 106. If the determination result in step S301 shows that any section that can be layer-merged is present among the dynamic adjustment sections output from the adjustment section determination unit 106 (step S301; Yes), in step S302, the CPU 11, as the process optimization unit 107, extracts that section as a layer-merging section and excludes it from the search targets for subsequent processing. If the determination result in step S301 shows that any section that can be layer-merged is present among the dynamic adjustment sections output from the adjustment section determination unit 106 (step S301; No), the CPU 11 skips the processing of step S302.

[0071] Here, the process optimization unit 107 may use any method for determining whether layer merging is possible, but the process optimization unit 107 may use a method that combines pattern matching and condition determination using mathematical expressions. For example, the process optimization unit 107 may extract subgraphs that match patterns that are registered in advance as combinations of convolutional layers that can be merged based on the layer configuration of each of the convolutional layers, and determine, as a layer merging interval, an interval that corresponds to a subgraph that satisfies a condition that is registered in advance as a condition for layer merging based on the specifications of the inference hardware 1000.

[0072] Next, in step S303, the CPU 11, functioning as the process optimization unit 107, determines whether all patterns that can be layer-merged have been checked. If all patterns that can be layer-merged have not been checked (step S303; No), the CPU 11 returns to the processing of step S301. If all patterns that can be layer-merged have been checked (step S303; Yes), in step S304, the CPU 11, functioning as the process optimization unit 107, determines whether all dynamic adjustment sections have been checked. If all dynamic adjustment sections have not been checked (step S304; No), the CPU 11 returns to the processing of step S301.

[0073] If all dynamic adjustment sections have been checked (step S304; Yes), in step S305, the CPU 11, as the processing optimization unit 107, determines whether layers in the dynamic adjustment section that are determined not to be layer-integrated and layers not included in the dynamic adjustment section match a pattern that allows layer integration.

[0074] If a layer determined not to be layer-merging possible within the dynamic adjustment section and a layer not included in the dynamic adjustment section match a layer-merging possible pattern (step S305; Yes), in step S306, the CPU 11, as the process optimization unit 107, extracts the section as a layer-merging section and excludes it from search targets for subsequent processing.If a layer determined not to be layer-merging possible within the dynamic adjustment section and a layer not included in the dynamic adjustment section do not match a layer-merging possible pattern (step S305; No), the CPU 11 skips the processing of step S306.

[0075] Next, in step S307, the CPU 11, functioning as the process optimization unit 107, determines whether all of the patterns that can be merged have been checked. If all of the patterns that can be merged have not been checked (step S307; No), the CPU 11 returns to the process of step S305. If all of the patterns that can be merged have been checked (step S307; Yes), the CPU 11 ends the series of processes.

[0076] In this way, by having the processing optimization unit 107 match the layer integration interval created by optimization with the dynamic control interval of the decimal point position, the inference preprocessing device 10 can achieve more efficient calculation processing compared to when the processing optimization unit 107 is not provided.

[0077] In addition, the inference preprocessing executed by the CPU after reading the software (program) in each of the above embodiments may be executed by various processors other than the CPU. Examples of processors in this case include programmable logic devices (PLDs) whose circuit configuration can be changed after manufacture, such as field-programmable gate arrays (FPGAs), and dedicated electrical circuits, such as application-specific integrated circuits (ASICs), which are processors having a circuit configuration specifically designed to execute specific processes. Furthermore, the inference preprocessing may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

[0078] In addition, in each of the above embodiments, the inference preprocessing program is pre-stored (installed) in the storage 14, but this is not limiting. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0079] The following supplementary note is further disclosed regarding the above embodiment: (Supplementary note 1) An inference pre-processing device including: a memory; and at least one processor connected to the memory, wherein the processor is configured to: convert an input inference model into a trial model capable of processing equivalent to that of the hardware by referring to a configuration of inference hardware; repeat an inference simulation while changing a fixed-point position using the trial model and sample data to be inferred; evaluate a result of the inference simulation using ground truth data of the sample data; and determine an initial fixed-point position for dynamic fixed-point position control of the inference hardware based on a result of the evaluation. (Addendum 2) A non-transitory storage medium storing a program executable by a computer to perform inference preprocessing, wherein the inference preprocessing converts an input inference model into a trial model capable of processing equivalent to that of the hardware by referring to the configuration of inference hardware, repeats inference simulation while changing the fixed-point position using the trial model and sample data to be inferred, evaluates the results of the inference simulation using ground truth data of the sample data, and executes processing to determine an initial fixed-point position for dynamic fixed-point position control of the inference hardware based on the results of the evaluation.

[0080] REFERENCE SIGNS LIST 10 Inference pre-processing device 101 Hardware model construction unit 102 Inference trial unit 103 Accuracy measurement unit 104 Fixed point position determination unit 105 Processing allocation unit 106 Adjustment interval determination unit 107 Processing optimization unit

Claims

1. An inference pre-processing device comprising: a hardware model construction unit that converts an input inference model into a trial model that can perform processing equivalent to that of the inference hardware by referring to the configuration of the inference hardware; an inference trial unit that repeats inference simulation while changing the fixed-point position using the trial model and sample data to be inferred; an accuracy measurement unit that evaluates the results of the inference simulation by the inference trial unit using correct data from the sample data; and a fixed-point position determination unit that determines the initial fixed-point position of dynamic fixed-point position control of the inference hardware based on the results of the evaluation by the accuracy measurement unit.

2. An inference preprocessing device as described in claim 1, further comprising a processing allocation unit that references the configuration of the inference hardware, allocates the computational processing of the inference model to each processor of the inference hardware, and outputs a model to be executed by the inference hardware.

3. An inference preprocessing device as described in claim 1 or claim 2, further comprising an adjustment interval determination unit that extracts an interval where dynamic fixed-point position control is to be performed intensively as a dynamic adjustment interval based on the intermediate output of each layer obtained by the inference trial unit and the initial fixed-point position obtained by the fixed-point position determination unit, and outputs the dynamic adjustment interval together with the initial fixed-point position.

4. An inference pre-processing device as described in claim 3, further comprising a processing optimization unit that searches for sections that can be layer-integrated among the dynamic adjustment sections output by the adjustment section determination unit, and the hardware model construction unit constructs the trial model that reflects the sections that can be layer-integrated obtained by the processing optimization unit.

5. An inference pre-processing method in which a processor converts an input inference model into a trial model that can perform processing equivalent to that of the inference hardware by referring to the configuration of the inference hardware, repeats an inference simulation using the trial model and sample data to be inferred while changing the fixed-point position, evaluates the results of the inference simulation using the correct data of the sample data, and determines the initial fixed-point position of the dynamic fixed-point position control of the inference hardware based on the results of the evaluation.

Images