A system used to perform mixed-precision inference, a program used to perform mixed-precision inference, and a method for performing mixed-precision inference.

Abstract

This system provides a mechanism to suppress the decrease in accuracy and the increase in computation time in inference using mixed precision. [Solution] The system 100 used to perform inference using mixed precision includes an information acquisition unit 121 that acquires environmental information, which is information about the surrounding environment of the object of inference, and a data type setting unit 122 that sets the data type used for each layer in inference according to the acquired environmental information.

Description

Technical Field

[0001] The present disclosure relates to a system used to perform inference using mixed precision, a program used to perform inference using mixed precision, and a method of performing inference using mixed precision.

Background Art

[0002] In inference using an NPU (Neural network Processing Unit), various techniques for reducing the computation time have been proposed. For example, Patent Document 1 discloses a technique using mixed precision. Mixed precision is a technique for reducing the computation time while maintaining the accuracy of inference by changing the data type for each layer in the inference. The data type used for each layer is preset.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When using mixed precision for object detection around a vehicle, the surrounding environment changes in real time. Therefore, the appropriate data type for each layer can also change moment by moment. As a result, the preset data type may no longer be the appropriate data type, leading to problems such as a decrease in object detection accuracy or an increase in computation time. Such problems may be common not only to vehicles but also to object detection used in an environment that changes in real time.

Means for Solving the Problems

[0005] The present disclosure can be realized in the following forms.

[0006] According to one embodiment of the present disclosure, a system (100) is provided for performing inference using mixed precision. This system comprises an information acquisition unit (121) that acquires environmental information, which is information about the surrounding environment of the object of the inference, and a data type setting unit (122) that sets the data type used for each layer in the inference according to the acquired environmental information.

[0007] According to the system described above, the system includes an information acquisition unit that acquires environmental information, which is information about the surrounding environment of the object of inference, and a data type setting unit that sets the data type used for each layer in inference according to the acquired environmental information. Therefore, even when the surrounding environment is constantly changing, the data type used for each layer can be appropriately set according to the environmental information. This makes it possible to suppress a decrease in inference accuracy and an increase in computation time.

[0008] This disclosure can be implemented in various forms. For example, it can be implemented in the form of a method for performing inference using mixed precision, a computer program for implementing this method, or a non-temporary recording medium on which such computer program is stored. [Brief explanation of the drawing]

[0009] [Figure 1] A diagram showing the schematic configuration of a system in one form of this disclosure. [Figure 2] A diagram showing an example of a mixed precision table. [Figure 3] A diagram showing an example configuration. [Figure 4] A flowchart illustrating the steps for performing inference using mixed precision. [Figure 5] A flowchart showing the steps for creating a configuration. [Figure 6] A diagram illustrating how to create a configuration. [Modes for carrying out the invention]

[0010] A. Embodiments: A1. Configuration of System 100: The system 100 shown in Figure 1 is used to perform inference using mixed precision. Specifically, the system 100 is used for inference of object detection in the environment surrounding a vehicle. In this embodiment, the system 100 is mounted on a vehicle. The system 100 is connected to the environmental information unit 110. The system 100 comprises a processor 120 and a storage unit 130.

[0011] <Configuration of Environmental Information Department 110> The environmental information unit 110 detects environmental information and outputs it to the processor 120. Environmental information is information about the surrounding environment of the object of inference. The environmental information unit 110 includes a camera 111, a solar radiation sensor 112, a weather sensor 113, a road information sensor 114, and a traffic volume sensor 115.

[0012] Camera 111 captures images of the area around the vehicle as environmental information. Camera 111 may capture images not only in front of the vehicle, but also to the sides and rear. The field of view of camera 111 includes objects to be detected. Camera 111 has, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The images are supplied to the NPU (Neural Network Processing Unit) 200 via processor 120. The NPU 200 performs inference to detect objects included in the images.

[0013] The solar radiation sensor 112 detects solar radiation information, which is information about the amount of solar radiation around the vehicle, as environmental information. The solar radiation sensor 112 is configured as a sensor that can measure, for example, brightness or lightness. Using the detected amount of solar radiation, the solar radiation sensor 112 outputs to the processor 120 whether it is daytime or nighttime around the vehicle.

[0014] The weather sensor 113 detects weather information, which is information about the weather around the vehicle, as environmental information. Weather information includes whether the weather is sunny or rainy. The weather sensor 113 detects, for example, rain adhering to the vehicle's windshield. Such a weather sensor 113 has a light-emitting element that illuminates the windshield and a light-receiving element that receives reflected light from the windshield. The weather sensor 113 detects rain by utilizing the property that the intensity of reflected light changes depending on whether there is rain adhering to the windshield. Using the detected rain, the weather sensor 113 outputs to the processor 120 whether the weather around the vehicle is sunny or rainy.

[0015] The road information sensor 114 detects road information, which is information about the type of road the vehicle is traveling on, as environmental information. Road types include, for example, general roads and expressways. The road information sensor 114 has a GPS (Global Positioning System) and a database that stores map information. The road information sensor 114 detects whether the road the vehicle is traveling on is a general road or an expressway and outputs it to the processor 120.

[0016] The traffic volume sensor 115 detects congestion information around the vehicle as environmental information. Congestion information includes whether the area around the vehicle is congested or not. The traffic volume sensor 115 also detects, for example, the number of other vehicles traveling around the vehicle. Using the detected number of other vehicles, the traffic volume sensor 115 outputs to the processor 120 whether or not there is congestion.

[0017] <Configuration of processor 120 and storage unit 130> Processor 120 executes various controls in system 100. Processor 120 is, for example, a CPU. Storage unit 130 stores various data in system 100. Storage unit 130 is constituted by a storage device such as, for example, an HDD (Hard Disc Drive) or an SSD (Solid State Drive). Processor 120 functions as information acquisition unit 121 and data type setting unit 122 by executing program 131 stored in storage unit 130.

[0018] Information acquisition unit 121 acquires environmental information from environmental information unit 110. Data type setting unit 122 sets the data type used for each layer in the inference according to the environmental information acquired by information acquisition unit 121. The layers in the inference are, for example, a Convolution layer, a ReLU layer, a Pooling layer, a SoftMax layer, etc. The data type is, for example, a floating point type (FP), an integer type (INT), etc. The number of bits in the data type is, for example, 8 bits, 16 bits, 32 bits. Hereinafter, the data type and its number of bits are combined and expressed as "FP32".

[0019] The data type setting unit 122 in the present embodiment determines the data type used for each layer using the environmental information and the mixed precision table 132 shown in FIG. 2. The mixed precision table 132 is stored in advance in the storage unit 130. In the mixed precision table 132, the environmental information and the data type used for each layer are associated with each other, and these correspondence relationships are stored as Config. In the present embodiment, the mixed precision table 132 stores 64 types of Config from Config0 to Config63. The black circles in the mixed precision table 132 shown in FIG. 2 indicate the acquired environmental information. For example, when the information acquisition unit 121 acquires that it is daytime, sunny, driving on a highway, and not congested as environmental information, the data type setting unit 122 uses Config0.

[0020] Figure 3 shows examples of three Config0, 1, and 2. In Config0 to 2, the data types used in each layer are stored. In the example of Figure 3, seven layers L1, L2, L3, L4, L5, L6, and L7 are shown. Among these layers L1 to L7, layer L1 is the input layer and layer L7 is the output layer. Therefore, mixed precision is applied to the intermediate layers L2 to L6. In this embodiment, the data types of layer L1 of the input layer and layer L7 of the output layer are FP32. In this embodiment, in layers L2 to L6, data types of FP16 and INT8 are used. Among the examples shown in Figure 3, for example, in Config0, FP16 is used in layer L2 and layer L5, and INT8 is used in layer L3, layer L4, and layer L6.

[0021] The data type setting unit 122 reads each Config from the mixed precision table 132 and transmits it to the NPU 200 shown in Figure 1. The NPU 200 executes inference with mixed precision using the data type for each layer determined by the Config.

[0022] A2. Method for executing inference using mixed precision: The procedure in the flowchart shown in Figure 4 is used for inference using mixed precision. Also, the procedure is repeatedly executed periodically at predetermined time intervals. For example, the procedure is executed every 0.1 seconds. Also, the procedure is not limited to every 0.1 seconds and may be executed every cycle of signal processing for vehicle control. First, the information acquisition unit 121 acquires environmental information (S110). The environmental information acquired in this embodiment is the amount of solar radiation around the object of inference, the presence or absence of rain, the road type, and the presence or absence of traffic jam information.

[0023] The data type setting unit 122 sets the data type used for each layer in the inference according to the acquired environmental information (S120). In this embodiment, the data type used for each layer is predetermined and stored in the storage unit 130 as the mixed precision table 132.

[0024] The NPU200 performs inference using the data type used for each layer determined by the method described above.

[0025] How to create A3.Config: The steps in the flowchart shown in Figure 5 are used to create the Config. The Config creation method is performed in advance before performing inference using mixed precision. The mixed precision table 132 is created by performing the Config creation method multiple times.

[0026] The method for creating a Config will be explained with reference to Figures 5 and 6. Below, we will describe an example where the data type used in each layer before applying mixed precision is FP32. Furthermore, below, we will explain using the case of creating Config0 from the Configs shown in Figure 2 as an example. As shown in S210 of Figure 5 and the upper part of Figure 6, all of the intermediate layers L2 to L6 to which mixed precision is applied are replaced with smaller data amounts. In this embodiment, FP32 is replaced with INT8. In the following, such a replacement will also be called "quantization".

[0027] As shown in S220 of Figure 5, the cosine similarity is calculated for each of the intermediate layers L2 to L6. Specifically, in S210, a comparison is performed between the data type before substitution and the data type after substitution. The cosine similarity is shown in the range of -1 to +1, with a value closer to -1 indicating low similarity and a value closer to +1 indicating high similarity. Below each of the layers L2 to L6 in the upper part of Figure 6, the cosine similarity for each layer is shown. For example, the cosine similarity of layer L3 is 0.8, indicating a relatively high cosine similarity and a relatively small quantization error when quantized. In contrast, the cosine similarity of layer L2 is -0.7, indicating a relatively low cosine similarity and a relatively large quantization error when quantized.

[0028] As shown in S230 of Figure 5, the order of cosine similarity of each layer L2 to L6 is determined. Here, the order is determined so that the layers are arranged from lowest to highest cosine similarity. In the example shown in the upper part of Figure 6, the layers are arranged in the order of layer L2, layer L4, layer L6, layer L5, and layer L3.

[0029] As shown in S240 of Figure 5, layers with low cosine similarity are replaced with larger data volumes. In this embodiment, the replacement is performed on the layer with the lowest cosine similarity among layers L2 to L6. Therefore, as shown in the middle of Figure 6, layer L2, which has the lowest cosine similarity, is replaced from INT8 to INT16.

[0030] As shown in S250 of Figure 5, the accuracy of the inference is calculated. More specifically, inference is performed using layers L2 to L6, including layer L2 which was replaced in S240, and the accuracy is calculated. The test data used for inference includes environmental information. When creating Config0 as shown in Figure 2, the test data includes the environmental information that it is daytime, sunny, the road being traveled is a highway, and there is no traffic congestion. The test data is prepared according to the Config to be created.

[0031] As shown in S260, it is determined whether the accuracy calculated in S250 meets the predetermined criteria. If the accuracy calculated in S250 meets the predetermined criteria (S260: YES), the Config settings are completed as shown in S270.

[0032] If the accuracy calculated in S250 does not meet the predetermined criteria (S260: NO), the process returns to S240, and the second-lowest similarity layer is replaced with a larger data volume. Specifically, as shown in the lower part of Figure 6, layer L4, which has the second-lowest cosine similarity of -0.3, is replaced from INT8 to INT16. Subsequently, as shown in Figure 5, the steps of S250 and S260 are executed. If the accuracy still does not meet the criteria even after performing inference using layers L2 to L6, including the replaced layer L4, the process returns to S240 again, and the replacement of layer L6, which has the third-lowest cosine similarity, is performed. In this way, the steps of S240 to S260 are repeated until the accuracy in S260 meets the predetermined criteria, or until all layers L2 to L6 have been replaced.

[0033] By repeatedly executing the steps S210 to S270 described above, multiple configurations are created. This results in the creation of the mixed precision table 132.

[0034] According to the system 100 of the embodiment described above, the system includes an information acquisition unit 121 that acquires environmental information, which is information about the surrounding environment of the object of inference, and a data type setting unit 122 that sets the data type used for each layer in inference according to the acquired environmental information. Therefore, even when the surrounding environment is constantly changing, the data type used for each layer can be appropriately set according to the environmental information. It can also be said that the data type used for each layer can be set dynamically. This makes it possible to suppress a decrease in inference accuracy and an increase in computation time.

[0035] Furthermore, according to the system 100 of this embodiment, the data type setting unit 122 sets the data type used for each layer using environmental information and the mixed precision table 132. Therefore, by pre-storing an appropriate mixed precision table 132, the data type used for each layer can be set more appropriately.

[0036] Furthermore, according to the system 100 of the embodiment, the environmental information includes at least one of solar radiation information, weather information, road information, and traffic congestion information, so the data type setting unit 122 can more appropriately set the data type used for each layer according to this information.

[0037] B. Other Embodiments 1: (B1) In the above embodiment, the data type setting unit 122 used a mixed precision table 132, but the disclosure is not limited thereto. The data type setting unit 122 may set the data type to be used for each layer in inference without using the mixed precision table 132. For example, the data type setting unit 122 may calculate an appropriate data type to be used for each layer depending on the surrounding environment.

[0038] (B2) In the above embodiment, environmental information included solar radiation information, weather information, road information, and traffic congestion information, but this disclosure is not limited thereto. Environmental information may be any information. Environmental information may include, for example, time information, building information, traffic signal information, pedestrian information, vehicle information, road information, visibility information, noise information, geographic information, obstacle information, temperature information, humidity information, and light environment information. Time information is information about the current time. Building information is information about the types of surrounding buildings. Traffic signal information is information about the color of the signal displayed by the traffic signal and the timing of the signal change. Pedestrian information is information about the position of pedestrians, the direction and speed of pedestrian movement, and the density of pedestrians. Vehicle information is information about the speed, direction, type, distance between the vehicle and other vehicles, and distance between other vehicles of surrounding vehicles. Road information is information about the condition of the road and the condition of the pavement. Visibility information is information about the quality of visibility and the light conditions. Noise information is information about the ambient noise level and the presence of specific sounds such as horns and sirens. Geographic information includes GPS data, elevation, and terrain topography. Obstacle information includes information on surrounding fixed obstacles, including buildings, guardrails, and trees, and moving obstacles, including animals and drones. Temperature information includes information on ambient air temperature and road surface temperature. Humidity information includes information on ambient humidity and precipitation. Light environment information includes information on sunlight intensity, shadow location, and reflected light. The various types of information included in the environmental information described above are illustrative and do not limit this disclosure.

[0039] C. Other Embodiments 2: (C1) In the embodiments described above, the weather information included whether it was sunny and whether it was raining, but the disclosure is not limited thereto. The weather information may be any information relating to the weather. For example, the weather information may only include whether it is sunny or not. Alternatively, the weather information may include whether it is sunny, rainy, cloudy, and snowy.

[0040] (C2) In the above embodiment, the system 100 was mounted on a vehicle, but the disclosure is not limited thereto. The system 100 may be located outside the vehicle. For example, the system 100 may be implemented as a server located outside the vehicle. In such a configuration, the environmental information unit 110 provides environmental information to the processor 120 using wireless communication or the like.

[0041] (C3) In the above embodiment, INT8, INT16, and FP32 were used as data types set by the data type setting unit 122, but the disclosure is not limited thereto. The data type setting unit 122 may use any data type.

[0042] (C4) In the above embodiment, the system 100 was used for object detection on a vehicle, but the disclosure is not limited thereto. The system 100 may be mounted on any moving object. Furthermore, the system 100 may be used for any inference, not just object detection.

[0043] (C5) The System 100 and Method described herein may be implemented by a dedicated computer provided by configuring a processor and memory programmed to perform one or more functions embodied by a computer program. Alternatively, the System 100 and Method described herein may be implemented by a dedicated computer provided by configuring a processor by one or more dedicated hardware logic circuits. Alternatively, the System 100 and Method described herein may be implemented by one or more dedicated computers configured by a combination of a processor and memory programmed to perform one or more functions and a processor configured by one or more hardware logic circuits. The computer program may also be stored as instructions executed by the computer on a computer-readable non-transitional tangible recording medium.

[0044] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in each embodiment corresponding to the technical features in the embodiments described in the summary of the invention can be replaced or combined as appropriate in order to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. This disclosure may be implemented, for example, in the following forms.

[0045] This disclosure may be implemented in the following forms: [Form 1] A system (100) used to perform inference using mixed precision, An information acquisition unit (121) acquires environmental information, which is information about the surrounding environment of the object of the aforementioned inference, A data type setting unit (122) sets the data type used for each layer in the aforementioned inference according to the acquired environmental information, A system that includes these features. [Form 2] The system described in Form 1, The storage unit (130) further stores a mixed precision table (132), which is a table in which the environmental information and the data type used for each layer are linked. The data type setting unit sets the data type to be used for each layer using the environmental information and the mixed precision table. system. [Form 3] A system according to form 1 or 2, The aforementioned inference is performed to detect objects around the vehicle. The aforementioned environmental information is, Solar radiation information relating to the amount of solar radiation around the vehicle, The weather information relating to the surrounding weather of the vehicle, Road information relating to the type of roads surrounding the vehicle, The information includes at least one of the following: traffic congestion information relating to the traffic congestion conditions around the vehicle, system. [Form 4] A program used to perform inference using mixed precision, An information acquisition function that acquires environmental information, which is information about the surrounding environment of the object of the aforementioned inference, A data type setting function that sets the data type used for each layer in the aforementioned inference according to the acquired environmental information, A program that causes the processor to execute. [Form 5] A method for performing inference using mixed precision, The steps include obtaining environmental information, which is information about the surrounding environment of the object of the aforementioned inference, The steps include setting the data type used for each layer in the aforementioned inference according to the acquired environmental information, A method that includes [a certain feature]. [Explanation of Symbols]

[0046] 100...System, 110...Environmental Information Unit, 111...Camera, 112...Solar Radiation Sensor, 113...Weather Sensor, 114...GPS, 115...Traffic Volume Sensor, 120...Processor, 121...Information Acquisition Unit, 122...Data Type Setting Unit, 130...Storage Unit, 131...Program, 132...Mixed Precision Table, L1, L2, L3, L4, L5, L6, L7...Layers

Claims

1. A system (100) used to perform inference using mixed precision, An information acquisition unit (121) acquires environmental information, which is information about the surrounding environment of the object of the aforementioned inference, A data type setting unit (122) sets the data type used for each layer in the aforementioned inference according to the acquired environmental information, A system equipped with these features.

2. The system according to claim 1, The storage unit (130) further stores a mixed precision table, which is a table in which the environmental information and the data type used for each layer are linked. The data type setting unit sets the data type to be used for each layer using the environmental information and the mixed precision table (132). system.

3. The system according to claim 2, The aforementioned inference is performed to detect objects around the vehicle. The aforementioned environmental information is, Solar radiation information relating to the amount of solar radiation around the vehicle, The weather information relating to the surrounding weather of the vehicle, Road information relating to the type of roads surrounding the vehicle, The information includes at least one of the following: traffic congestion information relating to the traffic congestion conditions around the vehicle, system.

4. A program used to perform inference using mixed precision, An information acquisition function that acquires environmental information, which is information about the surrounding environment of the object of the aforementioned inference, A data type setting function that sets the data type used for each layer in the aforementioned inference according to the acquired environmental information, A program that causes the processor to execute.

5. A method for performing inference using mixed precision, The steps include obtaining environmental information, which is information about the surrounding environment of the object of the aforementioned inference, The steps include setting the data type used for each layer in the aforementioned inference according to the acquired environmental information, A method that includes [a certain feature].

Images