Method for configuring a neural network
By determining noise parameters between training and inference hardware, the trade-off between chip area and performance in neural network configuration is resolved, ensuring consistency of output results on the inference hardware and improving training efficiency and accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ROBERT BOSCH GMBH
- Filing Date
- 2020-12-18
- Publication Date
- 2026-07-03
AI Technical Summary
Existing technologies involve a trade-off between chip area and performance when configuring neural networks, resulting in arithmetic operations that inference hardware cannot understand and whose responses are difficult to simulate on hardware, thus affecting training efficiency and accuracy.
By determining the noise parameters between the training hardware and the inference hardware, the output data on the training hardware is the same as the output data on the inference hardware. The noise parameters of the neural network are adjusted using the backpropagation method to ensure that the same output results are achieved on the inference hardware.
This achieves consistency between the output results on the inference hardware and the training hardware, reduces the need for understanding the characteristics of the inference hardware, saves development time, and improves training efficiency and accuracy.
Smart Images
Figure CN113011579B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method for configuring a neural network. It also relates to a method for training a neural network having noise parameters determined according to the proposed method. Furthermore, the invention relates to a computer program. Finally, the invention relates to a machine-readable storage medium. Background Technology
[0002] Current DNN (deep neural network) inference hardware primarily represents a trade-off between chip area and performance. Because floating-point computing devices are more expensive than digital signal processors (DSPs), DSPs are used for DNN computations in most cases. A common practice in this scenario is to train the DNN offline, where a fixed-point DNN is provided to the inference hardware. Because of these hardware trade-offs aimed at saving chip area and operating costs, arithmetic operations that are incomprehensible to the user of the inference hardware are often performed. This can lead to oversaturation or undersaturation effects, for example, during regular training of the DNN used for inference hardware.
[0003] This could be due to factors such as the fact that the functionality of the inference hardware can only be accessed through a framework, but if the inference hardware has random elements, the reason could also lie in the inference hardware itself. Here, there is a similarity to Deep Bayesian Neural Networks, which attempt to model the uncertainty of the data used. Summary of the Invention
[0004] The objective of this invention is to provide an improved method for configuring neural networks.
[0005] According to the first aspect, this task is solved using a method for configuring a neural network, which has the following steps:
[0006] - Feed image data to a neural network implemented on training hardware;
[0007] - Feed this image data to a neural network implemented on inference hardware;
[0008] - Determine the deviation between the output data of the training hardware and the output data of the inference hardware; and
[0009] - Determine the noise parameters of the neural network such that after feeding these image data to the neural network implemented on the training hardware and after feeding these image data to the neural network implemented on the inference hardware, the output data of the inference hardware is the same as the output data of the training hardware.
[0010] In this way, the noise parameters are determined such that the result on the training hardware is bit identical to the result on the inference hardware. As a result, the neural network can be trained on the training hardware immediately after the noise parameters are determined, for the purpose of implementation on the inference hardware.
[0011] Here, the error between training hardware with floating-point capabilities and inference hardware with fixed-point capabilities is considered. This allows the output response of the neural network on the training hardware to be as close as possible to the output response of the neural network on the inference hardware. Consequently, the typically unreproducible rounding errors of the inference hardware do not need to be addressed, allowing R&D resources to be focused on improving the neural network. As a result, this approach saves considerable time in studying the operational characteristics of the inference hardware in detail. According to the invention, the inherently unknown error is modeled using noise distribution.
[0012] According to the second aspect, the task is solved using a method for training a neural network implemented on training hardware, the neural network having noise parameters that are determined according to the proposed method for configuring the neural network.
[0013] According to the third aspect, the task is solved using a computer program that has instructions that, when implemented by a computer, cause the computer to implement methods for configuring the neural network.
[0014] According to the fourth aspect, the task is solved using a machine-readable storage medium on which the computer program is stored.
[0015] Preferred extensions of this method are the subject of the dependent claims.
[0016] An advantageous extension of this method is characterized by using at least one object category as output data of the neural network. In this way, simplified output data of the neural network can be compared with each other, where only the object category contributes to the determination of the bias. This supports a simple way and method for comparing the output data of the neural network.
[0017] Another advantageous extension of this method features the use of intermediate results from the layers of the neural network as the output data. In this way, the more complex output data of the neural network are compared with each other, which allows for a more precise identification of deviations in the output data. As a result, the response of the neural network regarding bit identity can be better realized.
[0018] Another advantageous extension of this method features that the deviation between the output data is backpropagated to the noise distribution of the layers of the neural network implemented on the training hardware. In this way, the noisy response of the neural network is used to model the response of the inference hardware, which is itself a black box. For this purpose, the backpropagation method, which is well known, is used.
[0019] Another advantageous extension of this method is characterized by configuring the neural network using the same parameters or different parameters. This advantageously supports a better determination of the discrepancy between the output data of the neural network on the training hardware and the output data of the neural network on the inference hardware.
[0020] Another advantageous extension of this method features altered image data fed to both the training and inference hardware. In this way, the discrepancy between the output data of the neural network on the training hardware and the output data of the neural network on the inference hardware can also be determined more effectively. Attached Figure Description
[0021] The invention will now be described in detail with reference to the accompanying drawings, which illustrate other features and advantages. Here, identical or functionally equivalent elements are referred to by the same reference numerals.
[0022] In the attached diagram:
[0023] Figure 1 A schematic diagram illustrating the working principle of the proposed method is shown.
[0024] Figure 2 The results of reasoning hardware without applying the present invention are shown;
[0025] Figure 3 The results of reasoning hardware under the application of this invention are shown; and
[0026] Figure 4 A schematic diagram illustrating an implementation of the proposed method is shown. Detailed Implementation
[0027] The core idea of this invention is to configure or organize a neural network so that it can be robustly trained for inference hardware. In this way, hardware / software errors in inference hardware, whose technical details are often unknown or incompletely known, can be advantageously compensated for. In this way, large-scale checks on the precise hardware characteristics of the inference hardware can be advantageously avoided.
[0028] Without the proposed method, it would be technically costly to also train the hardware to simulate the inference hardware's response.
[0029] This invention enables the training of neural networks on training hardware for dedicated embedded inference hardware that is not fully disclosed or whose high complexity has severely hampered technological development activities on inference hardware, for example, due to limited resources of hardware users.
[0030] A key advantage of the proposed method is that it ensures the response of the neural network on the inference hardware is identical to its response on the training hardware before training of the neural network for inference hardware begins. This ensures a certain level of security and guarantees the full usability of the neural network implemented on the inference hardware.
[0031] Without limiting the generality, inference hardware (e.g., in the form of vehicle ambient awareness sensors, such as cameras, lidar sensors, ultrasonic sensors, radar sensors, etc.) is constructed as hardware that is only partially public or known, in which the outputs (feature maps) of each layer can be read. In this case, it cannot be assumed that fast inference can be performed with the aid of inference hardware.
[0032] Furthermore, for inference hardware, for example, there are 16 bits available for weights and output control. The fixed-point format, that is, the number of bits available for the integer and comma parts, is not important because the inference hardware affects all bits, and thus errors accumulate.
[0033] Figure 1 A schematic system diagram illustrating the working principle of the proposed method is shown. It can be seen that the system 100 utilizes a training scenario, comprising training hardware 10 and inference hardware 20 (inference perception hardware, such as a camera). The inference hardware 20 is used for object recognition based on image data 1 and can be configured, for example, as a camera, a lidar sensor, a radar sensor, an ultrasonic sensor, etc., these ambient environment recognition sensors can preferably be built into a vehicle (not shown).
[0034] Inference hardware 20 is typically a system-on-a-chip (SoC) with limited, i.e., optimized area, computing power, and power consumption. Here, a different counting system is used on inference hardware 20, for example, than on training hardware 10; that is, integer format is used instead of floating-point format. However, this prevents the entire neural network 11 from being presented on inference hardware 20 at once.
[0035] It can be seen that the same image data 1 (e.g., in the form of detected images and / or random images) is fed not only to the training hardware 10 with the first neural network 11 but also to the inference hardware 20 with the second neural network 21. Furthermore, the neural networks 11 and 21 are configured using the same, unchangeable weights or parameters in the initial steps. The output data 21 of the second neural network 20 is fed to the error determination device 30, as is the output data 11 of the training hardware 10 with the first neural network 11.
[0036] Error determination device 30 determines the deviation or error between the output data 21 of inference hardware 20 and the output data 11 of training hardware 10, wherein the determined error is backpropagated to the various noise parameters R of the neural network 11 of training hardware 10 during the iteration process. This is performed until the error is substantially zero. As a result, this means that: now, training hardware 10 with the first neural network 11 and inference hardware 20 with the second neural network 21 provide almost identical output data or results with the same image data 1 and the same parameterized neural networks 11, 21.
[0037] For example, the well-known gradient descent method can be used to perform backpropagation of the error.
[0038] Advantageously, the determined noise parameter R or the parameters of the noise distribution can also be used for future training.
[0039] After the method concludes, the first neural network 11 on the training hardware is configured with appropriate noise parameters R, which remain unchanged from now on. The neural network 11 configured in this way can now be trained in a manner known per se, wherein at the end of training, the trained neural network 11 can be transferred to the inference hardware 20. This ensures that the inference response of the inference hardware 20, now having the neural network 11 configured according to the invention and subsequently trained, provides the same results as the neural network 11 trained on the training hardware 10.
[0040] This could be very advantageous for the safety-critical construction of inference hardware 20, for example, for an ambient environment detection sensor in a vehicle, in which, for legal reasons, the error response must be reproduced exactly.
[0041] In a simple variation of the proposed method, it can be specified that, as output data, only the classification is checked and these classifications are compared with each other, such as checking whether the object is identified by the inference hardware 20 and the training hardware 10 in the same way based on the image data 1 (e.g., identified as a human, infrastructure object, car, etc.).
[0042] In one variant, it can be specified that: intermediate results of the layers of the second neural network 21 are used as output data 22 of the inference hardware 20 having the second neural network 21, and these intermediate results are compared with output data 12 of the training hardware 10 having the neural network 11 in the form of intermediate results of the layers of the neural network 11.
[0043] The result is that, using the proposed method, the noise distribution is estimated based on the outputs of each layer of neural networks 11 and 21. For this purpose, it is preferable to use as much analysis as possible of the trained neural networks 11 and 21, including the outputs of all layers. It is also advantageous to use multiple independently trained networks 11 and 21 for analysis.
[0044] Now, the difference between the neural network 11 computed on training hardware 10 and the neural network transformed on inference hardware 20 can be calculated, and the error distribution can be estimated accordingly. One-sided testing can also be performed if only the most important images are of interest, which is advantageous due to limited data, where, for example, a uniform distribution can be chosen as the error distribution.
[0045] The advantages of the proposed method are: it does not require a precise understanding of the inference hardware 20, thereby saving development time. It also provides more accurate estimates of the error distribution for each layer of the neural networks 11 and 21, resulting in higher bit precision and consequently, better neural networks on the inference hardware 20.
[0046] Figure 2 and 3 The effects of the proposed method are shown. Figure 2 The image provided by the inference hardware 20 without using the proposed method can be seen. Regions A and B within this image are distinguished differently. Region A corresponds to the road direction and is well distinguished. However, region B is not accurately distinguished, which may be problematic given that region B occupies a large portion of the entire image.
[0047] Figure 3An image provided by inference hardware 20 using the proposed method is shown. In this case, region B is completely absent and only region A is resolved. As a result, this means that the operational response of inference hardware 20 is improved.
[0048] Figure 4 A schematic flowchart illustrating the implementation of the proposed method is shown.
[0049] In step 200, image data 1 is fed to neural network 11 implemented on training hardware 10.
[0050] In step 210, image data 1 is fed to neural network 21 implemented on inference hardware 20.
[0051] In step 220, the deviation between the output data 12 of the training hardware 10 and the output data 22 of the inference hardware 20 is determined.
[0052] In step 230, the noise parameter R of the neural network 11 is determined such that after the image data 1 is fed to the neural network 11 implemented on the training hardware 10 and after the image data 1 is fed to the neural network 21 implemented on the inference hardware 20, the output data 22 of the inference hardware 20 is the same as the output data 12 bits of the training hardware 10.
[0053] The proposed method is preferably constructed as a computer program having program code means for implementing the method on training hardware 10 and inference hardware 20.
[0054] Although the present invention has been described above with reference to specific embodiments, those skilled in the art can implement undisclosed or only partially disclosed embodiments in advance without departing from the spirit of the present invention.
Claims
1. A method for configuring a neural network (11), the method comprising the following steps: - The image data (1) is fed to the neural network (11) implemented on the training hardware (10). - The image data (1) is fed to a neural network (21) implemented on the inference hardware (20); - Determine the deviation between the output data (12) of the training hardware (10) and the output data (22) of the inference hardware (20); and - Determine the noise parameter R of the neural network (11) such that after the image data (1) is fed to the neural network (11) implemented on the training hardware (10) and after the image data (1) is fed to the neural network (21) implemented on the inference hardware (20), the output data (22) of the inference hardware (20) is the same as the output data (12) of the training hardware (10). wherein The neural network (11) implemented on the training hardware (10) is configured using the determined noise parameters R, wherein the configured neural network (11) is trained. The trained neural network (11) is transcribed onto the inference hardware (20), wherein the inference response of the inference hardware (20) with the trained neural network (11) provides the same result as that of the neural network (11) trained on the training hardware (10).
2. The method of claim 1, wherein at least one object category is used as output data (12, 21) of the neural network (11, 21).
3. The method according to claim 1 or 2, wherein intermediate results of the layers of the neural network (11, 21) are used as output data (12, 21) of the neural network (11, 21).
4. The method according to claim 1 or 2, wherein the deviation between the output data (12, 22) is backpropagated to the noise distribution of the layers of the neural network (11) implemented on the training hardware (10).
5. The method according to claim 1 or 2, wherein the neural network is configured using the same parameters or using different parameters (11, 21).
6. The method according to claim 1 or 2, wherein the image data (1) supplied to the training hardware (10) and the inference hardware (20) is modified.
7. A method for training a neural network (11) implemented on training hardware (10), the neural network having a noise parameter R, the noise parameter being determined according to any one of the preceding claims.
8. A computer program product comprising a computer program, the computer program including instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 6.
9. A machine-readable storage medium having a computer program stored thereon, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 6.