Method for verifying an implemented neural network
By using an activation coverage verification method on an embedded platform, which utilizes multiple verification images and synthesized images, the high cost and low efficiency of neural network verification on embedded platforms are solved, ensuring the correctness and security of the neural network on the target system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ROBERT BOSCH GMBH
- Filing Date
- 2021-12-14
- Publication Date
- 2026-05-29
Smart Images
Figure CN116615764B_ABST
Abstract
Description
Background Technology
[0001] Machine learning methods, especially deep learning, are becoming increasingly important in the field of autonomous driving. The use of neural networks is particularly prevalent in areas requiring environmental perception using sensors installed in vehicles. Typically, for embedded platforms, the development of neural networks involves various toolchains, including tools for training / evaluation and application. Using network training and evaluation toolchains, the generated and trained neural network can be defined, and finally, this neural network is transferred to the target system using instruction lists / execution descriptions. In target systems with embedded special-purpose processors (embedded hardware), this can be achieved using assembly language or machine instructions.
[0002] Requirements for such systems typically involve power consumption, required resources such as bandwidth, runtime, and storage needs, as well as permissible interactions with other systems or components sharing resources. Disruption-free operation and resource consumption can usually be checked directly on the embedded device through statistical analysis of the corresponding instruction list, while performance KPIs are typically calculated on the source system, in a simulation or development environment, using the original network definition. If such a system is used for safety-critical tasks, it must be ensured that the network used has the same characteristics (KPIs) as the original network definition; that is, the network implementation on both the target and source systems should provide the same results for each input. This is typically performed using toolchain evaluation. However, as security requirements for the networks used increase, the verification requirements for such toolchains also increase. Here, toolchain evaluation is not only very costly and cumbersome but can also slow down the development process. Summary of the Invention
[0003] According to aspects of the invention, a method for verifying an implemented neural network, a method for providing control signals, an apparatus, a computer program, and a machine-readable storage medium, as characterized in the independent claims, are provided. Advantageous designs are the subject of the dependent claims and the description that follows.
[0004] In this general description of the invention, the order of the method steps is presented to make the method easy to understand. However, those skilled in the art will recognize that several of these method steps can also be performed in another order and yield the same or corresponding results. In this sense, the order of these method steps can be changed accordingly. Some features are equipped with numbers to improve readability or make the assignment more explicit, but this does not imply the existence of specific features.
[0005] According to one aspect of the present invention, a method for verifying an implemented neural network is proposed. This method comprises the following steps:
[0006] In one step of the method, multiple verification images are provided. In another step, the multiple verification images are fed into an original neural network (source system). In another step, the multiple verification images are fed into an implemented neural network (target system). In another step, the activation coverage of the original neural network is determined using all the verification images provided. In another step, the determined activation coverage is compared with a target coverage. In yet another step, the corresponding output image of the original neural network, generated from the multiple verification images, is compared with the corresponding output image of the implemented neural network, generated from the multiple verification images, to verify the implemented neural network.
[0007] The term "verification image" should be understood broadly. Beyond the image in the narrow sense of optical imaging, it can generally be understood as a lattice of values, for example, that can be described by tensors. In particular, the term "verification image" also includes lattices of values that can be generated by RADAR (radar) sensor systems and / or LIDAR (lidar) sensor systems and / or ultrasonic sensor systems and / or video systems.
[0008] Similar to code coverage in classic unit testing, activation coverage determines the degree of validation of a given neural network, in order to demonstrate the proportion of the neural network that has been tested across a series of tests applied to the network.
[0009] The network under test consists of any number of layers. Each layer has L... i Both contain multiple feature maps F i，j Each of these feature maps has dimensions. Therefore, each feature map contains H items. i，j x W i，j pixel P i，j，k The feature map, which has layer index i, feature map index j, and pixel index k, has a height of H and a width of W.
[0010] The activation coverage is determined by the proportion of non-zero pixels to the total number of pixels. In other words, the activation coverage determines the proportion of pixels activated to validate the neural network. Let A... i，j，k (I) is the pixel P used for a specific verification image I. i，j，k The activation. Then, a set of verification images T = I1, ..., I are given to it. n The activation of a single pixel is defined as
[0011]
[0012] Finally, the activation coverage C(N, r) of network N on the test set with respect to the verification image T is:
[0013] in
[0014] For a set of validation images, the resulting activations can be accumulated for each layer and each feature map. The resulting activation coverage is determined by counting the non-zero pixels and dividing that number by the total number of pixels.
[0015] Similar to line coverage in classic programming, activation coverage quantifies the tested portion of a network as a degree of confidence in the corresponding verification process, and allows for a reduction in the number of verification images when it cannot be guaranteed that the network definition or the original neural network is completely identical to the implemented neural network.
[0016] Advantageously, this method allows for direct verification of the network's transfer to the target system without relying on a verified toolchain. In other words, the implemented neural network can be verified by repeatedly comparing the corresponding output images of the original neural network with those of the implemented neural network. Specifically, multiple input verification images can be used to compare the corresponding output images of the corresponding neural network, either in software and hardware or in another target system. If the two results are identical, the implemented neural network is considered correct relative to the test set of verification images. To determine whether the test set used is sufficient to determine the desired level of confidence in the verification, the activation coverage can be determined according to this method.
[0017] If the activation coverage is C(N, T) relative to the network definition N or the original neural network. a Multiple verification images T with ) = 1 a To test the implemented neural network, the network must at least be sufficiently matched on the source and target systems. This is achieved when using an activation coverage of C(N, T). b Multiple verification images T with ) < 1 b In the case of verifying neural network N, the corresponding network definition still has some untested regions, and therefore when the corresponding untested parts of the network definition are activated, the implemented neural network can produce unexpected output images, that is, results other than the original network definition.
[0018] The confidence that the implemented neural network behaves identically to the network definition corresponding to the original neural network increases with increasing activation coverage. Therefore, maximizing activation coverage is desirable.
[0019] Advantageously, this verification relies not only on creating the implemented neural network using a qualified and trustworthy toolchain, but also, if necessary, on the secure use of the implemented neural network through targeted verification of the network used. This method enhances security when using implemented neural networks already developed in a software environment by comparing individual images generated by the original neural network and the implemented neural network separately, and / or by using the determination of activation coverage to ensure that all pixels of the original neural network are addressed, thus determining a complete verification of the implemented neural network. Since feature maps may only be accessible in the case of the original neural network, activation coverage can be determined using the original neural network, because, as mentioned above, activation coverage is a characteristic of the network, not its implementation. In other words, in this method, the plurality of verification images can be selected such that the implemented neural network can be tested and / or compared to a desired extent corresponding to a specific activation coverage by selecting a plurality of verification images corresponding to the desired extent (achieved) of activation coverage. The actual testing / comparison is performed as follows: two networks, the original neural network and the implemented neural network, are used to process the selected verification images, and then the output images are compared, where the portions specified by coverage are addressed in both networks. By using multiple verification images in this way, it can be ensured that multiple paths can be mapped correspondingly through the networks when comparing coverage.
[0020] According to one aspect, it is proposed that the plurality of verification images have a first number of test images and / or a second number of synthesized images.
[0021] Here, the term "test image" should be interpreted broadly and can correspond to images generated in a real environment using sensor systems of different modalities. The term "synthetically generated image" should also be interpreted broadly and can correspond to images generated by computation and / or simulation. Having multiple verification images with random selection of images generated in a real environment and / or randomly synthesized can result in a range of verification images in which verification cannot be performed in practice. In particular, synthetic images can be determined by computation such that individual pixels defined by the network are activated. This allows for the generation and provision of a test set using an activation coverage of 1. Therefore, this is particularly advantageous because it is only possible to provide a test set with full activation coverage using real images in exceptional cases. Such exceptional cases can be due to, for example, very small and low-complexity networks. The larger the network, the less likely it is to achieve high coverage without targeted image selection and / or the generation of synthetic images. According to one aspect, it is proposed that, in order to determine the activation coverage using multiple verification images, inactive pixels are identified in at least one feature map of the original neural network and compared with the total number of pixels in that at least one feature map of the original neural network.
[0022] Advantageously, by utilizing this aspect of the method, the verification of the original neural network can be determined, controlled, and ensured in a targeted manner through this comparison, because, for example, some regions of the neural network, corresponding to the network topology, may be of greater importance to the output signal than other regions.
[0023] According to one aspect, it is proposed that at least one of the synthesized images in a plurality of synthesized images, particularly each synthesized image, activates at least one pixel in at least one feature map of the original neural network that was not activated by the plurality of test images.
[0024] When using arbitrary validation images, a very large set of validation images may be needed to achieve full activation coverage, i.e., activation coverage equal to 1. To systematically improve the validation coverage of the implemented neural network, an iterative approach can be used:
[0025] First, an initial set of validation images T0 is fed into the network to calculate the corresponding coverage C(N, T0).
[0026] -For the calculated activation coverage C(N, T) n This set of verification images T) n Identifier pixel S n =P i，j，k :δ(A i，j，k (T n )) = 0, meaning a pixel that has not yet been activated.
[0027] - For each pixel P among these pixels that have not yet been activated i，j，k ∈S n Calculate the synthetic image that activates the pixel and add it to the set of verification images T. n Based on this, an activation coverage C(N, T) is generated. n+1 )>C(N,T n A new set of verification images T n+1 .
[0028] -If the new activation coverage C(N,T) n+1 ) = 1 or if C(N, T) n+1 If the number of features is large enough to ensure the required reliability, the iteration can end; otherwise, as mentioned above, for each feature map, i.e., for a number of features H... i，j xW i，j pixel P i，j，k Identifier pixel S n =P i，j，k :δ(A i，j，k (T n )) = 0, these pixels are those that have not yet been used in this new set of verification images T. n+1 It is activated.
[0029] Using multiple verification images T determined in this way l The implemented neural network can be verified, and the multiple verification images not only have test images but also have synthetic verification images that are systematically generated in any way.
[0030] Advantageously, through these systematically determined synthetic verification images, multiple verification images can be created at a reduced cost for the provided verification of the implemented neural network, and the necessary number of said multiple verification images can be reduced. This iterative method allows for the controlled creation of said multiple verification images, which correspond to a specific (specified) activation coverage of the original neural network.
[0031] In one aspect, it is proposed to generate at least one synthetically generated verification image using the following steps:
[0032] Identify at least one pixel in at least one feature map that is not activated using these verification images;
[0033] The pixels of the synthesized verification image are determined by inverse operation of the original neural network, starting from at least one pixel identified by using these verification images that were not activated in the at least one feature map.
[0034] In other words, this means that: this can produce systematically generated verification images that activate at least one inactive pixel in at least one feature map of the original neural network by calculating a suitable input value from the identified (inactive) pixel to the input of the neural network.
[0035] One approach proposes that the original neural network is a convolutional neural network (CNN). Specifically, using this CNN, the systematically generated synthetic verification images can be determined through inverse computation.
[0036] In one aspect, it is proposed that the primitive neural network is implemented as a computer program on a computer. Advantageously, the primitive neural network implemented in software can perform activation coverage and computation on systematically generated synthetic alloy images, in order to verify neural networks that are typically implemented in hardware.
[0037] One approach proposes that the implemented neural network be implemented in assembly language and / or entirely in hardware for the processor. This implementation allows the neural network to be used for various purposes in a resource-efficient and speed-optimized manner.
[0038] According to one aspect of this method, it is proposed to use toolchains to verify the implemented neural network. The combination of described verification methods can be advantageously verified using toolchains to compare results and / or improve the verification.
[0039] A method is proposed in which control signals for manipulating at least partially automated vehicles are provided based on a verified implemented neural network of one of the above methods; and / or warning signals for warning vehicle occupants are provided based on a verified implemented neural network.
[0040] The term "based on" should be understood broadly regarding the characteristic of "providing control signals based on a verified, implemented neural network." This term should be understood to mean that a verified, implemented neural network is used for any determination or calculation of the control signals, which does not preclude the use of other input parameters for such determination. This applies correspondingly to the provision of warning signals.
[0041] In highly automated systems, such control signals can be used to introduce a transition to a safe state. For example, for at least partially automated vehicles, this could result in a slow stop on the shoulder.
[0042] An apparatus is proposed, configured to perform one of the methods described above. This apparatus enables the use of at least one of the described methods in different systems.
[0043] Proposal: The above-described apparatus is used to verify the implemented neural network, which is configured for use with at least partially automated mobile platforms.
[0044] A computer program is described, comprising instructions that, when executed by a computer, cause the computer to perform one of the methods described above. This computer program enables the use of the described methods in various systems.
[0045] A machine-readable storage medium is described, on which the aforementioned computer program is stored. This machine-readable storage medium facilitates the transport of the aforementioned computer program. Attached Figure Description
[0046] Reference to embodiments of the present invention Figures 1 to 3 This will be shown and elaborated upon in more detail below. Among them:
[0047] Figure 1 A flowchart is shown for the method used to verify the implemented neural network;
[0048] Figure 2 a and b illustrate the concept of activation coverage in this method; and
[0049] Figure 3 The activation coverage is shown when using systematically generated synthetic verification images to verify a two-layer neural network. Detailed Implementation
[0050] Figure 1 A flowchart of a method 100 for verifying an implemented neural network is outlined. In one step of the method 100, a plurality of verification images 110 are provided. In another step S10, the plurality of verification images 100 are fed into an original neural network. In another step S20, the plurality of verification images 100 are fed into the implemented neural network. In another step S30, the activation coverage of the original neural network is determined using all the verification images from the provided plurality of verification images 100. In another step S40, the determined activation coverage is compared with a target coverage. In another step S50, the corresponding output image 120 of the original neural network, generated from the plurality of verification images, is compared with the corresponding output image 130 of the implemented neural network, generated from the plurality of verification images 100, to verify the implemented neural network, for example, corresponding to bit identity.
[0051] Figure 2This paper outlines the concept of activation coverage for the original neural network 230 in the case of exemplary input image 212 and output image 222. It also discusses the use of feature maps L1: {F...} 1，1 F 1，2 F 1，3 F 1，4 The first layer 232 and feature map L2: {F 2，1 F 2,2 F 2,3 F 2,4 In the second layer 234 of the original neural network 230, the input image 212 activates 38 pixels out of 112 pixels in the entire feature map, meaning that the number of activated pixels is greater than zero. Therefore, the activation coverage of the original neural network 230 is: 38 / 112 = 33.9%.
[0052] Figure 2 b outlines the concept of activation coverage of the original neural network 230 in the case of multiple input images 210 and corresponding multiple output images 220. In a network with feature maps L1: {F 1，1 F 1，2 F 1，3 F 1，4 The first layer 232 and the second layer 234 and feature map L2: {F 2，1 F 2,2 F 2,3 F 2,4 In the original neural network 230, the input image 210 sequentially activates pixels in the corresponding feature maps, and these pixels are accumulated accordingly. The resulting activation coverage is determined by counting the pixels with non-zero activation and dividing that number by the total number of pixels. This example shows that the activation coverage is: (112-10) / 112 = 91.1%.
[0053] Figure 3 The schematic outline shows that each layer 232, 234 has four features L1: {F 1，1 F 1，2 F 1，3 F 1，4}、L2:{F 2，1 F 2，2 F 2，3 F 2，4 The activation coverage of the two-layer neural network 230. For example... Figure 2As illustrated in section a, an activation coverage of 33.9% is exemplarily obtained using a single verification image. Using multiple verification images 210, an activation coverage of 91.1% is obtained, which is determined in step S30. In step S60 of the method, inactive pixels of the feature map are identified. In step S70, synthetic verification images 310 are systematically generated to activate the identified inactive pixels of the corresponding feature maps. By applying these systematically generated synthetic verification images 310, a 100% activation coverage of the neural network 230 is achieved, and the neural network 230 is used to generate the corresponding output image 320. Scale 283 is used to interpret the activation of individual pixels of the feature map between activation zero and activation one.
[0054] In other words, in order to verify the implemented neural network, a total of multiple verification images T are generated. l These verification images not only contain test images but also systematically generated synthetic images. According to... Figure 3 A total of 10 pixels were not activated in the initial set of verification images, which, for example, only included the test image. For each pixel identified as inactive, a synthetic verification image can be generated that correspondingly activates these previously inactive pixels, and these pixels are added to the plurality of verification images. Here, the synthetic verification image activates one or more previously inactive pixels respectively. This achieves maximum activation coverage. Figure 3 In the example, this would result in 100% activation coverage.
Claims
1. A method for verifying an implemented neural network, the method comprising the following steps: Provide multiple verification images; The multiple verification images are fed into the original neural network; The plurality of verification images are fed into the implemented neural network; The activation coverage of the original neural network is determined using all of the provided multiple verification images. The activation coverage is defined as the proportion of the total number of pixels activated to verify the neural network. The determined activation coverage is compared with the target coverage; and The corresponding output image of the original neural network generated from the plurality of verification images is compared with the corresponding output image of the implemented neural network generated from the plurality of verification images to verify the implemented neural network.
2. The method according to claim 1, wherein the plurality of verification images have a first number of test images and / or a second number of synthesized images.
3. The method of claim 1 or 2, wherein, in order to determine the activation coverage using the plurality of verification images, inactive pixels are identified in at least one feature map of the original neural network and compared with the total number of pixels in the at least one feature map of the original neural network.
4. The method of claim 3, wherein the plurality of verification images are a plurality of synthesized images, wherein at least one of the synthesized verification images in the plurality of synthesized verification images activates at least one pixel in at least one feature map of the original neural network that was not activated by the plurality of test images.
5. The method of claim 4, wherein each of the plurality of synthesized verification images activates at least one pixel in at least one feature map of the original neural network that was not activated by the plurality of test images.
6. The method of claim 4, wherein at least one synthesized image is generated using the following steps: Identify at least one pixel in at least one feature map that is not activated using the verification image; The pixels of the synthesized verification image are determined by inverse operation of the original neural network, starting from at least one pixel identified by the at least one feature map that was not activated in the verification image.
7. The method according to claim 1 or 2, wherein the original neural network is a convolutional neural network.
8. The method according to claim 1 or 2, wherein the original neural network is implemented as a computer program on a computer.
9. The method according to claim 1 or 2, wherein the implemented neural network is implemented in assembly language and / or entirely in hardware for the processor.
10. The method of claim 1 or 2, wherein a toolchain is used to verify the implemented neural network.
11. A method for providing control signals for operating at least a partially automated vehicle and / or providing warning signals for alerting vehicle occupants, wherein, Based on the validated implementation of the neural network according to any one of claims 1 to 10, control signals for manipulating at least partially automated vehicles are provided; and / or based on the validated implementation of the neural network, warning signals for warning vehicle occupants are provided.
12. An apparatus for verifying an implemented neural network, the apparatus being configured to perform the method according to any one of claims 1 to 10.
13. The apparatus of claim 12 for verifying the application of the implemented neural network, wherein the implemented neural network is configured for use with a mobile platform that is at least partially automated.
14. An apparatus for providing control signals for operating at least a partially automated vehicle and / or providing warning signals for warning vehicle occupants, said apparatus being configured to perform the method according to claim 11.
15. A computer program product comprising a computer program, the computer program including instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 11.
16. A machine-readable storage medium having a computer program stored thereon, the computer program including instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 11.