Processing data with neural networks over limited hardware resources
Through the collaborative cooperation between network pruning and neuron selection, the efficiency of neural network processing in embedded systems with limited hardware resources is solved, and efficient and low-cost neural network processing is achieved, suitable for automated driving and robot applications.
Patent Information
- Application Number
- CN202410652158.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-26
- Filing Date
- 2024-05-24
- Publication Date
- 2024-11-26
AI Technical Summary
Implementing efficient neural network data processing on hardware platforms with limited resources is a challenge, especially in embedded systems, especially in automated driving of vehicles and robotic applications. Hardware resources are limited and it is difficult to achieve high-performance neural networks. Network processing.
Through a collaborative collaboration between network pruning and neuron selection, negative metrics are used to quickly evaluate and delete unimportant neurons, retain critical subsets of neurons, and further optimize the collection of neurons based on positive metrics for efficient data processing, At the same time, the architecture of the neural network is optimized under the limitation of hardware resources to reduce the number of neurons without affecting the output quality.
It significantly saves computing time and hardware resources, improves the efficiency and output quality of data processing, reduces hardware costs, and realizes high-quality neural network processing on a limited hardware platform, similar to the car sharing model that optimizes resource utilization.
Smart Images

Figure CN119026634A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to processing data using neural networks on hardware platforms with limited resources, for example using such data in control devices for vehicles and other embedded systems. Background Art
[0002] It is possible to learn to drive a vehicle in road traffic with relatively few driving hours and with manageable mileage in kilometers. Typically, a learner driver spends only a few dozen hours behind the wheel before the driving test and has driven less than 1000 km. The human driver is then still able to drive the vehicle safely even in many situations that were not seen during training. His brain is able to generalize these unseen situations from the training.
[0003] In order to precisely incorporate this capability into at least partially automated driving of vehicles, for example when evaluating measurement data obtained by monitoring the vehicle environment with sensors, neural networks are used. The greater the capability of such neural networks, the higher the implementation requirements on the hardware platform used. However, embedded hardware, especially for mobile use, is much more expensive per unit of performance than, for example, desktop or server hardware for stationary use. This is especially true for embedded hardware with a higher safety integrity level (SIL). Summary of the invention
[0004] The present invention provides a method for processing input data x using a neural network. The neural network includes a set N of neurons.
[0005] Within the scope of the method, firstly, a subset of neurons is determined, the use of which can be omitted without excessively impairing the performance of the neural network, according to a specified negative measure. Removing some of these neurons is also called "network pruning".
[0006] From the subset N\P that remains after pruning, determine the subset of neurons that play a key role in processing the specific input data x according to the specified positive metric Thus, the subset A can be different for each input data x, ie A=A(x), for example.
[0007] To process a specific input data x, a subset of neurons is selected This subset is a superset of set A In the simplest case, D = A. However, it is also possible to include additional neurons in the set D according to any other criteria, for example when only a small set of neurons A(x) is absolutely necessary for processing specific input data x and the specified processing power budget has not been exhausted.
[0008] It has been recognized that network pruning based on negative metrics on the one hand and neuron selection using positive metrics on the other hand work synergistically. Negative metrics can be evaluated particularly quickly, and neurons of the order of 80% and in extreme cases 99% can already be identified as unimportant. The amount of work required to check the set N\P of neurons that remain after pruning that are important for processing specific input data x is linearly related to the number of remaining neurons. Therefore, preselecting neurons by pruning saves a lot of computing time, and in return it is possible to spend more time for each remaining neuron to decide whether the neuron is also important for processing specific input data x. As a result, the processing can be performed using much fewer neurons overall than the neurons originally provided by the architecture of the neural network, without sacrificing the quality of the output y or causing these output y to be excessively delayed.
[0009] This is somewhat similar to the selection process of a university. Especially in courses such as information technology or physics that have no or only unclear high school graduation admission restrictions, a large number of students will initially flock in. Therefore, in the initial semester, "pruning" is performed by first conducting rigorous examinations in basic subjects such as mathematics. Using this negative measure, a large classroom that was originally full of people can be reduced to one-quarter to one-fifth with a relatively small amount of work. Then, from the remaining set of students, positive measures such as seminars and other more intensively supervised courses can be used to select those students who really have outstanding talents in information technology or physics. These students are hired as research assistants for professors after they graduate.
[0010] The input data x of the neural network can be, for example, pixel values of an image or values for annotating points of a point cloud. The image can be, for example, a camera image, a video image, a thermal image or an ultrasound image. On the other hand, radar measurements and lidar measurements usually provide point clouds as results, which assign one or more measurement variables to points in space. Ultrasonic measurements can also provide point clouds as results.
[0011] The output data y of the neural network may, for example, include classification scores associated with one or more classes of a specified classification. In particular in applications in vehicles, driver assistance systems and robots, these classes may, for example, represent types of objects in the environment of the vehicle or robot, the presence of which is indicated by the input data x.
[0012] In a particularly advantageous embodiment, the neurons of the set D are implemented on a hardware platform whose resources are not sufficient to implement the entire set N of neurons. Depending on the input data x, the neurons in the changing subset A(x) or D(x) can then, for example, share hardware resources. In this way, these hardware resources can be used efficiently and even redundant resources do not need to be installed on the hardware platform. The cost savings are comparable to car sharing, where not as many vehicles are maintained as there are users, but only as many vehicles are used simultaneously as are expected. In this way, all users can complete all their transport tasks and the costs for purchasing and maintaining vehicles are significantly reduced overall.
[0013] In mobile applications in vehicles, the size of the hardware platform often has additional constraints in terms of installation space, heat generation and / or power consumption, which further exacerbates the cost advantage.
[0014] In a further particularly advantageous embodiment, one or more neurons are selected from the subset (N\P)\A, whose use is approved by the negative metric but not by the positive metric, for additional inclusion in the set D. In this way, the selection made by the positive metric can be refined again, which in turn improves the quality of the output y. The neurons deleted by pruning usually do not yet have a significant effect on the quality of the output y, while the selection according to the positive metric can certainly represent a compromise between the number of selected neurons on the one hand and the quality of the output y on the other hand.
[0015] In a particularly advantageous embodiment, the number of neurons to be additionally incorporated into the set D is specified depending on the specified computing power budget. For example, if the hardware platform used has the capacity to process a certain number of neurons, and the set of neurons A(x) selected by the positive metric for processing specific input data x has not yet exhausted this capacity, this capacity can be filled with other neurons and thus fully utilized. The best quality of the output y can often be achieved when as many neurons as possible are used within the framework of the existing hardware resources. This is somewhat similar to: if the allocated processing time is fully utilized, the best results can be achieved in an exam.
[0016] It is therefore particularly advantageous to define the computing power budget as the total number of neurons |D| in the set D.
[0017] In another particularly advantageous embodiment, the neurons are selected from the subset (N\P)\A in descending order of importance for processing the specific input data x. In this way, the gain in the quality of the output y can be maximized for each additionally added neuron.
[0018] For example, the order of importance can be specified according to the values determined for these neurons by the positive metric. In this way, the positive metric can be based, for example, from the beginning on: determining the values of all neurons from the subset N\P. Then, for example, neurons whose values exceed a specified threshold or the first n neurons with the highest values can be included in the set A. Then, on this basis, the set D can be seamlessly expanded without fundamentally changing the working principle.
[0019] In another particularly advantageous embodiment, the negative metric evaluates the neuron independently of the specific input data x. In particular, a pruning performed once using the negative metric can then be used repeatedly for a plurality of specific input data x. In particular, the same pruning can also be used for completely different domains and / or distributions of the input data x.
[0020] In another particularly advantageous embodiment, a neural network is selected in which the inputs supplied to each neuron are aggregated by calculating a weighted sum to activate the neuron. The negative measure can then evaluate the neurons at least according to their weights in the weighted sum. If the neurons do not contribute to the weighted sum or contribute only very little, then these neurons cannot make a significant contribution to the output y.
[0021] In another particularly advantageous embodiment, the positive metric maps specific input data x to a hash value H(x) with reduced dimensionality. The positive metric then determines the hash value h that is most similar to H(x) from a specified lookup table. * , in which the hash value h is stored in association with the information about the participation of the neuron. Finally, the positive metric uses the hash value h * The information stored in the lookup table in association is used to evaluate the neurons from the subset N\P. In the table, for example, what can be stored in binary form is that some neurons are fully activated while some other neurons are not activated. However, in the table, for example, specific activation strengths can also be stored. Then, the hash function H is advantageously designed so that the hash function maps input data x similar to each other to similar hash values H(x). Therefore, this position-sensitive hash function H is of a different type from, for example, a cryptographically secure hash function, which even only maps minimally different input data x to completely different hash values. An example is a so-called locally sensitive hash function (Locally Sensitive Hash Functions), i.e., LSH, which ideally maps similar input data x to the same output (to the same "Bucket"). Searching for information through a lookup table can be much faster than direct calculation. Through the dimension of the hash value H(x), the degree of discretization of information about neuron evaluation can be set, for example, in particular.
[0022] However, the positive metric may also include, for example, any approximate nearest neighbor search (ANNS), or another metric that estimates the involvement of a neuron in processing a specific input data x without having fully performed that processing. The calculation of the positive metric takes computation time, which must be weighed against the time saved due to the reduction in the number of neurons that are ultimately calculated.
[0023] In another particularly advantageous embodiment, the specific input data x also belongs to a domain and / or a distribution X, based on a plurality of input data. The value of the positive measure of creates a preselection of those neurons that play a key role in processing these specific input data x. Then, the final check of which neurons are important for processing specific input data x can be limited to this preselection, which correspondingly reduces the computational effort.
[0024] In a further advantageous embodiment, a control signal is determined from output data y generated by the neural network from the input data. The vehicle, driver assistance system, robot, quality control system, area monitoring system and / or medical imaging system is controlled by means of the control signal. Then, for a given configuration of the hardware platform used to implement the neural network, the probability that the reaction executed by the correspondingly controlled technical system in response to the control signal is appropriate to the situation represented by the input data x is increased due to the improved quality of the output data y. The input data x can be, for example, measurement data recorded by at least one sensor.
[0025] The method can in particular be completely or partially computer-implemented. Thus, the invention also relates to a computer program having machine-readable instructions which, when executed on one or more computers, cause the one or more computers to carry out the described method. In this context, control devices for vehicles and embedded systems for technical devices which are likewise capable of executing machine-readable instructions are also to be regarded as computers.
[0026] Likewise, the invention also relates to a machine-readable data carrier and / or a download product having the computer program. A download product is a digital product that can be transmitted via a data network, that is to say can be downloaded by a user of the data network, which digital product can be sold for immediate downloading in an online store, for example.
[0027] Furthermore, a computer can be equipped with the computer program, the machine-readable data carrier or the download product. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Further measures for improving the invention are presented in more detail below together with the description of preferred exemplary embodiments of the invention based on the drawings.
[0029] in:
[0030] Figure 1 An embodiment of a method 100 for processing input data x using a neural network 1 is shown;
[0031] FIG. 2 shows an illustration of the dependency of the neurons 11 - 19 used to process the input data x on the input data x. DETAILED DESCRIPTION
[0032] Figure 1 A schematic flow chart of an embodiment of a method 100 for processing input data x using a neural network 1 is shown. The neural network 1 comprises a set N of nine neurons 11-19. The interaction of these neurons is explained in more detail in conjunction with FIG2.
[0033] In step 110, based on the specified negative measure 2, a subset of neurons 11-19 whose use can be omitted without excessively impairing the performance of the neural network 1 is determined. The criterion of non-significant impairment of performance may be, for example, that the neural network 1 achieves at least a specified accuracy when processing input data x from a specified set of test or validation data even without using neurons in the subset p. The accuracy may be determined, for example, based on a comparison of output data y determined from the test or validation data with a target output used to label these test or validation data.
[0034] According to block 111, the neurons 11-19 can be evaluated independently of the specific input data x by means of the negative measure 2, based on which, for example, the order of neurons 11-19 to be deleted can be determined. For this evaluation, for example, the following parameters and / or coefficients can be used, which are respectively assigned to the neurons 11-19 in the network 1 and characterize the behavior of the network 1.
[0035] exist Figure 1 In the example shown in , in step 105, a neural network 1 is selected in which the inputs delivered to each neuron 11-19 are aggregated by taking a weighted sum to activate the neuron 11-19. Next, according to block 112, the negative metric 2 can evaluate the neuron 11-19 at least according to the weight in the weighted sum.
[0036] In step 120, according to the specified positive metric 3, a subset of neurons 11-19 that play a key role in processing the specific input data x is determined from the subset N\P This means that the path that the input data x takes through the network 1 depends on the input data x.
[0037] According to block 121, the positive metric 3 may map the specific input data x to a hash value H(x) with reduced dimensionality. Next, according to block 122, the positive metric may determine the hash value h that is most similar to H(x) from a specified lookup table. * In this lookup table, a hash value h is stored in association with information about the participation of neurons 11-19. Next, according to block 123, the hash value h can be used to * The information stored in the lookup table is associated to evaluate neurons 11-19 from the subset N\P.
[0038] According to block 124, based on a plurality of input data from a domain and / or distribution X to which the specific input data x also belongs, The value of the positive measure (3) can create a pre-selection of those neurons (11-19) that play a key role in processing these specific input data x.
[0039] In step 130, a subset of neurons 11-19 is selected to process a specific input data x. This subset is a superset of set A In the simplest case, set A can simply be passed as set D. However, Figure 1 An example of how set D can also be meaningfully expanded relative to set A is shown.
[0040] According to block 131 , one or more neurons 11 - 19 can be selected for additional inclusion in set D, for example, from the subset (N\P)\A, whose use was approved by negative metric 2 but not by positive metric 3 .
[0041] According to block 131a, for example, depending on a specified computing power budget, the number of neurons 11-19 to be additionally incorporated into set D can be specified. According to block 131b, this budget can be specified, for example, as a total number |D| of neurons 11-19 in set D.
[0042] According to block 131c, neurons 11-19 can be selected from the subset (N\P)\A, for example, in descending order of importance for processing specific input data x. According to block 131d, this order of importance can be specified, for example, in dependence on the values determined for these neurons 11-19 by the positive measure 3.
[0043] In step 140 , the input data x are processed using neurons 11 - 19 of set D into the sought output data y.
[0044] According to block 141 , neurons 11 - 19 of set D may be implemented, for example, in particular on a hardware platform whose resources are insufficient to implement the entire set N of neurons 11 - 19 .
[0045] exist Figure 1 In the example shown in FIG. 1 , in step 150 , a control signal 150 a is determined based on the output data y. In step 160 , the control signal 150 a is used to control the vehicle 50 , the driving assistance system 51 , the robot 60 , the quality control system 70 , the area monitoring system 80 and / or the medical imaging system 90 .
[0046] Figure 2a The architecture of a neural network 1 having neurons 11-19, which together form a set N, is illustrated.
[0047] These neurons 11-19 are arranged in three layers a, b, c. The first layer a is an input layer, which receives input data x and contains neurons 11-14. The second layer b is a hidden layer within the neural network 1, which contains neurons 15-17. The third layer c is an output layer, which outputs output data y from the neural network 1 and contains neurons 18 and 19.
[0048] exist Figure 2a In the example shown in , it was found by means of negative metric 2 that the use of neurons 11, 14 and 16 drawn with dashed lines can be omitted without excessively impairing the performance of the neural network 1. These neurons 11, 14 and 16 form a set P. Therefore, after deleting these neurons 11, 14 and 16, only the paths drawn with arrows are available in the network 1 for processing the input data x into the output data y. In this case, Figure 2a The examples shown in are purely illustrative. In a real, complex neural network 1, it is possible to delete far more Figure 2a A third more neurons were mapped, 11-19, i.e. usually more than 80 percent and in extreme cases up to 99 percent.
[0049] Figure 2b It is shown by way of example how the set A of neurons 11-19 that play a key role in processing specific input data x depends on these input data x. The set A(x1) of neurons 12, 15 and 19 that play a key role in processing the first input data x1 is plotted, and the set A(x2) of neurons 13, 17 and 19 that play a key role in processing the second input data x2 is plotted.
[0050] From the set N\P of neurons 11-19 that have not been excluded from the processing by the negative measure 2, the sets A(x1) and A(x2) are selected respectively. These sets each only still contain three of the total nine neurons 11-19. Therefore, the input data x1 and x2 can also be processed into output data y on a hardware platform that can only implement a maximum of three neurons. In this case, for example, after determining the sets A(x1) and A(x2), the hardware platforms can be configured accordingly.
[0051] Figure 2c The exemplary illustration shows how the set A(x1) of neurons 12, 15 and 19 that play a key role in processing the first input data x1 can also be meaningfully expanded to a superset D(x1) of neurons that can process the input data x1 into output data y when using it. In particular, the neuron 18 is also included in this superset D(x1).
Claims
1. A method (100) for processing input data x using a neural network (1), the neural network comprising a set N of neurons (11-19), the method comprising the following steps: Determining (110) that subset of neurons (11-19) whose use can be omitted without unduly impairing the performance of the neural network (1) based on the specified negative measure (2) According to the specified positive metric (3), determine (120) from the subset N\P the subset of neurons (11-19) that play a key role in processing the specific input data x P); To process the specific input data x, select (130) a subset of neurons (11-19) The subset is a superset of the set A Using the neurons (11-19) of the set D, the input data x is processed (140) into output data y.
2. The method (100) according to claim 1, wherein: The neurons (11-19) of the set D are implemented (141) on a hardware platform whose resources are insufficient to implement the entire set N of neurons (11-19).
3. The method (100) according to any one of claims 1 to 2, wherein: From the subset (N\P)\A whose use is approved by the negative metric (2) but not by the positive metric (3), one or more neurons (11-19) are selected (131) for additional inclusion in the set D.
4. The method (100) according to claim 3, wherein: Depending on the specified computing power budget, the number of neurons (11-19) to be additionally incorporated into the set D is specified (131a).
5. The method (100) according to claim 4, wherein: The computing power budget is defined (131b) as the total number |D| of neurons (11-19) in the set D.
6. The method (100) according to any one of claims 3 to 5, wherein: From said subset (N\P)\A, neurons (11-19) are selected (131c) in descending order of importance for processing said specific input data x.
7. The method (100) according to claim 6, wherein: The order of importance is defined (131d) as a function of the values determined for the neurons (11-19) by the positive metric (3).
8. The method (100) according to any one of claims 1 to 7, wherein: The negative measure (2) evaluates (111) the neuron (11-19) independently of the specific input data x.
9. The method (100) according to any one of claims 1 to 8, wherein selecting (105) a neural network (1) in which the inputs to each neuron (11-19) are aggregated by taking a weighted sum to activate the neuron (11-19), and The negative measure (2) evaluates (112) the neurons (11-19) at least according to their weights in the weighted sum.
10. The method (100) according to any one of claims 1 to 9, wherein: The positive metric (3) Mapping (121) the specific input data x to a hash value H(x) with reduced dimensionality; Determine (122) the hash value h that is most similar to H(x) from the specified lookup table * , in said look-up table, a hash value h is stored in association with information about the participation of neurons (11-19); and Use (123) and the hash value h * The information stored in the lookup table is associated to evaluate neurons (11-19) from the subset N\P.
11. The method (100) according to any one of claims 1 to 10, wherein: According to the multiple input data from the domain and / or distribution X to which the specific input data x also belongs The value of the positive measure (3) creates (124) a preselection of those neurons (11-19) that play a key role in processing the specific input data x.
12. The method (100) according to any one of claims 1 to 11, wherein Determining (150) a control signal (150a) based on the output data y; and Utilizing the control signal (150a) to control (160) a vehicle (50), a driving assistance system (51), a robot (60), a quality control system (70), an area monitoring system (80) and / or a medical imaging system (90).
13. A computer program comprising machine-readable instructions which, when executed on one or more computers, cause the one or more computers to perform the method (100) according to any one of claims 1 to 12.
14. A machine-readable data carrier and / or a download product having a computer program according to claim 13.
15. One or more computers having a computer program according to claim 13 and / or having a machine-readable data carrier and / or a download product according to claim 14.
