Devices and methods for generating compressed networks from trained neural networks

Abstract

An apparatus and method for generating a compressed network from a trained neural network are disclosed, wherein the method includes: a model (402) generating a compressed map (412) from first training data (304), the compressed map (412) representing the influence of model components of the model (402) on first output data in response to the first training data (304); generating a compressed network (606) by compressing (604) the trained neural network according to the compressed map (412); the trained neural network (610) generating trained network output data (612) in response to second training data (306); the compressed network (606) generating compressed network output data (608) in response to the second training data (306); and training the model (402) by comparing the trained network output data (612) with the compressed network output data (608).

Description

Technical Field

[0001] Various embodiments generally relate to apparatuses and methods for generating compressed networks from trained neural networks. Background Technology

[0002] For example, in autonomous driving, imaging sensors such as camera sensors and / or video sensors can be used to provide digital images of the environment surrounding the vehicle. Neural networks can be trained to process these digital images in various environments, such as busy cities, snow-capped mountains, or barren plains, and can control the vehicle based on the situations depicted in the digital images. Therefore, training the neural network for various environments, situations, objects, contexts, etc., makes the trained neural network computationally intensive. However, if the vehicle is in one environment (e.g., a city), the vehicle does not need to react to situations or objects in other environments (e.g., barren plains) that the neural network was trained on. Therefore, for example in real-time and safety-critical systems, it is necessary to provide a model capable of generating a compressed network with low computational cost from a trained neural network for a specific environment.

[0003] Training various neural networks on large datasets to perform multiple tasks results in high computational costs for trained neural networks. For many applications, such as real-time or safety-critical applications, it may be necessary to provide neural networks with low computational costs. Furthermore, it may be necessary to perform only some of the multiple tasks. Therefore, it may be necessary to generate compressed networks from trained neural networks, where the compressed networks are capable of performing some of the multiple tasks (e.g., one task) at low computational cost.

[0004] In Hinton et al.’s “Distilling the Knowledge in a Neural Network” (arXiv: 1503.0253, 2015), a method for compressing resource-intensive neural networks into resource-efficient neural networks is described.

[0005] In "Model Compression" by Bucila et al. (KDD proceedings, 2006), a method for compressing resource-intensive neural networks into resource-efficient neural networks is described.

[0006] In Chen et al.’s “You Look Twice: GaterNet for Dynamic Filter Selection in CNNs” (arXiv: 1811, 11205, 2019), a method for improving the performance of neural networks using scaling masks is described.

[0007] In Finn et al.’s “Model-Agnostic Meta-Learning for Fast Adaptation of DeepNetworks” (International Conference on Machine Learning, 2017), a meta-learning method for generalizing to unknown tasks is described. Summary of the Invention

[0008] The method and apparatus having the features of the first example and the thirty-first example enable the model to be trained to generate a compressed network from a trained neural network for performing a specific task of the trained neural network.

[0009] A model can be any kind of algorithm that provides output data in response to input data. For example, a model can be a neural network.

[0010] The model may include a first model part and a second model part. Generating a compressed graph may include the first model part generating an influence graph. The influence graph may represent the influence of a first model component in response to associated first training data on each first output data in the first output data. Generating a compressed graph may further include generating a combined influence graph of multiple influence graphs. Generating a compressed graph may include the second model part generating the compressed graph from the combined influence graph. Illustratively, the influence graph may represent the importance or influence of a corresponding first model component in response to the first training data on the first output data. The features mentioned in this paragraph, combined with the first example, provide a second example.

[0011] Each of the plurality of first model components may include a bias and multiple weights. A first model component may further include an activation function. The features mentioned in this paragraph, combined with the second example, provide a third example.

[0012] Each of the plurality of first model components may further include a first model component output. The influence graph may include the outputs of multiple first model components from one of the first training data. The features mentioned in this paragraph, combined with the second or third example, provide a fourth example.

[0013] Training the model may include training a first model part and / or training a second model part. The first and / or second model parts can be trained by comparing the output data of the trained network with the output data of the compressed network. The features mentioned in this paragraph, combined with any of the second through fourth examples, provide a fifth example.

[0014] Generating a compressed network may include removing network components from the trained neural network based on the compressed graph if the corresponding value in the compressed graph meets a predefined criterion. The predefined criterion is met if the corresponding value in the compressed graph is below a predefined threshold. A sixth example is provided by combining the features mentioned in this paragraph with any of the first through fifth examples.

[0015] Training the model can include training the model to increase the overall compression. The overall compression can be increased by reducing the sum of each value in the compressed graph. In other words, multiple values ​​in the compressed graph can be added together, and the model can be trained to reduce the sum. The seventh example is provided by combining the features mentioned in this paragraph with any of the first through sixth examples.

[0016] Comparing the trained network output data with the compressed network output data can involve determining the loss value by comparing each trained network output data with its associated compressed network output data. A loss function can be used to determine the loss value among multiple loss values. The loss function can be the cross-entropy loss function. The features mentioned in this paragraph, combined with the fifth example, provide the eighth example.

[0017] The method may further include determining a total loss value from multiple loss values. The total loss value can be determined by a regularization term and the sum of the multiple loss values. The first and second model parts can be trained using the total loss value and the gradient of the loss values ​​relative to the first model component through backpropagation. The regularization term can be any term preferred for the sparse solution. Therefore, the regularization term has the effect of extracting or compressing the trained neural network. In other words, the regularization term has the effect of increasing the number of compression factors with values ​​of "0" or close to "0" in the compressed graph. The features mentioned in this paragraph, combined with the eighth example, provide a ninth example.

[0018] The first model portion may include at least a portion of a trained neural network. A first model component of the first model portion may correspond to a trained network component of the trained neural network. The features mentioned in this paragraph, combined with any of the second through ninth examples, provide a tenth example.

[0019] A trained neural network may include a first part and a second part of trained network components. The trained network component associated with the first part may be different from the trained network component associated with the second part. A first model component of the first model portion may correspond to the first part of the trained network components. The features mentioned in this paragraph, combined with the tenth example, provide an eleventh example.

[0020] The first part of the trained network component can provide intermediate output data for the first training data, and the second part of the trained network component can provide the first output data for the intermediate output data. This has the effect that the first model component initially (i.e., before training) corresponds to the first layer of the trained neural network, where the first layer of the trained neural network is important for analyzing the features of the processed data. The features mentioned in this paragraph, combined with the eleventh example, provide the twelfth example.

[0021] An influence graph can represent the effect of a trained network component's response to associated first training data on a first output data. The features mentioned in this paragraph, combined with the eleventh or twelfth example, provide a thirteenth example.

[0022] The first output data can be generated by a trained neural network from the first training data. The features mentioned in this paragraph, combined with the thirteenth example, provide the fourteenth example.

[0023] The combined influence map generated from the plurality of influence maps may include the sum or average of the plurality of influence maps. The feature mentioned in this paragraph, combined with any of the second through fourteenth examples, provides the fifteenth example.

[0024] A trained neural network can be trained to provide first output data for a first input data of multiple tasks. A compressed network can provide second output data for a second input data of at least one of the multiple tasks. The features mentioned in this paragraph, combined with any of the first through fifteenth examples, provide a sixteenth example.

[0025] Generating a compressed network may include multiplying a compressed graph with a trained neural network. The seventeenth example is provided by combining the features mentioned in this paragraph with any of the first through sixteenth examples.

[0026] Each of the multiple trained network components may include biases and multiple weights. The trained network components may further include activation functions. The features mentioned in this paragraph, combined with any of the first through seventeenth examples, provide the eighteenth example.

[0027] Generating a compressed network may involve multiplying multiple weights and / or biases of each trained network component of the trained neural network with a compression factor associated with the compressed graph. Therefore, combined with the regularization term in Example 9, the number of multiple weights and / or biases with values ​​of "0" or close to "0" in the compressed network increases. This has the effect that the compressed network has lower computational cost and that the computations performed by the compressed network require less time. The compressed network generated using the compressed graph as described above has the effect that it does not have a predetermined network architecture, but rather a task-specific, efficient network architecture. The features mentioned in this paragraph, combined with Examples 17 and 18, provide Example 19.

[0028] Each trained network component may further include batch normalization. Each of the multiple weights in each trained network component may be normalized before being multiplied with the compression factor associated with the compressed graph. The features mentioned in this paragraph, combined with the nineteenth example, provide the twentieth example.

[0029] Before multiplying with the compression factor associated with the compressed graph, a bias offset can be added to the bias of each trained network component. The bias offset can be determined using batch-normalized batch variance, batch mean, batch compression factor, and batch offset. The features mentioned in this paragraph, combined with the twentieth example, provide the twenty-first example.

[0030] First training data and / or second training data are selected from multiple datasets. The multiple datasets may include multiple tasks, and the first and / or second training data may include at least one of the multiple tasks. The first training data may differ from the second training data. A twenty-second example is provided by combining the features mentioned in this paragraph with any of the first through twenty-first examples.

[0031] A selection model can be used to select first and / or second training data from the plurality of data. The selection model can be or may include a neural network. The features mentioned in this paragraph, combined with the twenty-second example, provide the twenty-third example.

[0032] The first and / or second training data may include one of the following data types: digital images, time series, or point clouds. The data type of the first training data may correspond to the data type of the second training data. The twenty-fourth example is provided by combining the features mentioned in this paragraph with any of the first through twenty-third examples.

[0033] The model may include a neural network. The twenty-fifth example is provided by combining the features mentioned in this paragraph with any of the first through twenty-fourth examples.

[0034] The first and / or second model components can be neural networks. The features mentioned in this paragraph, combined with the twenty-fifth example, provide the twenty-sixth example.

[0035] The trained network components can be neurons of a trained neural network. The twenty-seventh example is provided by combining the features mentioned in this paragraph with any of the first through twenty-sixth examples.

[0036] The method may further include generating digital output data from a compressed network generated by a trained model for digital input data. The features mentioned in this paragraph, combined with any of the first through twenty-seventh examples, provide a twenty-eighth example.

[0037] The method may further include a sensor detecting digital input data. The sensor may be a camera sensor, video sensor, radar sensor, LiDAR sensor, ultrasonic sensor, motion sensor, or thermal sensor. The features mentioned in this paragraph, combined with the twenty-eighth example, provide a twenty-ninth example.

[0038] The method may further include generating third training data from a compressed network generated by the trained model. The method may include training another model using the third training data. A thirtieth example is provided by combining the features mentioned in this paragraph with any of the first through twenty-ninth examples.

[0039] At least a portion of the model can be implemented by one or more processors. The thirty-first example is provided by combining the features mentioned in this paragraph with any of the first through thirtieth examples.

[0040] A computer program product may store program instructions configured to, if executed, perform a method of any one of the first through thirty-first examples. The features mentioned in this paragraph provide a thirty-second example.

[0041] A device may include a compressed network generated by a model trained using any of the methods in the first through thirty-first examples. The device mentioned in this paragraph provides a thirty-fourth example.

[0042] A system may include the device of the thirty-fourth example, wherein the device may be configured to process digital input data. The system may further include at least one sensor. The sensor may be configured to provide digital input data to the device. The system mentioned in this paragraph provides a thirty-fifth example.

[0043] A vehicle may include at least one sensor. The at least one sensor may be configured to provide digital input data. The vehicle may further include a driving assistance system. The driving assistance system may include a compressed network generated by a model trained using any of the methods in the first through thirty-first examples. The compressed network may be configured to provide digital output data in response to the digital input data. The driving assistance system may be configured to use the digital output data to control the vehicle. The thirty-sixth example is provided for the vehicle mentioned in this paragraph. Attached Figure Description

[0044] Various embodiments of the present invention are described with reference to the following figures, wherein: Figure 1 Devices according to various embodiments are shown; Figure 2 An imaging device according to various embodiments is shown; Figure 3 A processing system for selecting training data according to various embodiments is illustrated; Figure 4 A processing system for generating compressed maps according to various embodiments is shown; Figure 5 Model components according to various embodiments are shown; Figure 6 A processing system for generating a compressed network from a trained neural network, according to various embodiments, is shown; Figure 7 A processing system for training a model is illustrated according to various embodiments; Figure 8 Methods for generating compressed networks from trained neural networks according to various embodiments are illustrated; Figure 9 Systems including a compression network according to various embodiments are shown; Figure 10 Vehicles according to various embodiments are shown. Detailed Implementation

[0045] In one embodiment, "circuit" can be understood as any kind of logical implementation entity, which can be hardware, software, firmware, or any combination thereof. Thus, in one embodiment, "circuit" can be hardwired logic circuitry or programmable logic circuitry such as a programmable processor, for example, a microprocessor (e.g., a Complex Instruction Set Computer (CISC) processor or a Reduced Instruction Set Computer (RISC) processor). "Circuit" can also be software implemented or executed by a processor, such as any kind of computer program, for example, a computer program using virtual machine code (such as, for example, Java). Any other kind of implementation of the corresponding functionality, which will be described in more detail below, can also be understood as a "circuit" according to alternative embodiments.

[0046] Neural networks are used in many applications and can be trained to perform multiple tasks. However, this leads to, for example, high computational costs. In many applications or situations, only some of the multiple tasks for which the neural network is trained are necessary. Illustratively, a model is trained to generate a compressed network from the trained network, wherein the compressed network is capable of performing at least one of the multiple tasks at a lower computational cost.

[0047] Figure 1 A device 100 according to various embodiments is illustrated. Device 100 may include one or more sensors 102. Sensors 102 may be configured to provide (digital) data 104. Sensors 102 may be any kind of sensor capable of providing (digital) data, such as imaging sensors (e.g., camera sensors or video sensors), radar sensors, LiDAR sensors, ultrasonic sensors, motion sensors, thermal sensors, etc. Multiple sensors may have the same sensor type or different sensor types. Device 100 may further include a memory device 106. Memory device 106 may include memory used, for example, in processing performed by a processor. The memory used in the embodiments may be volatile memory, such as DRAM (Dynamic Random Access Memory), or non-volatile memory, such as PROM (Programmable Read-Only Memory), EPROM (Erasable PROM), EEPROM (Electrically Erasable PROM), or flash memory, such as floating gate memory, charge-trapped memory, MRAM (Magnetoresistive Random Access Memory), or PCRAM (Phase Change Random Access Memory). Memory device 106 may be configured to store digital data 104 provided by one or more sensors 102. The device 100 may further include at least one processor 108. As described above, the at least one processor 108 can be any kind of circuit, i.e., any kind of logic implementation entity. In various embodiments, the processor 108 may be configured to process digital data 104.

[0048] In the following description, the embodiment will be based on a digital image as digital data 104. Note that digital data of any data type can be used, such as time series or point clouds.

[0049] Figure 2 An imaging apparatus 200 according to various embodiments is illustrated. The imaging apparatus 200 may include one or more imaging sensors 202. The imaging sensors 202 may be configured to provide a plurality of digital images 204. The plurality of digital images 204 may include a plurality of tasks. In other words, each of the plurality of digital images 204 may be associated with a task among the plurality of tasks. For example, the plurality of digital images 204 may include a plurality of first digital images 206 associated with a first task, a plurality of second digital images 208 associated with a second task, and a plurality of third digital images 210 associated with a third task, wherein each task may be different from any other task among the plurality of tasks. Each of the plurality of digital images 204 may be associated with a class among a plurality of classes, and at least one class among the plurality of classes may be associated with a task among the plurality of tasks. Illustratively, each task among the plurality of tasks may include more than one class among the plurality of classes. For example, in autonomous driving, one of the tasks could be autonomous driving in an urban area, where associated categories could be streets, pedestrians, buildings, sky, trees, road signs, etc.; or one of the tasks could be autonomous driving in a rural area, where associated categories could be main roads, pedestrians, animals, sky, trees, road signs, etc. Sensor 102 can be any kind of sensor capable of providing (digital) data, such as imaging sensors (e.g., camera sensors or video sensors), radar sensors, LiDAR sensors, ultrasonic sensors, motion sensors, thermal sensors, etc. Multiple sensors can have the same sensor type or different sensor types. Device 200 may further include a memory device 106 to store multiple digital images 204. Device 200 may further include at least one processor 108. Processor 108 can be configured to process the multiple digital images 204.

[0050] Figure 3A processing system 300 for selecting training data according to various embodiments is illustrated. The processing system 300 may include a memory device 106. The memory device 106 may store a plurality of digital images 204, such as a plurality of first digital images 206, a plurality of second digital images 208, and a plurality of third digital images 210. The processing system 300 may further include at least one processor 108. The processor 108 may be configured to process the plurality of digital images 204. The processor 108 may implement at least a portion of a trained neural network, wherein the trained neural network may be configured to process the plurality of digital images 204. In other words, the trained neural network may have been trained to provide output data for the plurality of digital images 204. The processor 108 may be configured to implement at least a portion of a selection model 302. The selection model 302 may be a neural network. The selection model 302 may be configured to process the plurality of digital images 204. The selection model 302 may be configured to select first training data 304 from the plurality of digital images 204. The first training data 304 may include any number of digital images. The first training data 304 may include digital images associated with at least one of a plurality of tasks of the plurality of digital images 204, wherein each task may include at least one of a plurality of classes. The first training data 304 may include at least one digital image 206A from a plurality of first digital images 206 associated with the first task, such as a first portion of the plurality of first digital images 206. The selection model 302 may be further configured to select second training data 306 from the plurality of digital images 204. The second training data 306 may include any number of digital images. The second training data 306 may include digital images associated with at least one task of the first training data 304. The second training data 306 may include at least one digital image 206B from a plurality of first digital images 206 associated with the first task, such as a second portion of the plurality of first digital images 206, wherein the second portion of the plurality of first digital images 206 may be different from the first portion of the plurality of first digital images 206. Illustratively, model 302 can select several digital images from a plurality of digital images 204 associated with at least one task as first training data 304, and can select several digital images from the plurality of digital images 204 associated with at least one task as second training data 306, wherein the digital images of the first training data 304 are different from the digital images of the second training data 306. In other words, the plurality of digital images 204 can represent data of multiple tasks including multiple classes, wherein the trained neural network can be configured to perform multiple tasks, and wherein the first training data 304 and / or the second training data 306 can represent at least one of the multiple tasks. Illustratively, the first training data 304 and / or the second training data 306 describe at least one of the multiple tasks.

[0051] Figure 4 A processing system 400 for generating compressed maps according to various embodiments is illustrated. The processing system 400 may include a memory device 106. The memory device 106 may store first training data 304. The processing system 400 may further include at least one processor 108. The processor 108 may be configured to process the first training data 304. The processor 108 may be configured to implement at least a portion of a model 402. The model 402 may include a first model portion 404. The first model portion 404 may be configured to process the first training data 304. The first model portion 404 may be configured to generate a plurality of influence maps 406 for the training data 304. In other words, the first model portion 404 may be configured to generate an influence map for each first training data in the first training data 304. In yet another way, the first model portion 404 may be configured to generate an influence map for each digital image (e.g., at least one digital image 206A) of the first training data 304. The plurality of influence maps 406 may represent the influence of a first model component on first output data in response to the first training data 304. The influence map associated with the digital image of training data 304 can represent the influence of a first model component on a first output data point in the first output data in response to the associated first training data. In other words, the first model portion 404 can include a first model component for processing data such as the first training data 304. The first model portion 404 can be a neural network, and the first model component can be a neuron of the neural network. Figure 5 Model components according to various embodiments are illustrated. A first model portion 404 may include a prior layer 502, wherein the prior layer 502 may include a plurality of prior model components 506 and may include a prior bias 508. The first model portion 404 may include a posterior layer 504, wherein the posterior layer 504 may include a plurality of posterior model components 510 and may include a posterior bias 512. The prior model components 506 and posterior model components 510 may include a plurality of connections 514. Each connection of the posterior model components among the plurality of posterior model components 510 may include a weight 506A or a prior bias 508. Each posterior model component may include an activation function. Each of the plurality of prior model components 506 may provide a prior model component output 516. Each posterior model component may determine a posterior model component output 518. The posterior model component output 518 may be determined using the prior model component output 516, the weight 506A, the prior bias 508, and optionally the activation function. The influence graph in multiple influence graphs 406 may include multiple first model component outputs for one of the first training data (such as digital image 206A) in the first training data 304.

[0052] The first model portion 404 may include at least a portion of a trained neural network. A first model component of the first model portion 404 may correspond to a trained network component of the trained neural network. In other words, the architecture of weights, biases, and activation functions comprising at least a portion of the first model portion 404 may correspond to at least a portion of the trained neural network. Illustratively, the first model component of the first model portion 404 may be initialized—i.e., before training the first model portion 404—as at least a portion of the trained neural network. The trained neural network may include a first portion and a second portion of the trained network components, wherein the first model component may correspond to the first portion of the trained network components. The first portion of the trained network components may provide intermediate output data for first training data 304, and the second portion of the trained network components may provide first output data for the intermediate output data. In other words, the first model portion 304 may include a first model component, wherein the first model component may include at least a portion of the trained network components of the trained neural network. Illustratively, the trained neural network includes multiple network layers, and the first model portion may include the first portion of multiple network layers. Therefore, the output generated by each of the first model components for a given first training data in the first training data 304 can correspond to the output generated by the corresponding trained network component. In other words, if the trained neural network processes the first training data, the trained neural network can output first output data for the first training data, and each trained network component can include an output, i.e., a trained network component output. Therefore, the first model component output of the first model component for the first training data can represent the importance or influence of the first model component on the first output data in response to the first training data. Illustratively, each first model component can have a different influence on generating the corresponding first output data in order to process the first training data. In other words, some first model components may have no influence on processing the corresponding first training data, i.e., the output of the corresponding first model component may have a value of "0". The influence graph can include multiple first model component outputs for a given first training data in the first training data 304. Therefore, the influence graph can represent the influence of the trained neural network component given by the first model component on a given first output data in response to the associated first training data.

[0053] like Figure 4As shown, processor 108 can be further configured to generate a combined influence map 408 of multiple influence maps 406. The combined influence map 408 can be generated using the sum or average of the multiple influence maps 406. In other words, each influence map in the multiple influence maps 406 can include multiple first model component outputs, and the combined map can be generated by adding (i.e., summing) or averaging (i.e., averaging) the corresponding first model outputs of each influence map in the multiple influence maps 406. Therefore, generating the combined influence map 408 can be a transformation of the multiple influence maps 406, wherein model 402 can include this transformation.

[0054] Model 402 may further include a second model portion 410. The second model portion 410 may be a neural network. According to various embodiments, model 402 is a neural network, i.e., the first model portion 404 and the second model portion 410 are neural networks. The second model portion 410 may be configured to generate a compressed graph 412 from the combined influence graph 408. Therefore, the compressed graph 412 may represent the influence of model components of the model on the first output data in response to the first training data 304. The compressed graph 412 may include a plurality of compression factors, each of which may be assigned to a trained network component of the trained neural network. Each trained network component may include a bias and at least one weight, and the compressed graph 412 may include a compression factor associated with a corresponding trained network component, wherein the compression factor may be a factor used to scale or compress the bias and at least one weight of the corresponding trained network component.

[0055] Figure 6A processing system 600 for generating a compressed network from a trained neural network, according to various embodiments, is illustrated. Processing system 600 may substantially correspond to processing system 400, wherein processor 108 is further configured to generate a compressed network 606. Memory device 106 may further store the specification of a trained neural network 602. The specification of the trained neural network 602 may include multiple weights and / or multiple biases of the trained network components of the trained neural network. Processor 108 may be configured to process the specification of the trained neural network 602. Processor 108 may be configured to scale (e.g., compress 604 or extract) the specification of the trained neural network 602 using compression graph 412. In other words, processor 108 may be configured to scale (e.g., compress 604 or extract) multiple weights and / or multiple biases of the trained network components using compression graph 412. Processor 108 may be configured to generate a compressed network 606 by compressing the specification of the trained neural network 602 according to compression graph 412. In other words, processor 108 can be configured to generate compressed network 606 by compressing trained neural network 604 according to compression graph 412. The trained neural network 604 can be compressed into compressed network 606 by multiplying compression graph 412 with the specifications of trained neural network 602. In other words, trained neural network 604 can be compressed by multiplying each of the multiple weights and / or each of the multiple biases of the trained network components with the associated compression factor of compression graph 412. As described above, the trained neural network can be trained to perform multiple tasks, i.e., processing data of multiple classes, and the first training data 304 can represent at least one of the multiple tasks, wherein the at least one task can include at least one class, such as multiple classes. Therefore, compression graph 412 generated by model 402 using the first training data 304 can represent the effect of model components (such as a first model component) on the first training data 304, and illustratively, compressed network 606 generated using compression graph 412 can be configured to perform at least one of the multiple tasks. In other words, the first training data 304 describes the task description, and the task description is a constraint on the trained neural network.

[0056] Each trained network component may further include or be associated with batch normalization, and modified weights may be determined for each of the multiple weights of the trained network component before multiplying with the compression factor associated with compression graph 412.

[0057] Modified weights ( W m ) can be determined by equation (1): (1) in W These are the corresponding weights, and where c is determined by... Confirmed, among which It is the batch compression factor of the batch normalization layer, and It is the batch variance of the batch normalized layer.

[0058] Before multiplying with the compression factor associated with compression diagram 412, the bias offset ( b offset The bias is added to each trained network component. The bias offset can be determined by equation (2): (2) in It is the batch average of the batch normalized layer, and among them It is the batch offset of the batch normalization layer.

[0059] Before multiplying by the compression factor associated with compressed graph 412, multiple modified weights can be normalized. This can be achieved through prediction. The norm of each row is used to normalize the multiple modified weights.

[0060] Using the compressed graph 412 to generate the compressed network 606 may change the batch statistics, and thus make it more difficult to train the model 402; determining the modified weights according to the bias offset and / or normalizing the modified weights can avoid the above problems.

[0061] Figure 7A processing system 700 for training a model is illustrated according to various embodiments. Processing system 700 may substantially correspond to processing system 600, wherein a memory device may further store second training data 306, including, for example, a digital image 206B. Processor 108 may be configured to implement at least a portion of compressed network 606. Compressed network 606 may be configured to process the second training data 306 and may be configured to generate compressed network output data 608 in response to the second training data 306. Processor 108 may be further configured to implement at least a portion of trained neural network 610. Trained neural network 610 may be configured to process the second training data 306 and may be configured to generate trained network output data 612 in response to the second training data 306. Processor 108 may be configured to train model 402 by comparing trained network output 612 with compressed network output 608. Model 402 may be trained by training a first model portion 404 and / or a second model portion 410. Processor 108 can be configured to determine a loss value by comparing each trained network output data in trained network output data 612 with the associated compressed network output data in compressed network output data 608. For example, trained neural network 610 may be a segmentation network capable of generating segmented images for digital images of second training data 306, and compressed network 606 may be a segmentation network capable of generating segmented images for digital images of second training data 306. Processor 108 can be configured to compare each segmented image generated by trained neural network 610 with a segmented image generated by compressed network 606 and can be configured to determine a loss value based on this comparison. A loss function can be used to determine the loss value among multiple loss values. The loss function may be a cross-entropy loss function. Processor 108 can be further configured to determine a total loss value 614 among the multiple loss values. The total loss value 614 can be determined by a regularization term and the sum of the multiple loss values. The regularization term can be any term of a preferred sparse solution, such as... L 1 or L 0.5 Loss. The first model part 404 and / or the second model part 410 can be trained using the total loss value 614. The first model part 404 and / or the second model part 410 can be trained by reducing (e.g., minimizing) the total loss value 614.

[0062] The total loss value of 614 can be determined by equation (3): (3) in S It is the first training data 304, and T It is the second training data 306, in which It is a trained neural network 610, and It uses the first training data 304 ( S The generated compression network 606, in which Y The trained network outputs 612 data points, and among them... It is a regularization term.

[0063] Illustratively, due to the regularization term, not only are the components of the trained network (such as the associated weights) scaled, but most of the compression factors in the compressed graph 412 are equal to or close to the value "0". This has the effect of omitting multiple weights associated with the trained neural network 610, which implies that the trained neural network is compressed into a compressed network 606.

[0064] Figure 8 A method 800 for generating a compressed network from a trained neural network according to various embodiments is illustrated. Method 800 may include a model 402 generating a compressed graph 412 (in 802) from first training data 304. The compressed graph 412 may represent the effect of model components of model 402 on first output data in response to the first training data 304. Method 800 may include generating a compressed network 606 (in 804) by compressing the trained neural network 604 according to the compressed graph 412. Method 800 may include a trained neural network 610 generating trained network output data 612 (in 806) in response to second training data 306. Method 800 may include the compressed network 606 generating compressed network output data 608 (in 808) in response to the second training data 306. Method 800 may further include training model 402 (in 810) by comparing the trained network output 612 with the compressed network output 608.

[0065] Figure 9 A system 900 including a segmentation model according to various embodiments is illustrated. System 900 may include a first device 902. The first device 902 may be a computer-controlled device, such as a robot, vehicle, home appliance, power tool, manufacturing machine, personal assistant, access control system, etc. The first device 902 may be a device for transmitting information, such as a surveillance system or medical (imaging) system. System 900 may be a defect detection system, and the first device 902 may be a moving part on an assembly belt. System 900 may be a system for autonomous driving, and the first device 902 may be a vehicle. System 900 may be an embedded system. System 900 may be a real-time and / or safety-critical system, i.e., a system with time and computational constraints. System 900 may further include a sensor 904. Sensor 904 may be configured to detect digital input data associated with the first device 902, such as digital images, for example, digital images showing the environment around the vehicle or showing parts on an assembly belt.

[0066] System 900 may include a second device 906. The second device 906 may be configured to process digital input data provided by sensor 904. The second device 906 may include a compression network, such as compression network 606. Compression network 606 may be generated from a trained neural network using method 800 for generating a compression network from a trained neural network. The trained neural network may be configured to perform multiple tasks, and compression network 606 may be configured to perform at least one of the multiple tasks. In other words, the trained neural network may be configured to process digital data associated with multiple tasks comprising multiple classes, and compression network 606 may be configured to process digital input data associated with at least one of the multiple tasks, wherein the at least one task may include at least one class, such as multiple classes. Illustratively, digital output data generated by compression network 606 for digital input data may substantially correspond to digital output data that the trained neural network would generate for digital input data. System 900 may further include a control device 908. Control device 908 may be configured to control first device 902 using the digital output data provided by second device 906.

[0067] Figure 10A vehicle 1000 according to various embodiments is illustrated. The vehicle 1000 may be a vehicle with an internal combustion engine, an electric vehicle, a hybrid vehicle, or a combination thereof. Further, the vehicle 1000 may be a car, truck, ship, drone, or aircraft, etc. The vehicle 1000 may include at least one sensor 1002. The sensor 1002 may be any kind of sensor capable of providing (digital) input data, such as an imaging sensor (e.g., a camera sensor or video sensor), a radar sensor, a LiDAR sensor, an ultrasonic sensor, a motion sensor, a thermal sensor, etc. The vehicle 1000 may include a driver assistance system 1004. The driver assistance system 1004 may include a memory device 106. The driver assistance system 1004 may further include at least one processor 108. The processor 108 may implement a compression network 606, wherein the compression network 606 may be generated from a trained neural network by a model 402 via method 800. The compression network 606 may be configured to provide digital output data in response to digital input data. The driver assistance system 1004 can be configured to control the vehicle 1000 using digital output data. In other words, the driver assistance system 1004 can be configured to process the digital output data generated by the compression network 606 and use the generated digital output data to output control commands to one or more actuators of the vehicle 1000. Therefore, the driver assistance system 1004 can use the digital output data to influence the current vehicle handling, for example, to maintain or change the current vehicle handling. Changing the current vehicle handling can be, for example, an intervention in vehicle handling due to safety reasons (such as emergency braking).

Claims

1. A method for generating a compressed network from a trained neural network for a computer-implemented model of autonomous driving, the method comprising: • The first model part of the model generates an influence map representing the effect of the model component in response to the associated first training image on each of the first output data in the first output data, wherein each generated influence map includes the effect on the processing of the corresponding first training image for each model component; • Generate a combined impact map using each of the generated impact maps; • The second model part of the model generates a compressed map from the combined influence map; • A compressed network is generated by compressing a trained neural network according to a compression graph, wherein generating the compressed network includes deleting network components from the trained neural network according to the compression graph if the corresponding values ​​in the compression graph satisfy a predefined deletion criterion. • The trained neural network responds to a second training image to generate output data from the trained network; • The compressed network responds to the second training image to generate compressed network output data; • The model is trained by comparing the output data of the trained network with the output data of the compressed network; and The first training image and the second training image are digital images of the environment surrounding the vehicle; and The trained neural network is trained to provide first output data for first input data of multiple tasks, each task being associated with a driving environment, and the compressed network provides second output data for second input data of at least one of the multiple tasks.

2. The method of claim 1, wherein training the model comprises training a first model portion and / or training a second model portion by comparing trained network output data with compressed network output data.

3. The method according to claim 1 or 2, wherein if the corresponding value in the compressed graph is lower than a predefined threshold, the predefined deletion criterion is satisfied.

4. The method of claim 1 or 2, wherein training the model includes training the model to increase total compression by reducing the sum of each value of the compressed graph.

5. The method of claim 1 or 2, wherein the first output data is generated by a trained neural network for a first training image.

6. The method of claim 1, wherein a selection model is used to select a first training image and / or a second training image from a plurality of data.

7. The method according to claim 1 or 2, further comprising: • A compressed network generated by a trained model responds to input data to generate a third training image; • Train another model using a third training image.

8. A computer program product storing program instructions, said program instructions being configured, when executed, to perform the method of any one of claims 1 to 7.

9. An apparatus configured to perform the method of any one of claims 1 to 7.

10. A system comprising: A device comprising a compressed network generated by a model trained by any one of claims 1 to 7, the device being configured to process digital input data; At least one sensor is configured to provide digital input data to the device.

11. A means of transport, comprising: At least one sensor is configured to provide digital input data; A driving assistance system includes a compressed network generated from a trained neural network using a model trained by any one of claims 1 to 7, wherein the trained neural network is configured to process digital input data, wherein the compressed network is configured to provide digital output data for the digital input data provided by the at least one sensor, and wherein the driving assistance system is configured to use the digital output data to control the vehicle.

Images