Convolutional neural network and methods for operating the convolutional neural network

Abstract

The disclosure relates to a convolutional neural network (10) and a method for operating the convolutional neural network (10). It is proposed that the convolutional neural network (10) comprises a layered structure, wherein a first convolutional layer (12) is provided that processes a feature map with dimensions C x H x W and a second convolutional layer (14) that processes a subset (X̂) of the preceding feature map with dimensions k x H x W, where k < C.

Description

State of the art

[0001] The disclosure relates to a convolutional neural network and a method for operating the convolutional neural network.

[0002] Neural networks, especially convolutional neural networks (CNNs), typically require extensive computing and storage resources.

[0003] So-called pruning methods are known from the prior art, which are used to reduce the resources required. For example, static methods are known that are based on eliminating weights or filters of the neural network to reduce the computational effort. However, the complexity of the neural network's inputs is not taken into account in the known static methods. This leads to inefficiencies in these methods.

[0004] Dynamic computing methods are also known, which aim to adapt computing resources to the complexity of the neural network's inputs in order to increase efficiency. However, dynamic approaches have proven disadvantageous in practice when integrated into hardware optimized for static computational patterns.

[0005] The present disclosure relates to a method for reducing the resources required, which overcomes the disadvantages of the prior art. Disclosure of the invention

[0006] One embodiment relates to a method for operating a convolutional neural network to solve a specified task, comprising the following steps: - Providing a first feature map by means of a first convolutional layer of the convolutional neural network, wherein the first feature map comprises a first set of channels of number C, - Determining a subset of the first feature map based on a selection of a second set of channels of number k from the first set of channels of number C of the first feature map, where k < C, wherein the selection of the second set of channels of number k is based on a relevance of the channels with respect to the intended task, - Processing the subset of the first feature map using a second convolutional layer of the convolutional neural network, where the subset of the first feature map comprises the second set of channels of number k, - Providing a second feature map by adding the second set of channels of the number k of the processed subset of the first feature map to the channels of the number C of the first feature map, wherein the second feature map comprises the first set of channels of the number C.

[0007] The first convolutional layer of the convolutional neural network processes an input and, based on this input, generates the first feature map. This first feature map might, for example, have the dimensions C × H × W, where C is the number of channels in the first set of the feature map and H and W describe the spatial dimensions of the feature map.

[0008] To effectively reduce computing resources, the second convolutional layer is designed to process not the entire first feature map, but a subset of it. This subset might include, for example, the dimensions k × H × W, where k is the number of selected channels in the second set and H and W describe spatial dimensions of the feature map.

[0009] This selection helps to effectively reduce and concentrate computing resources by processing only the channels of the number k that are highly relevant to the intended task in the second convolutional layer.

[0010] The second convolutional layer is designed to improve the properties of the selected channels of number k. The second convolutional layer can be based, for example, on existing architectures such as ResNets; see Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., Zhang, L.: CvT: Introducing Convolutions to Vision Transformers. In: 2021 IEEE / CVF International Conference on Computer Vision (ICCV). pp. 22-31. IEEE Computer Society, Los Alamitos, CA, USA (October 2021). https: / / doi.org / 10.1109 / ICCV48922.2021.00009, https: / / doi.ieeecomputersociety.orQ / 10.1109 / ICCV48922.2021.00009 or ConvNext, cf. Liu, Z., Mao, H., Wu, CY, Feichtenhofer, C., Darrell, T., Xie, S.: A ConvNet for the 2020s. In: 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11966-11976 (2022). https: / / doi.org / 10.1109 / CVPR52688.2022.0116, be trained.

[0011] The proposed method synergistically combines the advantages of static pruning and dynamic computing, utilizing static computation graphs that improve both the training and inference of the neural network. Computing resources are dynamically allocated based on channel relevance, effectively optimizing inference speeds. The proposed method requires no user-defined specifications; in particular, it eliminates the need for CUDA kernel modifications, additional loss functions, or extensive fine-tuning.

[0012] After the second convolutional layer processes a subset of the first feature map, channel integration is performed. This process generates a second feature map by adding the second set of channels (number k) from the processed subset of the first feature map to the channels (number C) of the first feature map. The second feature map comprises the first set of channels (number C). Therefore, the second feature map also includes, for example, the dimensions C × H × W. Channel fusion combines processed channels from the processed subset with unprocessed channels from the first feature map while maintaining the dimensionality required for subsequent convolution operations.

[0013] Advantageously, determining the subset of the first feature map includes: determining the relevance of channels of the first set of channels of number C with respect to the intended task based on a weight vector, and selecting the second set of channels of number k from the first set of channels of number C with the greatest relevance with respect to the intended task.

[0014] Channel selection is achieved, for example, by compressing the first feature map and processing the compressed first feature map using a two-layer neural subnetwork. Compression includes, for example, reducing the dimensions through adaptive average pooling, so that the compressed first feature map comprises the dimensions C × 1 × 1. The first compressed feature map is processed by a two-layer, and in particular fully connected, subnetwork, for example, comprising a bottleneck structure. Subsequent sigmoid activation generates a weight vector w, where the elements, i.e., the individual weights of the weight vector w, quantify the respective relevance of each channel.Based on the weights, the second set of channels of the number k can be selected by choosing the k most relevant channels, thereby determining the subset of the first feature map.

[0015] According to an advantageous embodiment, the subset of the first feature map, after processing by the second convolutional layer, is multiplied by the weight vector w of the channels in the second set of channels of number k. The weight vector w is derived from the preceding adaptive calculation of the relevance of the individual channels. Multiplying the channels of the processed subset of the first feature map by the weight vector allows for improved channel selection during training by providing feedback of the gradients of the processed subset of the first feature map to the channel selection. This enables the channel selection to be optimized based on the relevance of the channels during training.

[0016] To provide a second feature map, the second set of channels, number k of which are part of the processed subset of the first feature map, is added to the channels of number C in the first feature map. For example, it is possible to add the channels of number k from the processed subset to the respective channels at their original positions in the first feature map, or to replace the channels at their original positions in the first feature map with the channels of number k from the processed subset.

[0017] According to an advantageous embodiment, generating the second feature map involves adding the second set of channels (number k) from the processed subset of the first feature map to a specific group of channels from the first set (number C). Thus, for channel integration, it is proposed that the channels (number k) from the processed subset are added to a predetermined group of channels from the first set (number C). This ensures that the processed channels (number k) from the processed subset of the first feature map remain at the same position in the generated second feature map throughout the entire operation of the convolutional neural network. The generated second feature map comprises the dimensions of the first feature map, namely C × H × W.

[0018] According to one embodiment, the second feature map is processed by means of a third convolutional layer of the convolutional neural network. The construction of the third convolutional layer can be analogous to that of the first convolutional layer. Using the third convolutional layer, a third feature map, also with dimensions C × H × W, can again be provided, analogous to the first convolutional layer.

[0019] Following the third convolutional layer, a fourth convolutional layer can be provided, analogous to the second convolutional layer, and structured analogously to the second layer, thus processing a subset of the third feature map. The subset of the third feature map can be determined analogously to the subset of the first feature map.

[0020] It is therefore possible for the convolutional neural network to have a layered structure, alternating between a convolutional layer that processes a feature map with dimensions C × H × W and a convolutional layer that processes a subset of the preceding feature map with dimensions k × H × W, where k < C. Between the two layers, channel selection is performed to determine the subset of the preceding feature map, or channel integration is performed to provide the subsequent feature map.

[0021] Alternatively or additionally, a different layer structure can be implemented in the convolutional neural network. For example, a layer structure with one or more of the following features can be implemented: - alternating arrangement of one or more convolutional layers processing a feature map with dimensions C x H x W and one or more convolutional layers processing a subset of the first or the preceding feature map with dimensions k × H × W, where k < C. - after the first and second layers, each further convolutional layer is designed in such a way that it processes a subset of the first or a further feature map with dimensions k × H × W, where k < C. - Two or more convolutional layers, processing a subset of the first or preceding feature map with dimensions k × H × W, are arranged sequentially, followed by one or more convolutional layers processing a feature map with dimensions C × H × W. - Two or more convolutional layers processing a feature map with dimensions C × H × W are arranged consecutively, with one or more convolutional layers subsequently processing a subset of the first or the preceding feature map with dimensions k × H × W.

[0022] One embodiment relates to a convolutional neural network comprising at least a first convolutional layer configured to provide a first feature map comprising a first set of channels of number C, wherein a second convolutional layer is provided after the first convolutional layer, wherein the second convolutional layer is configured to process a subset of the first feature map, wherein the subset of the first feature map comprises a second set of channels of number k, where k < C.

[0023] The convolutional neural network includes a channel selector. The channel selector is designed to determine a selection of channels k from the second set by compressing the first feature map and processing the compressed first feature map using a two-layer neural subnetwork.

[0024] The number k of channels in the subset of the first feature map is determined such that it matches the number of channels in the kernels of the second convolutional layer, or conversely, the number of channels in the kernels of the second convolutional layer is matched to the number k of channels in the subset. The second convolutional layer therefore comprises k channels in the kernels. For comparison, the first convolutional layer comprises C channels in the kernels.

[0025] The convolutional neural network further includes a channel integrator. This channel integrator is configured to provide a second feature map by adding the second set of channels of number k from the processed subset of the first feature map to the channels of number C from the first feature map, where the second feature map comprises a first set of channels of number C. Following the second convolutional layer, a third convolutional layer is provided, which is configured to process the second feature map.

[0026] Following the third convolutional layer, a fourth convolutional layer can be provided, analogous to the second convolutional layer, which is structured analogously to the second layer and therefore processes a subset of the third feature map.

[0027] The convolutional neural network can therefore be designed to have a layered structure, alternating between a convolutional layer that processes a feature map with dimensions C × H × W and a convolutional layer that processes a subset of the preceding feature map with dimensions k × H × W, where k < C. A channel selector or a channel integrator is provided between the layers.

[0028] Alternatively or additionally, a different layer structure can be implemented in the convolutional neural network. For example, a layer structure with one or more of the following features can be implemented: - alternating arrangement of one or more convolutional layers processing a feature map with dimensions C × H × W and one or more convolutional layers processing a subset of the first or the preceding feature map with dimensions k × H × W, where k < C. - after the first and second layers, each further convolutional layer is designed in such a way that it processes a subset of the first or a further feature map with dimensions k × H × W, where k < C. - Two or more convolutional layers, processing a subset of the first or preceding feature map with dimensions k × H × W, are arranged sequentially, followed by one or more convolutional layers processing a feature map with dimensions C × H × W. - Two or more convolutional layers processing a feature map with dimensions C × H × W are arranged consecutively, with one or more convolutional layers subsequently processing a subset of the first or the preceding feature map with dimensions k × H × W.

[0029] Further embodiments relate to a method for training a convolutional neural network according to the described embodiments in a method for operating the convolutional neural network according to the described embodiments.

[0030] Advantageously, the method for training the convolutional neural network involves multiplying the subset of the first feature map, after processing by the second convolutional layer, by the weight vector w of the channels in the second set of k channels. The weight vector w is derived from the preceding adaptive calculation of the relevance of each channel. Multiplying the channels of the processed subset of the first feature map by the weight vector allows for improved channel selection during training by providing feedback of the gradients of the processed subset of the first feature map to the channel selection process. This enables the optimization of channel selection based on channel relevance during training.

[0031] Further embodiments relate to the use of a convolutional neural network according to the described embodiments or a convolutional neural network trained according to the described training method to solve an intended task according to a method according to the described embodiments, wherein the intended task includes at least one task from the field of computer vision or audio signal processing, for example image recognition, object recognition, image segmentation or audio recognition.

[0032] A device for processing digital data, in particular digital sensor data, especially for machine learning, classification, or controlling a computer-controlled machine, comprises a processor and a memory for at least one artificial neural network configured to execute the method. This device can be prepared independently of the specific application and subsequently trained depending on the specific application.

[0033] The digital sensor data includes, for example, digital images such as video data, radar, LiDAR, ultrasound, motion, thermal images, or audio signals.

[0034] Convolutional neural networks can be used to classify sensor data, detect the presence of objects within the data, or perform semantic segmentation. This is applied, for example, in environmental perception and / or recognition, particularly in the fields of automated or (semi-)autonomous driving and robotics, where data is analyzed specifically with regard to traffic signs, ground conditions, road surfaces, pedestrians, vehicles, and animals. Another application is the analysis of image data in the medical field.

[0035] The convolutional neural network can be used to control a technical system, in particular to calculate a control signal for that system. The technical system could be, for example, a computer-controlled machine, such as a robotic system, a vehicle, a household appliance, a power tool, a manufacturing machine, or an access control system, or a system for transmitting information, such as a surveillance system or a medical system, especially an imaging system. The control is based on the analysis of sensor data, particularly scalar time-series data, and on controlling the technical system based on this analysis. The analysis can also include regression analysis to determine a continuous value, for example, regarding distance, velocity, or acceleration, or to track an element, such as an object, within the data.

[0036] The convolutional neural network can also be used to detect anomalies in a technical system. During training, the convolutional neural network can be trained to recognize features of the technical system and store these features during inference.

[0037] Further advantages will become apparent from the description and the accompanying drawings. Exemplary embodiments of the invention are shown in the drawings and are explained in more detail in the following description. Here, the same reference numerals in different figures denote identical or at least functionally comparable elements. When describing individual figures, reference is also made, where appropriate, to elements from other figures. The drawing shows: Fig. 1 a schematic representation of a section of a convolutional neural network; Fig. 2 a schematic representation of another section of the convolutional neural network from Fig. 1;

[0038] Fig. Figure 1 shows a schematic representation of a section of a convolutional neural network 10, hereinafter also referred to as neural network 10.

[0039] In the example, the neural network 10 comprises a first convolutional layer 12 and a second convolutional layer 14, whereby possible further layers, in particular further convolutional layers, are not shown.

[0040] The first convolutional layer 12 processes an input E from the neural network 10 and provides the first feature map X based on the input E. In this example, the first feature map X comprises the dimensions C × H × W, where C is the number of channels in the first set of the first feature map.

[0041] Following the first convolutional layer 12, a channel selector 16 is provided. The channel selector 16 determines a subset X̂ of the first feature map based on a selection of a second set of channels of the number k from the first set of channels of the number of the first feature map, where k < C, wherein the selection of the second set of channels of the number k is based on a relevance of the channels with respect to the intended task.

[0042] The number k of channels k to be processed in the second convolutional layer 14 is determined by the formula k = C / c, where C is the number of channels in the first set of the first feature map and c is a hyperparameter that determines the extent of the channel reduction. In the example, the number C, with C=16, of channels is reduced from 16 to k, with k=4.

[0043] The second convolutional layer 14 no longer processes the entire first feature map X, but only the subset X̂. In this example, the subset X̂ comprises the dimensions k × H × W, where k is the number of selected channels of the second set and H and W describe spatial dimensions of the feature map.

[0044] This selection helps to effectively reduce and concentrate computing resources by processing only the k channels that are highly relevant to the intended task in the second convolutional layer 14. The second convolutional layer 14 can, for example, be designed based on known architectures and is configured such that processing the subset X̂ improves the properties of the selected k channels.

[0045] Following the second convolutional layer 14, a channel integrator 18 is provided. The channel integrator, also called channel fusion, combines processed channels of the processed subset X̂' with unprocessed channels of the first feature map X while maintaining the dimensionality required for subsequent convolution operations.

[0046] The channel integrator 18 is configured to provide a second feature map X̅ by adding the second set of channels of number k from the processed subset X̂' of the first feature map X to the channels of number C from the first feature map X, where the second feature map X̅ comprises the first set of channels of number C. The second feature map X̅ therefore also includes, for example, the dimensions C × H × W.

[0047] In this example, channel integration is performed by adding the k channels of the processed subset X̂' to a predetermined group of channels from the first set of channels C. This ensures that the k processed channels of the processed subset X̂' of the first feature map X are located at the same position in the second feature map X̅ throughout the entire process of operating the neural network 10. The second feature map X̅ thus generated comprises the dimensions of the first feature map X, namely C × H × W.

[0048] The number k of channels in the subset X̂ of the first feature map is determined such that it is adapted to a number of channels in the kernels of the second convolutional layer 14, or conversely, the number of channels in the kernels of the second convolutional layer 14 is adapted to the number k of channels in the subset X̂. The second convolutional layer 14 therefore comprises k channels in the kernels. For comparison, the first convolutional layer 12 comprises C channels in the kernels.

[0049] According to one embodiment, the second feature map X̅ is processed by means of a third convolutional layer (not shown) of the neural network 10. The construction of the third convolutional layer can be analogous to the first convolutional layer 12. Using the third convolutional layer, a third feature map, also with dimensions C × H × W, can again be provided analogously to the first convolutional layer 12.

[0050] Following the third convolutional layer, a fourth convolutional layer can be provided, analogous to the second convolutional layer 14, which is structured analogously to the second layer 14 and therefore processes a subset of the third feature map. The subset of the third feature map can be determined analogously to the subset of the first feature map.

[0051] It can therefore be provided that the convolutional neural network 10 comprises a layered structure, with alternating convolutional layers processing a feature map with dimensions C × H × W and convolutional layers processing a subset of the preceding feature map with dimensions k × H × W, where k < C. Between the two layers, channel selection is performed to determine the subset of the preceding feature map, or channel integration is performed to provide the subsequent feature map.

[0052] Alternatively or additionally, a different layer structure can be implemented in the convolutional neural network. For example, a layer structure with one or more of the following features can be implemented: - alternating arrangement of one or more convolutional layers processing a feature map with dimensions C × H × W and one or more convolutional layers processing a subset of the first or the preceding feature map with dimensions k × H × W, where k < C. - after the first and second layers, each further convolutional layer is designed in such a way that it processes a subset of the first or a further feature map with dimensions k × H × W, where k < C. - Two or more convolutional layers, processing a subset of the first or preceding feature map with dimensions k × H × W, are arranged sequentially, followed by one or more convolutional layers processing a feature map with dimensions C × H × W. - Two or more convolutional layers processing a feature map with dimensions C × H × W are arranged consecutively, with one or more convolutional layers subsequently processing a subset of the first or the preceding feature map with dimensions k × H × W.

[0053] Based on Fig.Section 2 describes the channel selection in detail. Channel selection is performed, for example, by compressing the first feature map X and processing the compressed first feature map X_comp using a two-layer neural subnetwork. The compression includes, for example, reducing the dimensions by adaptive average pooling 20, so that the compressed first feature map X_comp has the dimensions C x 1 x 1. The first compressed feature map X_comp is processed by a two-layer, in particular fully connected, subnetwork 22, for example, comprising a bottleneck structure, with a reduction factor r, for example, r = 16. A weight vector w is generated by means of subsequent sigmoid activation, where the elements, i.e., the individual weights, of the weight vector w quantify the respective relevance of each channel.Based on the weights, the second set of channels of the number k can be selected by choosing the k most relevant channels, cf. 24, and thereby the subset X̂ of the first feature map X can be determined.

[0054] According to an advantageous embodiment, the subset X̂ of the first feature map is multiplied, after processing by the second convolutional layer 14, by the weight vector w of the channels of the second set of channels of number k. The weight vector w is derived from the preceding adaptive calculation of the relevance of the individual channels. Multiplying the channels of the processed subset X̂' of the first feature map by the weight vector allows for improved channel selection during training by providing feedback of the gradients of the processed subset X̂' of the first feature map to the channel selection. This enables the channel selection to be optimized based on the relevance of the channels during training. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., Zhang, L.: CvT: Introducing Convolutions to Vision Transformers. In: 2021 IEEE / CVF International Conference on Computer Vision (ICCV). pp. 22-31. IEEE Computer Society, Los Alamitos, CA, USA (oct 2021). https: / / doi.org / 10.1109 / ICCV48922.2021.00009, https: / / doi.ieeecomputersociety.orQ / 10.1109 / ICCV48922.2021.00009

[0010] Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A ConvNet for the 2020s. In: 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11966-11976 (2022). https: / / doi.org / 10.1109 / CVPR52688.2022.0116

[0010]

Claims

[1] Method for operating a convolutional neural network (10) for solving a designated task, comprising the following steps: - Providing a first feature map (X) by means of a first convolutional layer (12) of the convolutional neural network (10), wherein the first feature map (X) comprises a first set of channels of number C, - Determining a subset (X̂) of the first feature map (X) based on a selection of a second set of channels of number k from the first set of channels of number C of the first feature map (X), where k < C, wherein the selection of the second set of channels of number k is based on a relevance of the channels with respect to the intended task, - Processing the subset (X̂) of the first feature map (X) using a second convolutional layer (14) of the convolutional neural network (10), wherein the subset (X̂) of the first feature map (X) comprises the second set of channels of the number k, - Providing a second feature map (X̅) by adding the second set of channels of number k of the processed subset (X̂') of the first feature map (X) to the channels of number C of the first feature map (X), wherein the second feature map (X̅) comprises a first set of channels of number C. [2] The method of claim 1, wherein determining the subset (X̂) of the first feature map (X) comprises: Determining the relevance of channels of the first set of channels of number C with respect to the intended task based on a weight vector (w) and selecting the second set of channels of number k from the first set of channels of number C with the greatest relevance with respect to the intended task. [3] Method according to one of claims 1 or 2, wherein the subset (X̂) of the first feature map (X) is multiplied by the weight vector (w) of the channels of the second set of channels of number k after processing by means of the second convolutional layer (14). [4] Method according to one of the preceding claims, wherein the second set of channels of number k of the processed subset (X̂') of the first feature map (X) is added to a specific group of channels of the first set of channels of number C. [5] A method according to any of the preceding claims, comprising: Processing the second feature map () using a third convolutional layer of the convolutional neural network (10). [6] Convolutional neural network (10) comprising at least a first convolutional layer (12) and a second convolutional layer (14), wherein the convolutional neural network (10) is configured to perform a method according to any one of claims 1 to 5. [7] Convolutional neural network (10) according to claim 6, wherein the number k of channels of the subset (X̂) of the first feature map (X) is adapted to a number of kernels of the second convolutional layer (14). [8] Convolutional neural network (10) according to one of claims 6 or 7, wherein the convolutional neural network (10) is configured to provide a second feature map (X̅) by adding the second set of channels of number k of the processed subset (X̂') of the first feature map (X) to the channels of number C of the first feature map (X), wherein the second feature map (X̅) comprises a first set of channels of number C, and wherein a third convolutional layer is provided after the second convolutional layer (14), wherein the third convolutional layer is configured to process the second feature map (X̅). [9] Training a convolutional neural network (10) according to one of claims 6 to 8, in a method according to claims 1 to 5. [10] Using a convolutional neural network (10) according to any one of claims 6 to 8 or a convolutional neural network (10) trained according to the method of claim 9 to solve a intended problem according to a method according to any one of claims 1 to 5, wherein the intended problem comprises at least one problem from the field of computer vision or audio signal processing, for example image recognition, object recognition, image segmentation or audio recognition.

Images