Apparatus and computer-implemented method for training an artificial neural network

Abstract

To provide a device and a computer implemented method which make a training procedure for an artificial neural network easier and more efficient.SOLUTION: A computer implemented method for training an artificial neural network includes the steps of: preparing a convolutional neural network layer; grouping channels of a plurality of output channels into a predetermined number of distinct groups of the neural network layer; preparing, for each of the output channels of each individual distinct group of the predetermined number of distinct groups, a distinct normalization function; determining output of an artificial neural network depending on a result of each of the distinct normalization functions; and training hyper-parameters in training of the artificial neural network.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The present invention relates to an apparatus and computer-implemented method for training an artificial neural network. [Background technology]

[0002] Prior art neural networks are known to be over-parameterized and require vast amounts of data for training. It is therefore desirable to make the training procedure simpler and more sample-efficient. Summary of the Invention [Means for solving the problem]

[0003] Disclosure of the Invention This is achieved by the device and method according to the independent claims.

[0004] A computer-implemented method for training an artificial neural network includes: preparing a neural network layer, particularly a convolutional neural network layer, for an equivariant feature map having a plurality of output channels; grouping the plurality of output channels of the neural network layer into a predetermined number of distinct groups, wherein the output channels of each distinct group are structured into a respective grid that defines the spatial location of each of the output channels of each distinct group within the grid for each distinct group; preparing a separate normalization function for each of the output channels of each distinct group of the predetermined number of distinct groups, wherein the separate normalization function is defined depending on the spatial location of the output channel within the grid in which the output channels are structured and depending on adjustable hyperparameters for the normalization function; determining an output of the artificial neural network depending on the result of each separate normalization function; and training the hyperparameters in training the artificial neural network. In this way, data symmetry is used to improve the effort of collecting training data, sample efficiency, and generalization accuracy of the training. Explicit knowledge or analytical expression of the transformations that define the data symmetry is not required. This is achieved by using the concept of maintaining a transformed copy of the learned filter. The transformed copy of the filter is automatically learned directly from the data. This is achieved by inducing an arbitrary topology for one or more convolutional filters in each layer of the artificial neural network, which essentially aligns the filters in a one-, two-, or N-dimensional grid, and further promoting filters that are closer to each other in this topology to be more correlated than filters that are further apart. This results in a smooth transformation of the filters, which leads to better generalization and improved isomorphism. The normalization function is a local normalization function applied to the output of each convolutional layer as a means of promoting correlation.The result is a topological feature mapping structure for learning equivariant representations, improving generalization. This method allows for training networks that are approximately equivariant to unknown transformations by learning them directly from the data, and that are approximately equivariant to transformations for which no analytical representation may exist, such as lighting changes.

[0005] Advantageously, the method comprises the steps of: preparing a dimension parameter, grouping the channels of a plurality of output channels of a neural network layer into a predetermined number of distinct groups each having a predetermined size, where the size is defined depending on the total number of output channels of the neural network layer, and structuring, for each group, the output channels of this group into a grid having a dimension according to the dimension parameter, whereby the dimension parameter is parameterized for training with groups of variable size.

[0006] Advantageously, the method comprises determining a rescaling parameter for each output channel and rescaling the output for the output channel after normalization in dependence on the rescaling parameter for this output channel, thereby introducing a further trainable parameter.

[0007] Advantageously, the method comprises preparing, for one output channel, a neighborhood function defining a set of output channels, wherein a normalization function for this output channel is defined depending on the output channels of the neural network layer in the set of output channels and ignoring other output channels, the neighborhood function providing a parameterized size of the set of neighboring output channels.

[0008] Advantageously, the set of output channels is defined by a distance parameter that defines the distance between the output channels of the neural network layer, and in the training, this distance parameter is trained, which is a particularly efficient parameter for training.

[0009] Advantageously, the set of output channels is defined by a spatial distance parameter that defines the distance between the output channels of the neural network layer in the grid, and this spatial distance parameter is trained during training. Preferably, the neighborhood function operates not only on the channel but also on the spatial extent, which enhances the regularization effect and improves generalization ability in situations where data is severely limited.

[0010] Advantageously, the method comprises preparing sensor data, in particular sensor data describing features of a digital image, and processing the sensor data and determining output signals relying on an artificial neural network thus trained in order to classify the sensor data, to detect, locate or segment objects in the sensor data and / or to detect anomalies in the sensor data. Thus, the topology of the grid is learned from the sensor data during training.

[0011] Advantageously, the method comprises operating a physical system in dependence on the output signal or generating input data for a generative adversarial neural network or a variational autoencoder for data synthesis. The layer may be formed by a normalization function, which may be incorporated into the generative adversarial neural network or variational autoencoder as a layer normalization.

[0012] Advantageously, the artificial neural network is configured to segment, classify or detect pedestrians, road signs and / or vehicles in digital images, the method comprising: collecting a set of digital images from a database; applying one or more transformations to each digital image, including reflection, rotation, smoothing or contrast reduction, to generate a modified set of digital images; generating a first training set comprising the collected set of digital images, the modified set of digital images and a set of digital images not relating to pedestrians, road signs and / or vehicles; training the artificial neural network in a first stage using the first training set; generating a second training set for a second training stage comprising the first training set and digital images that were erroneously detected after the first training stage as images depicting pedestrians, road signs and / or vehicles; and training the neural network in the second stage using the second training set.

[0013] An apparatus for processing data has an input for data, an output for an output signal, at least one processor and a memory for an artificial neural network configured to perform the method described above.

[0014] Further advantageous embodiments can be derived from the following description and drawings. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 shows a schematic diagram of an apparatus for processing data. [Figure 2] FIG. 1 illustrates steps in a method for an artificial neural network. [Figure 3] FIG. 1 illustrates an example of a filter for processing data. DETAILED DESCRIPTION OF THE INVENTION

[0016] 1 shows an embodiment of an apparatus 100 for processing data. In this embodiment, the data is sensor data. Specifically, in this embodiment, a digital image represented by the data is processed.

[0017] The device 100 has at least one processor 102 and memory 104 for an artificial neural network 106. The device 100 has an input 108 for data and an output 110 for an output signal. These elements of the device 100 interact via a data connection 112.

[0018] In general, data from video, radar, LiDAR, ultrasonic sensors, thermal imaging sensors or motion sensors can be processed.

[0019] The sensor data is processed relying on an artificial neural network 106 to determine an output signal. The sensor data can be processed to classify the sensor data, to detect, locate or segment objects in the sensor data, and / or to detect anomalies in the sensor data. The output signal can be used to operate a physical system depending on the output signal, or to generate input data for a generative adversarial network (GAN) or a variational autoencoder (VAE) for data synthesis purposes.

[0020] In this embodiment, the artificial neural network 106 comprises a convolutional neural network. More generally, the methods described below apply to artificial neural networks 106 having neural network layers for equivariant feature mapping f:X→Z, in particular convolutional neural network layers.

[0021] This method will be described below with reference to FIG.

[0022] The method includes, in step 202, generating a plurality of output channels z h,w,g,dIn this embodiment, a convolutional neural network layer is prepared. For example, f:X→Z is a convolutional neural network layer. Here,

number

[0023] Step 202 includes providing a dimension parameter M and a plurality of output channels z of the neural network layer. h,w,g,d The channels are each of size

number

[0024] Step 202 is to determine for each group g ∈ G its output channel z h,w,g,d to be a grid having one dimension according to a dimension parameter M.

[0025] The method includes, in step 204, selecting a plurality of output channels z of a neural network layer. h,w,g,d The method includes grouping the channels g into G distinct groups g.

[0026] Each distinct group g has its own output channel z h,w,g,d is structured to be a distinct M-dimensional grid, which allows for the distribution of output channels z for each distinct group g within the grid. h,w,g,d Each spatial location h,w is defined.

[0027] The method includes, in step 206, generating an output channel z for each of the G distinct groups g. h,w,g,d A separate normalization function for each

number

[0028] This output channel z h,w,g,d The normalization function LS(z h,w,g,d ) is the output channel z h,w,g,d The output channels z of the neural network layer in the set h,w,g,d is defined depending on and ignoring other output channels.

[0029] In this example, output channel z h,w,g,d A neighborhood function N that defines a set of k (d) is prepared.

[0030] Output channel z h,w,g,d The set of output channels z h,w,g,d The distance parameter k defines the distance between the neighbor functions N k In one embodiment, each output channel is identified by an index, and the distance is defined as the numerical distance between the indices of two output channels.

[0031] In one embodiment, for a grid with d∈D for each output channel and dimensionality parameter M=1, i.e., a one-dimensional grid, if the neighborhood size for one output channel is d=10 and k=2, then the neighborhood function is N k (d)={8,9,10,11,12}.

[0032] In another embodiment, the spatial neighborhood function N l,m (h,w) can be specified. Output channel z h,w,g,d Let the set of output channels z of the neural network layer in the grid h,w,g,d It can be defined by the spatial distance parameters l and m that define the distance between them.

[0033] Any boundary conditions on the neighborhood function are a modeling choice, and one can choose either circular, zero-padding, or other common padding techniques.

[0034] In one embodiment, the following parameters may be used:

number

[0035] In general, a separate normalization function LS(z h,w,g,d ) is the output channel z h,w,g,d This output channel z in a grid where h,w,g,d It depends on the spatial location of,h,w,.

[0036] A separate normalization function LS(z h,w,g,d ) is, in this example, dependent on the spatial locations h, w and the normalization function LS(z h,w,g,d ) is specified depending on the adjustable hyperparameters for

[0037] This method involves the use of a separate normalization function LS(zh,w,g,d ) and includes a step 208 of determining the output of the artificial neural network 106 depending on the results of each.

[0038] The method includes a step 210 of training parameters and hyperparameters in training the artificial neural network 106. In particular, the distance parameter k and the spatial distance parameters l and m are trained. The artificial neural network 106 is trained in dependence on the output.

[0039] In this embodiment, step 202 is performed prior to training to determine the grid. Steps 204 through 208 are performed iteratively to process the training data in step 210. Parameters of the artificial neural network 106 are determined and updated during training, for example, by gradient descent to minimize a loss criterion.

[0040] After training is complete, step 212 is performed.

[0041] Step 212 involves preparing sensor data, particularly sensor data that describes the characteristics of the digital image.

[0042] Step 214 is then executed.

[0043] In step 214, the sensor data is processed relying on the thus trained artificial neural network 106 to determine an output signal, in this embodiment, to classify the sensor data, to detect, locate or segment objects in the sensor data, and / or to detect anomalies in the sensor data.

[0044] Step 216 is then executed.

[0045] In step 216, a physical system is operated depending on the output signal.

[0046] Alternatively or additionally, sensor data can be processed to generate input data for a generative adversarial neural network or a variational autoencoder for the purpose of data synthesis.

[0047] The method further comprises: h,w,g,d For each rescaling parameter η g,d and determine this output channel z h,w,g,d The rescaling parameter η for g,d After normalization depending on h,w,g,d In this case, the output of the artificial neural network 106 is scaled to the output channel z h,w,g,d Rescaled output for

number

number

[0048] The normalization function described above is a local squash function, which is used as a standard layer of a convolutional neural network and is placed after the convolution. If the convolutional neural network has any or a combination of batch normalization, pooling, activation function, and dropout layer, the local squash function is placed after the convolution, optional batch normalization, and optional pooling, and before the optional activation function and optional dropout layer.

[0049] The local squash function can be implemented as a layer in the artificial neural network 106. This layer can be located anywhere between two convolution operations and can be applied to all layers or only to desired layers. Preferably, this layer is located between the batch normalization layer and the activation function layer. Preferably, this local squash function is applied to more than a single layer of the artificial neural network 106.

[0050] In this way, a topology, or grid, is induced for the multiple output channels of the CNN layer described above. The artificial neural network 106 is encouraged to learn intermediate transformations between the existing filters. Local squash function layers are "intermediate" filters that increase the number of transformations that the artificial neural network 106 captures in its representation, and are therefore isovariant.

[0051] The desired induced topology is achieved by the normalization function described above. The artificial neural network 106 can have multiple CNN layers and median filter or local squash function layers.

[0052] Figure 3a shows the 17x17 pixel output of a filter with a local squash function on a one-dimensional grid. The resulting filter, which produces the output of Figure 3a, is structured by a representation of the original grid, which may be unknown in some cases. Figure 3b shows the 17x17 pixel output of a filter without a local squash function. Figure 3c shows the 17x17 pixel output of a filter with a well-defined one-dimensional grid and a smooth transition learned using a local squash function. The trained weights of the artificial neural network 106 in this case represent a topological structure structured in the learned grid of intermediate filters in each layer.

[0053] The artificial neural network 106 can be used in processing image data in vehicles or robots, or for medical devices or machines that process image data in order to determine what action to take by their actuators.

[0054] In particular, the artificial neural network 106 can be configured to segment, classify, or detect pedestrians, road signs, and / or vehicles in digital images.

[0055] The method may include collecting a set of digital images from a database. The method may include applying one or more transformations to each digital image, including reflection, rotation, smoothing, or contrast reduction, to generate a modified set of digital images. The method may include generating a first training set having the collected set of digital images, the modified set of digital images, and a set of digital images not related to pedestrians, road signs, and / or vehicles. The method may include training the artificial neural network 106 in a first stage using the first training set. The method may include generating a second training set for a second training stage, the second training set including the first training set and digital images erroneously detected as depicting pedestrians, road signs, and / or vehicles after the first training stage. The method may include training the artificial neural network 106 in a second stage using the second training set.

Claims

1. A computer-implemented method for training an artificial neural network (106), comprising: A neural network layer, in particular a convolutional neural network layer, for equivariant feature mapping having multiple output channels is provided; grouping (204) the channels of the plurality of output channels of the neural network layer into a predetermined number of distinct groups, wherein the output channels of each distinct group are structured into a distinct grid that defines a spatial location of each of the output channels of the distinct group within the grid for the distinct group; providing (206) a separate normalization function for each of the output channels of each respective one of the predetermined number of separate groups, wherein the separate normalization function is defined in dependence on the spatial location of the output channel within the grid on which the output channels are structured and in dependence on adjustable hyperparameters for the normalization function; determining (208) an output of the artificial neural network depending on the results of each of the separate normalization functions; training (210) the hyperparameters in training the artificial neural network (106); providing a neighborhood function for one output channel that defines a set of output channels, wherein the normalization function for the output channel is defined depending on the output channels of the neural network layer in the set of output channels and ignoring other output channels; A computer-implemented method for training an artificial neural network (106).

2. providing dimensional parameters (202); grouping (204) the channels of the plurality of output channels of the neural network layer into a predetermined number of separate groups each having a predetermined size, wherein the size is defined depending on the total number of output channels of the neural network layer; and for each group, structuring the output channels of that group into a grid having dimensions according to the dimensioning parameters. The method of claim 1.

3. Determine the rescaling parameters for each output channel, rescaling the output for the output channel after normalizing in dependence on the rescaling parameter for the output channel; 3. The method according to claim 1 or 2.

4. the set of output channels is defined by a distance parameter that defines a distance between the output channels of the neural network layer, and the distance parameter is trained in the training.

4. The method according to any one of claims 1 to 3.

5. the set of output channels is defined by a spatial distance parameter that defines a distance between the output channels of the neural network layer within the grid, and the training comprises training the spatial distance parameter.

5. The method of claim 4.

6. providing (212) sensor data, in particular sensor data describing features of a digital image; processing (214) the sensor data and determining an output signal relying on the trained artificial neural network (106) to classify the sensor data, to detect, locate or segment objects in the sensor data, and / or to detect anomalies in the sensor data; 6. The method according to any one of claims 1 to 5.

7. Operating a physical system in dependence on the output signals (216), or generating input data for a generative adversarial neural network or a variational autoencoder for data synthesis. The method of claim 6.

8. the artificial neural network (106) is configured to segment, classify, or detect pedestrians, road signs, and / or vehicles in a digital image; The method comprises: collecting a set of digital images from a database; applying one or more transformations to each digital image, including mirroring, rotating, smoothing, or contrast reduction, to generate a set of modified digital images; generating a first training set comprising the collected set of digital images, the modified set of digital images, and a set of digital images not relating to pedestrians, road signs, and / or vehicles; training the artificial neural network (106) in a first stage using the first training set; generating a second training set for a second training stage comprising the first training set and digital images that are erroneously detected as depicting pedestrians, road signs, and / or vehicles after the first training stage; training the artificial neural network (106) in a second stage using the second training set; Including, 8. The method according to any one of claims 1 to 7.

9. An apparatus (100) for processing data, comprising at least one processor (102) and a memory (104) having an artificial neural network (106) stored therein, The at least one processor (102) A neural network layer, in particular a convolutional neural network layer, for equivariant feature mapping having multiple output channels is provided; grouping (204) the channels of the plurality of output channels of the neural network layer into a predetermined number of distinct groups, wherein the output channels of each distinct group are structured into a distinct grid that defines a spatial location of each of the output channels of the distinct group within the grid for the distinct group; providing (206) a separate normalization function for each of the output channels of each respective one of the predetermined number of separate groups, wherein the separate normalization function is defined in dependence on the spatial location of the output channel within the grid on which the output channels are structured and in dependence on adjustable hyperparameters for the normalization function; determining (208) an output of the artificial neural network depending on the results of each of the separate normalization functions; training (210) the hyperparameters in training the artificial neural network (106); providing, for one output channel, a neighborhood function defining a set of output channels, wherein the normalization function for the output channel is defined depending on the output channels of the neural network layer in the set of output channels and ignoring other output channels; The present invention is characterized in that the present invention is configured as follows: An apparatus (100) for processing data.

10. A computer program comprising machine-readable instructions that, when executed by a computer, cause the computer to carry out the method of any one of claims 1 to 8.

11. A computer-readable storage medium having the computer program according to claim 10 stored thereon.

Images