Method and device for processing sensor data by means of a neural convolutional network

Abstract

Methods and devices for processing sensor data by means of a neural convolutional network. The computer-implemented method comprises processing sensor data by a plurality of successive layers of a convolutional network, wherein the convolutional network has a convolutional filter layer receiving at least one input matrix having input data values, a first filter matrix being given by a sum of filter basis functions weighted by first weights, at least one second weight being calculated from the first weights by applying a transformation formula parameterized by a transformation parameter to the first weights for a respective value of the transformation parameter, for each second weight a respective second filter matrix being determined by calculating a sum of filter basis functions weighted by the second weight, and convolving the input matrix with the first filter matrix and each second filter matrix, thereby producing for each filter matrix an output matrix having output data values, and an aggregation layer merging the output matrices.

Description

Technical Field

[0001] Various embodiments generally relate to devices and methods for processing sensor data by means of neural convolutional networks. Background Technology

[0002] Neural networks now have a wide range of applications, such as recognizing objects in image data or controlling robots and autonomous vehicles. Due to their large number of parameters, neural networks can handle very complex datasets and are typically trained with the goal of making good predictions on future, unknown input data, such as classifying objects in an image as accurately as possible. A particularly successful type of neural network for this purpose is the convolutional neural network (CNN).

[0003] Recent advances in machine learning and machine vision have demonstrated that models, such as neural networks, can benefit from the introduction of prior knowledge. One possibility for this prior knowledge is the assumption that a particular transformation will not change the model's predictions. Corresponding to this prior knowledge, models can be designed to be equivalent to or invariant to the corresponding transformations. Summary of the Invention

[0004] According to an embodiment (Example 1), a computer-implemented method for processing sensor data using a convolutional network is provided. The method includes: processing the sensor data through a plurality of consecutive layers of the convolutional network, wherein the convolutional network has a convolutional filter layer that receives at least one input matrix having input data values; implementing a first filter matrix, the first filter matrix being given by a sum of filter basis functions weighted by a first weight; calculating at least one second weight from the first weight for a corresponding value of a transformation parameter by applying a transformation formula parameterized by the transformation parameter to the first weight; determining a corresponding second filter matrix for each second weight by calculating a sum of filter basis functions weighted by the second weight; and performing a convolution operation on the input matrix with the first filter matrix and each second filter matrix to generate an output matrix having output data values ​​for each filter matrix, wherein the convolutional network also has an aggregation layer that combines the output matrices.

[0005] The method enables convolutional networks to be trained and function equivalently or invariantly with respect to parameter transformations, wherein the transformations can be estimated from a given set of training data. This allows overfitting to be avoided or reduces the amount of data required for training, since not all variables need to be included in the training data.

[0006] Example 2 is based on the method of Example 1, wherein the transformation parameter is an angle, and the convolutional filter layer calculates the second weight by applying the transformation formula, such that the second filter matrix is ​​a first filter matrix rotated by the angle.

[0007] This allows convolutional networks to be invariant or equivalent to rotations (e.g., rotations of the objects to be classified).

[0008] Example 3 is based on the method of Example 1, wherein the transformation parameter is a scaling parameter, and the convolutional filter layer calculates the second weight by applying the transformation formula, such that the second filter matrix is ​​a scaling of the first filter matrix, wherein the strength of the scaling is given by the scaling parameter.

[0009] This allows convolutional networks to remain invariant or equivalent to size changes (e.g., changes in the size of the object to be classified), such size changes as those caused by the distance of the object.

[0010] Example 4 is a method according to one of Examples 1 to 3, wherein the aggregation layer determines the corresponding value of a predefined evaluation variable for each output matrix and combines these output matrices by outputting the identifier of the output matrix that maximizes the evaluation variable.

[0011] Thus, the network identifies which transformation parameters (e.g., which rotation angle) best correspond to the input matrix of the convolutional filter layer. For example, the convolutional network identifies the orientation or distance at which objects exist in the image. Layers following the convolutional filter layer can then use this information for regression or classification.

[0012] Example 5 is a method according to any one of Examples 1 to 4, comprising training the convolutional network by comparing the values ​​predicted by the convolutional network for the training data with predetermined reference values ​​for the training data, wherein the coefficients of the first weights and / or the coefficients of the transformation formula are trained.

[0013] Training the weights and / or coefficients allows the convolutional network to adapt to transformations that may occur in the sensor data.

[0014] Example 6 is a method according to one of Examples 1 to 5, including controlling the actuator based on the output of the convolutional network.

[0015] Example 7 is a convolutional network having multiple consecutive layers, the multiple consecutive layers having convolutional filter layers and aggregation layers, wherein the convolutional network is designed to perform the method according to one of Examples 1 to 6.

[0016] Example 8 is a software or hardware agent, particularly a robot, having sensors designed to provide sensor data and a convolutional network according to Example 7, wherein the convolutional network is designed to perform regression or classification of the sensor data.

[0017] Example 9 is a software or hardware agent according to Example 8, which has an actuator and a control device, the control device being designed to use the output of a convolutional network to control at least one actuator.

[0018] Example 10 is a computer program having program instructions designed to perform a method according to one of Examples 1 to 6 when executed by one or more processors.

[0019] Example 11 is a machine-readable storage medium on which program instructions are stored, the program instructions being designed to perform a method according to one of Examples 1 to 6 when executed by one or more processors. Attached Figure Description

[0020] Embodiments of the invention are illustrated in the accompanying drawings, and these embodiments are explained in more detail below. In the drawings, the same reference numerals generally refer to the same parts in multiple views. The drawings are not necessarily drawn to scale, but generally focus on illustrating the principles of the invention.

[0021] Figure 1 An example of object recognition in autonomous driving is shown.

[0022] Figure 2 An example of a neural network is shown.

[0023] Figure 3 An example of a neural convolutional network is shown.

[0024] Figure 4 This demonstrates how to apply a convolutional filter to two-dimensional input data.

[0025] Figure 5 An example of a filter function is provided.

[0026] Figure 6 An example of rotation for the filter function with different rotation angles is shown.

[0027] Figure 7 A flowchart is shown illustrating a method for processing sensor data using convolutional networks. Detailed Implementation

[0028] In machine learning, a function is learned that maps input data to output data. During learning (e.g., training the neural network or other model), this function is determined from a set of input data (also called the training data set), which predetermines the desired output (e.g., the desired classification of the input data) for each input, such that the function assigns the inputs to the outputs in the best possible way.

[0029] An example application of this machine learning capability is object classification for autonomous driving, such as... Figure 1 As shown.

[0030] It should be noted that, in the following text, an image or image data is very generally understood as a collection of data representing one or more objects or patterns. The image data may be provided by a sensor that measures visible or invisible light (e.g., infrared or ultraviolet light, ultrasonic or radar waves, or other electromagnetic or acoustic signals).

[0031] exist Figure 1 In the example, vehicle 101 (e.g., a passenger car or a truck) is equipped with vehicle control device 102.

[0032] The vehicle control device 102 has data processing components, such as a processor (e.g., a CPU (central processing unit)) 103 and a memory 104 for storing control software and data. The vehicle control device 102 operates according to the control software, and the data is processed by the processor 103.

[0033] For example, the stored control software has (computer program) instructions that, when executed by the processor, cause the processor 103 to implement one or more neural networks 107.

[0034] The data stored in memory 104 may include, for example, image data detected by one or more cameras 105. The one or more cameras 105 may, for example, take one or more grayscale or color photographs of the environment surrounding vehicle 101.

[0035] The vehicle control device 102 can determine the presence and types of objects based on the image data, such as stationary objects like traffic signs or road markings, or moving objects like pedestrians, animals, and other vehicles in the surrounding environment of the vehicle 101.

[0036] Then, the vehicle control unit 102 can control the vehicle 101 based on the result of the object determination. Thus, the vehicle control unit 102 can, for example, control the actuator 106 (e.g., the brake) to control the speed of the vehicle, for example, to brake the vehicle.

[0037] According to one implementation, in Figure 1In the example, the control is based on image classification performed by a neural network.

[0038] Figure 2 An example of a neural network 200 is shown, which is designed to map input data to output data. For example, the neural network may be designed to classify an image into a predetermined number of categories.

[0039] In this example, the neural network 200 includes an input layer 201, multiple "hidden" layers 202, and an output layer 203.

[0040] It should be noted that neural network 200 is a simplified example of an actual neural network, which may contain more processing nodes and hidden layers.

[0041] The input data corresponds to input layer 201 and can generally be viewed as a multidimensional arrangement of values; for example, an input image can be viewed as a two-dimensional arrangement of values ​​corresponding to image pixel values. The input of input layer 201 is connected to processing node 204.

[0042] If layer 202 is a "fully connected" layer, then processing node 204 multiplies each input value of the layer's input data by a weight and sums the calculated values. Additionally, node 204 may add a bias to this sum. Alternatively, in the case of a fully connected layer, processing node 204 is followed by a non-linear activation function 205, such as a ReLU unit (f(x) = max(0, x)) or a sigmoid function (f(x) = 1 / (1 + exp(-x))). The resulting value is then output to the next layer.

[0043] However, at least a portion of layer 202 may also have layers that are not fully connected, such as convolutional layers or pooling layers in the case of convolutional neural networks.

[0044] Output layer 203 receives values ​​from the last layer 202 (of the sequence of layers 202). Typically, output layer 203 processes these received values ​​and then outputs them for further processing. For example, in the case of using the neural network for image classification, output layer 203 converts the received values ​​into probabilities, where these probabilities indicate that the image corresponds to one of predefined categories. The category with the highest probability can then be output by output layer 203 as the predicted category for further processing. To train neural network 200, it can be determined, on training data with known category assignments, whether the category predicted by neural network 200 is consistent with the known category, or whether the probability that the neural network has already determined for the known category is evaluated, typically using a loss function. A similar process can be performed when training neural network 200 for segmentation or regression, etc.

[0045] It should be noted that image classification can be viewed as equivalent to the classification of objects displayed in the image. If the original image contains multiple objects, such as in autonomous driving, segmentation can be performed (possibly via a separate neural network) so that each segment represents one object, and these segments are used as input to a neural network that classifies the image.

[0046] A convolutional neural network (CNN) is a special type of neural network that is particularly well-suited for analyzing and classifying image data.

[0047] Figure 3 An example of a neural convolutional network 300 is shown.

[0048] It should be noted that, Figure 3 The diagram only shows the input or output data of each layer of the neural network 300, and these layers are represented only by dashed lines. These layers may have the following characteristics as shown in the reference diagram. Figure 2 The shape described.

[0049] The input data corresponds to input layer 301. For example, the input data is an RGB image, which can be viewed as three two-dimensional matrices (these matrices correspond to the pixel values ​​of the image). These three matrices can also be viewed as a single three-dimensional field, also called a tensor. A tensor can be viewed as an n-dimensional field (array), or it can be understood as a generalization of matrices; for example, numbers are zero-dimensional tensors, vectors are one-dimensional tensors, matrices are two-dimensional tensors, a cube of numbers is a three-dimensional tensor, a vector of cubes is a four-dimensional tensor, a matrix of cubes is a five-dimensional tensor, and so on.

[0050] Neural convolutional networks typically use three-dimensional and four-dimensional tensors. For example, multiple RGB images can be understood as a four-dimensional tensor (number of images × number of channels (e.g., three) × height × width).

[0051] The input data is processed by a first convolutional layer 302. In the convolutional layer, the input data is modified by a convolutional filter, which can be viewed as a (two-dimensional or three-dimensional) arrangement of values.

[0052] The convolutional filter takes a subset of the input data and convolves that subset. The convolutional filter can be interpreted as representing possible features in the input data, such as a specific shape. The output of each convolutional filter is a feature map.

[0053] Typically, the convolutional filter moves "pixel-by-pixel" across the entire input data (of the layer 202 to which the convolutional filter belongs). For example, in the case of an RGB image as input data, the convolutional filter corresponds to a three-dimensional tensor (or three filter matrices "overlapping" each other) and moves across all "pixels" (elements) of the image. However, an alternative stride can also be chosen for the filter, such as a stride of 2, which considers only each second value.

[0054] Figure 4 The convolution filter is shown being applied to two-dimensional input data 401.

[0055] The input data 401 of the filter is shown as a two-dimensional matrix. Similarly, the output data 402 of the filter is shown as a two-dimensional matrix. In the case of multiple channels (e.g., RGB images), multiple such matrices can "overlap" (and form a tensor), but for simplicity, only one channel is considered.

[0056] By applying filter matrix 405 to input value 403 and its surrounding environment 406 (in Figure 4 The output value 404 for input value 403 is obtained by showing the components of the matrix other than input value 403 (entries are shown for these components). The size of the surrounding environment 406 is determined by filter matrix 405: the surrounding environment 406, together with input value 403, forms a submatrix of input data, which has the same size as filter matrix 405. Filter matrix 405 is applied to input value 405 and its surrounding environment 406 by forming an inner product with the submatrix (both matrices are understood as vectors). The result of the inner product is output value 404. Figure 4 The example shown yields the following values:

[0057] 1*1+2*3+1*0

[0058] +2*1+4*2+2*(-1)

[0059] +1*0+2*4+1*(-3)=20.

[0060] Therefore, for an image or feature map as input data, each entry in the filter matrix can be understood as a weighted sum of pixel values ​​in a submatrix of the input data. Thus, each entry in the filter matrix corresponds to a pixel position (relative to the center of the filter matrix, which is aligned with the current input value 405).

[0061] In this example, filter matrix 306 is a 3×3 matrix, but it can also have different sizes. By shifting the filter matrix over the input data 401, the entire output value of output data 402 can be clearly produced, so that output data 402 ultimately corresponds to the convolution of input data 401 with filter matrix 405.

[0062] Here, at the edges of the input data, the values ​​for all the values ​​of the filter are not necessarily present in the input, for example, at the edges of a matrix, where a 3x3 filter is moved on the matrix.

[0063] One possibility for edge processing is to move the filter only when it is completely within the input, but this can reduce the output dimension (i.e., the dimension of the matrix corresponding to the output data) compared to the input dimension (i.e., the dimension of the matrix corresponding to the input data). For example, moving a 3x3 filter on the matrix reduces the output dimension by 2x2 compared to the input dimension.

[0064] To avoid reducing the output dimensionality, this paper explores another edge processing possibility: expanding the input data through padding. Typically, this edge padding is done with zeros. For example, zeros are padded on all four sides of the input data matrix 401 (i.e., bounding), so that the output of the 3x3 filter (which moves across the bounding matrix) has the same dimension as the original input matrix. Padding can also be performed such that the output dimension is greater than the input dimension.

[0065] In convolutional layer 302, a nonlinear activation function is typically applied after the convolutional filter. Figure 3 (not shown in the image), for example, the ReLU unit.

[0066] The data is then passed to pooling layer 303. In pooling layer 303, a filter also moves across the input data, typically outputting the maximum value or the average of multiple input values. Therefore, in... Figure 4 In the example, an average or maximum value is formed on the values ​​of the submatrix (with input value 403 in the middle) to produce the output value 404. Typically, the step size of this filter is greater than 1, for example, a step size of 2 or 3. For example, a 2x2 filter with a step size of 2 is moved across the input data matrix, providing the maximum value for every four input values. In other words, the pooling layer combines multiple input values, i.e., it aggregates multiple data values, which is why pooling layers are also called aggregation layers. The operation of the pooling layer can also be viewed as subsampling, which is why pooling layers are also called subsampling layers.

[0067] In other words, a pooling layer can be viewed as a form of non-linear downsampling, in which the amount of data is reduced by combining the outputs of multiple nodes in the next layer into a single node, for example by receiving the maximum value of the output.

[0068] Typically, there is no activation function in pooling layer 303, so this pooling can also be considered as part of a convolutional layer (or another layer) (typically, each layer of a neural network has an activation function).

[0069] After pooling layer 303 comes the second convolutional layer 304, and after the second convolutional layer 304 comes the second pooling layer 305.

[0070] It should be noted that there can be another convolutional layer after a convolutional layer, and further convolutional layers and / or pooling layers can be part of a convolutional neural network.

[0071] The input to the first convolutional layer 302 is, for example, a digital image, while the input to the subsequent convolutional layer 304 is a feature map output from the previous convolutional layer (or the previous pooling layer 303).

[0072] Following the second pooling layer 305 are one or more fully connected layers 306. Prior to this, the tensors obtained from the second pooling layer 305 are flattened into (one-dimensional) vectors.

[0073] The output layer receives data from the last fully connected layer 306 and outputs output data 307.

[0074] The output layer may include processing, such as converting values ​​into probabilities or probability vectors, for example, by using the Softmax function ( ), where v i (i=1,...,K are the received values) or a sigmoid function is applied to the values, and then the category with the highest probability is output for further processing.

[0075] It should be noted that neural convolutional networks do not necessarily have to have one (or more) fully connected layers (or must have them at the ends). Furthermore, neural convolutional networks can process multiple layers in parallel.

[0076] When training a neural network, it may be desirable to train the network such that the function it is learning is invariant with respect to a particular transformation. For example, the neural network should always recognize a dog as a dog, even if the dog is in a different location in the digital image (translation), smaller (scaling) (e.g., because the dog is far away), or if the dog is tilted in the image (rotation).

[0077] Methods for training neural networks to be invariant to such transformations (i.e., the neural network “learns invariants”) particularly involve imprinting the invariants in the training data, network architecture, or loss function, for example, by...

[0078] • The training data set is supplemented by generating additional training data using known transformations, and the neural network should remain invariant to the known transformations, such as translation, scaling, rotation, etc.

[0079] • Introduce an equivalent filter (e.g., in terms of transformation, such as rotating the filter by 90, 180, or 270 degrees) or use a Gaussian filter.

[0080] • Similar to the Tangent-Prob method, invariance is imprinted in the case of small transformations by adding a term to the loss function, where (local) invariance is achieved by adapting the loss function (or gradient step size) during training.

[0081] The following describes examples of the design and training of convolutional networks that result in certain transformations of the convolutional network being invariant or equivalent with respect to its input data.

[0082] For reference Figure 4 As explained, the filters of a convolutional network can be represented by a filter matrix of 405. In the following text, the set of real-valued invertible matrices is denoted by GL(n, R). Each closed subset of GL(n, R) is a Liezekiel set (matrix).

[0083] function Through formation The vector space V of the function is defined by a linear combination of the functions. The L elements of the Liskov set are represented as differential operators. a For functions Perform the calculation. If L a Applied to functions The result is in V, that is If L is an element of V, then L a The application described can be understood as matrix multiplication. In other words, the following control equation (hereinafter also called the transformation equation) holds:

[0084] .

[0085] Matrix B[a] is also called the control matrix. In the embodiments described below, L a The parameter 'a' represents the transformation of the convolutional filter, and the parameter 'a' that is used to parameterize this transformation can take different values.

[0086] A filter matrix 405 can be generated from a filter kernel (or convolution kernel) κ by inserting the pixel position (e.g., as pixel coordinates x, y relative to the center of the filter matrix) into the convolution kernel (e.g., κ(x, y)) for the pixel position corresponding to the entry of the filter matrix 405. However, the pixel position can also be scaled (e.g., it can be viewed as widening or narrowing the convolution kernel).

[0087] The transformation of the (spatial) convolution kernel using the Liskov group can be written as a linear combination. Therefore, it is assumed that the convolution kernel κ can be written as a basis function. A linear combination of . Therefore, it is a vector space. Elements:

[0088] .

[0089] If we take the element L of the Li group a When applied to the convolution kernel κ, the following equation holds:

[0090] .

[0091] In other words, this means that once a parameter value (e.g., rotation angle) is selected for the parameter a of the Lie group, the convolution kernel can be controlled accordingly (e.g., rotated) by multiplying it with the following vector of the basis function using a matrix corresponding to the parameter, and forming the vector from the basis function through a linear combination.

[0092] As an example, the filter should be rotated, for instance, to form the convolution kernel of filter matrix 405. In this case, the special orthogonal group SO(3) and its Lie algebra so(3) are considered as a Lie set. This Lie set has only one (infinitesimal) generator, which is generated by the differential operator. Given. As an example, suppose L operates on a vector space V of functions, spanned by the set of polynomials v1 = x², v2 = 2xy, v3 = y². Then the following holds:

[0093] (1)

[0094] From this, the following equation can be derived (from the exponential equations of the Li group):

[0095] B[ϕ]= (2)

[0096] Here, ϕ is the rotation angle and corresponds to the aforementioned parameter of transformation a.

[0097] In particular, based on the above equation, the following formula holds true:

[0098] (3)

[0099] Therefore, matrix B[ϕ] illustrates how the basis vectors are transformed by ϕ when the convolution kernel (i.e., the filter) is rotated. The vector.

[0100] If we choose a filter function f from the function space V: R 2 →R (i.e., the convolution kernel), then the weight vector w∈R is selected in the final effect. 3 And calculate the inner product:

[0101]

[0102] Therefore, a filter matrix can be generated by selecting weights w1, w2, w3 and inserting discrete grid values ​​(e.g., pixel positions scaled relative to the center of the filter matrix if necessary) as x and y in the convolution kernel.

[0103] For example, a 5x5 convolution matrix can be generated by setting x = -0.2, -0.1, 0, 0.1, 0.2 and y = -0.2, -0.1, 0, 0.1, 0.2, where the entries of the convolution matrix correspond to... f(x, y, w) 25 values ​​for 25 combinations of x and y.

[0104] Figure 5 An example of a filter function for a combination of w1, w2, and w3 is shown, where the base functions v1, v2, and v3 are also multiplied by... .

[0105] Such a filter function can be rotated by an angle ϕ using equation (3), where B[ϕ] comes from equation (2). To do this, one only needs to calculate cos2ϕ and sin2ϕ, multiply the matrices in (2) correspondingly and sum them, and apply (3) to the basis function vector.

[0106] Therefore, we can then derive the following formula: f(x, y, w) new value

[0107]

[0108] in

[0109] As a simple example, for = (1, 0, 0), meaning when rotated 45°... ,

[0110] B[ϕ]= Therefore, the rotating filter core

[0111] = .

[0112] Figure 6 The diagram shows the angles 0°, 45°, 90°, and 135° from left to right. The filter function.

[0113] According to various implementation methods, controllable and scalable convolutional layers of the neural network are provided, for example, as layer 202 of neural network 200.

[0114] Set a maximum degree for this layer. The set of functions consists of homogeneous polynomials ranging from degree 0 to the maximum degree. For example, if the maximum degree is 3, then these functions consist of 1, x, y, x... 2 ,xy,y 2 x 3 x 2 y, xy 2 y 3 Given.

[0115] Then, basis functions v1, ..., v are formed from these functions by dividing each function by a power of the scaling parameter s. L Here, the exponent equals the degree of the corresponding function. For the example above, we obtain the basis functions 1, x / s, y / s, x 2 / s 2 ,xy / s 2 y 2 / s 2 x 3 / s 3 x 2 y / s 3 , xy 2 / s 3 y 3 / s 3 .

[0116] Additionally, choose the scaling and grid size N. For each variable x, y, a corresponding centered grid is derived. For example, N=2 results in a spatial 5x5 (= (2N-1) x (2N-1)) filter (i.e., a 5x5 filter matrix).

[0117] For example, with N=2 and a scaling factor of 0.5, the following centered grid is obtained:

[0118] and .

[0119] Each basis function v1, ..., v is evaluated at points on a scaled grid given by these matrices. L This results in matrices V1, ..., V L .

[0120] For example, for the basis function xy / s with the above-mentioned centered grid for x and y and scaling parameter s=1. 2 The following matrix is ​​obtained:

[0121]

[0122] These matrices are stacked together to obtain a tensor V of dimensions Lx(2N+1)x(2N+1).

[0123] The basis function derived matrix K describes the differential operator L. a =L ϕ Operations on the basis function set are similar to those on the right-hand side of equation (1).

[0124] The control equation can be calculated using matrix K in a similar manner to (2).

[0125] Equation (2) is the basis function x 2 2xy and y 2 (Scaling parameter s=1), an example of the control equation for a homogeneous polynomial of degree 2. For a homogeneous polynomial of degree 3, the differential operator L... a For the set of polynomials v1=x 3 v2=x 2 y, v3=xy 2 v4=y 3 Perform the calculation.

[0126] The following equation holds true:

[0127] L a .

[0128] The following control equations can be derived from them:

[0129] (4)

[0130] Furthermore, trainable weight tensors W1, ... W are defined for this layer. M A set of.

[0131] Two embodiments are described below.

[0132] According to the first embodiment, a set of discrete angles and / or scaling ratios is selected, for example, ϕ∈{0, 10, 20, ..., 350}, s∈{1, 2, 3, 4, 5}.

[0133] For each angle and / or each scaling factor, along one or more axes, each weight tensor W1, ..., W MMultiply by V. This is, for example, the inner product between a vector of the weight matrix and a vector of the basis function matrix. If the basis function matrix is ​​a tensor (higher dimension), then the weight matrix is ​​also a tensor and correspondingly adapts to shrinkage (inner product), that is, shrinking multiple axes.

[0134] This provides a set of convolutional kernels for (controllable) layers.

[0135] This layer convolves its input data with each of the provided convolutional kernels (as shown in the reference). Figure 4 The result is pooled over an angle ϕ. This makes the layer invariant at ϕ and / or s.

[0136] The neural network is trained to perform, for example, classification or regression.

[0137] According to the second embodiment, after the neural network is trained for a task (e.g., classification or regression), all weights preceding this layer (i.e., all weights used to weight the input to this layer) are set, and the input data is passed through this layer. Additionally, an independent variable (e.g., angle) that takes the maximum value during pooling is output by this layer. This output can be used as input to a second network with trainable weights, which is trained for classification or regression. This independent variable can also be used elsewhere. For example, the size of an object (represented by the value of a scaling parameter) can be used to estimate the distance to the object.

[0138] In the above process, a transformation group (SO(2)) is predetermined, and then a filter is learned from the space of filter functions, which can be scale-invariant / rotation-invariant, depending on how the filter layers are further used. Since the transformation group is known, the control equation (for v1=x) can be directly formulated. 2 v2=2xy, v3=y 2 Equation (2) for the filter function space spanned by a homogeneous polynomial of degree 3, and equation (3) for the filter function space spanned by a homogeneous polynomial of degree 3.

[0139] An embodiment is described below, in which the neural network learns the set of transformations during training. Thus, a set of transformations adapted to the training data is learned. However, in this case, the control equation is unknown beforehand. For example, a convolutional layer 202 is configured to learn the control matrix as given by equation (2) during training. This allows equivalence or invariance to be imprinted in the neural network by selecting a general polynomial basis and a general set of parameters in the matrix equation and learning angles ϕ and / or rotating the convolutional kernels for a set of angles (where common weights are used), and then performing pooling (e.g., by means of maximization) (e.g., by arranging a pooling layer after the convolutional filter layer, which selects the value for each angle by maximization).

[0140] Therefore, as in the above embodiment, the filter matrices V1, ..., V that belong to the basis functions are determined. L And stack them into a tensor V, and define trainable weight tensors W1, ..., W M Furthermore, a set of trainable tensors C1, ..., Cn of size L x L is defined. k .

[0141] In the first embodiment, a set of discrete angles is selected, for example, ϕ∈{0, 10, 20, ..., 350}.

[0142] For trainable tensors C1, ..., C2 k The group, for each selected angle ϕ i Calculate the following tensors:

[0143] ,

[0144] (When k is odd, one term can be omitted accordingly when k is even.) Therefore, for example, in the case of k=9...

[0145]

[0146] For each angle, assign each weight tensor W1, ... W M Multiply by V along one or more axes. This provides a set of convolutional kernels for the (controllable) layers.

[0147] This layer convolves its input data with each of the provided convolutional kernels (as shown in the reference). Figure 4 (as described above). The result is pooled at angle ϕ. i Above. This makes the layer invariant under ϕ.

[0148] According to the second embodiment, a second trainable function (e.g., a second neural network) is provided, the input data is fed into the second trainable function, and the second trainable function outputs ϕ to the convolutional layer.

[0149] For trainable tensors C1, ..., C2 k For the group, calculate the following tensor for angle ϕ (for the example of k=9, see the formula above for smaller or larger k as well):

[0150] .

[0151] For each angle, assign each weight tensor W1, ... W M Multiply by V along one or more axes. This provides a set of convolutional kernels for the (controllable) layers.

[0152] Instead of pooling (such as max pooling, summation, sorting, etc.) to create invariance, create equivalent layers (or equivalent networks) that output parameters (such as the angle in the example above) whose associated convolutional kernels produce the maximum output value.

[0153] The neural network is trained to perform classification or regression.

[0154] In embodiments where a neural network is trained for classification or regression, the input data is, for example, a set of data points such as images or audio signals, and identifiers, such as categories or output parameters, are assigned to the input data for the training. The training data set can contain a large number of such input data sets.

[0155] In summary, various embodiments have provided, such as Figure 7 The method shown.

[0156] Figure 7 Figure 700 illustrates a method for processing sensor data using convolutional networks.

[0157] The sensor data is processed through multiple consecutive layers 701 to 704 of the convolutional network, wherein the convolutional network has a convolutional filter layer 702.

[0158] - Receive at least one input matrix with input data values;

[0159] - Implement (and, for example, determine) a first filter matrix, which is given by a sum of filter basis functions weighted with a first weight;

[0160] - For the corresponding values ​​of the transformation parameters, at least one second weight is calculated from the first weight by applying a transformation formula parameterized by the transformation parameters to the first weight;

[0161] - For each second weight, the corresponding second filter matrix is ​​determined by calculating the sum of the filter basis functions weighted by that second weight; and

[0162] - Convolve the input matrix with the first filter matrix and each of the second filter matrices to produce an output matrix with output data values ​​for each filter matrix.

[0163] The convolutional network also has an aggregation layer 703 that combines the output matrices.

[0164] In other words, according to various embodiments, convolutional layers are used in convolutional networks to apply parameterized filters for multiple parameter values. Subsequently, pooling can be performed on the output data at these parameter values. Therefore, the arrangement of convolutional layers and subsequent pooling layers is invariant to changes in the parameters (typically, the more different parameter values ​​used, the better).

[0165] "Through design," the convolutional network can be made invariant to transformations. Therefore, the convolutional network is also invariant to corresponding transformations (changes) (e.g., scaling and rotation) of the training data. This, in turn, allows the convolutional network to be trained with a smaller amount of training data, as there is no risk of the convolutional network overfitting to specific transformation parameter values ​​(e.g., overfitting to a specific orientation or scaling of an object) (i.e., avoiding "overfitting" with specific transformation parameter values). For example, in the case of classification networks, this avoids the convolutional network recognizing objects (e.g., cars) by the fact that the object always appears in a specific orientation in the training data.

[0166] Alternatively, the output data produced by the convolutional filter layer is combined over most parameter values ​​to create equivalence. To this end, in the case of maximizing (i.e., max pooling) the parameters, instead of (only) taking the output data with the maximum value, the parameter value with the maximum value is output to other processing instances, such as other neural networks or other layers. These other processing instances can then take this parameter value into account during further processing. For example, these other processing instances could consider rotation or scaling of a particular object and take this into account during classification.

[0167] According to one implementation, the transformation is controllable in that the convolutional filter can be transformed according to the value of a parameter, for example, by means of a control equation. This parameter can be, for example, an angle or a scaling parameter. Specifically, the scaling parameter in the convolutional layer can be used to scale the convolutional filter, and the output of the convolutional layer can be combined at different scaling parameter values. This combination (pooling) at scaling parameter values ​​keeps the layer scaling of the convolutional network invariant. Similarly, combining at angles keeps the layer angle of the convolutional network invariant.

[0168] Figure 7 This method enables a reduction in the number of parameters of the convolutional network. For example, robustness can be improved and the number of parameters of the convolutional network can be reduced by using a large number of possible parameter values ​​(e.g., a large number of scaling values, such as all scaling values ​​from a predetermined range) (e.g., compared to using convolutional filters with different scaling without subsequent pooling).

[0169] According to one implementation, the transformations are learned from training data during the training of the convolutional network. For example, a set of transformations is learned from the training data under which the training data remains invariant.

[0170] The convolutional network can be used to process any type of sensor data, such as video data, radar data, LiDAR (light detection and ranging) data, ultrasonic data, motion data, etc. The output of the convolutional network can be control data (or at least the basis of control data generated by a further processing system), such as for computer-controlled machines—like robots, vehicles, home appliances, power tools, machines for manufacturing products, personal assistants, or access control systems, or information transmission systems—like surveillance systems or medical (imaging) systems. According to one embodiment, the convolutional network is trained for such an application. The convolutional network can also be used to generate training data for training a second neural network.

[0171] Combining the output data generated by applying the convolutional filter layer across multiple parameter values ​​includes, for example, determining parameter values ​​that maximize the metric of the filter's output data. To apply the filter at the location of the input data, this involves, for example, determining parameter values ​​that maximize the filter's output value (i.e., max pooling). Then... Figure 4 In the example, and at an angle, i.e., a rotation angle in filter matrix 405 that makes output value 404 the maximum, the maximum output data (e.g., the maximum output value) is then output (e.g., for each position of the input data where the filter is applied as the filter moves across the input data). Other types of aggregation are also possible, such as averaging the output values ​​of the filter over parameter values ​​(e.g., angles). This can be performed independently for each position of the input data of this layer where the filter is applied, or it can be performed collectively (e.g., determining parameter values ​​and outputting associated output data for which the average output value is maximized at multiple or all positions of the input data where the filter is applied).

[0172] According to one embodiment, the selected output data itself is not further processed (i.e., no further processing is done on the maximum value), but only the parameter value determined by maximizing the output value is output to control other convolutional layers. However, this should be understood as aggregating the output data on the parameter values. Alternatively, the output data and the determined parameter values ​​can be output. Since the parameter values ​​are determined by comparing the output data, they are still considered as a combination of the output data (because the output data is obviously already combined in the parameter values ​​that take the maximum value, or the parameter values ​​are determined by combining the output data through comparison).

[0173] This method can be implemented using one or more circuits. In one implementation, "circuit" can be understood as any type of logic implementation entity, which can be hardware, software, firmware, or a combination thereof. Thus, in one implementation, "circuit" can be a hardwired logic circuit or a programmable logic circuit, such as a programmable processor, such as a microprocessor. "Circuit" can also be software implemented or executed by a processor, such as any type of computer program. Consistent with alternative implementations, any other type of implementation of the corresponding functionality, which will be described in more detail below, can be understood as a "circuit".

[0174] "Robot" can be understood as any physical system (with mechanical parts whose movement is controlled), such as computer-controlled machines, vehicles, household appliances, power tools, manufacturing machines, personal assistants, or access control systems.

[0175] The convolutional network can be used for regression or classification of data. Here, the term "classification" also includes semantic segmentation, such as semantic segmentation of images (which can be viewed as pixel-by-pixel classification). Similarly, the term "classification" includes detection, such as object detection (which can be viewed as classification based on the presence or absence of an object).

[0176] Although the invention has been shown and described primarily with reference to specific embodiments, those skilled in the art will understand that many changes in design and detail may be made without departing from the spirit and scope of the invention as defined by the claims. Therefore, the scope of the invention is determined by the appended claims and is intended to cover all changes within the literal meaning or equivalent scope of the claims.

Claims

1. A computer-implemented method for processing image data using a convolutional network to classify images into a predefined number of categories, wherein the image data is processed through a plurality of consecutive layers of the convolutional network, wherein the convolutional network includes an input layer, a convolutional filter layer, an aggregation layer, and an output layer, wherein the method comprises: - Receive the input image through the input layer; - Receive at least one input matrix having input data values ​​through the convolutional filter layer, wherein the input matrix corresponds to the pixel values ​​of the input image; - Convolve the input matrix with a first filter matrix and each second filter matrix such that for each filter matrix an output matrix with output data values ​​is produced, wherein the first filter matrix is ​​given by a sum of filter basis functions weighted with a first weight, wherein at least one second weight is calculated from the first weight by applying a transformation formula parameterized by the transformation parameters to the first weight for the corresponding values ​​of the transformation parameters, and wherein for each second weight, a corresponding second filter matrix is ​​determined by calculating a sum of filter basis functions weighted with that second weight; - The output matrix is ​​combined through the aggregation layer; as well as - The output layer outputs the category with the highest probability. The transformation parameter is a scaling parameter, and the convolutional filter layer calculates the second weight by applying the transformation formula such that the second filter matrix is ​​a scaling of the first filter matrix, wherein the strength of the scaling is given by the scaling parameter.

2. The method according to claim 1, wherein, The aggregation layer determines the corresponding value of a predefined evaluation variable for each output matrix and combines the output matrices by outputting the identifier of the output matrix that maximizes the evaluation variable.

3. The method according to any one of claims 1 to 2, comprising: The convolutional network is trained by comparing the values ​​predicted by the convolutional network for training data with predetermined reference values ​​for the training data, wherein the coefficients of the first weights and / or the coefficients of the transformation formula are trained.

4. The method according to any one of claims 1 to 2, comprising controlling the actuator based on the output of the convolutional network.

5. An apparatus for processing image data to classify images into a predefined number of categories, wherein the apparatus is designed to perform the method according to any one of claims 1 to 4.

6. A system for processing image data to classify images into a predefined number of categories, comprising: The sensor is designed to provide image data; and The device according to claim 5.

7. The system of claim 6, comprising an actuator and a control device, the control device being designed to control at least one actuator using the output of the device.

8. The system of claim 6, wherein the system is a robot.

9. A computer program product having program instructions designed to perform the method according to any one of claims 1 to 4 when executed by one or more processors.

10. A machine-readable storage medium having stored thereon program instructions designed to perform the method according to any one of claims 1 to 4 when executed by one or more processors.

Images