Methods for using devices and computers to train artificial neural networks
By grouping and organizing the output channels of neural network layers in a grid, and training filter transformation copies using unique normalization and neighborhood functions, the problem of over-parameterization of neural networks is solved, improving training efficiency and generalization ability, making it particularly suitable for data-constrained scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-16
- Publication Date
- 2026-03-13
AI Technical Summary
Existing neural networks are over-parameterized, require a large amount of data for training, and the training process is not efficient enough, making it difficult to achieve effective data symmetry and generalization accuracy.
By grouping the output channels of neural network layers into unique groups and organizing them in a grid, each group provides a unique normalization function. Filter transform copies are trained using tunable hyperparameters and neighborhood functions to learn equivariant representations, inducing topological relationships between filters, and applying local normalization functions to improve generalization ability.
It improves the efficiency of training data and generalization accuracy, reduces the dependence on data symmetry transformation, and achieves better filter smoothing transformation and generalization performance, especially in data-constrained scenarios.
Smart Images
Figure CN112529173B_ABST
Abstract
Description
Background Technology
[0001] This invention relates to devices and computer-based methods for training artificial neural networks.
[0002] It is known that existing neural networks are over-parameterized and require large amounts of data for training. Therefore, it is desirable to make the training process both easier and more sample-efficient. Summary of the Invention
[0003] This is achieved through the apparatus and method of the independent claims.
[0004] A computer-implemented method for training an artificial neural network includes providing a neural network layer, particularly a convolutional neural network layer, with multiple output channels for isovariant feature mapping; grouping the channels of the multiple output channels of the neural network layer into multiple unique groups, wherein the output channels of each individual unique group are organized into a separate grid, the separate grid defining the spatial position of each output channel of the individual unique group within the grid; providing a unique normalization function for each output channel of each individual unique group, wherein the unique normalization function is defined depending on the spatial position of the output channel within the grid in which the output channel is organized and on tunable hyperparameters of the normalization function; determining the output of the artificial neural network based on the result of each of the unique normalization functions; and training the hyperparameters during the training of the artificial neural network. This improves training data collection effort, sample efficiency, and generalization accuracy by using data symmetry for training. No explicit knowledge or analytical representation of the transformation defining the data symmetry is required. This is achieved by using the concept of maintaining a transformed copy of the learning filter. The transformed copy of the filter is learned automatically directly from the data. This is achieved by inducing an arbitrary topology above one or more convolutional filters in each layer of the artificial neural network, essentially arranging them in a 1D, 2D, or ND grid, and encouraging filters that are closer to each other in this topology to be more correlated than those that are farther apart. This results in a smooth transformation of the filters, leading to better generalization and improved equivariance. The normalization function is a local normalization function applied to the output of each convolutional layer as a means of encouraging correlation. This results in a topological feature map organization for learning equivariant representations and improved generalization. The method enables training networks that learn approximate equivariances of these rather unknown transformations directly from the data, as well as approximate equivariances for transformations for which analytical representations, such as changes in illumination, may not exist.
[0005] Advantageously, the method includes providing a dimensionality parameter, grouping channels from a plurality of output channels of a neural network layer into a plurality of unique groups, each unique group having a size, wherein the size is defined depending on the total number of output channels of the neural network layer, and organizing the output channels of each group into a grid with dimensions according to the dimensionality parameter. This utilizes the variable size of the groups for parameterization of the dimensionality parameter during training.
[0006] Advantageously, the method includes determining a rescaling parameter for each output channel, and rescaling the output of the output channel after normalization based on the rescaling parameter used for that output channel. This introduces additional trainable parameters.
[0007] Advantageously, the method includes providing a neighborhood function for defining a set of output channels, wherein the normalization function of the output channels is defined based on the output channels of the neural network layers in the set of output channels and without regard to other output channels. The neighborhood function provides a parameterized size of the set of nearby output channels.
[0008] Advantageously, the set of output channels is defined by a distance parameter, which defines the distance between the output channels of the neural network layer, and this distance parameter is trained during training. This is a very efficient parameter for training.
[0009] Advantageously, the set of output channels is defined by a spatial distance parameter, which is defined as the distance between the output channels of the neural network layers in the grid, and the spatial distance parameter is trained during training. Preferably, the neighborhood function operates not only on the channels but also on the spatial extent. This makes the regularization effect stronger and improves generalization performance in highly data-constrained scenarios.
[0010] Advantageously, the method includes providing sensor data, particularly sensor data characterizing digital images, which is processed by an artificial neural network trained to determine an output signal used to classify, detect, locate, or segment objects in the sensor data, and / or detect anomalies in the sensor data. Thus, the topology of the mesh is learned from the sensor data during training.
[0011] Advantageously, the method includes inducing a physical system dependent on the output signal, or generating input data for a generative adversarial neural network or variational autoencoder used for data synthesis. A normalization function can form a layer, which can be integrated as a layer normalization into the generative adversarial neural network or variational autoencoder.
[0012] Advantageously, the artificial neural network is suitable for segmenting, classifying, or detecting pedestrians, road signs, and / or vehicles in digital images. The method includes: collecting a set of digital images from a database; applying one or more transformations, including mirroring, rotation, smoothing, or contrast reduction, to each digital image to create a modified set of digital images; creating a first training set comprising the collected set of digital images, the modified set of digital images, and a set of digital images unrelated to pedestrians, road signs, and / or vehicles; training the artificial neural network using the first training set in a first phase; creating a second training set for a second phase of training, comprising the first training set and digital images that were incorrectly detected as depicting pedestrians, road signs, and / or vehicles after the first phase of training; and training the neural network using the second training set in a second phase.
[0013] An apparatus for processing data includes an input for data, an output for an output signal, a memory for an artificial neural network adapted to perform the method, and at least one processor.
[0014] Further advantageous embodiments can be derived from the following description and accompanying drawings. In the drawings:
[0015] Figure 1 The diagram schematically depicts a device used for processing data.
[0016] Figure 2 The steps in the method used for artificial neural networks are described.
[0017] Figure 3a , 3b 3c is an example of a filter used for processing data.
[0018] Figure 1 Various aspects of the device 100 used for data processing are depicted. The data in the example is sensor data. In particular, in this example, a digital image represented by the data is processed.
[0019] Device 100 includes a memory 104 for an artificial neural network 106 and at least one processor 102. Device 100 includes an input 108 for data and an output 110 for output signals. These components of device 100 interact via a data connection 112.
[0020] Generally speaking, it can process video, radar, LiDAR (Light Detection and Ranging), ultrasound, thermal imaging, or motion sensor data.
[0021] The sensor data is processed by an artificial neural network 106 to determine the output signal. The sensor data can be processed for classification, detection, localization, or segmentation of objects within the sensor data, and / or detection of anomalies. The output signal can be used to induce physical systems dependent on the output signal, or to generate input data for generative adversarial networks (GANs) or variational autoencoders (VAEs) used for data synthesis.
[0022] The artificial neural network 106 in the example includes a convolutional neural network. More generally, the method described below applies to the artificial neural network 106, which has features for isovariant feature mapping. Neural network layers, especially convolutional neural network layers.
[0023] The following is for reference. Figure 2 Describe the method.
[0024] The method includes providing multiple output channels in step 202. Used for equivariant feature mapping The neural network layer. In this example, a convolutional neural network layer is provided. Let For example, a convolutional neural network layer, where and These are the height and width of the input and output spaces, respectively. , These are the number of input and output channels, respectively. Therefore, a neural network layer includes... There are n convolutional kernels, each convolutional kernel has... There are trainable parameters, where n indicates the filter size determined by the modeler of the neural network layer.
[0025] Step 202 may include providing the dimension parameter M and multiple output channels of the neural network layer. The channels are grouped into G unique groups g, each unique group g having a size D: In this example, the size D depends on the output channels of the neural network layer. The total number To define, for example, define as .
[0026] Step 202 may include each group Its output channel Organized into a grid with dimensions according to the dimension parameter M.
[0027] The method includes, in step 204, processing multiple output channels of the neural network layer. The channels are grouped into G unique groups g.
[0028] Each unique group g's output channel Organized into separate M-dimensional grids, each M-dimensional grid defines a unique output channel for a unique group g. Spatial location in the grid h , w .
[0029] The method includes, in step 206, providing each output channel for each individual unique group g among the G unique groups g. Provides unique normalization functions
[0030]
[0031] in It is a tunable hyperparameter.
[0032] This output channel normalization function It depends on the output channel. The output channels of neural network layers in the set And it is defined without considering other output channels.
[0033] In this example, the output channel is defined. neighborhood function of the set .
[0034] Output Channel The set can be in the neighborhood function The distance parameter k Define this distance parameter k Define the output channels of a neural network layer The distance between them. In the example, each output channel is identified by an index, and this distance is defined as the numerical distance between the indices of two output channels.
[0035] In the example, for each output channel For those with dimension parameters The grid (i.e., a 1D grid) output channels The size of the neighborhood, for The neighborhood function is .
[0036] In another example, a spatial neighborhood function can be defined. Output channel The set can be determined by the spatial distance parameter l , m Define this spatial distance parameter l , mThe output channels of a neural network layer defined in a mesh The distance between them.
[0037] Any boundary condition for the neighborhood function is a modeling choice and can be selected as circular, zero-fill, or other commonly used filling techniques.
[0038] In the example, the following parameters can be used:
[0039] .
[0040] k The optimal value depends on the size of and the neural network layer to which it is applied. In the example, using .
[0041] Generally speaking, unique normalization function Depends on the output channel Organize the output channel within it. Spatial position in the grid h , w .
[0042] In this example, the unique normalization function It depends on the spatial location. h , w And it depends on the normalization function. It is defined by tunable hyperparameters.
[0043] This method includes a unique normalization function. Step 208 determines the output of the artificial neural network 106 based on the results of each step.
[0044] The method includes step 210, which trains parameters and hyperparameters during the training of the artificial neural network 106. Specifically, it trains the distance parameter. k and spatial distance parameters l , m The training of the artificial neural network depends on the output.
[0045] In this example, step 202 is performed before training to fix the grid. In step 210, steps 204 through 208 are repeated to process the training data. For example, the parameters of the artificial neural network 106 are determined and updated during training using gradient descent, which minimizes the loss criterion.
[0046] After training is complete, proceed to step 212.
[0047] Step 212 includes providing sensor data, particularly sensor data characterizing digital images.
[0048] Next, proceed to step 214.
[0049] In step 214, the artificial neural network 106, trained in this way, processes the sensor data to determine an output signal. In this example, the output signal is processed to classify the sensor data, detect, locate, or segment objects within the sensor data, and / or detect anomalies in the sensor data.
[0050] Next, proceed to step 216.
[0051] In step 216, the physiological system is induced based on the output signal.
[0052] Instead of or in addition to this, sensor data can be processed to generate input data for generative adversarial neural networks or variational autoencoders used for data synthesis.
[0053] The method may further include determining each output channel. rescaling parameters And after normalization, it depends on the output channel used. rescaling parameters To rescale the output channel The output of the artificial neural network 106 depends on the output channel. Rescaling output (For example (This is) determined.
[0054] The normalization function described above is a local squeezing function, which is used as a standard layer in a convolutional neural network and is placed after the convolution. If the convolutional neural network includes any one or a combination of batch normalization, pooling, activation functions, and dropout layers, the local squeezing function is placed after the convolution, optional batch normalization, and optional pooling, and before the optional activation functions and optional dropout layers.
[0055] The local squeezing function can be implemented as a layer in the artificial neural network 106. This layer can be placed anywhere between two convolutional operations and can be applied to every layer, or only to the desired layers. Preferably, the layer is placed between the batch normalization layer and the activation function layer. Preferably, the local squeezing function is being applied to more than one layer of the artificial neural network 106.
[0056] In this way, a topology, i.e., a mesh, is generated above the output channels of the aforementioned CNN layers. This encourages the artificial neural network 106 to learn intermediate transforms between existing filters. The local squeezing function layer is an "intermediate" filter that increases the number of transforms, which the artificial neural network 106 captures in its representation and is therefore equivariant.
[0057] The desired induced topology is achieved through the normalization function mentioned above. The artificial neural network 106 may include multiple CNN layers and intermediate filters, namely, local squeezing function layers.
[0058] Figure 3a The output of a 17×17 pixel filter, comprising a local compression function with a 1D mesh, is depicted. The results are... Figure 3a The output filter is organized in the original latent unknown grid formula. Figure 3b The filter output of a 17×17 pixel filter without a local squeezing function is depicted. Figure 3c The output of a 17×17 pixel filter with a clear 1D grid and a smooth transition learned using a local squeezing function is depicted. In this case, the training weights of the artificial neural network 106 represent the topology organized in the learning grid of the intermediate filters at each layer.
[0059] Artificial neural network 106 can be used in image data processing in vehicles or robots, or in medical devices or machines that process image data to determine actions to be performed by their actuators.
[0060] In particular, the artificial neural network 106 can be adapted to segment, classify, or detect pedestrians, road signs, and / or vehicles in digital images.
[0061] The method may include collecting a set of digital images from a database. The method may include applying one or more transformations, including mirroring, rotation, smoothing, or contrast reduction, to each digital image to create a modified set of digital images. The method may include creating a first training set, which includes the collected set of digital images, the modified set of digital images, and a set of digital images unrelated to pedestrians, road signs, and / or vehicles. The method may include training an artificial neural network 106 using the first training set in a first phase. The method may include creating a second training set for a second phase of training, which includes the first training set and digital images that were incorrectly detected as depicting pedestrians, road signs, and / or vehicles after the first phase of training. The method may include training the artificial neural network 106 using the second training set in a second phase.
Claims
1. A computer-implemented method for training an artificial neural network (106), wherein, The artificial neural network (106) is configured to accept sensor data and to determine an output signal for classifying the sensor data, detecting, localizing or segmenting objects in the sensor data, and / or detecting anomalies in the sensor data, and wherein the artificial neural network (106) comprises a neural network layer configured for determining an equivariant feature map having a plurality of output channels, wherein a set of output channels is defined by a distance parameter defining distances between output channels of the neural network layer in a grid, wherein the method comprises the steps of grouping (204) channels of the plurality of output channels of the neural network layer into a plurality of distinct groups, wherein the output channels of each individual distinct group are organized into an individual grid defining for the individual distinct group a spatial position of each output channel of the individual distinct group in the grid, providing (206) a distinct normalization function for each output channel of each individual distinct group of the plurality of distinct groups, wherein a distinct normalization function is defined depending on a spatial position of an output channel in a grid in which the output channel is organized and depending on a tunable hyperparameter of the normalization function, providing a neighborhood function defining a set of output channels for an output channel, wherein the normalization function of the output channel is defined depending on output channels of the neural network layer in the set of output channels and without considering other output channels, determining (208) an output of the artificial neural network depending on a result of each of the distinct normalization functions, training (210) the hyperparameters and training the distance parameter in a training of the artificial neural network (106), the sensor data representing a digital image.
2. The method of claim 1, wherein, providing (202) a dimension parameter, grouping (204) channels of the plurality of output channels of the neural network layer into a plurality of distinct groups, each distinct group having a size, wherein the size is defined depending on a total number of output channels of the neural network layer, and organizing the output channels of each group into a grid having dimensions according to the dimension parameter.
3. The method according to claim 1 or 2, characterized in that, determining a rescaling parameter for each output channel, rescaling an output of the output channel depending on the rescaling parameter for the output channel after the normalization.
4. The method of claim 3, wherein, the set of output channels is defined by a spatial distance parameter defining distances between output channels of the neural network layer, and the spatial distance parameter is trained in the training.
5. The method according to claim 1 or 4, characterized in that, the neural network layer is a convolutional neural network layer.
6. The method of claim 1, wherein, actuating (216) a physical system depending on the output signal, or generating input data for a generative adversarial neural network or a variational autoencoder for data synthesis.
7. The method according to one of the preceding claims, characterized in that, An artificial neural network (106) is adapted to segment, classify or detect pedestrians, road signs and / or vehicles in digital images, the method comprising collecting a set of digital images from a database; applying one or more transformations to each digital image, including mirroring, rotation, smoothing or contrast reduction, to create a set of modified digital images; creating a first training set comprising the collected set of digital images, the set of modified digital images and a set of digital images unrelated to pedestrians, road signs and / or vehicles; training the artificial neural network (106) in a first phase using the first training set; creating a second training set for a second phase of training, the second training set comprising the first training set and digital images of images that were falsely detected as depicting pedestrians, road signs and / or vehicles after the first phase of training; and training the artificial neural network (106) in a second phase using the second training set.
8. An apparatus (100) for processing data, characterized in that An input (108) for data, an output (110) for an output signal, a memory (104) for an artificial neural network (106) adapted to perform the method according to one of claims 1 to 7 and at least one processor (102).
9. Computer program product, characterized in that, A computer-readable storage medium comprising machine-readable instructions which, when executed by a computer, cause the computer to perform the method according to one of claims 1 to 7.
Citation Information
Patent Citations
An airport runway foreign matter material identification method based on adversarial neural network data enhancement
CN109902602A
Compact models for object recognition
US20190220653A1
Reducing architectural complexity of convolutional neural networks via channel pruning
US20190251441A1
Method and apparatus for object recognition
US6038337A