Method for generating a pair of digital images as training data for a neural network

Abstract

Method for generating a pair of digital images as training data for a neural network. A method for creating a pair of digital images to train a neural network to correct a noise-disturbed image portion of a noise-disturbed image, the method having the following steps: determining the degree of movement of an object within an overlapping region of a first digital image and a second digital image stored in the surrounding environment of a mobile platform; determining the respectively detected solid angles of the surrounding environments of the mobile platform, the first and second digital images; creating a pair of digital images composed of the first digital image and the second digital image as long as the respectively detected solid angles of the surrounding environments of the first and second digital images deviate from each other by no more than a specified difference and the degree of movement of the object within the overlapping region of the first and second digital images is lower than a specified value.

Description

Technical Field

[0001] The present invention relates to a method for generating a pair of digital images as training data for a neural network, which learns to restore a noise-disturbed image to its original state by means of the noise-disturbed image. Background Art

[0002] When generating a representation of the surroundings for autonomous driving by means of, for example, an imaging sensor, the result may be negatively affected due to various effects. Here, in addition to the regular measurement noise caused by the randomness of the measurement method of the imaging sensor and its detector design, there are also various environmental influences, such as direct sunlight, rain, snow or fog, which affect the measurement result. These influencing factors lead to many false detections (False-Positives) or a complete stop of detections (False-Negatives). As a result, the performance of the downstream algorithm is reduced. If the result is degraded due to such influencing factors in the case of the imaging method used, this data must, if necessary, be detected and corrected or discarded. Thus, methods that can identify or remove these incorrect measurements lead in particular to an improvement of the representation of the surroundings in the field of autonomous driving. Summary of the Invention

[0003] The present invention discloses a method for creating a pair of digital images to train a neural network to correct a noise-disturbed image part of a noise-disturbed image, a corresponding method for training a neural network, a corresponding method for correcting a noise-disturbed image part of a noise-disturbed image, a method for representing the surroundings of at least a partially automated mobile platform, a device, a computer program product and a machine-readable storage medium. Advantageous design alternatives are the subject of the subsequent description.

[0004] For noise suppression, a convolutional neural network can be used. For training such a network, "clean" training data, i.e., images without noise or only slightly noisy, is used. Unfortunately, such training data is not accessible in reality or can only be accessed very difficultly and expensively. If the noise distribution in the image relates to the static mean or median of the relevant pixels, i.e., if the pixel values of a plurality of mutually independent noise-disturbed images correspond to the noise-free image in terms of the mean or median, it is possible to successfully remove the noisy part of the digital image even without ground-truth annotations in the training image data. However, for this purpose, correspondingly suitable image pairs are required.

[0005] A method according to the present invention creates pairs of digital images for training a neural network, where the neural network corrects the image parts affected by noise in the images affected by noise. Here, in one step, the degree of movement of an object within the overlapping region of a first digital image stored in the surroundings of a mobile platform and a second digital image stored is determined. In the next step, the respectively detected solid angles of the surroundings of the mobile platform, the first and second digital images are determined. In the next step, as long as the respectively detected solid angles of the surroundings of the first and second digital images deviate from each other by no more than a specified difference and the degree of movement of the object within the overlapping region of the first and second digital images is lower than a specified value, a pair of digital images composed of the first digital image and the second digital image is created.

[0006] Neural networks provide a framework for a variety of different algorithms for machine learning, for collaboration, and for processing complex data inputs. Such neural networks learn to perform tasks based on examples and are not typically programmed with task-specific rules.

[0007] Such neural networks are based on an aggregation of different units or nodes, which are called artificial neurons. Each connection can transmit a signal from one artificial neuron to another artificial neuron. The artificial neuron that receives the signal can process the signal and then activate other artificial neurons connected to that artificial neuron.

[0008] In a conventional implementation of a neural network, the signal at the connection of an artificial neuron is a real number, and the output of the artificial neuron is calculated by a non-linear function of the sum of the inputs of the artificial neuron. The connections of these artificial neurons usually have weights, which are adapted as learning progresses. The weights increase or decrease the strength of the signal at the connection. An artificial neuron may have a threshold such that a signal is only output when the total signal exceeds the threshold. Generally, multiple artificial neurons are combined in layers. Different layers may perform different types of transformations on their inputs. The signal may reach the last layer, i.e., the output layer, from the first layer, i.e., the input layer, after traversing these layers multiple times.

[0009] The architecture of this artificial neural network can be a neural network constructed according to a multi-layer perceptron (MLP) network. The multi-layer perceptron (MLP) network belongs to the family of artificial feed-forward neural networks. In principle, an MLP consists of at least three neuron layers: an input layer, a hidden layer, and an output layer. This means that all neurons of the network are layered, where a neuron in one layer is always connected to all neurons in the next layer. There are no connections to the previous layer and no connections that skip a layer. Except for the input layer, there are also different neuron layers, where the neurons have non-linear activation functions and are connected to the neurons in the next layer. A deep neural network may have four such hidden layers.

[0010] The processing of an image can be achieved by using an encoder-decoder neural network, which is further developed similarly to the neural network mentioned above. The architecture of such an encoder-decoder neural network generally consists of two parts. The first part corresponds to an autoencoder and is a sequence of layers that calculate the input pattern downward to a lower resolution with respect to the data volume in order to obtain the desired information and reduce redundant information. The second part is a sequence of layers that calculate the output of the first part upward and reconstruct the desired output resolution, such as the input resolution. Optionally, there may be additional skip connections that directly connect some layers in the first part and in the second part.

[0011] Such encoder-decoder networks must be trained for their specific tasks. Here, each neuron of the corresponding architecture of the neural network, for example, obtains random initial weights. Then, the input data is sent into the network, and each neuron weights the input signal with its weights and passes the result on to the neurons in the next layer. Then, the total result is provided at the output layer. The magnitude of the error can be calculated, and the share of each neuron in this error can be calculated, and then the weights of each neuron are changed in the direction of minimizing this error. Then, it is traversed recursively, repeatedly measuring the error and adapting these weights until the error is below a pre-given limit.

[0012] Digital images can be negatively affected with respect to the use of these images due to various effects. In addition to the regular measurement noise caused by the randomness of the measurement method and its detector design, images of the surrounding environment of a mobile platform can also be affected by environmental effects. In particular, fog, snow, and rain significantly affect the image quality, but reflections or direct sunlight can also impair the image quality. This applies similarly to digital images, RADAR (radio detection and ranging) images, LIDAR (light detection and ranging) images, or other imaging methods in the optical field to varying degrees.

[0013] For example, in the case where an image is generated by a LIDAR (light detection and ranging) measurement method, these influencing factors lead to false detections (false positives) or a complete stop of the detection (false negatives). Due to this, the performance of downstream analysis algorithms is reduced.

[0014] Now, for example, consider the range measurement of a LiDAR (light detection and ranging) sensor. In the case of good environmental conditions (without weather effects or strong ambient light), almost no false positive or false negative measurements occur. However, in adverse conditions caused by the environment, both phenomena occur more and more frequently. For example, the laser pulses emitted by a LIDAR (light detection and ranging) scanner are reflected by raindrops or snowflakes before they reach the object to be measured. The measured distance is less than the true distance to the object to be measured. Due to a high level of ambient light (such as direct sunlight), detection also occurs before the reflected measurement pulse reaches the LIDAR (light detection and ranging) detector. Then, the measured distance is less than the true distance to the object being measured. If the emitted laser pulse is deflected from its path, for example, due to water droplets, the reflected signal may not reach the detector and the measurement is invalid or the object is outside the sensor range.

[0015] Thus, in the case of LIDAR (light detection and ranging) images, the noise distributions of measurements ("images") carried out continuously in time can be assumed to be independent of each other. Thus, the range measurement carried out at a certain solid angle does not give a statistically significant conclusion about the result of the next measurement. Therefore, knowing the noise distribution of one image does not allow the noise distribution of other images to be deduced. Thus, in the case of the LIDAR (light detection and ranging) measurement method and the images obtained thereby, the preconditions for applying the method described above exist.

[0016] In order for the created image pair to be suitable for the method described above, it must be ensured that: the scene is only superimposed with the noise from the noise sources mentioned above, and the changes in the scene do not cause any additional changes in the image at all.

[0017] This means that: in addition to the requirements for noise, on the one hand, only the scene whose image content is superimposed is analyzed, and it also means that: within the time range of creating the image pair, the changes caused by the movement of the imaged object are less than the specified value in terms of degree. With the help of the inertial navigation system available for the mobile platform, it can be ensured, if necessary, that the positions of the two images suitable for such an image pair deviate from each other by no more than a pre-defined value. However, for example, other imaging methods that are applied simultaneously can also be used to ensure the overlap of the two images. Sensor data, such as radar data, can also be used to determine the movement range of the object in this scene, and it can be ensured by comparison that such movement is below the specified limit. In particular, the offset of the image data in the overlapping area should not exceed a few pixels, that is, the offset should be more precisely limited to less than 5 pixels.

[0018] In order for the digital image pair to be suitable for this training of the neural network, it must also be ensured that the images are taken from very similar perspectives. This can be ensured in the following way: the respectively detected solid angles of the surroundings of the first and second digital images deviate from each other by no more than the specified difference. Here, the solid angle describes the share of the entire space that, for example, lies inside a given conical or pyramidal shell.

[0019] This can be ensured in particular in the following way: the footholds from which the images are taken deviate from each other by no more than the specified value. This can be ensured by a variety of different methods, such as by using a radar system for ranging, by triangulation, by a positioning system such as GPS (Global Positioning System), or by the measured values from an inertial navigation system connected to the following mobile platform, and such an imaging system can be installed on this mobile platform. Optionally, other ambient sensors, such as radar systems or cameras, and tracking algorithms are also used in combination with each other to find out whether there are other moving traffic members in this scene, and these traffic members have changed their positions to an unacceptable degree between the taking of the two images and thus changed the scene. Correspondingly, this also applies to the simultaneous taking using two imaging systems, and the two imaging systems are related to each other such that only an acceptable change in the solid angle of the associated images is caused. The acceptable offset of the shooting footholds caused by the movement of the mobile platform and / or the offset between the two imaging systems respectively is usually in the range of a few centimeters, and in particular, this offset should be less than 15 cm.

[0020] A mobile platform can be understood as a mobile, at least partially automated system and / or driver assistance system. Examples can be an automatically or partially automated vehicle or a vehicle with a driver assistance system. That is, in this context, an at least partially automated system encompasses a mobile platform with respect to the functional aspects of at least partial automation, but a mobile platform also includes vehicles and other mobile machinery including driver assistance systems. Other examples of mobile platforms can be; a driver assistance system with multiple sensors; a mobile multi-sensor robot, such as a robotic vacuum cleaner or a lawn mower; a multi-sensor monitoring system; a manufacturing machine; a personal assistant; or an access control system. Each of these systems can be a fully or partially automated system.

[0021] If, for example, a sequence of 20 photos now meets the requirements mentioned above, then 2 to the power of 20 (corresponding to the binomial coefficient) different image pairs can be generated from it as training data for a neural network. Since this method for obtaining training data requires no manual annotation at all, large amounts of data, such as large amounts of data from a vehicle fleet, can be obtained very simply and applied to the training of a neural network.

[0022] According to one design of the present invention, it is proposed that the degree of movement of an object within the overlapping region of the first and second digital images is determined using data from the inertial navigation system of the mobile platform. The inertial navigation system integrates the measurement data of multiple sensors mounted on the mobile platform in order to improve the quality of object detection and object tracking.

[0023] By analyzing multiple images that seem suitable as image pairs for correspondingly training a neural network and for which data from the inertial navigation system is respectively available, depending on the scope and complexity of the data of the vision tracking system, the degree of movement of an object within the overlapping region of these two images can be determined very simply. For example, in such a way that the positions and orientations of the systems that captured the corresponding images are compared with each other. Additional sensors belonging to the tracking system can also be used to determine the degree of movement of the object.

[0024] According to one design of the present invention, it is proposed that the respectively detected solid angles of the surroundings of the mobile platform, the first and second digital images are determined using data from the inertial navigation system of the mobile platform. For this purpose, the inertial navigation system can be used in the same way as described when analyzing the degree of movement of an object within the overlapping region.

[0025] According to a design of the present invention, it is proposed that the degree of movement of an object is determined by means of the signal of a radar system or a radar sensor or the image analysis of an optical system or a tracking system. As already described above, a radar system or a radar sensor is particularly suitable for detecting the movement of an object, because the radar signal can directly indicate the movement and speed within its detection area.

[0026] According to a design of the present invention, it is proposed that data of the inertial navigation system of a mobile platform are used to determine the deviation of the respectively detected solid angles of the surroundings of the mobile platform and the first and second digital images. In particular, through the data of the inertial navigation system (which can, if necessary, recognize the movement of the imaging system between the first and second images by means of inertial effects), conclusions can be drawn not only about the overlap of the image areas but also about the detected solid angles.

[0027] According to a design of the present invention, it is proposed that the stored first and second digital images are generated by signal transformation of a LIDAR (Light Detection and Ranging) system. The signal of a LIDAR system that scans the surroundings of a mobile platform with a laser beam, for example, can be converted into a two-dimensional image by means of transformation and then used to train a neural network in order to correct the noise-disturbed image parts of the image thus generated. As already explained above, the LIDAR system also has errors caused especially by environmental influences. By applying such a neural network to the images of this system, the LIDAR system can be used for further applications of an automated vehicle, such as being used in the decision-making system of an automated vehicle.

[0028] According to a design of the present invention, it is proposed that the stored first and second digital images are detected by means of the imaging systems of a plurality of at least partially automated mobile platforms. In order to quickly generate image pairs suitable for training a neural network to correct noise-disturbed images, image pairs of a plurality of at least partially automated mobile platforms equipped with corresponding imaging systems can be taken and then analyzed centrally in order to train the neural network in the described manner.

[0029] A method for training a neural network to correct the noise-disturbed image parts of a noise-disturbed image is proposed, wherein the image pair is generated according to the method for creating digital image pairs described above and the neural network is trained using the first digital image of the digital image pair as an input quantity to generate the second digital image of the digital image pair.

[0030] The method for generating suitable digital image pairs described above can be used to train a neural network in such a way that the first image of the image pair is used as the input quantity of the neural network and the second image of the pre-given image pair is used as the target quantity of the neural network. Here, for example, a neural network having the architecture described above precisely learns to filter noise from a noise-disturbed image by the neural network correcting the noise-disturbed image part.

[0031] A method for correcting a noise-disturbed image part of a noise-disturbed image by means of a neural network is proposed, and the neural network is trained according to the method for training a neural network. Here, the noise-disturbed image is given to the trained neural network as the input quantity, and the noise-disturbed image part is then corrected in the image generated by the neural network. By using the method for correcting the noise-disturbed image part, it is achieved that images generated by different imaging methods and possibly disturbed by noise are used for further applications in combination with an automation system, and these images are better suitable for, for example, being analyzed to recognize the surroundings of a mobile platform or actions triggered by a downstream decision system.

[0032] A method for representing the surroundings of an at least partially automated mobile platform by means of images is described, the images being detected by an imaging system of the surroundings of the at least partially automated mobile platform and the noise-disturbed image parts of these images being corrected according to the method for correcting the noise-disturbed image parts of a noise-disturbed image. By correcting the noise-disturbed image parts, the representation of the surroundings of the at least partially automated mobile platform can be used with higher quality for presentation or for further analysis, for example for a decision system, and can thus be used with higher safety with respect to actions derived from this representation of the surroundings.

[0033] According to a design of the method for representing the surroundings, the representation of the surroundings is used to control an at least partially automated vehicle and / or to send the presentation of the representation of the surroundings to a vehicle passenger. By means of this representation of the surroundings, for example, the actions of the mobile platform can be introduced or adjusted, or the vehicle passenger can use the representation to derive information that may affect the subsequent behavior of the vehicle passenger.

[0034] A device is described which is set up to carry out one of the methods described above. By means of such a device, it is possible, for example, to make the methods described above available for being embedded in a mobile platform.

[0035] A computer program product is described, which includes instructions that, when executed by a computer, cause the computer to perform one of the methods described above. With such a computer program product, the methods described above can be used, for example, in a simple manner on a mobile platform.

[0036] A machine-readable storage medium is described, on which the computer program described above is stored. With such a machine-readable storage medium, the computer program product described above can be transported. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Embodiments of the present invention are illustrated with reference to Figures 1 to 2 and are described in detail below. Among them:

[0038] Figure 1 a shows an input image without noise;

[0039] Figure 1 b shows the input image superimposed with the first synthetic noise;

[0040] Figure 1 c shows the input image superimposed with the second synthetic noise;

[0041] Figure 1 d shows an image with a corrected image portion of the input image affected by noise; while

[0042] Figure 2 shows a method for correcting an image portion affected by noise. DETAILED DESCRIPTION

[0043] Figure 1 a schematically shows an exemplary input image without noise. This image is an image of the surrounding environment of the platform created using a LIDAR (Light Detection and Ranging) system, where the road extends deep into the image. A car is detected deep in the image, there is a sidewalk along the road on the right, and vegetation is detected further to the right. On the left side of the road, a traffic sign stands in the distance. Here, all images 1a, b, c, and d are schematically transformed from the original image with the help of hatching for this presentation.

[0044] In Figure 1 b, the input image shown in Figure 1 a is superimposed with synthetic noise, where noise values with distances between 0 m and the measured distance are used for 25% of the pixels.

[0045] Figure 1 c is created according to Figure 1 b, where however the noise is withFigure 1 is superimposed on the image of a regardless of the noise of b. Figure 1 of the image of a.

[0046] Figure 1 d shows the result of a neural network trained using a plurality of noise-corrupted image pairs as described above. Figure 1 The result is obtained with the noise-corrupted image of b as the input quantity.

[0047] Therefore, Figure 1 The images of b and 1c are examples of noise-corrupted image pairs for training the neural network. Figure 1 d shows how unexpectedly well the neural network learns to correct the noise-corrupted image portions.

[0048] An exemplary neural network is an encoder-decoder network constructed of two parts. The first part is a sequence of layers such that the layers in the sequence scan the input grid downward to a lower resolution in order to obtain the desired information and store redundant information. The second part is a sequence of layers such that the layers in the sequence rescan the output of the first part into fully connected layers and produce the desired output resolution, such as a classification vector having the same length as the signals characterizing the various hazardous situations to be classified.

[0049] The first part is, for example, an encoder having three or more layers:

[0050] · An input layer, such as a 2D image;

[0051] · A plurality of significantly smaller layers that form an encoding for reducing data;

[0052] · An output layer, the dimension of which corresponds to the dimension of the input layer, that is, each output parameter in the output layer has the same meaning as the corresponding parameter in the input layer.

[0053] The encoder is constructed corresponding to an artificial Convolutional Neural Network and has one or more convolutional layers, optionally followed by a Pooling Layer after the convolutional layer. This layer sequence can be used with or without normalization layers (such as batch normalization), Zero-Padding layers, Dropout layers, and activation functions such as the Rectified Linear Unit ReLU, sigmoid function, tanh function, or softmax function. In principle, these units can be repeated arbitrarily frequently, and in the case of sufficient repetition, a Deep Convolutional Neural Network is then mentioned.

[0054] Here, the artificial convolutional neural network consists of an encoder-decoder architecture that achieves data reduction through multiple convolutions with a stride of 2. To rescale the compressed data representation, transposed convolutional layers ("Deconvolution") are used, as can be seen from Table 1. Table 1 describes the layers of the encoder convolutional network in more detail. The input to the encoder convolutional network is an 1800x32x1 image (range measurement for each "pixel") or a Tensor data pattern.

[0055] Layer (type) Initial form Input LiDAR distance map (1800,32,1) Conv2d1 (3x3, zero padding) (1800,32,32) Conv2d2 (3x3, zero padding) (1800,32,64) Conv2d3 (3x3, stride = 2) (900,16,96) Conv2d4 (3x3, stride = 2) (450,8,96) Conv2d5 (3x3, stride = 2) (225,4,96) Conv2d_transpose1 (3x3, stride = 2, zero padding) (450,8,96) Conv2d_transpose2 (3x3, stride = 2, zero padding) (900,16,64) Conv2d_transpose3 (3x3, stride = 2, zero padding) (1800,32,32) Conv2d6 (3x3) (1800,32,1)

[0056] Table 1.

[0057] During training, two different loss functions can be used to specify the parameters of the neural network. In the case of a LIDAR system, the noise distribution can be starting point, which can be described by the median. That is, the median of infinitely many captured images corresponds to the image without noise interference. For this kind of noise that can be described by the median, the error can be described by the L1 norm, where the variables of the error L1 are related to each other in the manner described by the following formula:

[0058]

[0059] Here, L is the loss function with the cursor index i, which is the pixel index from 1 up to the length times the width of the image; and is the prediction of the network, that is, applying the neural network to image A; and y: is image B.

[0060] For an image disturbed by noise whose noise can be described by an average value, the parameters of the neural network can be calculated using the L2 norm in combination with a variable as reproduced in Equation 2.

[0061]

[0062] Figure 2 A method for creating a pair of digital images for training a neural network to correct a noise-disturbed image portion of a noise-disturbed image is shown.

[0063] Here, in step S1, the degree of movement of an object within an overlapping region of a first digital image and a second digital image stored in the surrounding environment of the mobile platform is determined. In the next step S2, the solid angles respectively detected in the surrounding environments of the mobile platform, the first and second digital images are determined. In the next step S3, as long as the solid angles respectively detected in the surrounding environments of the first and second digital images deviate from each other by no more than a prescribed difference and the degree of movement of the object within the overlapping region of the first and second digital images is lower than a prescribed value, a pair of digital images composed of the first digital image and the second digital image is created.

Claims

1. A method for creating a pair of digital images to train a neural network to correct a noise - disturbed image part of a noise - disturbed image, the method comprising the following steps: (S1) Determine the degree of object movement within the overlapping region of a first digital image and a second digital image stored in the surrounding environment of a mobile platform, wherein, the neural network is trained using the first digital image of the pair of digital images as an input quantity to generate the second digital image of the pair of digital images; (S2) Determine the respectively detected solid angles of the surrounding environment of the mobile platform, the first and the second digital images; As long as the respectively detected solid angles of the surrounding environment of the first and the second digital images deviate from each other by no more than a specified difference and the degree of object movement within the overlapping region of the first and the second digital images is lower than a specified value, then (S3) create a pair of digital images composed of the first digital image and the second digital image.

2. The method according to claim 1, wherein the degree of object movement within the overlapping region of the first and the second digital images is determined using data from the inertial navigation system of the mobile platform.

3. The method according to claim 1 or 2, wherein the respectively detected solid angles of the surrounding environment of the mobile platform, the first and the second digital images are determined using data from the inertial navigation system of the mobile platform.

4. The method according to claim 1 or 2, wherein the degree of object movement is determined by means of the signal of a radar sensor or the image analysis of an optical system or a tracking system.

5. The method according to claim 1 or 2, wherein the deviation of the respectively detected solid angles of the surrounding environment of the mobile platform, the first and the second digital images is determined using data from at least one inertial sensor of the mobile platform.

6. The method according to claim 1 or 2, wherein the stored first and second digital images are generated by signal transformation of a LIDAR system.

7. The method according to claim 1 or 2, wherein the stored first and second digital images are detected by means of imaging systems of a plurality of at least partially automated mobile platforms.

8. A method for training a neural network to correct a noise - disturbed image part of a noise - disturbed image, wherein the image pair is generated according to one of claims 1 to 7, and the neural network is trained using the first digital image of the pair of digital images as an input quantity to generate the second digital image of the pair of digital images.

9. A method for correcting a noise - disturbed image part of a noise - disturbed image by means of a neural network trained according to claim 8, in such a way that the noise - disturbed image is given to the trained neural network as an input quantity, and the noise - disturbed image part is corrected in the image generated by the neural network.

10. A method for representing the surroundings of at least a partially automated mobile platform by means of an image, the image being detected by an imaging system of the surroundings of the at least partially automated mobile platform, and the image portions of the image affected by noise being corrected according to the method of claim 9.

11. The method according to claim 10, wherein the surroundings representation is used to control at least a partially automated vehicle and / or the presentation of the representation is sent to a vehicle passenger.

12. A device which is set up to carry out the method according to one of claims 1 to 11.

13. A computer program product which comprises instructions which, when the program is executed by a computer, cause the computer to carry out the method according to one of claims 1 to 11.

14. A machine-readable storage medium on which the computer program product according to claim 13 is stored.

Images