Method and device for generating training data for an artificial neural network

Abstract

The present invention relates to a method for generating training data for an artificial neural network for classifying anomalies in image data, in particular for classifying anomalies in image data from an environmental sensor of a motor vehicle.According to the invention, the method comprises the following steps: • Generating visual blocks on a substantially horizontal plane to replicate anomaly structures; capturing a labeling image using a visual sensor, which includes the replicated anomaly structures; creating labels for the replicated anomaly structures based on the captured labeling image; displaying environmental sensor data from a motor vehicle's environmental sensor using a display device; capturing a superimposed image using the visual sensor, which includes the replicated anomaly structures and the displayed environmental sensor data; creating a training dataset with labeled anomalies based on the superimposed image and the created labels for the replicated anomaly structures.Furthermore, the invention relates to a device configured to carry out the corresponding method, as well as a corresponding computer program.

Description

[0001] The present invention relates to a method for generating training data for an artificial neural network for classifying anomalies in image data, in particular for classifying anomalies in image data of an environmental sensor of a motor vehicle.According to the invention, the method comprises the following method steps: • Creating visual blocks on a substantially horizontal plane to simulate anomaly structures; Recording a label image using a visual sensor, which includes the simulated anomaly structures; Creating labels for the simulated anomaly structures based on the recorded label image; Displaying an environmental sensor datum from an environmental sensor of a motor vehicle using a display device; Recording an overlay image using the visual sensor, which includes the simulated anomaly structures and the displayed environmental sensor datum; Creating a training data set with labeled anomalies based on the overlay image and the created labels for the simulated anomaly structures.Furthermore, the invention relates to a device which is designed to carry out the corresponding method, as well as to a corresponding computer program. State of the art

[0002] Approaches are known from the prior art in which automatic image corrections are proposed when the input image data is at least partially impaired by rain, incident light, and / or dirt. Reference is made, for example, to WO 2022 / 128014.

[0003] First, however, the detection of such an impairment is required. If such detection is to be supported by artificial intelligence, data is needed to train the artificial intelligence. Generating training data for a blockage classifier is more difficult than for other common classification problems. Blocked patterns are rare in ordinary recordings, and the visual edges of the blocked regions are blurred. Consistent labeling by humans is therefore difficult. A common workaround for this problem is the creation of artificially generated blockages.

[0004] DE 10 2020 215 656 A1, for example, is known from the prior art. This document relates to a method for generating training data for an artificial neural network for object and degradation detection in autonomously driving vehicles, comprising: recording environmental sensor data by an environmental sensor of a vehicle; classifying environmental sensor data exhibiting an environmental anomaly as relevant environmental sensor data, wherein an environmental anomaly represents an environmental impairment of the environmental sensor data for object detection; and creating a training data set with environmental anomalies for an artificial neural network from the relevant environmental sensor data. Disclosure of the invention

[0005] Advantageously, the method and device according to the invention enables the generation of realistic anomalies in image data. A further advantage is that the necessary information regarding the anomalies, in particular the exact position of the generated anomalies, can be determined simply and reliably. This is made possible according to the invention by the features specified in the independent patent claims. Further embodiments of the invention are the subject of subclaims.

[0006] The method for generating training data for an artificial neural network for classifying anomalies in image data, in particular for classifying anomalies in image data from an environmental sensor of a motor vehicle, comprises the following method steps according to the invention: generating visual blocks on a substantially horizontal plane to simulate anomaly structures; capturing a label image using a visual sensor, which includes the simulated anomaly structures; creating labels for the anomaly structures based on the captured label image; displaying an environmental sensor datum from an environmental sensor of a motor vehicle using a display device; capturing an overlay image using the visual sensor, which includes the simulated anomaly structures and the displayed environmental sensor datum;Creating a training dataset with labeled anomalies based on the overlay image and the created labels for the anomaly structures.;

[0007] An anomaly in image data is understood to mean, in particular, missing or incorrect data that has arisen due to a visual, at least partial, blockage of the sensor during image generation. A blockage of a motor vehicle camera can, for example, be caused by water droplets on the windshield in front of the camera. Alternatively, dirt, mud, or other contaminants, possibly in combination with lighting effects, can lead to such blockages. In the case of such visual blockages, no or incorrect data or information is present in the corresponding image area of ​​the recorded image. This can also cause ambiguities and / or distortions in the corresponding image area of ​​the recorded image.

[0008] According to the invention, a blockage of the visual path is physically simulated. For this purpose, visual blockages (e.g., water droplets) are applied to a substantially horizontal plane, which, if captured by a sensor, could lead to an anomaly in the recorded image data. In this way, a first set of visual blockages is created, which lead to a first set of anomalies in an image. The terms "anomalies" and "visual blockages" are therefore to be distinguished from one another, but due to their interrelated nature, they may be used analogously in some cases.

[0009] This described, essentially horizontally mounted plane is located in the visual path between a camera (visual sensor) and a display device (screen). The camera can therefore record both the generated visual blockages and the images displayed on the display device. This creates an overlay image in which the displayed image (e.g., a previously captured image of a real driving scene) is overlaid with the simulated anomaly structures (originating in the image data). To create a training dataset, a large number of displayed images can be modified with the first set of visual blockages. It is also advantageous to generate several different sets of visual blockages (and thus anomaly structures) and use them to create the training dataset.

[0010] It is further provided that the reproduced anomalies are marked (labeled). The creation of the labels using the marking image can take place in a prior process step (with regard to the creation of the overlay image). For this purpose, labels for the anomalies are created in a separate step for each set of visual blockages (i.e., reproduced anomaly structures). For this purpose, a so-called marking image is first recorded using a visual sensor, which includes the first set of reproduced anomaly structures. For example, the marking image is recorded with a black or white background, i.e., the display device shows a monochrome image. In an alternative embodiment, the display device can also display a defined pattern, e.g., a grid. Based on the marking image, the labels for the reproduced anomalies are determined in the image data.Based on these anomaly labels, a so-called disturbance map is created, on which all identified and relevant anomalies are entered. The process advantageously includes the creation of a disturbance map in a subsequent step.

[0011] Subsequently, a training dataset with labeled anomalies is created based on the recorded overlay image, which includes the simulated anomaly structures and the recorded image of the real driving scene, as well as the labels for the anomaly structures created based on the labeling image. A training dataset typically consists of a plurality of images (data). The training dataset can consist entirely of the labeled images generated according to the invention or, alternatively, can also contain additional images (datasets), which, for example, include real anomalies recorded from the vehicle. Furthermore, the training dataset can contain images that do not contain any corresponding anomalies – and therefore do not have any such labels.

[0012] In an advantageous embodiment, the method comprises the method step: merging the recorded overlay image and the created labels for the simulated anomaly structures.

[0013] This means that the information regarding the simulated anomaly structures (which were extracted, in particular, from the labeling image) is transferred to the overlay image. This results in a correlation between the labels created for the anomaly structures in the labeling image and the surrounding anomalies in the acquired overlay image. For example, a spatial correlation is performed, so that the labels of one image (labeling image) are related to the same spatial position in the other image (overlay image).

[0014] In one possible embodiment, the method is characterized in that the creation of labels for the simulated anomaly structures is carried out on the basis of the recorded labeling image, wherein the labeling image comprises the simulated anomaly structures and an auxiliary datum displayed by means of the display device.

[0015] This means that the identification image contains both the simulated anomaly structures and another visual means. For this purpose, an auxiliary data item (i.e. a data set that generates an auxiliary image) is displayed using the display device. Such an auxiliary image is designed to support the identification of anomalies. Based on the displayed auxiliary image, both the detection and classification of the anomalies can be improved. The following can be used as an auxiliary image, for example: a monochrome background on which the simulated anomalies stand out clearly. Alternatively or additionally, defined structures can also be used as an auxiliary image. For example, the auxiliary image can show a grid structure. In this sense, the identification image comprises a grid structure overlaid with the simulated anomaly structures.As an additional method step, it is advantageously provided that an auxiliary date is displayed by means of the display device.

[0016] In a preferred embodiment, the method is characterized in that the creation of a training data set with labeled anomalies is carried out on the basis of a plurality of recorded overlay images and the created labels for the simulated anomaly structures.

[0017] This means that the visual blockages (or the resulting simulated anomaly structures) are used not only to modify a single image of, for example, a real driving scene, but to modify a multitude of different images of real driving scenes. In this sense, the method provides for: displaying a plurality of environmental sensor data from an environmental sensor of a motor vehicle using the display device, capturing a plurality of overlay images using the visual sensor, which include the simulated anomaly structures and the displayed environmental sensor data, and creating a training data set with labeled anomalies based on the captured overlay images and the labels created for the first anomaly structures.

[0018] In an alternative development, the method is characterized in that a training data set with labeled anomalies is created on the basis of a plurality of different, simulated anomaly structures.

[0019] This means that several different visual blocks (or the resulting different sets of anomaly structures) are used to create a training dataset. Accordingly, the method comprises generating a first set of visual blocks (or anomaly structures) and generating a second (e.g., x-th) set of visual blocks (or anomaly structures). In this sense, the method further provides for creating a training dataset with labeled anomaly structures based on a plurality of different sets of image data, wherein each set of image data is based on its own simulated anomaly structure. Furthermore, each set of image data advantageously comprises several overlay images. For each set of image data, the required label information can also be generated using a separate labeling image.

[0020] In an advantageous embodiment, the method is characterized in that the creation of the visual blocks for simulating the anomaly structures is carried out by applying water and / or a suspension and / or a solid.

[0021] In one possible embodiment, the method is characterized in that the generation of the visual blocks for simulating the anomaly structures is carried out in such a way that the simulated anomaly structures cover only a defined part of the horizontal plane.

[0022] This means that the visual blockages (or anomalous structures) do not cover the entire visual path of the visual sensor. Rather, the visual blockages are intended to occupy only a defined area. Furthermore, the visual blockages can be reproduced only in a defined region or at a defined position.

[0023] The approach presented here further provides a device configured to perform, control, or implement the steps of a variant of a method presented here in corresponding devices. This embodiment of the invention in the form of a device also allows the problem underlying the invention to be solved quickly and efficiently.

[0024] In an advantageous embodiment, the device for generating training data for an artificial neural network for classifying anomalies in image data, in particular for classifying visual blockages of an environmental sensor of a motor vehicle, is characterized in that the device comprises: a substantially horizontal plate with generated visual blockages for simulating anomaly structures; a display device configured to display environmental sensor data of an environmental sensor of a motor vehicle; a visual sensor configured to record a label image comprising the simulated anomaly structures and a second overlay image comprising the simulated anomaly structures and the displayed environmental sensor data;an evaluation device configured to create labels for the simulated anomaly structures based on the recorded labeling image and to create a training data set with labeled anomalies based on the recorded overlay image and the created labels for the simulated anomaly structures;

[0025] This means that the surface of the plate on which the visual blockages are simulated is essentially horizontal. Furthermore, it is advantageously a transparent or at least semi-transparent plate, so that both the visual blockages on the plate (or the resulting simulated anomaly structures) and the image displayed by the display device can be recorded by the visual sensor. The plate is advantageously made of glass or Plexiglas.

[0026] In one possible embodiment, the device is characterized in that the device comprises: a mirror which is mounted and arranged such that it directs an image displayed by the display device into a detection area of ​​the visual sensor.

[0027] In a preferred embodiment, the device is characterized in that the visual sensor and the substantially horizontal plate are mounted and arranged such that a distance between the visual sensor and the substantially horizontal plate does not exceed a defined maximum distance, in particular is not greater than 10 centimeters, preferably not greater than 5 centimeters.

[0028] Also advantageous is a computer program product or computer program with program code that can be stored on a machine-readable carrier or storage medium such as a semiconductor memory, a hard disk memory or an optical memory and is used to carry out, implement and / or control the steps of the method according to one of the embodiments described above, in particular when the program product or program is executed on a computer or a device. Embodiments

[0029] It should be noted that the features listed individually in the description can be combined with one another in any technically reasonable manner and demonstrate further embodiments of the invention. Further features and usefulness of the invention will become apparent from the description of exemplary embodiments with reference to the accompanying figures.

[0030] From the figures shows: Fig. 1 shows a representation of a first embodiment of the invention; and Fig. 2 shows a representation of a second embodiment of the invention; and Fig. 3 generating the labels in an overlay image; and Fig. 4 shows the method steps of an embodiment of the invention.

[0031] Fig. 1 shows a representation of a first embodiment of the invention. It shows an overall device for generating training data for an artificial neural network for classifying anomalies in image data. The overall device comprises a visual sensor 1 (camera). Furthermore, the overall device comprises a display device 4 (screen). Environmental sensor data (images) recorded by an environmental sensor (e.g., a front camera) in a motor vehicle in the actual driving environment are displayed on the display device 4. The visual sensor 1 and the display device 4 are positioned and aligned with one another in such a way that the visual sensor 1 can re-record the images displayed on the display device 4.

[0032] A transparent plate 2 is also shown. This is, for example, a glass pane or a Plexiglas pane. The transparent plate 2 is positioned between the visual sensor 1 and the display device 4. The plate 2 is essentially horizontally aligned. The horizontal alignment of the plate 2 requires a corresponding positioning and alignment of the visual sensor 1 and the display device 4, so that the visual sensor 1 can also accommodate the plate 2 largely in its entirety.

[0033] Visual blockages 3 are applied to the plate 2, resulting in the desired simulated anomaly structures 8 in the image data. For example, water droplets are applied as visual blockages 3 on the plate 2. These visual blockages represent obstructions in the field of view of the visual sensor 1. This means that the simulated anomaly structures resulting from the visual blockages 3 overlay part of the image fully displayed by the display device 4 when the image is captured by the visual sensor 1.

[0034] The overall device further comprises a computer 5. The computer 5 can be configured to perform various tasks. Firstly, the computer 5 can control the display device 4 in order to display an auxiliary image which is suitable for detecting anomaly structures in the image data. Alternatively, the computer 5 can, for example, determine existing anomalies in the image data from an identification image (i.e., from an auxiliary image with superimposed anomaly structures) and define corresponding labels and / or create a so-called disturbance map based thereon. Furthermore, the computer 5 can control the display device 4 in order to display existing environmental sensor data from an environmental sensor. Alternatively or additionally, the computer 5 can, for example, combine the recorded overlay image and the created labels or the disturbance map.In addition, the computer 5 can be configured to train the neural network based on the generated training data.

[0035] Fig. Figure 2 shows a representation of a second embodiment of the invention. It shows an overall device for generating training data for an artificial neural network for classifying anomalies in image data. Reference is made in particular to the above explanations. Fig. 1. In contrast, the execution in Fig. 2 has an additional reflection device 6 (e.g., a mirror). The reflection device 6 makes it possible to advantageously change the positioning and orientation of the display device 4. The reflection device 6 allows the field of view of the visual sensor 1 to be varied, so that the orientation of the visual sensor 1 and the display device 4 can be decoupled from each other. This makes it possible, for example, as in Fig. 2, to align the display device 4 vertically.

[0036] Fig. 3 shows the generation of labels in an overlay image. In Fig. 3a shows a marking image captured by the visual sensor 1. The grid structure 7 is visible in the background. The grid structure 7 is displayed by the display device 4 and captured by the visual sensor 1. However, a device (transparent plate) with applied visual blockages (water droplets) is located in the optical path between the display device 4 and the visual sensor 1. The visual sensor 1 accordingly captures the displayed marking image, which reproduces the grid structure 7, as well as the visual blockages that result in the anomaly structures 3a to 3e in the image data. The grid structure 7 supports the detection and classification of the simulated anomalies 3a to 3e.

[0037] Based on the identified anomalies 3a to 3e, a so-called disturbance map (overview of the existing anomalies) is created. Fig. 3b. The disturbance map can be used as a template to label the anomalies present in the overlay images.

[0038] This is in Fig. 3c. This shows an overlay image along with the labels added according to the disturbance map. The overlay image was generated by displaying an image of a real driving scene using the display device 4, wherein the image was captured, for example, with a front camera of a motor vehicle. The displayed image is recaptured with the visual sensor 1, with the anomaly structures 3 still being introduced in the optical path between the visual sensor 1 and the display device 4. Accordingly, the representation of the real driving scene with the introduced anomalies is overlaid in the overlay image. This overlay image is then correlated with the disturbance map so that the labels created from the disturbance map are transferred to the overlay image.

[0039] In Fig.Figure 4 shows a representation of the method steps of one embodiment of the invention. The method is started in a first step S1. In a next step S2, a substrate with initial visual blocking devices 3 is created. For example, several water drops are applied to a transparent glass plate.

[0040] In a next step S3, the simulated anomalies arising in the image data are labeled. For this purpose, an auxiliary image is displayed, for example, in a step S3a using the display device 4. The auxiliary image consists, for example, of a known geometric structure, e.g., a grid. In a next step S3b, the auxiliary image is recorded again using the visual sensor 1 through the carrier with the visual blockages 3. The image thus created in the visual sensor 1 shall be referred to as the labeling image. This labeling image is then evaluated to identify and classify the anomalies in the image. The so-called disturbance map can be created from the information obtained in a step S3c.

[0041] In step S4, an overlay image is generated. For this purpose, the environmental sensor data of an environmental sensor available on the display device 4 is displayed. This means that images depicting recordings of real driving scenes are displayed on the display device, which were previously recorded, for example, by a front camera of a motor vehicle. These displayed images are re-recorded by the visual sensor 1 through the carrier with the first visual blockages 3. The image thus created in the visual sensor 1 shall be referred to as the overlay image. It corresponds to the image of the recorded driving scene, but with the introduced anomalies present.

[0042] In step S5, the overlay image and the generated labels are merged. For example, the overlay image and the generated disturbance map are overlaid. This allows the corresponding labels from the disturbance map to be transferred to the overlay image.

[0043] In a condition B1, it is checked whether further environmental sensor data is available that should be converted into overlay images. If this is the case (Y-branch), the process continues with step S4. If this is not the case (N-branch), a further condition B2 can be checked whether further, alternative visual blockages are present or should be created. If this is the case (Y-branch), the process continues with step S2 and a carrier with second (or x-th) anomaly structures 3 is created. If this is not the case (N-branch), the training data set can, for example, be finalized in a step S6. Furthermore, on the basis of the training data set, training for an artificial neural network for classifying anomalies in image data can take place, in particular for classifying visual blockages of an environmental sensor of a motor vehicle. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] WO 2022 / 128014

[0002] DE 10 2020 215 656 A1

[0004]

Claims

[1] Method for generating training data for an artificial neural network for classifying anomalies in image data, in particular for classifying anomalies in image data of an environmental sensor of a motor vehicle, comprising the method steps: • Creating visual blocks (3) on a substantially horizontal plane to replicate anomaly structures (8); • Recording a marking image by means of a visual sensor (1) which comprises the simulated anomaly structures (8); • Creating labels for the simulated anomaly structures (8) based on the recorded labeling image; • Display of an environmental sensor data of an environmental sensor of a motor vehicle by means of a display device (4); • Recording an overlay image by means of the visual sensor (1), which includes the simulated anomaly structures (8) and the displayed environmental sensor data; • Creating a training dataset with labeled anomalies based on the overlay image and the created labels for the simulated anomaly structures (8). [2] Method according to claim 1, comprising the step of: • Merging the acquired overlay image and the created labels for the simulated anomaly structures (8). [3] Method according to one of the preceding claims, characterized by that the creation of labels for the simulated anomaly structures (8) is carried out on the basis of the recorded identification image, wherein the identification image comprises the simulated anomaly structures (8) and an auxiliary datum displayed by means of the display device (4). [4] Method according to one of the preceding claims, characterized by that the creation of a training data set with labeled anomalies is based on a plurality of recorded overlay images as well as the created labels for the simulated anomaly structures (8). [5] Method according to one of the preceding claims, characterized by that the creation of a training data set with labeled anomalies is based on a plurality of different, simulated anomaly structures (8). [6] Method according to one of the preceding claims, characterized by that the creation of the visual blocks (3) for simulating the anomaly structures (8) is carried out by applying water and / or a suspension and / or a solid. [7] Method according to one of the preceding claims, characterized bythat the creation of the visual blocks (3) for simulating the anomaly structures (8) is carried out in such a way that the simulated anomaly structures (3) cover only a defined part of the horizontal plane. [8] Device (1, 2, 4, 5, 6) arranged to carry out the method according to one of claims 1 to 7. [9] Device according to claim 8 for generating training data for an artificial neural network for classifying anomalies in image data, in particular for classifying anomalies in image data of an environmental sensor of a motor vehicle, characterized by that the device comprises: • a substantially horizontal plate (2) with generated visual blocks (3) for simulating anomaly structures (8); • a display device (4) which is configured to display an environmental sensor data item of an environmental sensor of a motor vehicle; • a visual sensor (1) configured to capture a label image comprising the simulated anomaly structures (8) and to capture an overlay image comprising the simulated anomaly structures (8) and the displayed environmental sensor data; • a computer (5) which is configured to create labels for the simulated anomaly structures (8) on the basis of the recorded labeling image and to create a training data set with labeled anomalies on the basis of the recorded overlay image and the created labels for the simulated anomaly structures (8). [10] Device according to claim 9, characterized by that a mirror which is mounted and arranged so that it directs an image displayed by the display device into a detection area of ​​the visual sensor. [11] Device according to claim 9 to 10, characterized bythat the device comprises: the visual sensor and the substantially horizontal plate are mounted and arranged such that a distance between the visual sensor and the substantially horizontal plate does not exceed a defined maximum distance, in particular is not greater than 10 centimeters, preferably not greater than 5 centimeters. [12] A computer program which, when executed by a device according to any one of claims 8 to 11, is adapted to carry out the method according to any one of claims 1 to 7. [13] A machine-readable storage medium on which the computer program according to claim 12 is stored.

Images