Method, computer program, data storage unit and system for the automatic detection of damage events using image material

Abstract

Methods for the automatic detection of damage events (1, 11) using image material, comprising the steps: a) Creating image material of at least part of an area (2, 12) exhibiting damage (1, 11), b) Analyzing the image material using an artificial neural network, whereby c) a plurality of bounding boxes (3, 4, 7) is generated, d) a two-dimensional density function is created from the bounding boxes (3, 4, 7), and e) the damage is located and classified using the two-dimensional density function, whereby The classification of the damage (1, 11) is based on reference damage data stored in a database and includes a technical categorization of the damage (1, 11).

Description

[0001] The present invention relates to a method, a computer program, a data storage unit and a system for the automatic detection of damage events based on image material.

[0002] In the event of damage, it is usually necessary to record and assess the damage in order to estimate the prospects and costs of repairs. Such recording and assessment of damage is usually carried out through an expert assessment, which involves considerable effort (time, deadlines, documentation, etc.) and expense. By recording and documenting damage using visual material such as photos or videos, the deployment of experts can be effectively organized and optimized in terms of effort and duration, and this has long been common practice. The conventional localization and classification of body damage based on images or video sequences often requires specialist knowledge, is extremely time-consuming, and therefore not real-time capable.

[0003] A well-known option for (semi-)automatic damage recording and documentation is the use of programs and systems that use image processing to analyze and evaluate image material of a damage incident. One well-known method involves examining possible damage areas in the image material using so-called bounding boxes, with each of the bounding boxes being weighted according to the damage potentially present therein. Such a bounding box generally provides a method or option for marking or enclosing objects using a tuple of 2D coordinates (x1, y1, x2, y2). From a plurality of such weighted bounding boxes, the bounding box with the highest probability of containing damage is then determined using non-maximum suppression.A disadvantage of this known method is that a model used for the calculation has to select a specific bounding box within a specific region for the calculation and any additional damage that may be present, even if it is of a lesser extent, cannot be detected and processed or can only be detected with difficulty.

[0004] Real-time capability is relevant wherever either people or machines, e.g. drones or vehicle scanners, document damage, especially bodywork damage, and are to be guided when recording the damage images.

[0005] The article "INSPECT-ONLINE: Assistance Systems for Assessing the Sewer System - Perspectives from Image Analysis and Machine Learning in Image Analysis, March 7, 2019" describes an inspection system for monitoring and detecting damage in sewer networks. A robot equipped with a multi-camera system travels through the sewer network and generates 3D data. The inspection robot records the data cyclically as it travels through the sewer network, generating images that largely overlap and are then combined into a 3D model. A bounding box is generated around the damage for damage detection and classification, as well as for training the neural network.

[0006] The Wikipedia article "Heatmap" (edited on October 16, 2018, 3:54 p.m. UTC. URL: https: / / de.wikipedia.org / w / index.php?title=Heatmap&oldid=181851051) describes a heatmap as a diagram for visualizing data based on a function that can be used to map a two-dimensional set of definitions onto the number line and display it in color.

[0007] The Wikipedia article “Minimum bounding box” (edited on August 1, 2019, 10:05 UTC. URL: https: / / en.wikipedia.org / w / index.php? title=Minimum_bounding_box&oldid=908839052) describes a bounding box with the smallest possible size (area or volume) that nevertheless encloses all points or geometries.

[0008] DE 10 2009 040 755 A1 describes a method for automatically detecting a defect in an electronic representation of an object. In a training phase, the dimensions of an analysis section are determined from a defect-free electronic representation. The defect-free representation is overlaid with the analysis section to determine a reference equivalent representation of the overlay area. In a subsequent test phase, the representation of an object to be tested is overlaid with the analysis section, and a test equivalent representation of the overlay area is determined. By comparing the test equivalent representation with the reference equivalent representation, a deviation between the two equivalent representations is determined.

[0009] The article "SAGAR, R.: How The Deep Learning Approach For Object Detection Evolved Over The Years. In: AIM Developers Corner. August 26, 2019" covers advances in object detection using deep learning. It explains how image processing algorithms have evolved significantly to tackle complex tasks such as depth detection in videos and object detection in ambiguous images. Traditional methods based on support vector machines (SVMs) have been gradually replaced by deeper neural networks that can generate hierarchical features and process various scale information. Two-stage detectors such as R-CNN and Faster R-CNN offer better detection accuracy, while single-stage detectors such as YOLO and SSD are optimized for real-time applications. The paper also explains how newer architectures such as RetinaNet solve the class imbalance problem and how mask R-CNNs are used for segmentation.Finally, future challenges such as the development of anchor-free methods and improving data augmentation are discussed. The text also highlights the importance of contextual reasoning and the optimization of deep learning models.

[0010] Based on the above-mentioned prior art, the object of the invention is to provide a method with which the disadvantages of the prior art described above can be eliminated or at least reduced, in particular to provide a method, a computer program, a data storage unit and a system for the automatic detection, evaluation and documentation of damage events using image material, which enables a safe, accurate and complete recording of even complex damage cases without the mandatory involvement of an expert.

[0011] The object is achieved according to the invention by the features of the independent claims. Advantageous embodiments of the invention are specified in the subclaims.

[0012] Within the scope of the method according to the invention, a set / plurality of possible bounding boxes are generated for the available image material with the aid of a computer unit / computer system and the neural network, each as a suggestion for possible damage areas. This is done based on the analysis of the image material, for example by subjecting the image material to image recognition / image processing. The image material can be analyzed using various filters, with at least one image feature being examined. As an image feature, for example, a statistical brightness distribution in the image can be analyzed and, if necessary, a histogram can be determined. Other image features of the image material examined within the scope of the invention can be its entropy and / or average brightness. In addition, color distribution, color gradient, contrasts, contrast gradients and / or brightness or darkness and their gradients can be analyzed.The image material can also be subjected to object detection and / or object tracking, for example with OpenCV.

[0013] While it is known to select the bounding box most likely to cause damage from the generated set of the majority of bounding boxes using non-maximum suppression and to suppress other, neighboring bounding boxes, according to the invention all generated bounding boxes (suggestions for possible damage areas) are processed by the system. Since the position of each bounding box relative to the image material and thus also to the actual area to be assessed is known, all bounding boxes can preferably be summed position-related. The result of such processing is then a type of two-dimensional, i.e. position-related, superposition of the bounding boxes, from which a two-dimensional density function, also referred to as a 2D density function, is created for the majority of the initially determined bounding boxes. In particular, the system can return a score value 0<=s<=1 for each of the bounding boxes.The bounding box area can then be replaced by a suitable kernel, such as a Gaussian kernel, which can then be weighted with the associated score. The weighted kernels can then be summed and, after normalization, yield the 2D density function. In this way, a 2D density function can be generated even though the system has never been trained on it.

[0014] In summary, it can be said that with the help of the invention, damage to almost any structure, in particular body damage, can be automatically localized in images or video sequences and the type(s) of damage can be classified. It is particularly advantageous that the localization of the damage and the probability of the presence of damage can be realized with the help of a neural network or neural networks and, if necessary, can also be displayed graphically. In particular, the invention can be used to localize and display damage, in particular body damage, on a motor vehicle using a photo or a video sequence. Since neural networks are capable of learning complex facts such as body damage and require far less than a second to predict them, they are extremely well suited to solving the problem described above.A particular advantage of the invention is that any type of damage, particularly to machinery, buildings, and municipal infrastructure, especially vehicle body damage, can be easily and standardized recorded using a guided damage recording process. Another particular advantage of the invention is that it enables automatic, secure, accurate, and complete detection, assessment, and documentation of damage events based on image material, especially complex and extensive damage cases, without the mandatory involvement of an expert. Finally, the invention enables real-time damage detection, assessment, and documentation.

[0015] The invention is therefore particularly well suited not only for assessing damage in order to estimate its extent and associated costs, but in particular for the (semi-)automatic technical detection and technical assessment of damage, for example in order to be able to estimate failure risks (e.g. the probability of future failure of a component) and repair options.

[0016] In an advantageous embodiment of the invention, at least the method steps b), c), d) and optionally e) are carried out by means of a computer unit, in particular automatically by means of a computer unit.

[0017] A particular advantage of the invention is that it can be used to detect and analyze damage to a structure such as a building, a machine, or municipal infrastructure. In preferred embodiments, the invention can be used in particular to locate and analyze damage to a part of a motor vehicle, an aircraft, or a watercraft, particularly a body.

[0018] In an advantageous embodiment of the invention, the image material can comprise at least one photograph or a video sequence. The image material is preferably in the form of image data. However, the invention also encompasses the processing of conventional analog image material in the form of photo or film prints, which can then be read / scanned within the scope of the invention, thus creating processable data using the neural network.

[0019] According to a further advantageous embodiment, a heat map is generated in the method of the invention after step d). Such a heat map offers a particularly user-friendly way of displaying the density function of bounding boxes determined by means of the invention. In particular, the heat map can be displayed superimposed on the image material underlying the calculation, so that a user can immediately associate the determined imaginary damage pattern with the real situation. In particular, the heat map display can be designed such that the brightness of the colors relates to the probability of damage being present (the brighter the display, the higher the probability of damage).A user can thus check the result of the damage detection according to the invention in a particularly simple manner either by examining the image material on which the calculation is based or by examining the original situation.

[0020] In an advantageous embodiment of the invention, in a further method step following method step c), each bounding box of the plurality of bounding boxes is weighted. The weighting can be determined, in particular, with regard to the respective probability of damage occurring. According to the invention, the result of this weighting is then analyzed and summarized. In particular, weighting values ​​created for the bounding boxes can be summed. In addition or alternatively to the representation of the density function of the bounding boxes, such weighting values ​​can be used as the basis for a heat map to be created and used for the calculation.

[0021] The invention involves classifying the damage based on reference damage data. Such reference damage data is stored in a database and includes a technical categorization of the damage. In particular, analyzed and classified image data from previous damage can be used as reference damage data. For example, body damage can be classified according to type and / or severity, such as light scratches, heavy scratches, light dents, deep dents, creases in the body panel, or cracks in the body panel.

[0022] Such reference damage data as described above can also be used in one embodiment of the invention to determine a degree of confidence that the damage is indeed present using the neural network. In this way, deficiencies in the image material underlying the damage assessment, such as blurriness, insufficient exposure and / or overexposure, and reflections, can be taken into account, and resulting errors can be compensated for or avoided.

[0023] In an advantageous embodiment of the invention, the image material can be subjected to an object recognition process. In this way, damage determination can be simplified, for example, by the system recognizing that the area depicted in the image material is a vehicle tire or a specific part of the vehicle body, such as a front fender or a vehicle door or a hood or tailgate. Based on such object recognition, the system can then access a corresponding object-related database, in particular an object-related damage database. Such object recognition also offers the possibility of tailoring the invention to specific user-specific requirements, for example, to specific types (vehicle types, tire types) and / or designs of a specific manufacturer.

[0024] The invention will be explained in more detail below with reference to preferred embodiments and the accompanying drawings. The features presented may represent an aspect of the invention both individually and in combination. Features of various embodiments are transferable from one embodiment to another.

[0025] It shows Fig. 1 a view of a photograph of damage to a motor vehicle tyre, Fig. 2 a view of the photo of the Fig. 1 in superposition with a plurality of bounding boxes, Fig. 3 a view of the photo of the Fig. 1 in superposition with a single bounding box according to a known method, Fig. 4 a view of the photo of the Fig. 1 in overlay with a heatmap created using the majority of bounding boxes within the scope of the invention, Fig. 5 a processed view of the Fig. 4 with contours of a density function, Fig. 6 a view of another photograph of damage to a motor vehicle body, Fig. 7. a view of the photo of the Fig. 6 in superposition with a heat map created using the majority of bounding boxes within the scope of the invention, Fig. 8 a processed view of the Fig. 7, and Fig. 9 a representation of an embodiment of the method according to the invention in a schematic diagram.

[0026] Fig. Figure 1 shows a photo of damage 1 on a motor vehicle wheel 2 as an example of a damage event that can be fully or at least partially automatically detected by the invention. Within the scope of the method according to the invention, the photo of the wheel 2 is taken in method step a). Fig. 1 was created as image material. Image data from the photo was stored in a computer unit and is used for the further implementation of the method according to the invention.

[0027] The photo of the Fig. 1 or its image material data are analyzed according to the invention in a method step b) using an artificial neural network. In a method step c), a plurality of bounding boxes 3, 4 and 7 are generated. In the present exemplary embodiment, each of the bounding boxes 3, 4, 7 is generated based on certain image features, such as changes in brightness in the photo. It can be seen that a first bounding box 4 is caused by lightened areas 5 of the rubber on the wall of a tire 6 of the wheel 1, which areas 5 are characterized by a relatively strong light-dark contrast between the color of the rubber and the lightened areas 5. A number of further bounding boxes 7 (see Fig. 2) is caused by image features such as relatively dark scratches 8 on the edge of a rim 9 of the wheel 1, which are characterized by a relatively strong light-dark contrast with the color of the areas of the rim adjacent to the scratch 8. Finally, another plurality of bounding boxes 3 is caused by a crack 10 in the sidewall of the tire 6, which, due to its deep black color, stands in a strong light-dark contrast with the gray color of the adjacent areas of the tire 6.

[0028] Fig. Figure 3 shows further processing of the bounding boxes 3, 4, 7 in a known method, wherein the bounding boxes 3, 4, 7 are evaluated according to their probability of indicating damage, and the bounding box 3 with the highest damage probability is output as detected damage, while the remaining bounding boxes 3, 4, 7 with a lower damage probability are suppressed within the framework of a non-maximum suppression. Differently, within the framework of the method according to the invention, in a method step d), a two-dimensional density function is created from the bounding boxes 3, 4, 7. This is calculated using a Fig. 4 is visualized in the heatmap 18 shown. According to the invention, the localization and classification of the damage is carried out using the created two-dimensional density function. Fig. 4 clearly shows that in significant improvement over the Fig. 3, all possible damages are displayed and visualized, namely the crack 10 in the tire 6 as a bright spot 19 as well as the scratches 8 on the rim 9 as a bright spot 20. The heatmap provides more and more gradual information than is the case with known methods using non-maximum suppression. A weighting of the damages is shown in the heatmap 18 of the Fig. 4 also included, namely the size of the colored display as well as the brightness / color of the marked areas. Fig. Figure 5 shows a graphical representation of the Fig. 4, where only contours of the density function are shown.

[0029] The Fig. Figures 6 to 8 show another example of damage 11 to a rear fender 12 of a motor vehicle. Damage 11 exhibits a crack 13 in the sheet metal, a scratched area 14, and a dented area 15. A missing fender cover also reveals a fastener 16.

[0030] Fig. 7 shows a heatmap 17 and Fig. 8 a graphical representation of the Fig. 7, where only contours of the density function are shown. It is clearly visible that the fastening 16 is marked as a less probable damage event with a relatively dark spot 21, while the dented area 15 and the scratched area 14 are marked as probable damage events with light spots 22 and 23, respectively. The crack 13 was not detected as damage due to the algorithm used to determine the damage. This can be remedied by a different algorithm that, for example, only considers dark contrasts.

[0031] Fig.9 shows a representation of an embodiment of the method according to the invention in a schematic diagram, wherein the step “localizing and classifying the damage” necessarily includes that the classification of the damage 1, 11 is carried out on the basis of reference damage data which are stored in a database and which comprises a technical categorization of the damage 1, 11. List of reference symbols 1 damage 2 motor vehicle wheel 3 Bounding Box 4 Bounding Box 5 brightened area 6 tires 7 Bounding Box 8 scratches 9 rim 10 crack 11 Damage 12 rear fenders 13 crack 14 scratched area 15 dented area 16 Fastening 17 Heatmap 18 Heatmap 19 spots in heatmap 20 spots in heatmap 21 Spot in Heatmap 22 Fleck in Heatmap 23 Fleck in Heatmap

Claims

[1] Method for the automatic detection of damage events (1, 11) based on image material, comprising the steps: a) creating image material of at least part of an area (2, 12) exhibiting damage (1, 11), b) analyzing the image material using an artificial neural network, whereby c) a plurality of bounding boxes (3, 4, 7) are generated, d) a two-dimensional density function is created from the bounding boxes (3, 4, 7), and e) the damage is localized and classified using the two-dimensional density function, where the classification of the damage (1, 11) is carried out on the basis of reference damage data stored in a database and includes a technical categorisation of the damage (1, 11). [2] Method according to claim 1, characterized bythat at least the method steps b), c), d) and optionally e) are carried out by means of a computer unit, in particular are carried out automatically by means of a computer unit. [3] Method according to claim 1 or 2, characterized by that the damage (1, 11) is present on a structure such as a building, a machine, municipal infrastructure, in particular on a part of a motor vehicle, an aircraft or a watercraft, in particular a body (12) or a vehicle wheel (2). [4] Method according to one of the preceding claims, characterized by that the image material includes at least one photo or video sequence. [5] Method according to one of the preceding claims, characterized by that after step d) a heatmap (17, 18) is generated. [6] Method according to one of the preceding claims, characterized bythat in a further method step after method step c) a plurality of the bounding boxes, preferably each bounding box (3, 4, 7) of the plurality of bounding boxes (3, 4, 7) is weighted. [7] Method according to one of the preceding claims, characterized by that the neural network determines a confidence that there is really damage (1, 11). [8] Method according to claim 7, characterized by that the image material is subjected to an object recognition process. [9] Computer program for locating and classifying damage (1, 11), designed to carry out a method according to one of the preceding claims. [10] Data storage unit for locating and classifying damage (1, 11), designed to carry out a method according to one of the preceding claims. [11] System for locating and classifying damage (1, 11), designed to carry out a method according to one of the preceding claims.

Images