Method and device for annotating images of vehicles, in particular of motor vehicles

Abstract

A device (2) for annotating images of vehicles (4), in particular of motor vehicles, comprises at least one stationary image capturing device (6) which is designed to capture at least one global image of a vehicle (4); at least one mobile image capturing device (8) which is designed to capture at least one local image of at least one sub-region (10a, 10b) of the vehicle (4); and an annotation device (12). The annotation device (12) is designed to receive the at least one global image and the at least one local image and to identify and annotate, in the global image, the at least one sub-region (10a, 10b) of the vehicle (4) contained in the at least one local image.

Description

[0001] Description

[0002] title

[0003] Method and device for annotating images of vehicles, in particular motor vehicles

[0004] The invention relates to a method and a device for annotating images of vehicles, in particular images of motor vehicles.

[0005] State of the art

[0006] Commercial and / or rental vehicles, in particular, must be visually inspected regularly to document their condition and detect damage early. This inspection can be automated using artificial intelligence methods, particularly neural networks. To obtain high-quality results, annotated training data is required for training neural networks.

[0007] Software programs are used to annotate images. They present a human operator with an image to be annotated. These programs provide the operator with tools that allow them to mark parts of the image, which are then saved as annotations for that image. The tools also allow them to zoom into specific areas of the image and / or make adjustments to the image itself, e.g., to increase contrast. Algorithms can also be implemented that, for example, automatically detect an edge and suggest certain prominent annotations, which the operator can confirm or reject.

[0008] US 2020 / 019 799 A1 discloses a method for annotating image sequences, and DE 10 2016 203 850 A1 discloses a method for annotating images of plants. It is an object of the invention to improve, in particular simplify, the annotation of images of motor vehicles and to increase the quality of the annotations.

[0009] Disclosure of the invention

[0010] The invention comprises a method for annotating images of vehicles, in particular motor vehicles, wherein the method comprises capturing at least one global image of a vehicle with at least one stationary image capturing device; capturing at least one local image of a partial area of ​​the vehicle with a mobile image capturing device; and identifying and annotating the partial area of ​​the vehicle contained in the at least one local image in the global image.

[0011] The invention also encompasses a device for annotating images of vehicles, in particular motor vehicles, comprising at least one stationary image recording device configured to record at least one global image of a vehicle, a mobile image recording device configured to record at least one local image of at least one partial area of ​​the vehicle, and an annotation device. The annotation device is configured to receive the at least one global image and the at least one local image and to identify and annotate the at least one partial area of ​​the vehicle contained in the at least one local image in the global image.

[0012] The partial area of ​​the vehicle contained in the at least one local image can in particular contain a damaged area of ​​the vehicle, which is marked as a damaged area or damaged area by the annotation in the global image.

[0013] The inventive provision of an annotation by manually selecting an image area with a mobile image capture device is more intuitive than an annotation performed with the aid of software on an image of the vehicle, e.g., by dragging a box; particularly if the image was taken at an earlier time and possibly by another person. When implementing a method according to the invention, the vehicle in question can be directly inspected. This simplifies the detection of damage, as it can be viewed from different angles and distances, and additional sensory impressions, such as hearing and touch, can be incorporated.

[0014] The annotations generated according to the invention are directly available in two qualities, thus eliminating the need for a transfer of image properties from learning to inference.

[0015] According to the present invention, the annotations are determined using a method based on known manual processes, eliminating the need for separate annotation of the image data. The annotation step is translated from digital into a concrete action that can be performed intuitively and more quickly with the same or better quality.

[0016] A method according to the invention can be carried out particularly advantageously in locations with a high vehicle density, such as in depots, workshops, transshipment points, rental stations, etc.

[0017] With a method and a device according to the invention, the annotation of images of vehicles can be considerably simplified and the quality of the annotations can be significantly improved.

[0018] The invention also includes a method for training a neural network, the method comprising annotating a plurality of images of a plurality of vehicles using a method according to the invention and using the images thus annotated to train the neural network.

[0019] The quality of a neural network trained in this way can be significantly improved compared to a neural network trained with conventional annotated image data. In one embodiment, a method according to the invention comprises acquiring a first local image of a first partial region of the vehicle; acquiring a second local image of a second partial region of the vehicle, wherein the first partial region is contained in the second partial region; identifying the second partial region of the vehicle contained in the second local image in the global image; and identifying the first partial region of the vehicle contained in the first local image in the second local image and / or in the global image.

[0020] The intermediate step of locating the first sub-area in the second sub-area can facilitate the assignment of the first sub-area in the global image of the vehicle and improve the quality of the assignment.

[0021] The intermediate step of locating the first sub-area in a second sub-area can be performed each time a sub-area is recorded or optionally by processing the first image directly and requesting a second image showing a second sub-area only if the first image alone does not allow for a clear assignment.

[0022] In one embodiment, the method includes identifying the vehicle. The vehicle can be identified, in particular, using an image of its license plate if the license plate is included in one of the captured images. Alternatively, the license plate or other information that uniquely identifies the vehicle can be manually entered by an operator. Identifying the vehicle ensures that only images of the same vehicle are correlated.

[0023] In one embodiment, the method comprises receiving additional information about the identified partial area and storing it together with the global image as an annotation of the identified partial area. The additional information can be provided by the operator and entered into the mobile image capture device via a corresponding input device, from where it is transmitted to the annotation device. Using the additional information, for example, the recorded damage can be classified as a scratch, a dent, a crack, or another type of damage. The quality of the annotation can be further improved in this way.

[0024] In one embodiment, the mobile image capture device has an optical output device, in particular a screen. An optical output device enables the mobile image capture device to output optical information to the operator. For example, an image captured by the mobile image capture device or a preview image can be output via the optical output device.

[0025] Instructions can also be issued to the operator via the optical output device. Such instructions may include, for example, taking a second image of the damaged area that covers a larger portion of the vehicle than a previously taken first image.

[0026] The mobile image recording device may also have an acoustic output device, for example a loudspeaker, in order to be able to output acoustic information and / or instructions to the operator.

[0027] In one embodiment, the mobile image recording device comprises at least one sensor that makes it possible to determine the position and / or orientation of the mobile image recording device in space, and the annotation device is designed to use information provided by the at least one sensor to identify the partial area of ​​the vehicle contained in the at least one local image in the global image.

[0028] Using information about the position and / or orientation of the mobile image capture device, the area of ​​the vehicle in which the partial area captured by the mobile image capture device is to be searched can be restricted. This simplifies, accelerates, and improves the identification of the partial area of ​​the vehicle contained in the local image in the global image. The at least one sensor can be, for example, an inclination sensor, a magnetic sensor, and / or a GPS sensor.

[0029] In one embodiment, the device according to the invention comprises at least one lighting device configured to illuminate the vehicle to improve the quality of the captured images. The lighting device may contain one or more LEDs.

[0030] In the following, an embodiment of the invention is described with reference to the accompanying drawings.

[0031] Short description of the characters

[0032] Figure 1 shows a schematic side view of a vehicle and a device according to the invention for annotating images of vehicles.

[0033] Figure 2 shows a global image of the vehicle with a damaged area.

[0034] Figure 3 shows a first local image containing a first partial area of ​​the vehicle that includes the damaged area.

[0035] Figure 4 shows an example of a second local image of a second partial area of ​​the vehicle, wherein the first partial area is contained in the second partial area.

[0036] Character description

[0037] Figure 1 shows a schematic side view of a device 2 according to the invention for annotating images of vehicles 4 together with a vehicle 4.

[0038] The vehicle 4 can be a truck, as shown in Figure 1. However, the vehicle 4 can also be a car or another commercial vehicle, e.g. a bus. The vehicle 4 can also be a trailer not shown in the figures. The vehicle 4 can also be a rail-bound vehicle or an aircraft. A device 2 according to the invention comprises at least one stationary image recording device 6, in particular a camera 6, which is designed to record at least one global image of a vehicle 4 while the vehicle 4 is positioned in front of the device 2 according to the invention or drives past the device 2 according to the invention.

[0039] A device 2 according to the invention can also comprise a portal ("scan gate") 3 on which the at least one stationary image recording device 6 is mounted, so that images of the vehicle 4 can be recorded while the vehicle 4 passes under the stationary image recording device 6. In this way, the image recording can be carried out particularly quickly and efficiently.

[0040] Such portals 3 can, for example, be positioned at the entrances and exits of depots and similar enclosed areas in order to be able to record all vehicles 4 entering and leaving.

[0041] A device 2 according to the invention may also comprise at least one lighting device 5 designed to illuminate the vehicle 4 in order to improve the quality of the recorded images.

[0042] A device 2 according to the invention further comprises a mobile image recording device 8, which is designed to be handheld and operated by a human operator to record local images of partial areas 10a, 10b of the vehicle 4. Such partial areas 10a, 10b of the vehicle 4 can, in particular, be areas of the vehicle 4 in which conspicuous areas, e.g., damage, are visible.

[0043] A device 2 according to the invention further comprises a machine annotation device 12.

[0044] The annotation device 12 can be arranged in or on the stationary image recording device 6. The annotation device 12 can also be formed in the mobile image recording device 8 or separately and at a spatial distance from the stationary image recording device 6 and the mobile image recording device 8.

[0045] The annotation device 12 is designed to receive the global images provided by the stationary image recording device 6 and the local images provided by the mobile image recording device 8.

[0046] The annotation device 12 may, for example, contain a microprocessor on which a program runs to provide the desired functions of the annotation device 12.

[0047] The transmission of images from the stationary image capture device 6 and from the mobile image capture device 8 to the annotation device 12 can be wired, i.e., via cable, or wirelessly, e.g., via a WLAN data connection. To enable wireless data transmission, the stationary image capture device 6, the mobile image capture device 8, and the annotation device 12 can be equipped with appropriate transmitters and receivers. The transmitters and receivers are not explicitly shown in Figure 1.

[0048] The annotation device 12 is also configured to identify the vehicle 4 of which the images were taken. The identification of the vehicle 4 can be performed, for example, based on information contained in the captured images. The identification of the vehicle 4 can be performed, for example, based on its license plate number if this is included in at least one of the captured images.

[0049] The annotation device 12 is further configured to use image recognition methods to identify and annotate at least one partial area 10a, 10b of the vehicle 4 contained in a local image captured by the mobile image recording device 8 in the global image, i.e., to provide it with at least one annotation. The annotation can in particular comprise marking a partial area 10a, 10b contained in the local image as a defective area 15 of the vehicle 4. In particular, if the annotation device 12 is not able to unambiguously identify a first local image containing a first partial area 10a of the vehicle 4 in the global image, the annotation device 12 can be configured to request a second local image of a second partial area 10b of the vehicle 4 from the mobile image recording device 8.The second local image is selected such that the first sub-area 10a, 10b is contained in the second sub-area 10b.

[0050] Figure 2 shows an example of a global image of the vehicle 4 with a damaged area 15, which will be referred to below as the damaged area 15. The damaged area 15 is schematically represented in the figures as a triangle. The damaged area 15 can, for example, include a scratch, a dent, a crack, or another type of visible damage.

[0051] The damaged area 15 is located in a first sub-area 10a, which in turn is contained in a second, larger sub-area 10b.

[0052] Figure 3 shows an example of a first local image containing a first partial area 10a of the vehicle 4, which includes the damaged area 15. Figure 4 shows an example of a second local image of a second partial area 10b of the vehicle 4, wherein the first partial area 10a with the damaged area 15 is contained in the second partial area 10b.

[0053] The annotation device 12 is capable of identifying the second sub-area 10b of the vehicle 4, which is contained in the second local image, in the global image (see Figure 2). Since the second sub-area 10b is larger than the first sub-area 10a, the second sub-area 10b may be easier to identify in the global image than the smaller first sub-area 10a.

[0054] The annotation device 12 is then able to identify the first partial area 10a of the vehicle 4 contained in the first local image in the second local image and, via the second image or via the second partial area 10b, in the global image. To simplify the location of a partial area 10a, 10b in the global image, the mobile image recording device 8 can contain at least one sensor 14 that makes it possible to determine the position and / or orientation of the mobile image recording device 8 in space.

[0055] The at least one sensor 14 may, for example, comprise an inclination sensor, a magnetic sensor and / or a GPS sensor.

[0056] If the mobile image recording device 8 is equipped with at least one sensor 14, the annotation device 12 can be configured to use information provided by the at least one sensor 14 to identify the partial area 10a, 10b of the vehicle 4 contained in the at least one local image in the global image.

[0057] From information about the inclination of the mobile image recording device 8 relative to the horizontal during image recording, for example, the height of the partial area 10a, 10b recorded by the mobile image recording device 8 above the ground can be deduced, and the search can be restricted to this area.

[0058] With information about the orientation of the mobile image recording device 8 in a horizontal plane and / or with information about the current position of the mobile image recording device 8, the area of ​​the vehicle 4 in which the partial area 10a, 10b recorded by the mobile image recording device 8 is to be searched can be restricted to the front, to one side, and / or to the rear of the vehicle 4, if necessary, and the search can be restricted to this area.

[0059] The mobile image capture device 8 may also include an input device 20 that allows an operator to enter additional information into the mobile image capture device 8. This additional information may be transmitted to the annotation device 12 along with the captured images.

[0060] The additional information may, for example, include information about the

[0061] Vehicle 4, e.g., its license plate number, and / or information about the recorded partial area 10a, 10b and / or about the damaged area 15. Using the manually entered additional information, the damaged area 15 can be classified by the operator, for example, as a scratch, a dent, a crack, or another type of damage.

[0062] The mobile image capture device 8 may, for example, comprise a keyboard and / or a touch-sensitive screen.

[0063] The mobile image capture device 8 can also have an optical output device 16, in particular a screen. An optical output device 16 enables the mobile image capture device 8 to output information to the operator. For example, an image captured by the mobile image capture device 8 or a preview image of an image captured by the mobile image capture device 8 can be output via the optical output device 16.

[0064] At least one marker can be displayed in the preview image, which indicates to the operator an area in which the damaged area 15 is to be positioned during image acquisition in order to enable good identification and annotation.

[0065] Optionally, instructions can also be output to the operator via the optical output device 16. Such instructions can include, for example, taking a second image of the damaged area 15, wherein the second image contains a larger partial area 10b of the vehicle 4.

[0066] The mobile image recording device 8 may also have an acoustic output device 18, for example a loudspeaker, in order to be able to output acoustic information and / or instructions to the operator.

Claims

1 . A method for annotating images of vehicles (4), in particular motor vehicles, the method comprising: capturing at least one global image of a vehicle (4) with at least one stationary image capturing device (6); capturing at least one local image of at least one partial area (10a, 10b) of the vehicle (4) with a mobile image capturing device (8); and identifying and annotating the at least one partial area (10a, 10b) of the vehicle (4) contained in the at least one local image in the global image.

2. The method according to claim 1, wherein the at least one partial area (10a, 10b) of the vehicle (4) contained in the at least one local image contains a defective area (15) of the vehicle (4), which is marked as a defective area (15) in the global image.

3. The method according to claim 1 or 2, wherein the method comprises recording a first local image of a first partial area (10a) of the vehicle (4); recording a second local image of a second partial area (10b) of the vehicle (4), wherein the first partial area (10a) is contained in the second partial area (10b); the second partial area (10b) of the vehicle (4) contained in the second local image vehicle (4) in the global image; and to identify the first partial area (10a) of the vehicle (4) contained in the first local image in the second local image and / or in the global image.

4. Method according to one of the preceding claims, wherein the method comprises identifying the vehicle (4); wherein the method in particular comprises identifying the vehicle (4) using an image of its license plate number.

5. Method according to one of the preceding claims, wherein the method comprises storing additional information about the identified partial area (10a, 10b) together with the global image.

6. A method for training a neural network, the method comprising annotating a plurality of images of a plurality of vehicles (4) using a method according to any one of the preceding claims and using the annotated images to train the neural network.

7. Device (2) for annotating images of vehicles (4), in particular motor vehicles, comprising: at least one stationary image recording device (6) configured to record at least one global image of a vehicle (4); at least one mobile image recording device (8) configured to record at least one local image of at least one partial area (10a, 10b) of the vehicle (4); an annotation device (12) configured to receive the at least one global image and the at least one local image; and to identify and annotate the at least one partial area (10a, 10b) of the vehicle (4) contained in the at least one local image in the global image.

8. Device (2) according to claim 7, wherein the partial area (10a, 10b) of the vehicle (4) contained in the at least one local image contains a defective area (15) of the vehicle (4), which is marked as a defective area (15) in the global image.

9. Device (2) according to claim 7 or 8, wherein the annotation device (12) is designed, in particular in the case that a first local image containing a first partial area (10a) of the vehicle (4) is not uniquely identifiable in the global image: to request a second local image of a second partial area (10b) of the vehicle (4) from the at least one mobile image recording device (8), such that the first partial area (10a, 10b) is contained in the second partial area (10b); to identify the second partial area (10b) of the vehicle (4) contained in the second local image in the global image; and to identify the first partial area (10a) of the vehicle (4) contained in the first local image in the second local image and / or in the global image.

10. Device (2) according to one of claims 7 to 9, wherein the at least one mobile image recording device (8) contains at least one sensor (14) which makes it possible to determine the position and / or the orientation of the at least one mobile image recording device (8) in space; and wherein the annotation device (12) is designed to use information provided by the at least one sensor (14) to to identify a partial area (10a, 10b) of the vehicle (4) in the global image.

11. Device (2) according to claim 10, wherein the at least one sensor (14) comprises an inclination sensor (14), and / or a magnetic sensor (14) and / or a GPS sensor (14).

12. Device (2) according to one of claims 7 to 11, wherein the annotation device (12) is designed to identify the vehicle (4); wherein the annotation device (12) is in particular designed to identify the vehicle (4) based on an image of its license plate number.

13. Device (2) according to one of claims 7 to 12, wherein the at least one mobile image recording device (8) has an optical output device (16), for example a screen.

14. Device (2) according to one of claims 7 to 13, wherein at least one of the mobile image recording device (8) has an acoustic output device (18), for example a loudspeaker.

Images