Device and method for testing a camera-based assistance system, and calibrating method

Abstract

The invention relates to a device (1) for testing a camera-based assistance system of a product (2), in particular a camera-based assistance system of a motor vehicle, comprising a main part (3) which is concave in some regions and can be attached to the product (2) and which has a display surface (5) in the curvature (4) of a concave region of the main part (3), said display surface being capable of displaying a test situation that can be detected by a camera of a camera-based assistance system to be tested. Advantageously, the display surface (5) is curved. The invention additionally relates to a method for testing a camera-based assistance system of a product (2) and to a method for calibrating a method for testing a camera-based assistance system of a product.

Description

[0001] Description:

[0002] SETES GmbH, 66121 Saarbrücken (Germany)

[0003] “Device and method for testing a camera-based assistance system and calibration method”

[0004] The invention relates to a device for testing a camera-based assistance system of a product, in particular a camera-based assistance system of a motor vehicle, which device comprises a partially concave base body which can be attached to the product and which has a display surface in a curvature of the concave region of the base body on which a test situation which can be captured by a camera of a camera-based assistance system to be tested can be displayed.

[0005] Furthermore, the invention relates to a method for testing a camera-based assistance system of a product, in particular for testing a camera-based assistance system of a motor vehicle, in which a test situation that can be captured by a camera of the camera-based assistance system to be tested is displayed on a curved display surface in a curvature of a concave region of a partially concave base body that is attached to the product.

[0006] The invention also relates to a method for calibrating a method for testing a camera-based assistance system.

[0007] A camera-based assistance system is a system that evaluates images or image sequences from a camera and uses these images or image sequences to decide on an assistance action. In a motor vehicle, an assistance action would, for example, be an emergency braking maneuver, which is initiated when an emergency braking assistant detects an object in an image that the vehicle is moving toward.

[0008] A product having a camera-based assistance system may be a motor vehicle, a motorcycle or a truck which is equipped, for example, with driver assistance systems for side collision warning, for collision warning, with a so-called adaptive cruise control or a parking assistant, that is to say in particular with a camera-based assistance system for person and / or object recognition.

[0009] Such a product may also be a military detection system that can be used, for example, to monitor the sky and warn of approaching flying objects, and that has a camera-based assistance system for detecting flying objects.

[0010] It is conceivable that a product is an industrial robot equipped with a camera-based assistance system that can, for example, detect and sort out reject parts in an industrial manufacturing process.

[0011] A test situation is an image or sequence of images that can be captured by the assistance system's camera.

[0012] ISO 26262:2018 (version 12 / 2018) is relevant for the development and validation of driver assistance systems, for example, for automated vehicles. Among other things, this standard specifies that standardized test situations must be reproducibly verified to test an assistance system. To do this, a standardized test situation is presented to an assistance system, and its response is tested. If the response matches the expected response, the assistance system has passed the test. If the response to the test situation does not match the expected response, the assistance system has failed the test.

[0013] A test situation can, for example, be a situation in which a camera-based emergency braking assistant of a motor vehicle must detect an obstacle in order to initiate an emergency braking maneuver.

[0014] To date, test procedures are known from the state of the art in which a camera of an assistance system to be tested is shown a film projected onto a flat screen, consisting of a sequence of virtual, affine images, as a test situation in which a dangerous situation such as a side collision occurs.

[0015] The disadvantages of this test procedure are the high complexity and the limited field of view of the camera of the camera-based assistance system being tested, which is limited horizontally to 50 to 70 degrees and usually deviates from the actual field of view of the camera. The field of view of a camera is the area covered by the camera, within which events or changes can be perceived. The field of view corresponds to the camera's detection range, which is defined by the camera's lens system.

[0016] It is also known to feed virtual test situations, such as virtual images, directly into the CAN bus of a driver assistance system and thus test the assistance system. However, this is not testing that involves the camera.

[0017] Previously known methods for testing camera-based assistance systems, for example, use virtual or affine images that do not take into account the actual field of view of a camera in an assistance system being tested. Furthermore, technical camera parameters are either ignored or merely estimated.

[0018] Particularly in the area of ​​driver assistance systems in motor vehicles, there may be significant deviations in the assessment of a dangerous situation by an assistance system that is tested using virtual images or image sequences but has to detect dangerous situations on real camera images when used in the operation of the motor vehicle.

[0019] The present invention is based on the object of creating a test device of the type mentioned above that is particularly compact. Furthermore, the invention is based on the object of creating a test device that is portable and particularly easy to handle.

[0020] The invention is also based on the object of creating a test device with which camera-based assistance systems with short-angle cameras or with wide-angle cameras can be tested using virtual or real images or a sequence of such images.

[0021] According to the invention, this problem is solved by providing a curved display surface. The partially concave base body can be attached to the product whose camera-based assistance system is to be tested in such a way that a camera of the camera-based assistance system to be tested is covered by the base body and captures a concave area.

[0022] The curved display surface creates a particularly compact test device in which the camera of the assistance system to be tested still has a detection field that advantageously enables a simulation of a real use of the assistance system for test purposes.

[0023] The partially concave base body is preferably 3D printed or injection-molded. It has proven advantageous for the base body to be made of a plastic.

[0024] The base body can in particular have the shape of a spherical shell section or a hemispherical shell section at least in some regions, whereby a particularly compact device with a thin-walled base body is advantageously created.

[0025] The present invention is also based on the object of creating a test method of the type mentioned above, by means of which a particularly simple and precise testing of a camera-based assistance system is possible.

[0026] Furthermore, the invention is based on the object of creating a standardized test procedure.

[0027] According to the invention, this object is achieved in that the test situation is displayed on the curved display surface in such a way that the test situation is captured by the camera of the assistance system to be tested as the original field of view of the camera.

[0028] The original field of view is the field of view of the camera without being covered by the partially concave base body, i.e. the field of view that the camera has when the assistance system is in operation.

[0029] In order to create a realistic test situation despite the limited, i.e. device-related reduced actual field of view of the camera, which is directed into the curvature of the concave area of ​​the base body, a test situation is shown on the display surface which gives the assistance system the impression that the camera is capturing its original field of view.

[0030] Because the display surface is curved and, due to the nature of the device, close to the camera, a transformation of a test situation is required before it is displayed on the display screen in order to compensate for the curvature of the display surface and its spatial proximity to the camera.

[0031] To create a transformed, for example, distorted test situation, a person skilled in the art can resort to known methods of so-called image processing. Image processing within the meaning of the present invention achieves, in particular, a perspectively correct representation of an image displayed on the curved display surface, taking into account the geometric properties of the base body and camera parameters. If a flat image is distorted by image processing, this is referred to as image warping, which belongs to the field of image processing.

[0032] The invention is further based on the object of creating a calibration method of the type mentioned at the outset, by means of which a test method can be calibrated particularly accurately.

[0033] This object is achieved according to the invention in that a spatial arrangement and / or orientation of the camera of the assistance system to be tested is determined relative to the curved display surface, wherein the spatial arrangement and / or orientation of the camera is used to create the digital output image.

[0034] For practical purposes, an image or a sequence of images is displayed on the curved display surface as a test situation.

[0035] It has proven useful to test an assistance system by playing a single image or a sequence of images like a film, with the single image or sequence of images depicting a real-life usage situation, such as a car ride, from a camera perspective. Images can be included in a sequence of images that depict a specific test situation in which intervention by the assistance system is required.

[0036] In one embodiment of the invention, a real or virtual image or a sequence of real images or virtual images is displayed on the curved display surface as a test situation.

[0037] Real images can, for example, be optical recordings of a camera car's movement, while virtual images can be computer-generated, artificially created images. Test situations can advantageously be created without real images. This makes the test method according to the invention particularly easy to adapt to different application scenarios.

[0038] Advantageously, the curved display surface is arranged in the curvature of the partially concave base body, or the curved display surface is formed by the curvature of the partially concave base body.

[0039] A separate display surface such as a screen is not absolutely necessary. The advantage is that a device is easy to manufacture and easy to handle.

[0040] It is understood that the display surface can also be formed by only one portion of the curvature. In one embodiment of the invention, the curvature of the partially concave base body is at least partially spherical segment-shaped, preferably partially hemispherical.

[0041] A partially spherical segment-shaped curvature is one that can be created, for example, when a spherical segment is pressed into a solid material to form the curvature.

[0042] For a partially spherical segment-shaped curvature forming a curved display surface, the high symmetry allows for the transformation of a flat image onto the curvature using image processing with relatively little computational and calibration effort. Advantageously, real-time processing of an image sequence with a frequency between 20 and 120 Hz is particularly well-suited.

[0043] If the curvature is partially hemispherical and forms a curved display surface, testing of a camera-based assistance system with a particularly wide field of view, such as a wide-angle camera or a fisheye camera, is possible. The partially hemispherical curvature enables a horizontal field of view of up to 180 degrees.

[0044] The curvature, which is preferably spherical segment-shaped or hemispherical, expediently forms a field of view of a camera of an assistance system to be tested during a test.

[0045] During the test, the camera does not capture its original field of view, but rather a test field of view that is limited by the curvature.

[0046] In a further embodiment of the invention, a digital output image is created from a digital input image, which is a digital image of the original field of view of the camera of the assistance system, based on parameters of the camera and / or parameters of the curved display surface, which digital output image is displayed on the display surface.

[0047] While test methods known from the prior art project a standardized test situation onto a flat screen, with a test method according to the invention the test situation is adapted to the curved display surface and / or camera parameters in such a way that the camera has the impression that it is capturing its original field of view. This means that the camera captures the test situation displayed on the curved display surface as if it were not using the device according to the invention. This advantageously enables a particularly reliable test of the camera-based assistance system, in particular because technical camera parameters such as focal length or its actual field of view are taken into account in addition to the geometric parameters of the partially concave base body and the curved display surface. A test under test conditions that are as realistic as possible is advantageously possible.Tests with distorting standard test situations with possibly restricted field of view are no longer necessary.

[0048] In a calibration method according to the invention, a calibration is expediently carried out based on technical parameters of a camera of the camera-based assistance system of a product and its relative position and / or orientation to the curved display surface.

[0049] For this purpose, in one embodiment of the invention, the device comprises a calibration device with which a position and / or orientation of a camera of the camera-based assistance system relative to the curved display surface can be determined, wherein the calibration device preferably has a stereo camera.

[0050] To ensure that a test can be adapted to a specific camera position, a calibration device determines the orientation and / or position of the camera relative to the base body. These values ​​are relevant for creating digital output images, as they ensure that the camera's actual, flat field of view is displayed during a test despite the test fixture. By taking into account the geometric parameters of the display surface, the camera parameters, and / or the geometric parameters of the partially concave base body, it can be ensured that the camera of the assistance system under test is shown its original, flat field of view on the curved display surface during a test.

[0051] In one embodiment of the invention, the device has a detachable fastening means by means of which the partially concave base body can be fixedly and detachably attached to a product to be tested, wherein the fastening means in particular has a suction cup.

[0052] The base body can advantageously be attached to various products. It doesn't matter whether the base body surrounds a front, rear, or side camera of a motor vehicle, a camera of an industrial robot, or a camera of another product. If the fastening device has a suction cup, it can be removed without leaving any residue after a test. A suction cup, especially one with an adjustable suction cup arm, can also be mounted and dismounted multiple times with precise positioning.

[0053] The device expediently comprises a digital projector, which is preferably arranged outside the curvature of the partially concave base body.

[0054] A digital projector can project a test situation onto the curved display surface using a standard product.

[0055] Preferably, the digital projector is mounted on the outside of the base body. This advantageously creates an easy-to-use device.

[0056] In a further embodiment of the invention, the device comprises an optical means for deflecting light from a light source of a digital projector, wherein the optical means is designed such that the light from the light source is deflected from an outer side of the partially concave base body into a curvature of the partially concave base body.

[0057] Advantageously, a particularly simple, modular device is created in which the digital projector can be arranged on the outside of the base body and can still display a test situation on a curved display surface in the curvature.

[0058] In one embodiment of the invention, the optical means comprises a deflecting mirror which is arranged in the region of the digital projector and preferably outside the curvature of the partially concave base body.

[0059] A device is created whose individual components are particularly easy to access. The device is also particularly easy to maintain.

[0060] In particular, the deflecting mirror can be mounted on a side facing away from the curvature, for example between the digital projector and the partially concave base body.

[0061] The optical means expediently comprises a curved mirror which is arranged in the curvature of the partially concave base body or in the region of the curvature outside the partially concave base body, and which is configured to redirect light from a light source of the digital projector onto the curved display surface in order to create the test situation to be captured by the camera of the camera-based assistance system.

[0062] By appropriately selecting the curved mirror, a horizontal field of view and a vertical field of view of up to 180 degrees can be created.

[0063] By arranging the curved mirror within the curvature, a compact device is created.

[0064] In one embodiment of a calibration method according to the invention, the field of view of the camera of the camera-based assistance system to be tested is adjusted during calibration.

[0065] Advantageously, it is ensured that the camera is shown an image on the curved display surface that corresponds to an image according to its original field of view.

[0066] A calibration according to the invention can create a particularly accurate test procedure.

[0067] The invention is explained in more detail below with reference to exemplary embodiments and the accompanying drawings relating to the exemplary embodiments. They show:

[0068] Fig. 1 shows an embodiment of a device according to the invention,

[0069] Fig. 2 shows an embodiment of a test and calibration method according to the invention.

[0070] A device 1, shown in an exploded view in Fig. 1 a, for testing a camera-based assistance system of a motor vehicle 2, shown in detail in Fig. 1 d, to which the device 1 according to the invention is attached, comprises a partially concave base body 3 which has a partially hemispherical curvature 4 which forms a curved display surface 5.

[0071] In this exemplary embodiment, the curved display surface 5 is a projection surface onto which a test situation can be projected by a digital projector 6, which is attached to the outside of the partially concave base body 3 by a holding device 7. The curved display surface 5 forms a field of view of a camera (not shown in Fig. 1a) of the assistance system to be tested during a test.

[0072] The base body 3 also includes a flat deflection area 8. Light emanating from the digital projector 6, not shown in Fig. 1, is deflected onto the curved display surface 5 by an optical means 9, which comprises a deflection mirror 10 and a curved mirror 11 mounted in the flat deflection area 10, and creates the test situation there. For this purpose, an opening 12 is provided in the partially concave base body 3, through which the light can enter the curvature 4 from outside the base body 2.

[0073] A detachable fastening means 13 is attached to the partially concave base body 3, wherein the detachable fastening means 13 comprises a plurality of suction cups 14, each of which has an adjustable suction cup arm 15. Using the suction cups 14, the partially concave base body 3 can be detachably fastened, for example, to a windshield 16 of the motor vehicle 2 shown in detail in Fig. 1 b, such that the curvature 4 is covered by a windshield 16 of the motor vehicle 2. For reasons of clarity, the suction cups 14 in Fig. 1 d are folded away from the windshield 18 and do not rest against it.

[0074] A calibration device 17, arranged in Fig. 1 b in a curvature 4 of the partially concave base body 3 and shown in detail in Fig. 1 c, has a convex base 18 and a partially convex support arm 19 integrally formed thereon, which extends perpendicularly from the base 18. A curvature 20 of the partially convex support arm 19 and that of a convex section 21 of the base 18 are designed such that the curvature 20 and the convex section 21 can be placed over their entire surface against the curvature 4 of the base body 3. This advantageously ensures their repeatable, positionally accurate arrangement.

[0075] A stereo camera 22 is attached to an end of the holding arm 19 facing away from the base 18, with the aid of which a position and an orientation of a camera of an assistance system of a product can be determined.

[0076] For calibration of the test device 1, the calibration device 17 is mounted in the flat deflection area 8 as shown in Fig. 1 b such that both the partially convex holding arm 19 and the convex section 21 of the base 18 lie fully against the curvature 4 of the base body 3.

[0077] With the aid of the stereo camera 22, a spatial position and an orientation as well as an arrangement of a front camera (not shown in Fig. 1d) of a camera-based assistance system of the motor vehicle 2 relative to the curved base body 3 and the curved display surface 5 are determined. A device 1 according to the invention is attached to the motor vehicle 2 shown in Fig. 1d. For this purpose, the partially concave base body 3 rests with a contact surface 23, which is formed by an end face 24 of the partially concave base body 3 and a rear side 25 of the curved mirror 12 and which delimits the curvature 4, against a windshield 16 of the motor vehicle 2 in such a way that a field of view of the front camera is delimited by the curved display surface 5, i.e. the front camera looks into the curvature 4.

[0078] Reference is now made to Fig. 2, where identical or equivalent parts are designated by the same reference number as in Fig. 1 and the letter a is added to the relevant reference number.

[0079] A calibration method according to the invention and a test method according to the invention are explained below using the example of a camera-based assistance system of a motor vehicle.

[0080] For calibrating a test method according to the invention according to Fig. 2a, in a first method step 200 a calibration device designated 17 in Fig. 1 is introduced into a curvature 4 of a partially concave base body 3 of a device 1 according to the invention and the base body is attached to a windscreen of a motor vehicle.

[0081] In a second method step 210, a digital, camera-parameter-independent test image, which may be a 180x180-degree test image, is displayed on a curved display surface. Using reference points of the test image, geometric boundaries such as the field of view of a camera of an assistance system under test can be determined within the curvature.

[0082] The test image was created by so-called image warping, in which a flat test image for a known camera position was converted into a distorted representation on a curved display surface.

[0083] A 180x180-degree test image is a reference test image that would extend completely over a hemispherical curvature and whose distortion is mathematically known compared to a flat version of the test image when displayed on a flat display surface. In a third method step 220, a field of view of the camera of the assistance system to be tested and its focal length are calibrated to the curved display surface from the digital test image with known distortion parameters and technical camera parameters. This method step is a so-called field of view adaptation, in which a field of view of the camera into the curvature and an original field of view of the camera when viewed on a flat surface are linked using so-called image warping.

[0084] In a fourth method step 230, a stereo camera of the calibration device captures an image of the front camera of the assistance system and transfers it to an evaluation unit, such as a PC, which uses this image to determine geometric parameters such as the position and orientation of the fixedly arranged front camera relative to the partially concave base body, in particular with respect to the relative position and orientation to the curved display surface. This step is particularly relevant when an assistance system comprises more than one camera, wherein the multiple cameras are arranged linearly offset from one another.

[0085] Based on the determined position and orientation data of the camera of the driver assistance system, the distortion parameters determined in method steps 210 and 220 are corrected by taking into account an actual camera position and orientation with respect to the curved display surface, i.e. the test image is adapted to the actual position of the camera of the driver assistance system.

[0086] Method step 230 ensures that the camera of the assistance system to be tested is shown a field of view, depending on its actual position when looking into the curvature of the partially concave base body, that corresponds to its original field of view when viewed outside the test situation.

[0087] The device is now ready for testing a camera-based assistance system.

[0088] In a test method according to the invention according to Fig. 2b, in order to create a test situation in a first method step 300, a digital input image, which is a digital image of an original field of view of a camera of an assistance system of the motor vehicle to be tested, is read into an evaluation unit.

[0089] In a second method step 310, the read digital input image is converted into a distorted digital output image by means of so-called image processing, using data from the calibration.

[0090] In a third method step 320, the distorted digital output image is projected onto the curved display surface by a digital projector. Taking into account the geometric parameters of the curved base body and the technical data of the camera, the distorted digital output image, when displayed on the curved display surface, is such that it is perceived by the camera of the assistance system under test as the original field of view, i.e., a flat image, in the test situation.In simple terms, by projecting a digital input image transformed into a distorted digital output image onto the curved display surface, a reverse transformation is carried out so that the camera, despite the restricted field of view by placing the base body on it, captures a field of view that corresponds to the original field of view of the camera, whereby a relative position and orientation of the camera with respect to the curved display surface is taken into account according to calibration.

[0091] The process steps 300 to 320 are repeated for each image of a sequence of images, so that the camera of the assistance system to be tested captures a sequence of images in the manner of a film.

Claims

Patent claims:

1. Device (1) for testing a camera-based assistance system of a product (2), in particular a camera-based assistance system of a motor vehicle, which device comprises a partially concave base body (3) which can be attached to the product (2) and which has a display surface (5) in a curvature (4) of a concave region of the base body (3), on which display surface a test situation which can be captured by a camera of a camera-based assistance system to be tested can be displayed, characterized in that the display surface (5) is curved.

2. Device according to claim 1, characterized in that the curved display surface (5) is arranged in the curvature (4) of the partially concave base body (3), or that the curved display surface (5) is formed by the curvature (4) of the partially concave base body (3).

3. Device according to claim 1 or 2, characterized in that the curvature (4) of the partially concave base body (3) is at least partially spherical segment-shaped, preferably partially hemispherical.

4. Device according to one of claims 1 to 3, characterized in that the curvature (4), which is preferably spherical segment-shaped or hemispherical, forms a field of view of a camera of an assistance system to be tested during a test.

5. Device according to one of claims 1 to 4, characterized in that the device (1) has a detachable fastening means (13) by means of which the partially concave base body (3) can be fixedly and detachably attached to a product to be tested, wherein the fastening means (13) in particular has a suction cup (14).

6. Device according to one of claims 1 to 5, characterized in that the device (1) has a digital projector (6) which is preferably arranged outside the curvature (4) of the partially concave base body (3).

7. Device according to one of claims 1 to 6, characterized in that the device (1) comprises an optical means (9) for deflecting light from a light source of a digital projector (6), wherein the optical means is designed such that the light of the light source is deflected from an outer side of the partially concave base body into a curvature (4) of the partially concave base body (3).

8. Device according to claim 7, characterized in that the optical means (9) has a deflecting mirror (10) which is arranged in the region of the digital projector (6) and preferably outside the curvature (4) of the partially concave base body (3).

9. Device according to claim 7 or 8, characterized in that the optical means (9) comprises a curved mirror (11) which is arranged in the curvature (4) of the partially concave base body (3) or in the region of the curvature (4) outside the partially concave base body (3), and which is designed to redirect light from a light source of the digital projector (6) onto the curved display surface (5) in order to create the test situation to be recorded by the camera of the camera-based assistance system.

10. Device according to one of claims 1 to 9, characterized in that the device (1) comprises a calibration device (17) with which a position and / or orientation of a camera of the camera-based assistance system relative to the curved display surface (5) can be determined, wherein the calibration device (17) preferably has a stereo camera (22).

11. Method for testing a camera-based assistance system of a product (2), in particular for testing a camera-based assistance system of a motor vehicle (2), in which a test situation that can be captured by a camera of the camera-based assistance system to be tested is displayed on a curved display surface (5) in a curvature (4) of a concave region of a partially concave base body (3) that is attached to the product (2), characterized in that the test situation is displayed on the curved display surface (5) in such a way that the test situation is captured by the camera of the assistance system to be tested as the original field of view of the camera.

12. Test method according to claim 1 1, characterized in that an image or a sequence of images is displayed as the test situation on the curved display surface.

13. Test method according to claim 1 1 or 12, characterized in that a real or a virtual image or a sequence of real images or virtual images is displayed as the test situation on the curved display surface.

14. Test method according to one of claims 11 to 13, characterized in that a digital output image is created from a digital input image, which is a digital image of the original field of view of the camera of the assistance system, on the basis of parameters of the camera and / or parameters of the curved display surface, which digital output image is displayed on the display surface.

15. A method for calibrating a test method according to one of claims 11 to 14, characterized in that a spatial arrangement and / or orientation of the camera of the assistance system to be tested is determined relative to the curved display surface, wherein the spatial arrangement and / or orientation of the camera is used to create the digital output image.

16. Calibration method according to claim 15, characterized in that a calibration is carried out based on technical parameters of the camera of the camera-based assistance system of a product and its relative position and / or orientation to the curved display surface (5).

17. Calibration method according to claim 15 or 16, characterized in that during calibration, a field of view adjustment of the camera of the camera-based assistance system to be tested takes place.

Images