Methods for fusing camera images and vehicle

Abstract

The invention relates to a method for fusing camera images (B) from several cameras (4, 5) that are at least partially movable relative to each other, wherein each independently movable camera (4, 5) or each group of jointly movable cameras (4, 5) is assigned at least one inertial measurement unit (6) and is movable together with it, wherein in a learning phase acceleration data (A) of the inertial measurement units (6) are acquired together with camera images (B) of the assigned cameras (4, 5), a homography matrix is ​​calculated and stored, wherein a look-up table is created from the homography matrix, wherein in an online phase the look-up table is retrieved for certain acceleration values ​​(A) determined by the inertial measurement units (6) and camera images (B) acquired by the assigned cameras (4, 5) and provides homography matrices to calculate fused images.

Description

The invention relates to a method for merging camera images according to the preamble of claim 1 and to a vehicle according to the preamble of claim 3.The fusion (stitching) of a plurality of camera images of vehicle cameras is complex if the camera positions can shift with respect to one another over time.US 2021 / 0179172 A1 describes methods and apparatus for determining a hitch angle including a camera for capturing a first image and a second image, a steering sensor configured to detect a steering angle, a speed sensor configured to detect a vehicle speed, a processor configured to generate a bird's eye view of the first image and the second image by a perspective transformation, generate a hitch model from the first image in response to the steering angle and the vehicle speed indicating that a vehicle is traveling straight, and generate a current hitch model from the second image, determine a hitch angle in response to an angle difference between the hitch model and the current hitch model, and a vehicle controller, controlling the vehicle in response to the hitch angle.DE 10 2019 202 269 A1 describes a method for calibrating a mobile camera unit of a camera system, wherein static position parameters (pus, pos, prs, pls) are fused with dynamic position parameters (pud, pod, prd, pld) of at least two edges and local coordinates (x, y, z) of the camera unit are calculated therefrom.DE 10 2013 209 156 A1 describes a camera having a housing for recording the environment of a motor vehicle for use in a motor vehicle surround view system, wherein image data of the environment of the motor vehicle can be generated by means of the camera and a first interface is present, via which the image data of the camera can be forwarded outside the housing for further processing. It is provided here that the camera has an inertial sensor which is arranged inside the housing and the data of the inertial sensor can be forwarded outside the housing via the first interface or a second interface for further processing.US 2017 / 0341583 A1 describes systems and methods for a towing vehicle towing a trailer having at least one imaging device. A method comprises: receiving a first image stream including a plurality of first images from a first imaging device connected to the trailer; receiving a second image stream including a plurality of second images from a second imaging device connected to the vehicle; determining at least one common feature between a first image of the first images and a second image of the second images; determining a first distance from the first imaging device to the at least one common feature and a second distance from the second imaging device to the at least one common feature; and determining a position of the first imaging device relative to the vehicle based on the first distance, the second distance, and a known position and posture of the second imaging device.The object of the invention is to specify a novel method for merging camera images and a novel vehicle.The object is achieved according to the invention by a method for fusing camera images having the features of claim 1 and by a vehicle having the features of claim 3.Advantageous embodiments of the invention are the subject matter of the dependent claims.A method for fusing camera images of a plurality of cameras that are at least partially movable relative to one another is proposed. This can be understood to mean two or more cameras that are movable relative to one another. Likewise, two or more cameras can be combined to form a group, wherein the cameras within a group are immovable with respect to one another, but movable with respect to a further camera or a further group of cameras.According to the invention, each independently movable camera or each group of jointly movable cameras is assigned at least one inertial measurement unit and is movable jointly therewith, wherein acceleration data of the inertial measurement units are captured together with camera images of the assigned cameras in a learning phase, a homography matrix is calculated and stored, wherein a look-up table is created from the homography matrix, wherein the look-up table is retrieved for specific acceleration values determined by the inertial measurement units and camera images captured by the assigned cameras in an online phase and supplies homography matrices in order to calculate fused images.In one embodiment, at least one of the cameras is configured as a mirror replacement camera on a cabin of a vehicle, wherein at least one further of the cameras is configured as a rear view camera on a chassis of the vehicle, wherein the cabin is movable relative to the chassis.According to one aspect of the present invention, a vehicle is proposed, comprising a chassis and a cabin which is movable relative to the chassis, wherein at least one mirror replacement camera and at least one inertial measurement unit is arranged on the cabin, wherein at least one rear view camera and at least one inertial measurement unit is arranged on the chassis. The vehicle has a control unit configured to carry out the method described above.In one embodiment, the vehicle is designed as a utility vehicle or bus.Furthermore, the vehicle may be configured as an autonomous or semi-autonomous vehicle.According to this, by taking account of acceleration data which are determined by at least one inertial measurement unit and by knowing the influence of specific accelerations on the ratio of the overlap of the camera images, the calculation for fusing the camera images can be accelerated.The method according to the invention does not require online image processing and calculation to dynamically merge images. Instead, historical data from the value range is used.Exemplary embodiments of the invention are explained in more detail below with reference to drawings.The following are shown: FIG. 1 shows a schematic view of a utility vehicle having a tractor and a trailer, and FIG. 2 shows a schematic view of a method for merging camera images of a plurality of camerasCorresponding parts are provided with the same reference numerals in all figures.FIG. 1 is a schematic view of a vehicle 1, in particular of a utility vehicle 1 with a tractor 2 and a trailer 3 or trailer. The tractor 2 has at least one mirror replacement camera 4 instead of an external mirror. In particular, the tractor 2 has a mirror replacement camera 4 on each side instead of a respective external mirror. Furthermore, the tractor 2 has at least one rear view camera 5, which, like the at least one mirror replacement camera 4, is directed against a direction of travel. The tractor 2 further has at least one inertial measurement unit 6. For example, an inertial measurement unit 6 is arranged in or on a cabin 7 and a further inertial measurement unit 6 is arranged on a chassis 8 of the tractor 2. The rear view camera 5 is likewise arranged on the chassis 8 and is therefore movable with respect to the mirror replacement camera 4 arranged on the air-sprung cabin 7.FIG. 2 is a schematic view of a method for fusing camera images B of a plurality of cameras 4, 5, for example the at least one mirror replacement camera 4 and the at least one rear view camera 5.For fusing a plurality of camera images B with an overlapping image section, a one-time calculation of the superposition and homography matrix is necessary with a static ratio of the positions of the cameras 4, 5.In a movable system with dynamic positions of the cameras 4, 5 (for example the position of a mirror replacement camera 4 on an air-sprung cabin 7 relative to the position of a rear view camera 5 on the chassis 8), no flat calculation can be used. By moving the vehicle 1, the cabin 7 can move differently with respect to the chassis 8 due to the accelerations. Overlapping image sections between the camera images B of the cameras 4, 5 are thus likewise dynamic and therefore increased computing effort is required for the merging.According to the present invention, it is proposed to accelerate the calculation for fusing the camera images B by taking into account acceleration data A that are determined by at least one inertial measurement unit 6 and on the basis of knowledge about the influence of specific accelerations on the ratio of the overlap of the camera images B.Merging (stitching) of images is effected via the determination of interest points in a plurality of images. These different interest points are compared and placed one above the other, a transformation then being calculated (see, e.g., https: / / courses.cs.washington.edu / courses / cse576 / 16sp / slides / 10_imageSti.pdf). If one determines in a learning phase how the images will roughly overlap, then later individual sections of the images can be viewed in order to perform a more rapid online method. If the transformations (homography matrices) are already known for different camera positions, then these can be reused if the exactly same camera positions are present again. If this is somewhat known, then a more rapid and accurate search algorithm can be achieved since it only needs to search and compare interest points in a relatively small area.In a learning phase, in a step S 1, historical data of the acceleration data A of the inertial measurement units 6 can be acquired together with the camera images B or image sections, in particular over the entire value range. A homography matrix can be calculated once from this in a step S 2 and stored in a step S 3. In this case, a look-up table is produced in a step S 4, which look-up table can be called up in an online phase in a step S 5 for specific acceleration values A and camera images B determined by the inertial measurement units 6 and supplies homography matrices in order to calculate the merged images in a step S 6. In this case, two images are merged into a single image after a transformation. The computationally intensive part of the image processing thus takes place once in the learning phase and can be transmitted to many vehicles. For example, steps S 1 to S 4 may be performed during the development of the vehicle 1. This also has the advantage that washed-out can be reduced by taking the acceleration data A into account at an early stage.The above-mentioned range of values results from possible extreme camera positions, for example in the case of a completely sprung driver's cab and in the case of a completely sprung driver's cab. In this range of values, an optimum homography matrix is calculated via optimizations. If it is known where the cameras are located relative to each other, then optimization can be performed in a restricted range of values. If the exact position is known, then a homography matrix already known for this position can be used directly without optimization.The look-up table has an input and an output, wherein the input is formed by position (or change of the relative position) of the two cameras based on acceleration data (integrated). The output is the corresponding homography matrix.

Claims

Method for merging camera images (B) of a plurality of cameras (4, 5) which are at least partially movable relative to one another, characterized in that each independently movable camera (4, 5) or each group of jointly movable cameras (4, 5) is assigned at least one inertial measurement unit (6) and is movable jointly therewith, wherein acceleration data (A) of the inertial measurement units (6) are captured together with camera images (B) of the assigned cameras (4, 5), a homography matrix is calculated and stored, wherein a look-up table is created from the homography matrix, wherein in an online phase the look-up table is retrieved for specific acceleration values (A) determined by the inertial measurement units (6) and camera images (B) captured by the assigned cameras (4, 5) and supplies homography matrices, to calculate fused images.Method according to Claim 1, characterized in that at least one of the cameras (4) is designed as a mirror replacement camera (4) on a cabin (7) of a vehicle (1), wherein at least one further of the cameras (5) is designed as a rear view camera (5) on a chassis (8) of the vehicle (1), wherein the cabin (7) is movable relative to the chassis (8).Vehicle (1) comprising a chassis (8) and a cabin (7) movable relative to the chassis (8), wherein at least one mirror replacement camera (4) and at least one inertial measurement unit (6) are arranged on the cabin (7), wherein at least one rear view camera (5) and at least one inertial measurement unit (6) are arranged on the chassis (8), characterized in that the vehicle (1) has a control unit configured to carry out the method according to claim 1 or 2.Vehicle (1) according to Claim 3, characterized in that the vehicle (1) is designed as a commercial vehicle (1).

Images