Method for fusing camera images and vehicle

Abstract

The invention relates to a method for fusing camera images (B) of a plurality of at least partially relatively movable cameras (4, 5), wherein at least one inertial measurement unit (6) is assigned to each independently movable camera (4, 5) or each jointly movable group of cameras (4, 5) and co-moves with the camera, wherein in a learning phase acceleration data (A) of the inertial measurement units (6) and corresponding camera images (B) of the cameras (4, 5) are recorded together, homography matrices are calculated and stored, wherein a look-up table is generated from the homography matrices, wherein in an online phase the look-up table is queried for a specific acceleration value (A) determined by the respective inertial measurement unit (6) and a camera image (B) recorded by the corresponding camera (4, 5) and provides a homography matrix in order to calculate a fused image.

Description

Technical Field

[0001] The present invention relates to a method for fusing camera images according to the preamble of claim 1 and a vehicle according to the preamble of claim 3. Background Technology

[0002] If the positional relationship between cameras may shift over time, fusing (stitching together) multiple camera images from vehicle cameras becomes quite complex.

[0003] US 2021 / 0179172 A1 describes a method and apparatus for determining the hinge angle of a trailer coupling, comprising: a camera for acquiring a first image and a second image; a steering sensor configured to identify a steering angle; a speed sensor configured to acquire a vehicle speed; a processor configured to generate a bird's-eye view of the first and second images by perspective transformation, generate a trailer coupling model from the first image in response to a steering angle indicating straight-line travel of the vehicle and the vehicle speed, generate a current-updated coupling model from the second image, determine the trailer coupling hinge angle in response to an angle difference between the trailer coupling model and the current coupling model; and a vehicle control device that controls the vehicle in response to the trailer coupling hinge angle.

[0004] DE 10 2019 202 269 A1 describes a method for calibrating a moving camera unit of a camera system, wherein static position parameters (pus, pos, prs, pls) are fused with dynamic position parameters (pud, pod, prd, pld) of at least two edges, and the local coordinates (x, y, z) of the camera unit are calculated therefrom.

[0005] DE 10 2013 209 156 A1 describes a camera with a housing for capturing images of the environment surrounding a vehicle for use in a vehicle surround-view system. The camera generates image data of the vehicle's surroundings and includes a first interface through which the image data can be forwarded for further processing outside the housing. The specification states that the camera has an inertial sensor disposed inside the housing, and that data from the inertial sensor can be forwarded via either the first or second interface for further processing outside the housing.

[0006] US 2017 / 0341583 A1 describes a system and method for a tractor unit used to tow a trailer having at least one imaging device. The method includes: receiving a first image stream having a plurality of first images from a first imaging device connected to the trailer; receiving a second image stream having a plurality of second images from a second imaging device connected to the vehicle; determining at least one common feature between one of the first images and one of the second images; determining a first distance from the first imaging device to the at least one common feature, and a second distance from the second imaging device to the at least one common feature; and determining the position of the first imaging device relative to the vehicle based on the first distance, the second distance, and a known position and orientation of the second imaging device. Summary of the Invention

[0007] The objective of this invention is to propose a novel method for fusing camera images and a novel vehicle.

[0008] According to the present invention, this task is accomplished by a method for fusing camera images having the features of claim 1 and a vehicle having the features of claim 3.

[0009] Advantageous designs of the present invention are the subject of the dependent claims.

[0010] A method is proposed for fusing camera images from multiple cameras that are at least partially movable relative to each other. This includes two or more cameras that are movable relative to each other. Similarly, two or more cameras can be combined into a group, wherein cameras within a group remain relatively fixed to each other but can move relative to another camera or another group of cameras.

[0011] According to the present invention, at least one inertial measurement unit is assigned to each independently movable camera or each group of cameras that can move together, and the inertial measurement unit moves together with the camera. During the learning phase, the acceleration data of the inertial measurement unit and the camera images of the corresponding / assigned camera are collected together, and the homography matrix is ​​calculated and stored. A lookup table is generated from the homography matrix. During the online phase, for a specific acceleration value determined by each inertial measurement unit and the camera image collected by the corresponding camera, the lookup table is called / queried and the homography matrix is ​​provided to calculate the fused image.

[0012] In one embodiment, at least one of the cameras is disposed on the driver's cab of the vehicle as a rearview mirror instead of a camera, wherein at least another of the cameras is disposed on the chassis of the vehicle as a reversing camera, wherein the driver's cab is movable relative to the chassis.

[0013] According to one aspect of the invention, a vehicle is provided comprising a chassis and a cab movable relative to the chassis, wherein at least one rearview mirror substitute camera and at least one inertial measurement unit are arranged on the cab, and wherein at least one reversing camera and at least one inertial measurement unit are arranged on the chassis. The vehicle has a control unit configured to perform the described method.

[0014] In one embodiment, the vehicle is configured as a commercial vehicle or a bus.

[0015] The vehicle can also be configured as an autonomous or semi-autonomous vehicle.

[0016] According to the present invention, by taking into account acceleration data determined by at least one inertial measurement unit and based on known information about the influence of a specific acceleration on the overlap relationship of camera images, the calculation of camera image fusion can be accelerated.

[0017] The method according to the invention eliminates the need for online image processing and computation to dynamically fuse images. Instead, historical data within a numerical range is used. Attached Figure Description

[0018] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0019] Figure 1 A schematic diagram of a commercial vehicle having a tractor and a semi-trailer or trailer is shown.

[0020] Figure 2 A schematic diagram illustrates a method for fusing camera images from multiple cameras.

[0021] Identical or corresponding parts have the same reference numerals in all the accompanying drawings. Detailed Implementation

[0022] Figure 1 This is a schematic diagram of a vehicle 1, particularly a commercial vehicle 1, which includes a tractor unit 2 and a semi-trailer 3 or trailer. The tractor unit 2 has at least one rearview mirror replacement camera 4 for replacing the exterior rearview mirrors. Specifically, the tractor unit 2 has one rearview mirror replacement camera 4 on each side for replacing their respective exterior rearview mirrors. The tractor unit 2 also has at least one reversing camera 5, which, like the at least one rearview mirror replacement camera 4, points in the opposite direction to the direction of travel. The tractor unit 2 also has at least one inertial measurement unit 6. For example, one inertial measurement unit 6 is arranged inside or on the cab 7, and another inertial measurement unit 6 is arranged on the chassis 8 of the tractor unit 2. The reversing camera 5 is also arranged on the chassis 8, and therefore can move relative to the rearview mirror replacement camera 4 arranged on the air-suspension cab 7.

[0023] Figure 2This is a schematic diagram of a method for fusing camera images B, which are derived from multiple cameras 4 and 5, such as at least one rearview mirror replacing camera 4 and at least one reversing camera 5.

[0024] When fusing multiple camera images B with overlapping image segments, if the positional relationship between cameras 4 and 5 is static, a single calculation of the overlap relationship and homography matrix is ​​required.

[0025] In motion systems where the positions of cameras 4 and 5 change dynamically (e.g., the position of the rearview mirror replacing camera 4 on the air-suspension cab 7 relative to the position of the reversing camera 5 on the chassis 8), a single-step calculation method is not feasible. Due to the movement of vehicle 1, the cab 7 may move to varying degrees relative to the chassis 8 due to the acceleration it experiences. Therefore, the overlapping image segment between the camera images B of cameras 4 and 5 is also dynamically changing, thus requiring more computation during fusion.

[0026] According to the present invention, the calculation of camera image B fusion is accelerated by taking into account acceleration data A determined by at least one inertial measurement unit 6 and based on known information about the influence of a specific acceleration on the overlap relationship of camera images B.

[0027] Image fusion (stitching) is performed by identifying interest points (OPPs) in multiple images. These different OOPs are compared and superimposed, and then a transformation is calculated (see, for example, https: / / courses.cs.washington.edu / courses / cse576 / 16sp / Slides / 10_ImageStitching.pdf). If the approximate overlap of the images is known during the learning phase, then individual segments of the images can be observed to perform a faster online processing procedure. If the transformations (homography matrices) are already known for different camera positions, these transformations can be reused when the exact same camera positions reappear. With a general understanding of this, a faster and more accurate search algorithm can be implemented, as it only needs to search and compare OOPs within a relatively small region.

[0028] Here, historical data of acceleration data A from the inertial measurement unit 6, along with camera images B or image segments, can be acquired simultaneously in step S1 of the learning phase, particularly across the entire numerical range. Based on this, a single calculation of the homography matrix can be performed in step S2, and stored in step S3. Here, a lookup table is generated in step S4, which can be queried in step S5 of the online phase for a specific acceleration value A determined by each inertial measurement unit 6 and camera image B, providing the homography matrix for calculating the fused image in step S6. Here, the two images are merged into a single image after transformation. Therefore, the computationally intensive part of image processing is completed in one go during the learning phase and can be transferred to many vehicles. For example, steps S1 to S4 can be performed during the development of vehicle 1. This also has the advantage that image blur artifacts can be reduced by considering acceleration data A in advance.

[0029] The aforementioned numerical range is derived from possible extreme camera positions, such as under conditions of full cab suspension compression and full cab suspension rebound. Within this range, the optimal homography matrix is ​​calculated through optimization. If the relative positions of the cameras are known, optimization can be performed within a finite numerical range. If the precise positions are known, the homography matrix known for that position can be used directly without optimization.

[0030] The lookup table has inputs and outputs. The inputs consist of the positions (or relative position changes) of two cameras, determined based on acceleration data (after integration). The output is the corresponding homography matrix.

[0031] List of reference numerals

[0032] 1. Vehicles, commercial vehicles

[0033] 2 tractor units

[0034] 3 semi-trailers

[0035] 4 cameras, rearview mirrors replace cameras

[0036] 5 cameras, including a reversing camera.

[0037] 6 Inertial Measurement Units

[0038] 7 cabs

[0039] 8 chassis

[0040] A acceleration data

[0041] B camera image

[0042] Steps S1 to S6

Claims

1. A method for fusing camera images (B) from a plurality of cameras (4, 5), said plurality of cameras (4, 5) being at least partially capable of relative motion with respect to each other, Its features are, At least one inertial measurement unit (6) is assigned to each camera (4, 5) capable of independent movement or to each group of cameras (4, 5) capable of joint movement, and the inertial measurement unit moves together with the camera. During the learning phase, the acceleration data (A) of the inertial measurement unit (6) and the camera images (B) of the corresponding camera (4, 5) are acquired together, and the homography matrix is ​​calculated and stored. A lookup table is generated from the homography matrix. During the online phase, the lookup table is queried and the homography matrix is ​​provided for a specific acceleration value (A) determined by each inertial measurement unit (6) and the camera image (B) acquired by the corresponding camera (4, 5) in order to calculate the fused image.

2. The method according to claim 1, characterized in that, At least one of the cameras (4) is disposed on the driver's cab (7) of the vehicle (1) as a rearview mirror instead of a camera (4), wherein at least another of the cameras (5) is disposed on the chassis (8) of the vehicle (1) as a reversing camera (5), wherein the driver's cab (7) is movable relative to the chassis (8).

3. A vehicle (1) comprising a chassis (8) and a cab (7) movable relative to said chassis (8), wherein, At least one rearview mirror replacement camera (4) and at least one inertial measurement unit (6) are arranged on the cab (7), and at least one reversing camera (5) and at least one inertial measurement unit (6) are arranged on the chassis (8). The vehicle (1) is characterized in that it has a control unit configured to perform the method according to claim 1 or 2.

4. The vehicle (1) according to claim 3, characterized in that, The vehicle (1) is configured as a commercial vehicle (1).

Images