Method for supplementing a vehicle environment image captured by a vehicle's vision assistance system

Abstract

The invention relates to a method for supplementing a vehicle environment image (130) captured by a vehicle vision assistance system (120) of a vehicle (100) with a rear portion of the vehicle environment (U) in a view obstruction area (150) where the vehicle environment (U) is obscured by a trailer (200) coupled to the vehicle (100). The trailer outline (210) of the trailer (200) is determined in the vehicle environment image (130) using a segmentation method. The view obstruction area (150) is determined as a contour (140) circumscribing the trailer outline (210). A vehicle-independent image data set of the current vehicle environment (U) is retrieved. Using a satellite image-to-street image transformation, a synthetic partial image (160) is generated from the image data set, which covers at least the section of the vehicle environment (U) assigned to the view occlusion area (150).The synthetic partial image (160) is adapted to the vehicle environment image (130) and superimposed with it to form a supplemented vehicle environment image (130').

Description

[0001] The invention relates to a method for supplementing a vehicle environment image captured by a vehicle vision assistance system according to the preamble of claim 1.

[0002] Modern vehicles are equipped with vision assistance systems that capture a field of vision which the driver cannot see directly or indirectly. For example, rear-view cameras (RVCs) capture the area behind the vehicle when traveling in the normal direction of travel. Surround view systems use one or more vehicle cameras to capture almost the entire area around the vehicle and can display it from various perspectives.

[0003] A trailer attached to the vehicle partially obstructs the field of vision of such a vision assistance system. This impairs its usability when operating the vehicle with a trailer attached.

[0004] To solve this problem, an additional secondary vehicle camera can be mounted on the trailer in such a way that its field of view covers the portion of the original field of view of the vision assistance system that is obscured by the trailer. By superimposing the images from the secondary vehicle camera and the images from the vision assistance system, the original field of view can be restored. A disadvantage of this solution is that an additional camera must be provided, connected to the vision assistance system, and calibrated relative to its coordinate system. In particular, correct superimposition of the images from the secondary vehicle camera requires calibration, which is determined by its pose (position and orientation). Therefore, a new calibration is required for each new trailer and / or for each new mounting position of the secondary vehicle camera on a trailer.

[0005] Accordingly, this procedure is very complex and prone to errors in the calibration of the secondary vehicle camera.

[0006] Furthermore, methods are known in which the vehicle's surroundings are continuously captured using the vision assistance system. If the vehicle is moving appropriately, and in particular sufficiently, those parts of the surroundings that are currently obscured by the attached trailer at the vehicle's current position are captured at previously traversed positions. Based on the already captured image data, and taking into account the vehicle's trajectory, a synthetic partial image can be generated that can replace the currently obscured portion of the vision assistance system's field of view.

[0007] The applicability of this method is limited by the trajectory already traveled and the image data acquired along that trajectory. If, for example, no image data has yet been acquired immediately after the vehicle starts, if the currently obscured area is blocked by obstacles, or if, for example when reversing into a parking space, this area is not within the captured field of view of the vision assistance system, then generating a synthetic partial image is not possible. Furthermore, the quality (fit) of such synthetic partial images depends heavily on the accurate estimation of the vehicle's own motion and the correct assignment of the image data to different positions along this trajectory.

[0008] Therefore, there is a need for a method to improve the supplementation of a section of a vehicle environment that is obscured by a coupled trailer and captured by a vision assistance system.

[0009] Document US 10,701,300 B2 describes a display system for generating a composite view of an area behind a vehicle towing a trailer. A first camera is designed to output initial image data corresponding to a first image and is configured to be mounted on the vehicle in a rearward-facing orientation. A second camera is designed to output secondary image data corresponding to a second image and is configured to be mounted on the trailer in a rearward-facing orientation. An image processor receives the initial image data and the secondary image data. The image processor is configured to combine the initial and secondary image data to generate composite image data corresponding to a composite image.

[0010] From US patent 2020 / 0207273A1, a vehicle with a controller is known that is configured to receive images from a camera on a drone and, based on the images received from the drone, creates a top view of the vehicle and displays it on a visual display.

[0011] The invention is based on the objective of providing an improved method for supplementing a vehicle environment image in a view obscuration area in which the vehicle environment is obscured by a trailer coupled to the vehicle.

[0012] The problem is solved according to the invention by a method having the features of claim 1.

[0013] Advantageous embodiments of the invention are the subject of the dependent claims.

[0014] In a method for supplementing a vehicle's surroundings image captured by a vehicle's vision assistance system with a view obscured area located behind the vehicle in the direction of travel, where the surroundings are obscured by a trailer attached to the vehicle, the obscured area is determined as a contour or bounding box that describes the trailer's outline in the vehicle's surroundings image. The trailer outline is determined using a region growing or optical flow method, or another segmentation method. The vehicle's surroundings are captured, at least partially, by a reversing camera, including the area opposite the forward direction of travel. The obscured area and the associated trailer outline are determined in the reversing camera's line of sight.

[0015] A preferably three-dimensional image dataset of the current vehicle environment, acquired beforehand independently of the vehicle (e.g., by flying or driving over it using satellites), is determined, i.e., retrieved. The image dataset of the vehicle environment can be determined based on the vehicle's geo-position.

[0016] Using a satellite image-to-street image transformation, a synthetic partial image is generated from the image dataset that covers at least the portion of the vehicle's surroundings corresponding to the field of view of the reversing camera. Satellite image-to-street image transformation methods are known from the prior art, for example, from the publications by Y. Shi, D. Campbell, X. Yu and H. Li, "Geometry-Guided Street-View Panorama Synthesis From Satellite Imagery," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 12, pp. 10009-10022, 1 Dec. 2022, doi: 10.1109 / TPAMI.2022.3140750, Weijia Li, Jun He, Junyan Ye, Huaping Zhong, Zhimeng Zheng, Zilong Huang, Dahua Lin, Conghui He, "CrossViewDiff: A Cross-View Diffusion Model for Satellite-to-Street View Synthesis", https: / / doi.org / 10.48550 / arXiv.2408.14765 and in Aysim Toker, Qunjie Zhou, Maxim Maximov, Laura Leal-Taixe, “Coming Down to Earth: Satellite-to-Street View Synthesis for Geo-Localization,” Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 6488-6497.

[0017] Another method for synthesizing streetscapes from satellite imagery is described in the publication "Geometry-Guided Street-View Panorama Synthesis from Satellite Imagery" by Yujiao Shi, Dylan Campbell, Xin Yu and Hongdong Li, see https: / / arxiv.org / pdf / 2103.01623. The methods described in this document allow a streetscape to be rendered from the image dataset within 0.2s to 0.02s.

[0018] The synthetic image located in the direction of view of the reversing camera is adapted to the vehicle surroundings image, which is also located in the direction of view of the reversing camera, and superimposed on it, at least within the bounding box area (i.e., the area obscured by the camera's view), to create a supplementary vehicle surroundings image. This adaptation can include geometric adjustments and / or adjustments to color and brightness values ​​such that edge noise ("seams") at the transition between the vehicle surroundings image and the synthetic image is minimized.

[0019] This method generates a vehicle environment image that is enhanced in the area where the trailer is normally obstructed, making it appear transparent and seemingly no longer blocking the view of the vehicle's surroundings. This significantly facilitates maneuvering the vehicle with a trailer attached and improves road safety.

[0020] The system requires no additional camera systems besides the reversing camera, making it particularly robust and easy to integrate. Furthermore, the system can be used regardless of the vehicle's previous travel distance, especially even directly from a stationary or parked position.

[0021] In one embodiment, the geoposition and geoorientation of a vehicle coordinate system assigned to the vehicle are recorded relative to a geographic coordinate system. Acquiring the data with reference to a common coordinate system, i.e., by transforming it into the geographic coordinate system, enables faster execution of the calculations.

[0022] In further development of the procedure -each vertex of the circumscribing contour is assigned a three-dimensional, vehicle-related vertex vector in the vehicle coordinate system originating from the optical center of the vision assistance system, i.e., the reversing camera, -for each vehicle-related corner point vector, a vehicle-independent corner point geovector is determined in the geo-coordinate system (G) based on a geo-position and a geo-orientation of a vehicle coordinate system and -Using satellite image-to-street image transformation, the synthetic partial image is generated from the vehicle-independently acquired image dataset, which covers at least that section of the vehicle's surroundings (U) enclosed by the vehicle-independent corner point geovectors assigned to the view obscuration area. Advantageously, by transforming the corner point vectors into the geocoordinate system using the satellite image-to-street image transformation method (Sat2Cam), a synthetic partial image can be generated that corresponds to the image area obscured by the trailer.

[0023] Exemplary embodiments of the invention are explained in more detail below with reference to drawings.

[0024] This shows: Fig. 1. Schematic perspective view of a vehicle with a vision assistance system, Fig. 2. Exemplary and schematic representation of the disruption of a vehicle's surroundings by a trailer, Fig. 3. An exemplary and schematic representation of a follower outline obtained using Region Growing, Fig. Four corner point vectors schematically transferred into a geographic coordinate system to define the boundaries of a visual obstruction area. Fig. 5. An exemplary and schematic supplementary vehicle environment image and Fig. 6. A schematic flowchart for the procedure.

[0025] Corresponding parts are marked with the same reference symbol in all drawings.

[0026] Fig. Figure 1 shows a purely schematic perspective view of a vehicle 100 in a vehicle environment U. A vehicle-related vehicle coordinate system 110 is assigned to the vehicle 100.

[0027] The pose (that is, the position and orientation) of the vehicle 100 in relation to the vehicle environment U is specified by reference to a vehicle-independent world or geo-coordinate system G, which may, for example, be selected in a unified form as World Geodetic System 1984 (WGS84).

[0028] The vehicle 100 includes a vision assistance system 120, which is simplified here as a rear vehicle camera (RVC) 120. The rear vehicle camera 120 captures a rearward part of the vehicle's surroundings U (relative to the vehicle coordinate system 110) in a vehicle surroundings image 130.

[0029] In a highly simplified manner and without taking into account the imaging optics (i.e., in the manner of a pinhole camera), the vehicle environment image 130 can be understood as a projection of the vehicle environment U onto the sensor surface of the reversing camera 120, which is not shown in detail here, in which the imaging rays intersect in the optical center 121 (which corresponds to the pinhole of the pinhole camera).

[0030] If the vehicle 100 has a (not shown here, in Fig. When the trailer 200 (as shown in Figure 2) is attached, part of the rear vehicle environment U captured by the vehicle's surroundings image 130 is obscured. The trailer 200 generally has an irregular shape, which, in its projection onto the sensor area of ​​the reversing camera 120, is simplified here to be represented only by a circumscribing contour 140. This circumscribing contour 140, also referred to below as the bounding box 140, does not necessarily have to be rectangular, although rectangular circumscribing contours 140 are particularly easy to detect and process.

[0031] The bounding box 140 can be specified (with reference to the vehicle coordinate system 110) in the present rectangular form by the endpoints of four vehicle-related corner point vectors 141, whose common origin is the optical center 121 of the reversing camera 120.

[0032] The camera pixels of the reversing camera 120, arranged within the bounding box 140, are disrupted by the projection of the trailer 200 and do not reproduce the vehicle's surroundings U. This area is subsequently referred to as the obscuration area 150.

[0033] Fig. Figure 2 shows the interference of the vehicle surroundings image 130 by a trailer 200 using a specific example. The trailer 200 appears in the vehicle surroundings image 130 as an irregular trailer outline 210, which depends on the orientation of the trailer 200 relative to the reversing camera 120 and can therefore change during steering movements.

[0034] The bounding box 140 is defined as the smallest possible circumscribing regular (preferably rectangular) contour 140 of the trailer outline 210. The area within the bounding box 140 forms the obscuration zone 150, in which the correct detection of the vehicle's surroundings is not guaranteed.

[0035] Fig. Figure 3 illustrates the determination of the pendant outline 210 using region growing. Region growing is a method known from image processing, for example from the publication Adams, R., Bischof, L.: Seeded region growing. IEEE Transactions on Pattern Analysis and Machine Intelligence 16(6), 641-647 (1994), in which, starting from a seed point 211, homogeneous or largely homogeneous image areas are merged into regions.

[0036] The task here is to merge the image areas lying within the trailer outline 210, which are largely homogeneous to each other but largely different from the image of the vehicle's surroundings U. For this purpose, a seed point 211 is chosen such that, regardless of the orientation of the coupled trailer 200 (i.e., irrespective of any prior steering movements), it lies within an image area onto which the trailer 200 is projected. For example, the seed point 211 can be placed at the geometric center of the vehicle's surroundings image 130. However, depending on the orientation of the reversing camera 120, other positions are also possible.

[0037] The bounding box 140 can be defined as the smallest rectangle that is aligned parallel to the edges of the vehicle environment image 130 and that encompasses all points of the trailer outline line 210.

[0038] The position of the optical center 121, the orientation of the reversing camera 120 (i.e., the orientation of the optical axis passing through the optical center 121), and the optical image scale are known with respect to the vehicle coordinate system 110, for example, as a result of a one-time calibration after mounting the reversing camera 120 on the vehicle 100. Thus, from the coordinates of the corner points of the bounding box 140 (relative to the pixel coordinate system of the vehicle environment image 130), the vehicle-related corner point vectors 141 with respect to the vehicle coordinate system 110 are also known, which limit the view obstruction area 150.

[0039] The invention further provides that the position and orientation (i.e., the pose) of the vehicle coordinate system 110 is continuously recorded with respect to a vehicle-independent geocoordinate system G, for example, using a Global Positioning System (GPS) and / or by recording the vehicle kinematics using accelerometers, gyroscopes, displacement sensors, or similar sensors. Preferably, the pose of the vehicle 100 is recorded with respect to a standardized geocoordinate system G such as the World Geodetic System 1984 (WGS84), since satellite-acquired image data is frequently referenced to such a standardized geocoordinate system G.

[0040] By determining the pose of the vehicle coordinate system 110 with respect to the geo-coordinate system G, the vehicle-related corner point vectors 141 can be transformed into vehicle-independent geo-vectors 141', as shown in Fig. 4 schematically represented. To simplify the representation, the solid angle area bounded by the vehicle-independent geovectors 141' (in the case of a rectangular bounding box 140, a pyramid extending infinitely into the object-side half-space of the reversing camera 120) is only shown in the two-dimensional projection onto the road surface traveled by the vehicle 100.

[0041] Methods are known from the prior art for preferably obtaining three-dimensional datasets from satellite-based, aerial, or road-based images of the vehicle's surroundings U. Using such datasets, views of the vehicle's surroundings U can be generated by transformations also known as satellite-to-street view, synthetic street view, or Sat2Cam, based on virtually any viewing axes referenced to a geographic coordinate system G.

[0042] The invention provides to apply these methods to the area of ​​visual axes limited by the vehicle-independent geovectors 141', in which the vehicle environment U is obscured by the trailer 200 for visibility in the vehicle environment image 130.

[0043] From the data sets about the vehicle environment U, a synthetic partial image 160 is obtained, which captures the situation hidden by the trailer 200 in the vehicle environment image 130. Fig. Figure 5 shows, purely as an example and schematically, a supplemented vehicle environment image 130' in which the obstruction area 150 within the bounding box 140 is replaced by the synthetic partial image 160. Although dynamic objects (for example, other moving road users or buildings that have changed since the satellite imagery was acquired) cannot be represented in the synthetic partial image 160, the driver still gains a sufficient overview of the vehicle environment U for orientation.

[0044] To adapt the synthetic sub-image 160 for the most seamless possible representation within the vehicle environment image 130, methods known as stitching, such as geometric adjustment and contrast, grayscale, and color value adjustment, can be applied. Geometric adjustment involves a transformation of the image coordinates, also known as warping, such that structures at the transition from the original vehicle environment image 130 to the synthetic sub-image 160 appear without interruptions and (at least in the case of linear structures) without significant changes in direction. Furthermore, grayscale or color values ​​can be adjusted to create a homogeneous image impression.

[0045] Fig. Figure 6 schematically shows a flowchart for an embodiment of the proposed method, comprising a first to tenth step S1 to S10 and a decision step E1.

[0046] In the first step S1, all system components involved in the process, in particular the vision assistance system 120 with its optical center 121 and its optical axis as well as its image scale, are calibrated to a common coordinate system, the vehicle coordinate system 110. In other words, it is determined which vector, referenced to the vehicle coordinate system 110 and emanating from the optical center 121, corresponds to a pixel at a certain position in a camera image of the vision assistance system 120.

[0047] In the second step S2, the driver's gaze direction is determined. Methods for gaze direction detection using at least one vehicle interior camera, which exploit the relative change in position of the Purkinje reflexes in at least one of the driver's eyes when their gaze direction changes, are known from the prior art. However, other methods for gaze direction detection and tracking can also be used. It is also possible, additionally or alternatively, to signal a need for a display of the part of the vehicle's surroundings U obscured by the trailer 200 through active user interaction (pressing a button, engaging reverse gear, or similar).

[0048] In the third step S3, the trailer outline 210 is detected in the vehicle environment image 130, for example using a region growing method, as shown by Fig. 3 has already been explained.

[0049] In the fourth step S4, the bounding box 140 is determined from the trailer outline line 210.

[0050] In the fifth step S5 of the presented procedure variant, the driver's gaze direction (already determined in the second step S2) is used to determine whether the driver is currently looking at a display showing the vehicle environment image 130. This makes it possible to execute the computationally and data-intensive subsequent steps S6 to S10 only if there is actually a need to display the vehicle environment U.

[0051] In decision step E1, the control flow of the procedure branches based on the determined direction of gaze. If, in the fifth step S5, it is determined that the driver is viewing the display with the presented vehicle environment image 130, the procedure continues with the sixth step S6. Otherwise, the procedure continues with a repeat of the fourth step S4.

[0052] In the sixth step S6, the field of view is determined using the calibration data of the vision assistance system 120, under which the bounding box 140 determined in the fourth step S4 appears (that is, the solid angle area of ​​the beam of all rays that, starting from the optical center 121, penetrate the image plane of the vision assistance system 120 at a point that lies inside or on the edge of the bounding box 140 is determined).

[0053] In the seventh step S7, the location of the optical center 121 and the orientation of the optical axis of the vision assistance system 120 in the geo-coordinate system G are determined based on the current vehicle pose (position and orientation of the vehicle coordinate system 110 relative to the geo-coordinate system G). Based on this location and orientation, the field of view determined in the sixth step S6 (the beam of light emanating from the optical center 121, bounded by the bounding box 140) is transformed into the geo-coordinate system G.

[0054] In the eighth step S8, a synthetic sub-image 160 is generated using a satellite image-to-street image transformation based on an image dataset derived from vehicle-independent images of the current vehicle environment U. The synthetic sub-image 160 captures how the current vehicle environment U appears within the field of view, limited by the bounding box 140 and emanating from the optical center 121.

[0055] Methods for creating views from image datasets that describe the view of the respective vehicle environment U from a multitude of viewing angles for a variety of possible vehicle positions are known from the prior art and are referred to as Synthetic Street View or Sat2Cam Imaging. Such methods can be implemented using machine learning techniques, for example, diffusion models, neural networks, and / or beam-optical projection modules. The preferably three-dimensional image datasets can be acquired systematically and independently of the vehicle, for example, by flying a photogrammetric camera (airborne imaging), by satellite-based photogrammetric surveys, and / or by driving a survey vehicle with a mounted camera system across the terrain.

[0056] Synthetic image 160 captures the situation of the vehicle's surroundings U at the time the image data set was acquired. Changing or moving objects (e.g., other road users or vegetation) may therefore be missing from synthetic image 160, appear additionally, or be shown in a changed position and / or extent, deviating from the current situation. However, synthetic image 160 is typically sufficient for the driver to orient themselves within the field of vision otherwise obscured by the trailer 200 in the area of ​​obstruction 150. It can also be assumed that satellite images will be available in or near real-time in the future, and that dynamic objects will then also be able to be displayed in the synthetic image using the Sat2Cam method.

[0057] In step nine, S9, the synthetic sub-image 160 is integrated into the vehicle environment image 130 to create the augmented vehicle environment image 130'. For this purpose, the synthetic sub-image 160 is geometrically adapted (in its pixel resolution) to the pixel resolution of the vehicle environment image 130 and positioned within the bounding box 140. In other words, the appropriately scaled image content of the synthetic sub-image 160 replaces the image content of the original vehicle environment image 130 within the bounding box 140.

[0058] Even with correct geometric adjustment of the synthetic sub-image 160, the sub-image 160 may differ in detail from the vehicle environment image 130 in its outer edge area. For example, straight structures such as road markings, traffic control devices, or building edges may be interrupted or kinked at the transition from the vehicle environment image 130 to the synthetic sub-image 160. Color differences are also possible, for example, due to altered lighting compared to the preferably three-dimensional image dataset.

[0059] To generate a seamless-looking augmented vehicle environment image 130', an optional tenth step S10 applies an image registration method to the synthetic partial image 160. Such image registration methods, for example for projective and nonlinear geometric rectification as well as for adjusting the color representation, are known from the prior art, for example from the publication by Mike Krainin and Ce Liu, "Seamless Google Street View Panoramas," available online at https: / / research.google / blog / seamless-google-street-view-panoramas.

[0060] With the enhanced vehicle environment image 130', the method according to the invention enables an adaptive display (adapted to the respective vehicle environment U) in which the trailer 200 appears transparent. No additional external camera system is required for this. The method (taking into account a calibration, as described in step S1) can be carried out independently of the vehicle 100, the design and mounting position of the vision assistance system 120, and the trailer 200. It enables a supplement covering the area of ​​obstruction 150, regardless of the vehicle's previous movement, and in particular, also directly from a parked position. Furthermore, it can be carried out independently of the current light and weather conditions and independently of the driver.

[0061] In one embodiment, the method can be controlled by detecting the driver's direction of view, so that in particular the computationally and data-intensive provision of the image data set and the acquisition and adaptation of the synthetic partial image 160 is only carried out if the driver has a need for a view or orientation that is not restricted by the trailer 200.

Claims

[1] Method for supplementing a vehicle environment image (130) captured by a vehicle vision assistance system (120) of a vehicle (100) with a rear part of the vehicle environment (U) in a view obstruction area (150) in which the vehicle environment (U) is obscured by a trailer (200) coupled to the vehicle (100), characterized by that continuously - the trailer outline (210) of the trailer (200) in the vehicle environment image (130) is determined using a segmentation method, - the area of ​​obstruction of view (150) in the vehicle environment image (130) is determined as a contour (140) describing the trailer outline (210), - a vehicle-independent image data set of the current vehicle environment (U) is determined, - a synthetic partial image (160) is generated from the image data set by means of a satellite image-to-street image transformation, which covers at least the section of the vehicle environment (U) assigned to the view occlusion area (150) and - the synthetic partial image (160) is adapted to the vehicle environment image (130) and superimposed on it at least in the viewing obstruction area (150) to form a supplementary vehicle environment image (130'). [2] Method according to claim 1, characterized by , that - each vertex of the circumscribing contour (140) is assigned a three-dimensional vehicle-related vertex vector (141) originating from the optical center (121) of the vision assistance system (120) in a vehicle coordinate system (110), - for each vehicle-related corner point vector (141) a vehicle-independent corner point geovector (141') is determined in a geo-coordinate system (G) based on a geo-position and a geo-orientation of the vehicle coordinate system (110), - by means of the satellite image-street image transformation from the vehicle-independently acquired image data set, the synthetic partial image (160) is generated which covers at least that section of the vehicle environment (U) which is enclosed by the vehicle-independent corner point geovectors (141') assigned to the view occlusion area (150). [3] Method according to claim 2, characterized by , that the geo-coordinate system (G) is chosen as World Geodetic System 1984 (WGS84). [4] Method according to any one of the preceding claims, characterized by , that the trailer outline (210) in the vehicle environment image (130) is determined by a method of region growing and / or optical flow. [5] Method according to any one of the preceding claims, characterized by , that the procedure is triggered when the driver's gaze falls within a predetermined range of gaze directions relative to a display designed to show the vehicle environment image (130). [6] Method according to claim 5 characterized by , that the direction of view is captured by at least one driver observation camera which is arranged in the interior of the vehicle (100) and is calibrated with reference to the vehicle coordinate system (110). [7] Method according to any one of the preceding claims, characterized by , that the synthetic partial image (160) is adapted to the vehicle environment image (130) by a process of image registration and / or contrast adjustment. [8] Method according to any one of the preceding claims, characterized by, that the image data set, captured by a satellite, includes images of the vehicle's surroundings (U) at the vehicle's current geo-position.

Images