Method and apparatus for operating a camera-monitor system for a motor vehicle

Abstract

Method for operating a camera-monitor system (100) for a motor vehicle (101), wherein the camera-monitor system (100) comprises two cameras (102, 103) assigned to a common side (106, 107) of the motor vehicle and each configured to provide an image (201, 202) of an environment (105) of the motor vehicle (101), the depicted environment (105) of the images (201, 202) partially overlapping, comprising: - providing a specification for a first image (201) of the first camera (102); - transforming the first image (201) into a first transformed image (301) such that the provided specification is fulfilled in the transformed image (301); - determining at least one image element (303) on a plane (309) in the first transformed image (301); - determining a corresponding Image element (304) in at least one second image (202) of the second camera (103) that corresponds to the at least one image element (303),- Transforming the second image (202) depending on the determined image elements (303, 304) of the first transformed image (301) and the second image (202), such that a further image element (305) in a border region (308) of a transformed second image (302) connects to a corresponding further image element (306) in a border region (307) of the transformed first image (301), wherein the transformation of the second image (202) is carried out depending on the determined image elements (303, 304) such that - the determined image elements (303, 304) in the transformed second image (302) and in the transformed first image (301) are identical, and / or - a line in the border region (308) of the second transformed image (302) connects to a corresponding line in the border region (307) of the first transformed image (301) immediately follows.

Description

The invention relates to a method for operating a camera-monitor system for a motor vehicle, in particular for a truck. The invention further relates to a device configured to carry out the method. Motor vehicles, such as trucks and buses, conventionally have exterior mirrors. Mirror replacement systems are also known, in which a camera and a monitor are used to display the fields of view of conventional exterior mirrors. US 2017 / 0091559 A1 concerns a camera-monitor system for a motor vehicle. DE 10 2014 213 536 A1 relates to a driver assistance system and a method for combining partial images to form an image of a continuous surrounding area of ​​a motor vehicle. Hua Lei et al - A robust improved Image stitching algorithm based on keypoints registration and Gao, Junhong et al. - Seam-driven Image stitching each concern stitching methods. Furthermore, the Wikipedia article "Image Stitching" (edited on November 6, 2017, accessed on April 21, 2026, URL: https: / / en.wikipedia.org / w / index.php?title=Image_stitching&ol did=809039739) describes how, in so-called "image stitching", corresponding image features between overlapping images are identified, projective transformations are determined from these, and the transformed images are assembled into a panorama. It is desirable to specify a method for operating a camera-monitor system for a motor vehicle that enables reliable operation. It is further desirable to specify a device for operating a camera-monitor system that enables reliable operation. The invention is characterized by a method for operating a camera-monitor system for a motor vehicle and by a corresponding device configured to carry out the method. According to at least one embodiment, the camera-monitor system comprises two cameras assigned to a common side of the vehicle. Each camera is configured to provide an image of the vehicle's surroundings. The depicted surroundings of the images overlap at least partially, particularly in a peripheral area. A specification for a first image from the first camera is provided. This first image is transformed into a first transformed image such that the specified specification is fulfilled in the transformed image. Specifically, the first image is transformed depending on the specified specification. At least one image element is identified on a plane in the first transformed image. This plane is specifically a plane in the 3D environment of the vehicle. A corresponding image element is identified in a second image from the second camera. This corresponding image element corresponds to the at least one image element of the first transformed image.The second image is transformed depending on the determined image elements of the first transformed image and the second image, such that another image element in a border area of ​​a transformed second image connects to a corresponding further image element in a border area of ​​the transformed first image, whereby the transformation of the second image takes place depending on the determined image elements, such that the determined image elements in the transformed second image and in the transformed first image match, and / or a line in the border area of ​​the second transformed image connects directly to a corresponding line in the border area of ​​the first transformed image. The first and second images are each transformed so that they can be stitched together in such a way that the additional image element is displayed coherently in the edge areas. The transformed first image serves as the reference image. The second image is transformed, for example, using homography, also known as projective transformation. The reference positions in the two images, whose correspondence is used as the optimality criterion, are obtained, for example, by feature recognition, feature comparison (e.g., SIFT), feature sorting (e.g., RANSAC), and / or by the linear least squares method. For this purpose, four image elements and four corresponding image elements are determined. This method makes it possible to display the images from two different cameras on the same page in such a way that the depicted environment is intuitively understandable to a user. Although the first image was taken by a different camera than the lower image, the two transformed images blend seamlessly at their edges. For example, the first camera is a camera with a narrow field of view and a long focal length, such as a telephoto lens. For example, the second camera is a camera with a short focal length and a wide field of view, such as a wide-angle lens. For example, the first camera is used to display the legally defined field-of-view classes 2 and 4 on a monitor. For example, the second camera is used to display the legally defined field-of-view classes 5 and partially 4. In particular, this method makes it possible to display larger areas of the environment than those defined by the legally defined field-of-view classes. This method allows the specified parameters to be implemented in the first image, and additionally, the two transformed images to be joined so that their edges seamlessly merge. Therefore, neither the first nor the second image needs to be defined as a reference image. Different parameters for the first image and / or the second image, or for the first transformed image and the second transformed image, are possible. Furthermore, the method can be reliably executed even if the first image and / or the second image are of comparatively poor quality, for example, in low light. Moreover, the two transformed images can be merged despite the different characteristics of the cameras used. According to at least one embodiment, the first image is lens-corrected before transformation. Alternatively or additionally, the second image is lens-corrected before transformation. The input images from the two cameras are, for example, rectified in such a way that known distortions due to the lenses used are compensated for. According to at least one embodiment, the at least one image element is determined on a ground plane associated with the vehicle. This is achieved, for example, by defining a restriction of the image region or by detecting the vehicle's movement while driving. This ensures that no object standing on the ground at the edge of the two images is lost during the transformation. Objects standing on the ground are always displayed in the transformed images. Objects standing on the ground always remain partially within the driver's field of vision. This increases safety. For example, the provided specification for the transformed first image is a horizontal representation of the horizon. Alternatively or additionally, the provided specification is a straight representation of the vehicle's ground edge. Alternatively or additionally, the provided specification is a straight representation of the vehicle's outer edge. Alternatively or additionally, the provided specification is a vertical representation of the vehicle's rear edge. Alternatively or additionally, the provided specification is a representation of a predefined element of the vehicle in the first transformed image point. For example, the upper corner of the trailer's rear edge should still be visible in the first transformed image. Other specifications are also possible, resulting, for example, from legal requirements and / or user comfort analyses, user settings, or depending on the driving situation. According to at least one embodiment, the first transformed image and / or the second transformed image are each further transformed to fulfill a further specified requirement. For example, this further transformation includes zooming, stretching, distorting, or similar operations to display a desired area of ​​the environment. According to embodiments, these further transformations are combined with the previous transformations and thus executed simultaneously. According to at least one embodiment, the first image is transformed using homography. Alternatively or additionally, the second image is also transformed using homography. This enables a reliable and sufficiently fast transformation. According to at least one embodiment, the second image is transformed based on the identified image elements such that the identified image elements are identical in the transformed second image and in the transformed first image. The coordinates of the image elements are identical in both the transformed first and second images. This allows for a seamless merging of the two edge regions. According to at least one embodiment, transforming the second image comprises transforming the determined image elements such that a line in the edge region of the second transformed image connects directly to a corresponding line in the edge region of the first transformed image, in particular a line on the ground plane. Thus, the two transformed images connect seamlessly. In particular, lines on the ground plane remain connected in the edge regions. According to at least one embodiment, the first transformed image and the second transformed image are displayed as a combined image on a common monitor of the camera-monitor system. For example, the first transformed image is displayed as the top image of the combined image, and the second transformed image is displayed as the bottom image. This method allows the user to intuitively grasp the combined image, even though the first transformed image is based on a different camera than the second transformed image. The device for the motor vehicle, which is designed to carry out the method according to at least one embodiment, is, for example, part of a motor vehicle control unit (ECU, electronic control unit). Advantages, features and further training described for the process also apply to the device and vice versa. Further advantages, features, and developments will emerge from the following examples, which are explained in conjunction with the figures. Identical, similar, and similarly effective elements can be marked with the same reference symbols across different figures. Figure 1 shows a schematic representation of a motor vehicle with a camera-monitor system according to an embodiment, Figure 2 shows a schematic representation of an image from a first camera according to an embodiment, Figure 3 shows a schematic representation of an image from a second camera according to an embodiment, Figure 4 shows a schematic representation of an overall image according to an embodiment, and Figure 5 shows a flowchart of a method according to an embodiment. Fig. 1 shows a schematic top-down view of a motor vehicle 101 according to an exemplary embodiment. The motor vehicle 101 is, for example, a truck with a trailer, a bus, and / or a passenger car. The motor vehicle 101 has a camera-monitor system 100 as a mirror replacement system. The camera-monitor system 100 has two cameras 102, 103 on each side 106, 107 of the motor vehicle 101. The cameras 102, 103 serve to record the surroundings 105 of the respective assigned side 106, 107. According to further embodiments, the two cameras 102, 103 are arranged only on one side of the pages 106, 107. The camera-monitor system 100 has two monitors 104, each assigned to one of the pages 106 and 107. During operation, a section of the environment 105 on page 106, recorded by the cameras 102 and 103 located there, is displayed as a complete image 400 on the assigned monitor 104. On the monitor 104 assigned to page 107, a section of the environment 105 is displayed as a complete image 400, recorded by the cameras 102 and 103 assigned to page 107. Thus, the complete image 400 differs on the two monitors 104. The overall image 400 on the monitor 104 is specifically designed to display a mirror image of a conventional external mirror and, in particular, to show further sections of the surroundings 105 that cannot be shown with a conventional external mirror. A device 110 is provided for the signal coupling of cameras 102, 103 and monitors 104. The device 110 is configured to control both cameras 102, 103 and monitors 104. For example, the device 110 is a control unit or part of a control unit of the motor vehicle 101. Fig. 2 shows a first image 201, recorded, for example, on page 106 using the first camera 102. The first camera 102 is, for example, a camera with a narrow field of view and a long focal length. The camera 102 captures a section of the surroundings 105 looking back. Due to the optics used, for example, the horizon 310 appears curved in image 201. A ground edge 311 of the vehicle 101 is shown curved in image 201, although it runs straight on the actual vehicle. For example, a rear edge 312 of the vehicle 101 is shown curved or slanted, although it runs vertically straight on the actual vehicle. Fig. 3 shows a second image 202, which, for example, was recorded on page 106 using the second camera 103. The second image 202 shows a section of the environment 105 that differs from the first image 201. However, the depicted areas of the environment 105 overlap at least partially. After carrying out the procedure described below in connection with Fig. 5, the overall image 400, which is based on the two images 201, 202, is displayed on the monitor 104, as shown by way of example in Fig. 4. The process begins with step 501, as shown in the embodiment of Fig. 5. In step 501, images 201 and 202 are each lens-corrected. This digitally compensates for distortions caused by the optics. In step 502, a first transformation is applied to one of the two images 201, 202. In the described embodiment, a transformation is applied to the first image 201. The transformation can be represented, in particular, by homography and / or does not change the lower image edge to be supported. The transformation in step 502 is performed to meet specifications for the representation of the first image. These specifications include, for example, at least one of the following: a horizontal representation of the horizon 310, a straight representation of the ground edge 311, a vertical representation of the rear edge 312, and a representation of a predefined element 313 (Fig. 4), for example, the upper corner of the trailer's rear edge, in a first transformed image 301. The first transformed image 301 is generated from the first image 201 by means of the transformation in step 502.The transformation in step 502 using homography enables a natural-looking representation of the first image 201. For example, the transformation can be performed manually, that is, by moving and modifying the transformed image 301 until a desired representation is achieved. Alternatively or additionally, the transformation can be performed automatically using calibration patterns, for example, four patterns. The calibration patterns are detected in image 201 and then transformed to defined positions. In step 503, four or more image elements 303 are annotated in the transformed first image 301. It is also possible to annotate fewer than four image elements 303 if further information about the image elements is available. In the still untransformed image 202 from the second camera 103, corresponding image elements 304 (Fig. 3) are annotated. For example, the annotation is performed manually by selecting the image elements 301 and the corresponding image elements 304. Alternatively or additionally, the image elements 303 and the corresponding image elements 304 are automatically detected. The image elements 303 and the corresponding image elements 304 lie in a plane 309, specifically at ground level with respect to the motor vehicle 101. In a calibration setup, it is possible that the plane 309 is a wall or a ceiling if the cameras 102, 103 are mounted differently than they will be later during operation in the motor vehicle 101. For example, annotation is carried out semi-automatically using one or more calibration patterns laid out on the floor. For example, the calibration patterns contain a minimum of four points that can be uniquely and automatically assigned to each other and are arranged in pairs as far apart as possible. Fully automated annotation using a sorting algorithm is also possible, which only allows feature points identified as matching to each other and located in layer 309. This can be achieved, for example, by a predefined restriction of the image region or by detection from the vehicle's movement while driving. In step 504, the second image 202 is transformed into a transformed second image 302. This is done primarily using a homography transformation. For this purpose, a transformation matrix is ​​calculated based on the image elements 303 and the corresponding image elements 304. This matrix is ​​applied to the second image 202 to obtain the transformed second image 302. The transformation is performed in such a way that the second image 202 is distorted so that it connects to the already transformed first image 301. The transformed first image 301 and the transformed second image 302 are connected in such a way that, in particular, all lines at ground level remain connected.In particular, in a boundary region 307 of the transformed first image 301 and in an immediately adjacent boundary region 308 of the transformed second image 302, another image element 305, for example a line, and a corresponding further image element 306 of the transformed first image 301 remain connected in the overall image 400 (Fig. 4). For example, the determined transformation matrix is ​​directly applied to the second image 202. In an optional step 505, further transformations are performed according to the exemplary embodiments. These can be carried out together with the transformation in step 504 or subsequently. The further transformations, for example, bilinear and / or generally polynomial, are applied to fulfill further requirements for the overall image 400 and / or the transformed first image 301 and / or the transformed second image 302. The transformations in step 505 are performed in such a way that the pixels in the adjacent border regions 307 and 308 remain in the same position. For example, the second image 202 is compressed so that a larger section of the surroundings 105 is displayed in the overall image 400. Other requirements are also possible. The initial homography, especially after an assumed ideal lens equalization, ensures, for example, that on the one hand the given constraints and specifications for the transformed first image 301 are met and on the other hand the second image 202 can still be stitched exactly by a ground plane homography 309. The use of image elements 303 and 304 for stitching, which are located on the ground plane 309, ensures in particular that the two transformed images 301 and 302 connect precisely on the ground plane 309. On the ground plane, consecutive straight lines remain straight. This ensures that objects on the ground remain at least partially within the driver's field of vision, thus guaranteeing safety. The use of homographic transformations ensures that, on the one hand, no image points are lost discontinuously and therefore unintuitively, and on the other hand, that straight lines remain straight, allowing for precise estimation of the direction of travel, for example, during maneuvering. Homography ensures, in particular, that straight edges are displayed straight, especially on a plane such as the ground plane 309. The advantage that straight edges remain straight and do not appear bent, according to the exemplary embodiments, exists in particular only before step 505, i.e., before the further transformation. After step 505, this can, for example, still be guaranteed for straight lines in the direction of travel, such as the median strip, depending on the transformation, but not generally for all lines at ground level. The camera-monitor system 100 also makes it possible to integrate additional camera images, for example to map the total image 400 to the legal field of view class 6. To obtain the overall image 400, the images 201, 202 arriving at the device 110 from the cameras 102, 103 are first corrected polynomially radially by a barrel lens correction with the parameters A, B, C and D at the motor vehicle 101. Homography is applied to the first image 201 from camera 102, which is shown above in the overall image 400. This homography ensures, for example, that the rear edge 312 is vertical, the horizon 310 is horizontally aligned, the upper corner 313 of the trailer's rear edge is just visible, not too much of the trailer is visible in the image, and the perspective distortion is not too extreme. This homography is determined, for example, by four manually defined displacements of four manually defined points in the image: the upper corner of the trailer's rear edge, the vehicle 101 in the lower part of the image, the horizon 310 at the outer edge of the image, and a point in the lower, relatively far outer part of the image to finally adjust the degree of perspective distortion. A calibration test image, which was taken beforehand, is used for this purpose. For example, the four pixels 303 and the corresponding pixels 304 in the two calibration test images 201 and 202 are annotated below. Pixels 303 and the corresponding pixels 304 are identical. For example, four test points are placed on the floor as far apart as possible so that they are still visible in both images 201 and 202. A homography transformation is applied to image 202, which is shown below in the overall image 400, so that the annotated image points 303 and 304 exactly match. For example, a vertical compression is applied to the second image 202 or the transformed second image 302, in which the top row of pixels in the edge area 308 remains fixed. The definition of image elements 303 and 304 can be performed automatically or semi-automatically as an alternative to manual annotation. For semi-automatic annotation, calibration patterns are designed, which are then automatically recognized in images 101 and 102 by the device 110 or its software. No extra calibration points are necessary for fully automatic annotation. The device 110 or its software is configured to identify suitable elements in images 201 and 202, particularly in plane 309, by means of which the homography transformation can be performed. Such elements could be, for example, road markings, manhole covers, or similar features.The method enables 400 precisely contiguous image edges and lines between all camera images 301, 302 displayed in the monitor 104 in the overall image, when displaying a section of each camera image 201, 202 that corresponds to certain criteria and specifications.

Claims

Method for operating a camera-monitor system (100) for a motor vehicle (101), wherein the camera-monitor system (100) comprises two cameras (102, 103) assigned to a common side (106, 107) of the motor vehicle and each configured to provide an image (201, 202) of an environment (105) of the motor vehicle (101), the depicted environment (105) of the images (201, 202) partially overlapping, comprising: - providing a specification for a first image (201) of the first camera (102); - transforming the first image (201) into a first transformed image (301) such that the provided specification is fulfilled in the transformed image (301); - determining at least one image element (303) on a plane (309) in the first transformed image (301); - determining a corresponding Image element (304) in at least one second image (202) of the second camera (103) that corresponds to the at least one image element (303),- Transforming the second image (202) depending on the determined image elements (303, 304) of the first transformed image (301) and the second image (202), such that a further image element (305) in a border region (308) of a transformed second image (302) connects to a corresponding further image element (306) in a border region (307) of the transformed first image (301), wherein the transformation of the second image (202) is carried out depending on the determined image elements (303, 304) such that - the determined image elements (303, 304) in the transformed second image (302) and in the transformed first image (301) are identical, and / or - a line in the border region (308) of the second transformed image (302) connects to a corresponding line in the border region (307) of the first transformed image (301) immediately follows. Method according to claim 1, comprising: lens equalization of the first image (201) and / or the second image (202) before transformation. Method according to claim 1 or 2, comprising: - Determining the at least one image element (303) on a ground plane associated with the motor vehicle (101). Method according to any one of claims 1 to 3, wherein the provided specifications comprise at least one of: - representing a horizon (310) horizontally, - representing a bottom edge (311) of the motor vehicle (101) straight, - representing a rear edge (312) of the motor vehicle (101) vertically, and - representing a specified element (313) of the motor vehicle (101) in the first transformed image (301). Method according to any one of claims 1 to 4, comprising: - further transforming the first transformed image (301) and / or the second transformed image (302) such that a further provided requirement is met. Method according to any one of claims 1 to 5, wherein at least one transformation comprises a homography transformation. Method according to any one of claims 1 to 6, comprising: - Displaying the first transformed image (301) and the second transformed image (302) as a combined image (400) on a common monitor (104) of the camera-monitor system (100). Device for a motor vehicle which is configured to perform a method according to any one of claims 1 to 7.

Images