A method for capturing surroundings of a motor vehicle by a camera device of the motor vehicle, a computer program product,

Abstract

The present invention relates to a method for capturing surroundings 16 of a motor vehicle (10, figure 1) by a camera device 14 of the vehicle, comprising the steps of providing a mirror device 22 of

Description

[0001] The present invention relates to the field of automobiles. In particular, the present invention relates to a method for capturing surroundings of a motor vehicle by a camera device of the motor vehicle according to the pending claim 1. Furthermore, the present invention relates to a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, as well as to a corresponding camera device. BACKGROUND INFORMATION

[0002] Stereo cameras in motor vehicles are often more expensive and bulkier than monocular cameras. Depth estimation with a monocular camera is challenging, because it only captures a 2D projection of the 3D world, lacking direct distance information. Monocular depth estimation algorithms typically use machine learning and require large data sets to recognize depth cues accurately. Despite advancements, these algorithms can still struggle with dynamic scenes, occlusions, and varying lighting conditions. SUMMARY OF THE INVENTION

[0003] It is an object of the present invention to provide a method, a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, as well as a corresponding camera device, by which an efficient way for capturing surroundings of a motor vehicle can be realized.

[0004] This object is solved by a method, a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, as well as a corresponding camera device according to the independent claims. Advantageous embodiments are presented in the dependent claims.

[0005] One aspect of the invention relates to a method for capturing surroundings of a motor vehicle by a camera device of the motor vehicle. A mirror device of the camera device is provided in a first time frame in a first position for capturing the surroundings in the first position. A first image in the first time frame is captured by a camera of the camera device, wherein in the first time frame an image reflected by the mirror device in the first position is captured. That mirror device is rotated in at least in one to the first position different second position in a second time frame for capturing the surroundings in the second position. At least a second image in the second time frame is captured by the camera, wherein in the second time frame an image reflected by the mirror device in the second position is captured. At least one stereo image is generated from at least the first image and the second image by an electronic computing device of the camera device.

[0006] Therefore, a rotating mirror is used to create at least two, in particular, multiple, pseudo camera locations for a single monocular camera. These pseudo camera frames enable the camera device to obtain stereo vision. By rotating a mirror in front of the monocular camera, capturing images of the surroundings from different angles is provided. The mirror effectively creates a pseudo camera behind it. With, for example, n different mirror angles, n pseudo camera images can be created. Using these n images along with the known locations of the pseudo cameras, an estimation of the three-dimensional coordinates of points can be provided.

[0007] Using this set up, an estimation of the depth with a single camera is provided. This method not only reduces costs by using only one camera, but also improves accuracy in depth estimation by capturing multiple view of the same surroundings. Instead of rotating the camera to capture multiple images from different angles, the fast-moving mirror is provided. This approach eliminates the risk of electronic component failure due to rapid motion.

[0008] According to another embodiment, a current mirror position of the mirror device is taken into consideration for generating the stereo image.

[0009] In another embodiment, the stereo image is transmitted to another electronic computing device for generating operation signals for the motor vehicle for an at least in part automated operation of the motor vehicle.

[0010] In another embodiment, from the at least one stereo image a bird’s eye view image is generated.

[0011] According to another embodiment, the camera is provided as a mono-camera.

[0012] In particular, the present method is a computer-implemented method. Therefore, another aspect of the invention relates to a computer program product comprising program code means for performing a method according to the preceding aspect.

[0013] A still further aspect of the invention relates to a non-transitory computer-readable storage medium comprising at least the computer program product according to the preceding aspect.

[0014] Furthermore, the present invention relates to a camera device for capturing surroundings of a motor vehicle, comprising at least one mirror device, one camera and one electronic computing device, wherein the camera device is configured for performing a method according to the preceding aspect. In particular, the method is performed by the camera device.

[0015] Furthermore, the present invention relates to an assistance system comprising at least the camera device for capturing the surroundings. Furthermore, the present invention relates to a motor vehicle comprising at least the assistance system and / or the camera device. The motor vehicle may be, for example, at least in part automated operated or fully automated operated.

[0016] Advantageous embodiments of the method are to be regarded as advantageous embodiments of the computer program product, the non-transitory computer-readable storage medium, the camera device, the assistance system and / or the motor vehicle.

[0017] In particular, the computing unit may include one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application-specific integrated circuits, ASIC, one or more field-programmable gate arrays, FPGA, and / or one or more systems on a chip, SoC. The computing unit may also include one or more processors, for example one or more microprocessors, one or more central processing units, CPU, one or more graphics processing units, GPU, and / or one or more signal processors, in particular one or more digital signal processors, DSP. The computing unit may also include a physical or a virtual cluster of computers or other of said units.

[0018] In various embodiments, the computing unit includes one or more hardware and / or software interfaces and / or one or more memory units.

[0019] A memory unit may be implemented as a volatile data memory, for example a dynamic random access memory, DRAM, or a static random access memory, SRAM, or as a non-volatile data memory, for example a read-only memory, ROM, a programmable read-only memory, PROM, an erasable programmable read-only memory, EPROM, an electrically erasable programmable read-only memory, EEPROM, a flash memory or flash EEPROM, a ferroelectric random access memory, FRAM, a magnetoresistive random access memory, MRAM, or a phase-change random access memory, PCRAM.

[0020] Further advantages, features, and details of the invention derive from the following description of preferred embodiments as well as from the drawings. The features and feature combinations previously mentioned in the description as well as the features and feature combinations mentioned in the following description of the figures and / or shown in the figures alone can be employed not only in the respectively indicated combination but also in any other combination or taken alone without leaving the scope of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The novel features and characteristic of the disclosure are set forth in the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and together with the description, serve to explain the disclosed principles. The same numbers are used throughout the figures to reference like features and components. Some embodiments of system and / or methods in accordance with embodiments of the present subject matter are now described below, by way of example only, and with reference to the accompanying figures.

[0022] The drawings show in:

[0023] Fig. 1 a schematic top view according to an embodiment of a motor vehicle comprising an embodiment of an assistance system comprising an embodiment of a camera device; and

[0024] Fig. 2 a schematic top view according to an embodiment of a camera device.

[0025] In the figures the same elements or elements having the same function are indicated by the same reference signs. DETAILED DESCRIPTION

[0026] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0027] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawing and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0028] The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion so that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus preceded by “comprises” or “comprise” does not or do not, without more constraints, preclude the existence of other elements or additional elements in the system or method.

[0029] In the following detailed description of the embodiment of the disclosure, reference is made to the accompanying drawing that forms part hereof, and in which is shown by way of illustration a specific embodiment in which the disclosure may be practiced. This embodiment is described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

[0030] Fig. 1 shows a schematic top view according to an embodiment of a motor vehicle 10. The motor vehicle 10 may comprise at least an assistance system 2. Therefore, the motor vehicle 10 may be configured as an at least in part automated motor vehicle or a fully automated motor vehicle. The assistance system 2 may comprise at least a camera device 14. The camera device 14 is configured for capturing surroundings 16 of the motor vehicle 10. Therefore, the camera device 14 according to the shown embodiment comprises at least one electronic computing device 18 as well as one camera 20, wherein the camera 20 is in particular a mono-camera 20.

[0031] Fig. 2 shows a schematic top view according to an embodiment of the camera device 14. In particular, Fig. 2 shows the electronic computing device 18, the camera 20 as well as a mirror device 22.

[0032] In particular, Fig. 2 shows a method for capturing the surroundings 16 of the motor vehicle 10. The mirror device 22 of the camera device 14 is provided in a first time frame in a first position P1 for capturing the surroundings 16. A first image 24 in the first time frame is captured by the camera 20, wherein the first time frame, an image reflected by the mirror device 22 in the first position P1 is captured. The mirror device 22 is rotated in at least in one to the first position P1 different second position P2 in a second time frame for capturing the surroundings 16 in the second position P2. At least a second image 26 is captured in the second time frame by the camera 20, wherein in the second time frame an image reflected by the mirror device 22 in the second position P2 is captured. At least one stereo image 28 is generated from at least the first image 24 and the second image 26 by the electronic computing device 18.

[0033] In particular, a current mirror position P1, P2 of the mirror device 22 is taken into consideration for generating the stereo image 28. Furthermore, the stereo image 28 is transmitted to another electronic computing device for generating operation signals for the motor vehicle 10, for example the assistance system 12. Furthermore, from the at least one stereo image 28 a bird’s-eye-view image is generated.

[0034] In particular, the shown invention uses a single rotary motor that can rotate the mirror device 22 quickly to capture the surroundings 16 from different vantage points. This camera device 14 allows for rapid image capture, creating a sweep of images 24, 26 from a particular scene at a high rate. This rapid capture facilitates easy feature matching across multiple images 24, 26, making the camera device 14 ideal for high-speed applications such as environmental mapping in autonomous vehicles.

[0035] Furthermore, the camera device 14 requires much less space than a multi-arm robot, making it ideal for installation in motor vehicles 10. The high-speed rotation of the mirror device 22 generates multiple images 24, 26 of the same scene from various vantage points. It is well-established that increasing the number of images 24, 26 for the same scene, significantly reduces errors in the three-dimensional reconstruction. Since the method only changes the sweep angle of the camera image 24, 25, constructing a bird’s-eye-view image (BEV) of the scene around the motor vehicle 10 is simplified. Each mirror position BEV only needs to be rotated by a single angle. The short time lack between images 24, 26 also simplifies finding correspondences, allowing the search region to be localized based on the mirror’s rotation speed, thus significantly reducing computational effort.

[0036] The approach employs a single monocular-camera 20 and the rotating mirror device 22 to capture multiple images 24, 26 of the scene from different vantage points. This method is not limited to calibration, but it is designed for efficient three-dimensional reconstruction by capturing a series of images 24, 26 with varied perspectives. Unlike the state of the art, which uses a stereo-camera setup for calibration purposes, the camera device 14 focuses on depth estimation and three-dimensional mapping using a monocamera 20. By leveraging a rotating mirror, precise depth estimation can be achieved without the need for multiple cameras, reducing hardware complexity and cost.

[0037] The camera device 14 maintains the full resolution of images 24, 26 and can capture multiple images 24, 26 of the same scene from different vantage points. The number of images 24, 26 is determined by the camera’s frame rate, which can easily reach 100 frames per second for typical cameras 20. This large set of images 24, 26, each with slight variations in vantage point, significantly reduces errors in three-dimensional reconstruction. Additionally, since consecutive images 24, 26 only have slight changes in perspective, feature search regions can be localized, thus reducing computation time significantly.

[0038] The proposed solution captures multiple images 24, 26 at the high framerate of the camera 20, typically up to 100 frames per second. It is well-established that increasing the number of vantage point images 24, 26 enhances depth estimation accuracy in three-dimensional reconstruction. Unlike prism-based methods, which split the image and reduce its quality and resolution, the mirror configuration preserves the full image quality and resolution. In prism-based methods, the image is split, which effectively halves the information captured per image. The proposed camera device 14 avoids this issue, ensuring that each image retains all its original details and quality, leading to more accurate depth estimation and improved three-dimensional reconstruction.

[0039] Since the camera device 14 captures the images 24, 26 at high rate, the features’ locations in the two images 24, 26 are closed. Hence, by localization the search region for finding a correspondence between features, for example a point 30, the computational time is significantly reduced.

[0040] In particular, the invention addresses potential discrepancies or errors in depth perception by rotating the mirror device 22 to capture multiple images 24, 26 of the same scene from vantage points. Each pair of vantage points provides a unique depth estimation, and while individual depth estimates may have some uncertainty, combining multiple depth estimations from various vantage points pairs significantly reduces this uncertainty. As the number of measurements increase, the average of these depth estimations converts towards the true depth value, enhancing the accuracy of the overall depth perception. This approach effectively mitigates errors and discrepancies that could arise from using a single stereo camera with mirrors, resulting in more reliable and precise depth estimation. The rotary mirror device 22 allows to capture images 24, 26 at a high frame rate ensuring that the location of feature points 30 between two consecutive images 24, 26 is bounded within a small region. This continuity of the scene means each feature follows a predictable trajectory through image frames.

[0041] Because a feature’s move in a bonded and predictable manner across frames, the electronic computing device 18 is able to localize the feature search in each subsequent image 24, 26 based on the feature’s location in previous frames. This reduces the computational demand from a global stretch to a localized stretch, significantly enhancing computational efficiency.

[0042] With n number of center points, the method can generate n (n-1) / 2 unique vantage point pairs. Each pair provides a separate depth estimation of the same scene, allowing for comprehensive aggregation of depth measurements.

[0043] By aggregating many depth estimations, the uncertainty and noise in individual measurements are significantly reduced. For instance, using a 60 frame per second camera 20, obtaining 60 vantage points, resulting in (60 x 59) / 2 = 1770 depth estimations. Averaging across these numerous estimations helps eliminating most of the noise and uncertainty, greatly improving depth and map accuracy. Reference signs 10 12 14 16 18 20 22 24 26 28 30 P1 P2 motor vehicle assistance system camera device surroundings electronic computing device camera mirror device first image second image stereo image point first position second position

Claims

1. A method for capturing surroundings (16) of a motor vehicle (10) by a camera device (14) of the motor vehicle (10), comprising the steps of:- providing a mirror device (22) of the camera device (14) in a first time frame in a first position (P1) for capturing the surroundings (16) in the first position (P1);- capturing a first image (24) in the first time frame by a camera (20) of the camera device (14), wherein in the first time frame an image reflected by the mirror device (22) in the first position (P1) s captured;- rotating the mirror device (22) in at least one to the first position (P1) different second position (P2) in a second time frame for capturing the surroundings (16) in the second position (P2);- capturing at least a second image (26) in the second time frame by the camera (20), wherein in the second time frame an image reflected by the mirror device (22) in the second position (P2) is captured; and- generating at least one stereo image (28) from at least the first image (24) and the second image (26) by an electronic computing device (18) of the camera device (14).

2. The method according to claim 1, characterized in thata current mirror position (P1, P2) of the mirror device (22) is taken into consideration for generating the stereo image (28).

3. The method according to claim 1 or 2, characterized in thatthe stereo image (28) is transmitted to another electronic computing device forgenerating operation signals for the motor vehicle (10) for an at least in part automated operation of the motor vehicle (10).

4. The method according to any one of claims 1 to 3, characterized in thatfrom the at least one stereo image (28) a birds eye view image is generated.

5. The method according to any one of claims 1 to 4, characterized in thatthe camera (20) is provided as a mono-camera.

6. A computer program product comprising program code means for performing a method according to any one of claims 1 to 5.

7. A non-transitory computer-readable storage medium comprising at least the computer program product according to claim 6.

8. A camera device (14) for capturing surroundings (16) of a motor vehicle (10), comprising at least one mirror device (22), one camera (20), and one electronic computing device (18), wherein the camera device (14) is configured for performing a method according to any one of claims 1 to 6.

Images