Method for calibrating support system of consumer automobile

Abstract

To provide a method for calibrating a support system of a consumer automobile and a consumer automobile for executing the method.SOLUTION: A method includes: capturing image data around a consumer automobile by a camera device in an image sensor coordinate system; capturing depth data showing a first distance between the consumer automobile and an object around by a depth sensor device; deriving a second distance between the consumer automobile and the object represented in the image data by image analysis based on artificial intelligence in a data processing device; creating three-dimensional image data from the image data in the three-dimensional image sensor coordinate system and the second distance; and mutually calibrating the image data and the depth data in the data processing device. The calibration includes determining a conversion rule. The method also includes applying the conversion rule to image data and / or depth data acquired subsequently by the data processing device.SELECTED DRAWING: Figure 2

Description

Detailed Description of the Invention

[0001] The present invention relates to a method for calibrating an assistance system of a civilian motor vehicle.

[0002] [background] In order to improve road traffic safety, consumer automobiles are increasingly equipped with devices that measure and analyze depth data. Depth data is used in particular to detect objects present in the automobile's surroundings. Depth data may be integrated into driver assistance systems, for example in terms of parking assistance, autonomous driving assistance, accident avoidance, etc. Depth data is measured by depth sensors. Examples of conventional depth sensors include lidar (light detection and ranging) and radar.

[0003] In addition to depth data, it is also known to use image data in civilian vehicles. The image data is acquired in the form of individual still images and also in the form of continuous image sequences (video). To record the images, one or more cameras may be used for the vehicle. The image data may be used to supplement or improve the driver's field of vision. In particular, it is known to combine depth data with image data and display these to the driver. For example, closer objects may be highlighted in the displayed representation to warn the driver of a dangerous situation.

[0004] In order to combine the image data and the depth data, they need to be calibrated. A known calibration method involves providing an object with known dimensions at a known distance from the vehicle. The object is recorded by the vehicle's depth sensor and camera. Software is then set up to analyze the image data and the depth data such that the recorded object's features, such as edges and corners, can be correlated with each other in the image data and the depth data.

[0005] [overview] It is an object of the present invention to provide a method for calibrating an assistance system in a civilian vehicle.

[0006] To achieve this, a method for calibrating an assistance system of a civilian vehicle is provided according to independent claim 1. Furthermore, a civilian vehicle is provided according to independent claim 14. Embodiments are subject matter of the dependent claims.

[0007] According to one aspect, there is provided a method for calibrating an assistance system of a consumer vehicle. The method includes capturing image data of a surrounding of the consumer vehicle in an image sensor coordinate system by a camera device disposed on the consumer vehicle. The method includes capturing depth data of a surrounding of the consumer vehicle by a depth sensor device disposed on the consumer vehicle. The depth data indicates a first distance between (i) the consumer vehicle and (ii) an object in the surrounding of the consumer vehicle in a depth sensor coordinate system. The method includes deriving, in a data processing device, by artificial intelligence based image analysis from the image data, a second distance between the consumer vehicle and the object in the surrounding of the consumer vehicle represented in the image data. The method includes generating, from the image data and the second distance, three-dimensional image data in a three-dimensional image sensor coordinate system. The method includes mutually calibrating, in the data processing device, the image data and the depth data. Mutually calibrating the image data and the depth data includes determining a transformation rule. According to the transformation rule, (i) the three-dimensional image data is transformed into the depth sensor coordinate system such that coordinates of a first distance coincide with coordinates of a second distance in the depth sensor coordinate system, (ii) the depth data is transformed into the three-dimensional image sensor coordinate system such that coordinates of a first distance coincide with coordinates of a second distance in the three-dimensional image sensor coordinate system, or (iii) the three-dimensional image data and the depth data are transformed into a predetermined coordinate system such that coordinates of a first distance coincide with coordinates of a second distance in the predetermined coordinate system. The method includes applying the transformation rule by the data processing device to the subsequently acquired image data and / or depth data.

[0008] Image data is usually present in a raster format. The coordinate system of the image sensor is two-dimensional. The origin of the coordinate system may be located, for example, at one of the corners of the raster (or grid), at the center of the raster, or at any point within the predefined raster. The horizontal axis of the image sensor coordinate system may correspond to the horizontal direction of the camera device. The vertical axis may correspond to the vertical direction.

[0009] The depth data is three-dimensional and exists as a point cloud and / or as a collection of vector data. In particular, the depth data may exist in the form of a line, where all points on the line have the same distance to the observer or the depth sensor. The origin of the depth sensor coordinate system may correspond to the position of the depth sensor. The horizontal and vertical axes may correspond to the horizontal and vertical directions of the depth sensor. The third axis may correspond to its optical axis.

[0010] When deriving the distance between a civilian vehicle and an object represented in the image data, three-dimensional data is generated from two-dimensional data. An artificial intelligence preferably used for this purpose is, for example, a neural network trained by training data to determine a specific distance from the origin of the image sensor coordinate system for each pixel (picture element) of the two-dimensional raster. This includes determining the distance (spacing) of the object based on stereo recording (when using two or more camera devices) or based on the change of successive images in the image sequence, employing other methods for determining the distance, or combining these methods. The derived distance may be presented in the same units as the two-dimensional data (for example, meters per pixel). The distance may also be presented in other units. Alternatively, the distance may be presented in an expanded or condensed representation format compared to the two-dimensional data. In any case, however, the distance values ​​match each other and correctly reproduce the relationship for the spacing. The distance values ​​can be matched to the two-dimensional data by setting or determining a conversion factor.

[0011] The actual calibration of image data and depth data with respect to each other is used to create rules by which points, vectors, lines and / or objects of both data are assigned to each other, which can then be applied to the recorded data (image data or depth data) so that they can be mapped to each other, for example to be displayed to the driver in an overlaid manner. In the context of the present disclosure, the coincidence of distance coordinates should be understood to mean the coincidence of points of coordinates where the respective distances end, which are assigned to each other by being in correspondence with each other.

[0012] In one embodiment, the calibration includes first determining matching characteristic object parts (characteristic parts of the object) from the distances determined in the image data and the distances determined in the depth data. The object parts include, for example, the three-dimensional coordinates of the corners or edges of the object, individual salient points, or the entire object. Characteristic object parts are parts of the object that are particularly representative of the object in the image data and / or depth data and may be sufficient to at least partially define the shape of the object. Examples of characteristic object parts include corners of a cube, a rectangular prism, or corners of other geometric bodies bounded by planes. In these examples, corners are more characteristic than edges and more characteristic than individual points or parts of the body surface. In the case of a sphere, a general solid of revolution, or any organic shape, the characteristic object parts may be given by their outline or parts. Matching characteristic object parts are characteristic object parts of both data sets that can be associated with each other within a predetermined tolerance. Matching characteristic object parts therefore represent the same object or the same parts of the object with a high probability.

[0013] The transformation rule may be determined according to one of at least three embodiments. In a first embodiment, the three-dimensional image data is transformed such that the second distance derived from the three-dimensional image data matches the distance derived from the depth data. Thus, the transformation rule is first determined in three-dimensional space based on the distances of both data. The transformation rule is then projected into two-dimensional space and then applied to the further image data. The transformation applied in this case may include translation, rotation, shear, scaling, etc. When the transformation rule is applied, the image data is transformed into the depth sensor coordinate system after the depth sensor coordinate system is projected into two dimensions. In the embodiment including the determination of the characteristic object part in the above description, the transformation rule is determined based on the characteristic object part instead of the distance of the image data. In this case, the characteristic object part is matched to the corresponding coordinate of the depth sensor data.

[0014] In a second embodiment, the three-dimensional depth data is transformed so that the distances in the three-dimensional image data match those in the depth data. In this case, the depth data is mapped into the coordinate system of the three-dimensional image data. In an embodiment relating to distinctive object parts, instead of distances, the determined object parts are matched to corresponding coordinates in the depth data.

[0015] In a third embodiment, the three-dimensional image data and the depth data are transformed into a predefined coordinate system. The image sensor distance coordinates and the depth sensor coordinates, or alternatively in one embodiment the characteristic object parts of both coordinate systems, coincide in the predefined coordinate system. In one embodiment, the predefined coordinate system is three-dimensional as well. In one embodiment, the predefined coordinate system is two-dimensional and the transformation rule maps from the two three-dimensional coordinate systems onto a two-dimensional space, for example by projection. The predefined coordinate system may be predefined by a user, for example by selecting an image part in the displayed image data. Alternatively, the predefined coordinate system may be created automatically as an average of the three-dimensional image sensor coordinate system and the depth sensor coordinate system, for example by determining the center of overlap of both coordinate systems (the three-dimensional image sensor coordinate system and the depth sensor coordinate system) as the origin of the predefined coordinate system.

[0016] In one embodiment, determining the transformation rule in the three-dimensional space may include projecting a number of points from the depth data to a two-dimensional image sensor coordinate system and assigning the number of said points to points or objects in the image data to determine the correspondence of data in both data sets. The assignment may be performed based on certain similarities of points / objects in both data sets (e.g. similar luminance values, similar luminance values ​​in a neighborhood, similar color values, similar color values ​​in a neighborhood). The assignment may be performed iteratively for each point of the depth data. The iterations may be performed independently of each other and therefore in parallel to speed up the assignment.

[0017] The transformation rule thus determined is preferably a matrix (transformation matrix), which may be a product of several matrices defining transformations (e.g. rotation, scaling, shear). Depending on the application, the matrix may perform a two-dimensional or three-dimensional transformation. In the three-dimensional case, the matrix may be expanded by parameters / coefficients depending on the third dimension. The relevant format of the transformation matrix for transformation to three dimensions is known to the person skilled in the art.

[0018] If the transformation matrix is ​​known, it can be applied to the subsequently acquired image data and / or depth data to perform further analysis on both data (image data and depth data) or to display the data in a combined form. As an example, the data can be displayed superimposed to give the viewer a sense of distance to certain objects. Alternatively, objects in the image data can be emphasized or attenuated based on their distance to draw the viewer's attention to closer objects and not to distract the viewer from more distant objects. For this purpose, different intensities or colors can be used. For example, a stronger intensity and / or color can be used for closer objects compared to more distant objects, and vice versa.

[0019] The mutual calibration of the image data and the depth data may be performed continuously in real time. Alternatively, the calibration may be performed at regular intervals or at a predetermined time. In one embodiment, the calibration may be performed when the lighting conditions change significantly. For example, the method may include (i) continuously measuring the brightness of the surroundings, and (ii) repeating the calibration when the brightness falls below a predetermined threshold or when the brightness rises above the threshold (or another threshold). In one embodiment, multiple conversion rules may be determined for different lighting conditions. Depending on the brightness, each conversion rule is applied to the image data and / or the depth data.

[0020] The method may include fusing the image data transformed by applying the transformation rule and / or the depth data transformed by applying the transformation rule with each other or with the remaining image data or depth data to which the transformation rule is not applied, where image pixels having matching coordinates and distances are assigned. The fusing of data may include overlaying, for example by adding or subtracting intensity or color values. Alternatively, as already explained, individual objects in the image data may be enhanced or attenuated based on their distance according to the depth data.

[0021] In particular, the fused data is used in driver assistance systems in consumer automobiles, which may include displaying the data. This may include determining which objects are within a predetermined minimum distance. In one embodiment, if an object is detected that is too close to the automobile, a warning may be displayed. The warning may include displaying the associated object and pointing to the object.

[0022] In certain embodiments, characteristic object parts are determined. Determining matching characteristic object parts in the image data and the depth data may include optimizing the image data by an image optimization algorithm. The optimization may include applying certain filter operations (e.g., applying edge detectors, contrast filters, filters to suppress image noise). The data filtered in this way may be further investigated for cohesive pixels, for example by segmentation and / or local or global maxima / minima analysis. Also, the image data and / or the depth data may be subjected to the above mentioned filter operations without using characteristic object parts.

[0023] Producing three-dimensional image data from the two-dimensional image data and producing from said image data determined distances between the consumer vehicle and objects around the consumer vehicle represented in the image data may include adding as a third dimension to the already existing two-dimensional image data the respective distances from the consumer vehicle, resulting in a three-dimensional point cloud, where each point is associated with the intensity and / or color of the corresponding image point.

[0024] Thereafter, applying the transformation rule to the acquired image data may include applying a two-dimensional projection of the transformation rule, and further, applying the transformation rule to the depth data may include performing a two-dimensional projection on the depth data before or after applying the transformation rule.

[0025] The method may be performed during driving operation of a civilian vehicle. In particular, the method may be performed without the use of a calibration standard, which would result in a calibration based on objects having known dimensions and known distances from the civilian vehicle. As mentioned above, the invention includes determining distances or measured distances in image data and depth data, and in some embodiments alternatively determining characteristic object parts, and automatically mapping them to each other. Thus, the invention eliminates the need to provide the above-mentioned objects. Any surroundings and any objects with any dimensions and distances may be used. Furthermore, the calibration method is repeatable at any time without setting up or visiting a given environment.

[0026] The depth data may be captured by at least one depth sensor device mounted on the consumer vehicle and selected from the group of a lidar measurement device, a time-of-flight measurement device, and a radar measurement device. In particular, the image data may be captured by at least one camera device mounted on the consumer vehicle and selected from the group of a video image camera device, a thermal image camera device, and an infrared camera device.

[0027] Image analysis in the case of deriving from image data the distance between a consumer vehicle and objects around the consumer vehicle represented in the image data may be based on artificial intelligence in the form of neural networks, in particular monodepth networks.

[0028] Further image data of the surroundings of the consumer vehicle in a further image sensor coordinate system may be captured by a further camera device provided in the consumer vehicle. When using image data of multiple camera devices, the calibration step may be performed as follows: First, image data of the further camera device is captured as second image data. In these image data, a step for deriving a distance between the consumer vehicle and an object represented in the second image data is performed similarly to the above-mentioned derivation example. In the first embodiment, the second image data is subsequently calibrated with the image data of the first camera device by determining a matching distance, in one embodiment alternatively or additionally by determining a characteristic object part and by determining a transformation rule. Unlike the above-mentioned steps performed on the basis of the first image data and the depth data, the calibration in this example uses the first image data and the second image data as well. In the second embodiment, the second image data is calibrated with the depth data. Unlike the above-mentioned steps, the calibration in this example uses the second image data and the depth data as well. In the third embodiment, the second image data is successively calibrated with the first image data and the depth data. Thus, in each of these embodiments, at least one transformation rule is determined that is applicable to the subsequently acquired (first and second) image data and / or depth data, such that the transformation rule can be used not only to calibrate the image data with the depth data, but also to calibrate image data originating from multiple sources with the depth data.

[0029] Further depth data about the surroundings of the consumer vehicle may be captured by a further depth sensor device provided in the consumer vehicle. The further depth data indicates a distance between the consumer vehicle and objects around the consumer vehicle in a further depth sensor coordinate system. Thus, similar to the example described above with respect to the plurality of (first and second) image data, a calibration may be performed based on the image data and the depth data, as well as the further depth data. Also in this example, multiple transformation rules may be determined, for example, between the depth data and the further depth data, between the image data and the further depth data, and between combinations of all three data.

[0030] A further aspect of the invention disclosed herein additionally relates to a consumer vehicle having a camera device, a depth sensor device, and a data processing device, where the consumer vehicle is configured to perform one or more of the methods described herein.

[0031] The embodiments described above with respect to a method for calibrating an assistance system of a consumer vehicle may be provided correspondingly for said consumer vehicle and vice versa.

[0032] Description of Exemplary Embodiments Further exemplary embodiments are described in more detail below with reference to the figures in the drawings, in which: FIG. 1 shows a schematic diagram of a known method for calibrating data; FIG. 2 shows a schematic diagram of a method according to the present invention for calibrating data; FIG. 3A shows an example of image data; FIG. 3B shows an example of depth data; FIG. 3C shows an example of fusion of image data and depth data; FIG. 4 shows a schematic diagram of the determination of the conversion rule.

[0033] 1 shows a schematic diagram of a known method for calibrating an assistance system of a consumer vehicle. In step 110, image data are determined by a camera device mounted on the consumer vehicle. These image data are subjected to a brightness / intensity analysis in step 115 in order to determine certain image features. These image features are for example edges or corners of the imaged object.

[0034] Step 125 concerns the acquisition of three-dimensional range data, for example by a lidar camera device. The range data, for example in the form of a point cloud, is also subjected to a brightness / intensity analysis in step 130. This analysis typically first involves a reduction of the range data to two-dimensional image data, in which features are detected in a similar manner to step 115.

[0035] In step 135, features in both data sets that can be assigned to each other are determined within the range of each detected feature. In step 140, a function is determined from these assignments. The function can be used to map the image data into two-dimensional aggregate distance data. The function can then be used to transform subsequent image data. The results of such a transformation can be combined with the distance data to indicate and / or highlight the distance of objects in the image data.

[0036] FIG. 2 shows a schematic diagram of a method according to the invention for calibrating an assistance system of a consumer vehicle. In step 210, the method captures two-dimensional image data by a camera device mounted on the consumer vehicle. In step 215, from these image data, distances between the consumer vehicle and objects represented in the image data are derived, thus generating three-dimensional data. This derivation is preferably performed by an image analysis based on artificial intelligence, in particular by using a neural network that derives information on the distance of the respective object or pixel of the image from specific image features. This derivation is performed by a data processing device, for example a component of the on-board electronics of the consumer vehicle. Alternatively, the data processing device is housed in a remote server and receives the image data to be processed via wireless communication. As a result of step 210, three-dimensional image data is present. The third dimension of the three-dimensional image data is given by the determined distance.

[0037] In one embodiment, in step 220, characteristic object parts are determined from the derived distances and optionally also from the underlying image data. Step 220 is not essential for the method 200 in one embodiment. Characteristic object parts are parts of the detected object or else the whole of said object. Characteristic object parts include in particular corners, edges and / or boundaries of the object. In particular, characteristic object parts each include a number of coordinates belonging to the same object. This determination may be performed, for example, by a 3D edge detector or a connected component analysis, or by other methods, in particular neural networks.

[0038] In step 225, depth data of the surroundings of the civilian vehicle is acquired. For this purpose, a laser scanner with lidar functionality or a radar device is preferably used. Every device for measuring meters of distance is in principle suitable for capturing depth data. The depth data is present in three dimensions.

[0039] In one embodiment, the depth data is also analyzed for distinctive object parts, similar to step 220. As previously explained, this involves determining groups of coordinates that belong to the same object.

[0040] In step 235, the mutually corresponding distances of both data sets are determined from the distance coordinates in the 3D image data and the distance coordinates in the depth data. This may be performed by correlating both data sets. Additionally or alternatively, statistical analysis and / or optimization may be performed. In one embodiment, the correspondence of these coordinates in both data sets is determined and the correspondence is maximized by transinformation, mutual information, and / or cinetropy. In one embodiment, instead of the distance coordinates, a number of matching object parts in both data sets may be determined as characteristic object parts for the distance derived from the image data and the depth data. For example, a correlation method can determine which of a number of cohesive coordinates of one data set match or correspond to a cohesive coordinate of the other data set. If the correspondence is above a predefined threshold, the coordinates of both data sets are marked as corresponding to each other or are stored together.

[0041] Step 240 concerns the determination of a transformation rule that associates the image data and the depth data with each other. As an example, the transformation rule includes one or more transformation matrices that transform the image coordinates and / or the depth coordinates into a different coordinate system. In addition to or instead of the transformation matrices, a non-linear transformation may be performed. As an example, the image data is transformed from a three-dimensional image sensor coordinate system to a depth sensor coordinate system. This transformation is performed on the condition that the three-dimensional coordinates of the distance in the three-dimensional image sensor coordinate system coincide with the distance coordinates in the three-dimensional depth sensor coordinate system after the transformation. The three-dimensional transformation rule thus obtained can be reduced to a two-dimensional transformation rule. The two-dimensional transformation rule can then be applied to the two-dimensional image data. For example, the three-dimensional transformation rule can be represented by one or more matrices that are transformed into the two-dimensional transformation rule by omitting matrix coefficients. Similar operations are possible for the non-linear transformation rules. The two-dimensional transformation rule can then be applied to the two-dimensional image data. This allows the two-dimensional image data to be transformed into the depth sensor coordinate system. In one embodiment, the image data may be transformed into a two-dimensional projection of the depth sensor coordinate system.

[0042] In a further example, step 240 includes transforming the depth data into a three-dimensional image sensor coordinate system such that the distance coordinates of both coordinate systems match. In this example, the transformation rule is again initially three-dimensional. In this form, the transformation rule may be applied to subsequent depth data so that the subsequent depth data matches the subsequent (two-dimensional) image data. This involves (i) first transforming the subsequent depth data three-dimensionally, and (ii) then projecting the transformed depth data into the two dimensions of the image data, or otherwise applying the two-dimensional projection of the transformation rule to the two-dimensionally projected depth data.

[0043] In a further example, step 240 includes transforming the three-dimensional image data and the depth data into a predefined coordinate system such that the distance coordinates of the image sensor coordinate system and the depth sensor coordinate system match in the predefined coordinate system. In one embodiment, the predefined coordinate system is also three-dimensional. And the determined transformation rule also maps from three dimensions to three dimensions. Alternatively, the predefined coordinate system may be two-dimensional. In this case, the transformation rule maps from three dimensions to two dimensions.

[0044] After step 240, the determined transformation rule may then be applied to the recorded image data and / or depth data.

[0045] 3A, 3B, and 3C show exemplary image data and depth data. FIG. 3A shows image data 300A, represented here as a grayscale image. Each pixel has a luminance value. The image data may be represented in color, for example as an RGB image. FIG. 3B shows depth data 300B. Each pixel has a luminance value that defines the distance of the respective position in the periphery of the consumer vehicle from the recording device. For example, a brighter value may represent closer, and vice versa. The depth data 300B is represented here as a grayscale image. However, the depth data may also include color data, for example with RGB coding. Similarly, distance may be represented by a color value. Alternatively, the color value may code further information, for example a change in relation to previously recorded depth data.

[0046] In the image data 300A and the depth data 300B, objects such as house facades, cars, cyclists, and pedestrians are identifiable. Furthermore, the image data and depth data show different parts of the surroundings of the recording device being used because the camera device and depth sensor are oriented differently and / or mounted in different positions.

[0047] The image data and depth data presented may be subjected to the above-mentioned analysis. The distance between the camera device and the represented object is derived from the image data. The distance is added as a third dimension to the image data. In one embodiment, characteristic object parts may be determined in both the three-dimensional image data and the depth data. Then, characteristic object parts that match in both coordinate systems may be determined. From these characteristic object parts that match, a transformation rule may be determined. The transformation rule is then applied to the image data and / or the depth data. This transforms these data into the respective corrected coordinate systems. As an example, regardless of whether all the distance data or only characteristic object parts are observed, the obtained data may be superimposed / fused. Such a superimposition is shown as image 300C in FIG. 3C. Image 300C shows both the luminance values ​​of the image data 300A and the luminance values ​​of the depth data 300B. The luminance values ​​of the depth data 300B can be recognized as a continuous line indicating the distance from the depth sensor by color and / or luminance value. As can be seen from image 300C, the difference in the orientation and the location of the recording device is improved by the conversion rule, so that the image data and the depth data can be properly superimposed.

[0048] FIG. 4 illustrates several coordinate systems used in the examples described herein. L4 shows a three-dimensional depth sensor coordinate system 410 and a two-dimensional image sensor coordinate system 420. The points are located around the camera device and the depth sensor or around the consumer vehicle to which they are mounted. FIG. 4 further shows a three-dimensional depth sensor coordinate system 410 and a two-dimensional image sensor coordinate system 420. From the image data that may be included in the coordinate system 420, distances are derived and then three-dimensional data is generated. These are illustrated by the three-dimensional coordinate system 430. In one embodiment, from the image data and distances on the one hand and from the depth data on the other hand, respective characteristic object parts may be determined. Furthermore, corresponding characteristic object parts may be determined in the coordinate system 410 and the coordinate system 430. In order to map the distance coordinates or characteristic object parts of both coordinate systems to each other, a transformation rule T CL may then be determined, which may then be collapsed, for example, to two dimensions and then applied to the image sensor coordinate system 420. The resulting modified image sensor coordinate system is illustrated by the displacement of the x and y axes in the coordinate system to the u and v axes.

[0049] The features disclosed in the above description, the claims and the drawings may be of importance both individually and in any desired combination to realise various embodiments. [Brief description of the drawings]

[0050] [Figure 1] 1 shows a schematic diagram of a known method for calibrating data. [Diagram 2] FIG. 2 shows a schematic diagram of a method according to the present invention for calibrating data. [Figure 3A] An example of image data is shown. [Figure 3B] 1 shows an example of depth data. [Figure 3C] 1 shows an example of fusing image data and depth data. [Figure 4] 1 illustrates a schematic diagram of determining a conversion rule.

Claims

1. 1. A method for calibrating an assistance system in a consumer vehicle, comprising: Capturing image data of a surrounding of a consumer vehicle in an image sensor coordinate system by a camera device provided on the consumer vehicle; capturing depth data about a surrounding of the consumer vehicle by a depth sensor device provided on the consumer vehicle, the depth data indicating a first distance between the consumer vehicle and objects about the consumer vehicle in a depth sensor coordinate system; deriving from the image data a second distance between the consumer vehicle and objects around the consumer vehicle represented in the image data by artificial intelligence based image analysis in a data processing device, and generating three-dimensional image data in a three-dimensional image sensor coordinate system from the image data and the second distance; and calibrating, in the data processing device, the image data and the depth data relative to each other; Calibrating the image data and the depth data relative to each other includes: (i) the three-dimensional image data is transformed into the depth sensor coordinate system such that coordinates of the first distance and coordinates of the second distance coincide in the depth sensor coordinate system; (ii) the depth data is transformed into the three-dimensional image sensor coordinate system such that coordinates of the first distance and the second distance coincide in the three-dimensional image sensor coordinate system; or (iii) the three-dimensional image data and the depth data are transformed into a predetermined coordinate system such that coordinates of the first distance and coordinates of the second distance coincide in the predetermined coordinate system; determining a transformation rule; The method comprises: applying, by the data processing device, the transformation rule to subsequently acquired image data and / or depth data.

2. The method of claim 1 , wherein the transformation rule specifies a matrix for coordinate transformation.

3. The method according to claim 1 or 2, wherein the mutual calibration of the image data and the depth data is performed continuously in real time.

4. the method comprising: (i) fusing the image data transformed by applying the transformation rule and / or the depth data transformed by applying the transformation rule with each other or (ii) with remaining image data or depth data to which the transformation rule has not been applied; The method according to claim 1 or 2, wherein image pixels having coordinates and distances that coincide with each other are assigned.

5. The method of claim 4 , wherein the fused data is used in a driver assistance system of the consumer vehicle.

6. Thereafter, applying the transformation rule to the acquired image data includes applying a two-dimensional projection of the transformation rule; The method of claim 1 or 2, wherein applying a transformation rule to the depth data comprises performing a two-dimensional projection on the depth data before or after applying the transformation rule.

7. The method of claim 1 or 2, performed during driving operation of the consumer vehicle.

8. The method of claim 1 or 2, performed without the use of calibration standards.

9. 3. The method according to claim 1 or 2, wherein the depth data is captured by at least one depth sensor device provided in the consumer vehicle and selected from the group of a lidar measurement device, a time-of-flight measurement device, and a radar measurement device.

10. 3. The method of claim 1 or 2, wherein the image data is captured by at least one camera device provided in the consumer vehicle and selected from the group of a video imaging camera device, a thermal imaging camera device, and an infrared camera device.

11. 3. The method according to claim 1 or 2, wherein the image analysis, in which a distance between the consumer motor vehicle and objects around the consumer motor vehicle represented in the image data is derived from the image data, is based on artificial intelligence in the form of a neural network, in particular a single-depth network.

12. The method of claim 1 or 2, wherein the method comprises capturing further image data of a surrounding of the consumer vehicle in a further image sensor coordinate system by a further camera device provided on the consumer vehicle.

13. 3. The method of claim 1 or 2, comprising capturing further depth data around the consumer vehicle by a further depth sensor device provided in the consumer vehicle, the further depth data indicating a distance between the consumer vehicle and objects around the consumer vehicle in a further depth sensor coordinate system.

14. A camera device; A depth sensor device; A consumer vehicle comprising: A consumer vehicle performing the method of claim 1.

Images