Method for arranging cameras detecting a mobile carrier platform relative to each other and for detecting the arrangement of the cameras relative to an object outside the mobile carrier platform

Abstract

The present invention relates to a method for detecting the relative arrangement of at least two cameras (103, 106) of a multi-camera system (109) for a mobile carrier platform (112) relative to each other. The method first has a reading step, in which at least a first image signal (130) and a second image signal (133) are read, wherein the first image signal (130) represents at least one image of a marker pattern (118) and first camera parameters (136) of a first camera (103), and the second image signal (133) represents at least one image of the marker pattern (118) and second camera parameters (139) of a second camera (106), wherein the second camera (106) is directed in a different viewing direction than the first camera (103), wherein the viewing areas of the first camera (103) and the second camera (106) at least partially overlap, and wherein a marker (115) with the marker pattern (118) is read as being arranged in a partial area of the viewing areas of the first camera (103) and the second camera (106). The method further has an extraction step, in which the first arrangement parameters (151) of the first camera (103) are extracted using the read first image signal (130) and / or the second arrangement parameters (154) of the second camera (106) are extracted using the read second image signal (133), wherein the first arrangement parameters (151) and the second arrangement parameters (154) represent the position and / or orientation of the first camera (103) and the second camera (106) relative to the marker (115). Finally, the method has a calculation step, in which the differential arrangement parameters (157) of the second camera (106) relative to the first camera (103) are calculated using the extracted first arrangement parameters (151) and / or second arrangement parameters (154) in order to obtain the relative camera position of the second camera (106) relative to the first camera (103).

Description

Field of the Invention

[0001] The present invention starts from an apparatus or a method according to the preamble of the independent claim. The subject matter of the present invention is also a computer program. Background Art

[0002] In order to obtain the highest possible accuracy of the data provided by these camera systems, multi-camera systems, in particular surround camera systems for vehicles, should be calibrated. Summary of the Invention

[0003] In this context, a method for detecting the relative arrangement of at least two cameras of a multi-camera system of a mobile carrier platform with respect to each other, a method for detecting the arrangement of a camera with respect to an object outside the mobile carrier platform, an apparatus using the method, and finally a corresponding computer program according to the main claim are proposed. Advantageous expansions and improvements of the apparatus described in the independent claim can be achieved by the measures listed in the dependent claims.

[0004] The method solution proposed here is based on arranging or recognizing a known two-dimensional marker pattern structure in the surroundings of a multi-camera system of a mobile carrier platform in order to detect the relative arrangement of at least two cameras of the multi-camera system with respect to each other and thereby obtain the relative camera position of a first camera with respect to a second camera. Thereby, the arrangement of a first camera of the multi-camera system with respect to an object outside the mobile carrier platform can also be detected in order to obtain the camera position of the first camera with respect to the object.

[0005] A method for detecting the relative arrangement of at least two cameras of a multi-camera system of a mobile carrier platform with respect to each other is proposed, wherein the method has the following steps:

[0006] Reading at least a first image signal and a second image signal, wherein the first image signal represents at least one image of a marker pattern and first camera parameters of a first camera, and the second image signal represents at least one image of a marker pattern and second camera parameters of a second camera, wherein the second camera is directed in a different viewing direction than the first camera, wherein the viewing areas of the first and second cameras at least partially overlap, and wherein a marker with a marker pattern is read as arranged in a partial area of the viewing areas of the first and second cameras;

[0007] Extract the first arrangement parameter of the first camera using the read first image signal and / or extract the second arrangement parameter of the second camera using the read second image signal - in particular, extract the first arrangement parameter of the first camera and / or the second arrangement parameter of the second camera using the read first and second image signals, where the first and second arrangement parameters represent the position and / or orientation of the first and second cameras relative to the marker; and

[0008] Calculate the differential arrangement parameter of the second camera relative to the first camera using the extracted first and / or second arrangement parameters, so as to obtain the relative camera position of the second camera relative to the first camera.

[0009] The camera may relate to the following photographic technology devices: which can record static or moving images on photographic film, or electronically record them on magnetic video tapes or digital storage media, or transfer the images through an interface. The camera may in particular relate to a camera of a surround camera system for a vehicle, where the camera is arranged on the vehicle and is configured to continuously detect and provide image information or images of the vehicle's surrounding environment. The multi-camera system may relate to a combination of multiple cameras, especially wide-angle cameras. The multi-camera system is applied, for example, in the automotive field, where the cameras of the multi-camera system have different viewing areas. Here, images of different viewing areas of the vehicle's surrounding environment recorded by the cameras can be combined into a 360-degree view, for example, to support the vehicle driver during difficult driving maneuvers, such as in narrow spaces. The mobile carrier platform may in particular relate to a vehicle for transporting people or goods or a working machine such as an excavator or an agricultural tractor. The camera parameters may in particular relate to the so-called internal parameters or intrinsic parameters of the camera. Here, the internal camera parameters can be understood as the internal geometry of the camera - for example, the focal length, distortion, etc. of the camera. The marker pattern may in particular relate to a known two-dimensional structure, which depicts, for example, a checkerboard pattern or a pattern with circles (symmetrically arranged, for example) and can be arranged on the marker. The marker may in particular relate to a sign for an optical marker or another object - for detecting the arrangement of at least two cameras of a multi-camera system for a mobile carrier platform and / or for detecting the arrangement of the cameras of a multi-camera system relative to an object outside the mobile carrier platform. The differential arrangement parameter can be understood, for example, as a vector of the distance or angular orientation between the first and second cameras, the respective orientations or respective positions of the first and second cameras. Thus, the relative positions or geographical arrangements of the first and second cameras relative to each other on the mobile support platform can be identified, such that this information does not need to be provided by the manufacturer of the mobile support platform. Especially when correcting or readjusting the first and second cameras, the solution proposed here has significant advantages over the prior art, because the solution for the arrangement or orientation of the first and second cameras is started with a technically easy-to-implement processing method.

[0010] There is a growing demand for multi-camera systems for vehicles, such as surround camera systems. These surround camera systems are used as visual aids for drivers by providing information about the vehicle's surrounding environment. Information about the driving surrounding environment of the vehicle is particularly useful in situations where the vehicle is maneuvered in narrow and / or enclosed spaces, or where a defined area of the vehicle's surrounding environment is not in the driver's direct line of sight due to a defined vehicle type and / or resulting different vehicle sizes (such as earthmoving construction machinery).

[0011] The advantage of the method proposed herein lies in particular in that, in order to detect the arrangement of at least two cameras of a multi-camera system of a mobile carrier platform (such as a vehicle) and / or in order to detect the arrangement of the cameras of the multi-camera system relative to an object outside the mobile carrier platform (such as a vehicle), no complex and expensive structure is required. This solution only requires a marker with at least one marker pattern, which is arranged in the visual surrounding environment of the mobile carrier platform (such as a vehicle) in order to perform the detection of the arrangement of at least two cameras relative to each other or the arrangement of the cameras relative to an object outside the mobile carrier platform, in order to obtain the respective positions of the cameras of the multi-camera system.

[0012] In the method proposed herein, for example, a plurality of markers can also be arranged in the overlapping observation areas of at least two adjacent cameras of the multi-camera system in order to perform the detection. Here, there are no restrictions regarding the arrangement of at least one marker and / or the markers relative to each other or relative to the mobile carrier platform, except that the markers should be clearly recognizable by the cameras. This situation enables the simple and time-saving implementation of this method, especially in the case of a large number of cameras, or in cases of time and / or space limitations, such as in a production enterprise. Since this involves an easily executable solution, no specialized knowledge is required for a technician to implement this method.

[0013] According to one embodiment, in the reading step, the third camera may read at least a third image signal representing at least one image of the marking pattern of the second marking and third camera parameters, wherein the third camera is oriented in an observation direction different from that of the first and / or second cameras, wherein the observation areas of the first, second and / or third cameras at least partially overlap, wherein the second marking is read as being arranged in a partial area of the observation areas of the first and third cameras and / or the second and third cameras, wherein, in the extraction step, the third image signal is used to extract third arrangement parameters of the third camera, and wherein, in the calculation step, the extracted third arrangement parameters are used to calculate differential arrangement parameters of the third camera relative to the second and / or first cameras in order to obtain the relative camera position of the third camera relative to the second and / or first cameras. This embodiment of the solution proposed herein offers the following advantages: The cameras of the multi-camera system can be arranged at any position on the vehicle, provided that there is sufficient overlap of the observation areas between adjacent cameras (i.e., the first and third cameras and / or the second and third cameras). In addition, the distortion in the overlapping observation areas of the first, second and / or third cameras should also be low enough to enable the successful detection of the arrangement of at least two cameras of the multi-camera system relative to each other. It is also conceivable that the same markings are used in this embodiment as in the embodiment of method 1 proposed herein. To this end, the spatial position of the markings can be changed after the steps of the embodiment of the method proposed herein have been carried out. In the method solution proposed herein, it is also possible to use only one marking as long as the images are recorded without changing the camera position and / or camera orientation.

[0014] According to one embodiment, in the reading step, the first and second markings may be read on the same side of the mobile carrier platform or on adjacent sides of the mobile carrier platform or on opposite sides of the mobile carrier platform. This embodiment of the solution proposed herein offers the following advantages: The first and second cameras of the multi-camera system observe the same marking or the same marking pattern from different observation angles, whereby the camera position of the first camera relative to the second camera can be calculated.

[0015] According to one embodiment, in the reading step, the first and / or second image signal may be read by the first and / or second wide-angle cameras with a field of view of at least 130 degrees, in particular at least 170 degrees. This embodiment of the solution proposed herein offers the following advantages: A wide-angle camera can, for example, provide a particularly representative spatial depth of the vehicle's surroundings. In addition, by using at least four cameras in the multi-camera system, each observation area of the vehicle's surroundings - such as in front of the vehicle, behind the vehicle, on the left side of the vehicle, on the right side of the vehicle - is covered, so that the driver of the vehicle has a 360-degree all-round view.

[0016] According to one embodiment, the extraction step can be carried out using a pre-known marker pattern structure, in particular where the marker pattern structure depicts a chessboard pattern and / or a pattern with arranged circles and / or a two-dimensional pattern. This embodiment of the method proposed here offers the following advantages: For example, the chessboard pattern represents a geometrically simple binary pattern with alternating black and white rectangles, where the pattern is rotation-invariant due to the arrangement of the rectangles and can thus be clearly recognized. Here, in principle, one marker pattern is sufficient to carry out the solution proposed here, and when using multiple marker patterns, time can be saved by reducing the steps involved in data detection.

[0017] A method for detecting the arrangement of a camera relative to an object outside a mobile carrier platform is proposed, where the method has the following steps:

[0018] Read an object image signal, where the object image signal represents at least one image of an object marker pattern and camera parameters of a first camera; and

[0019] Use the read object image signal and the first camera parameters to determine object arrangement parameters of the first camera relative to the object marker, where the object arrangement parameters represent the position and / or orientation of the first camera relative to the object marker in order to obtain the camera position of the first camera relative to the object marker.

[0020] The object marker can be a marker with a marker pattern. The marker used in the method for detecting the arrangement of at least two cameras in a multi-camera system can in particular also be used in the method for detecting the arrangement of a camera relative to an object outside a mobile carrier platform.

[0021] According to one embodiment, the method may include the following steps: providing reference arrangement parameters between a first reference point and a second reference point in the viewing area of a first camera, and using the provided reference arrangement parameters to determine at least the camera position of the first camera relative to the second reference point. The first reference point may refer to a point of a color marker on or at an object or just a virtual point. The first reference point is preferably arranged on a marker in the viewing area of the first camera. The first reference point may be, for example, a point on the marker pattern of the marker. The second reference point is preferably arranged at, on, or in a mobile carrier platform. The second reference point may be, for example, a point at the center or in the middle on an axis (especially the vehicle axis of the mobile carrier platform). The reference arrangement parameters between the first reference point and the second reference point may be understood as the spatial orientation of the reference points relative to each other, i.e., the relative position and the distance relative to each other - for example, in a coordinate system with the coordinate origin at the second reference point. The provision of the reference arrangement parameters may include determining the spatial distance or relative orientation of the reference points relative to each other by means of a suitable distance-detecting device. Preferably, the relative orientation of the reference points is determined by means of a rangefinder or a tape measure. It is particularly preferred that the reference points, especially the first camera, are arranged along an imaginary line. It is also advantageous that the symmetry axis of the marker is oriented parallel to the symmetry axis of the mobile carrier platform. This simplifies the determination of the relative orientation of the reference points and the determination of the camera position relative to the first reference point.

[0022] The determination of the camera position of at least the first camera relative to the second reference point using the provided reference arrangement parameters corresponds to a coordinate transformation from a coordinate system with the position of the first camera as the coordinate origin to a coordinate system with the position of the second reference point as the coordinate origin. By means of the method described herein, the coordinate system can be converted or transformed into a coordinate system with an arbitrarily pre-given coordinate origin. Thus, it is not necessary to accurately know the spatial position of the camera at the mobile carrier platform.

[0023] According to one embodiment, a method for detecting the arrangement of a camera relative to an object outside a mobile carrier platform may include the following steps: using differential arrangement parameters and the arrangement parameters of a first camera to determine an additional object arrangement parameter of a second camera relative to an object marker, where the additional object arrangement parameter represents the position and / or orientation of the second camera relative to the object marker in order to obtain the camera position of the second camera relative to the object marker. This embodiment of the solution proposed herein provides the following advantages: By knowing the camera position of the first camera relative to the object marker, knowing the camera position of the second camera relative to the object marker, and knowing the relative camera position of the second camera relative to the first camera, for example, calibration of a multi-camera system can be performed, and the multi-camera system can be readjusted.

[0024] According to one embodiment, in the reading step, a pattern corresponding to the marking pattern can be read as the object marking pattern. This embodiment of the solution proposed herein offers the following advantage: One marking pattern is sufficient to successfully execute the method solution proposed herein.

[0025] According to one embodiment, an object image signal can be read in the reading step, wherein the axis of the object marking pattern is oriented along the viewing axis of the first camera, and this viewing axis has less distortion compared to the other viewing axes of other viewing regions of the first camera. This embodiment of the solution proposed herein offers the following advantage: The read object image signal is processed to determine the misalignment or offset between the image patterns, or to determine the misalignment or offset between the reference point of the object marking and the reference point of the multi-camera system, so as to thereby obtain the camera position of the first camera relative to one of the object marking or the reference point.

[0026] One or more of the methods proposed herein can be implemented, for example, in software or hardware or in a hybrid form of software and hardware, such as in a device or a control device.

[0027] The solution proposed herein also implements a device that is configured to perform, control, or implement the steps of a variant of the proposed method in a corresponding apparatus. The task underlying the present invention can also be solved quickly and effectively by an implementation variant in the form of the device of the present invention.

[0028] For this purpose, the device can have at least one computing unit for processing signals or data, at least one storage unit for storing signals or data, at least one interface to a sensor or to an actuator for reading sensor signals from the sensor or outputting data signals or control signals to the sensor, and / or at least one communication interface for reading or outputting data embedded in a communication protocol. The computing unit can be, for example, a signal processor, a microcontroller, etc., wherein the storage unit can be a flash memory, an EEPROM, or a magnetic storage unit. The communication interface can be configured to read or output data wirelessly and / or wiredly, wherein a communication interface for reading or outputting wired data can read data from or output the data to a corresponding data transmission line, for example, electrically or optically.

[0029] In the present context, the device can be understood as an electrical device that processes sensor signals and outputs control signals and / or data signals based on the sensor signals. The device can have an interface constructed in hardware form and / or software form. In a hardware form construction, the interface can be, for example, part of a so-called system ASIC that includes various functions of the device. However, the interface can also be a separate integrated circuit or at least partially include discrete components. In a software form construction, the interface can be, for example, a software module that also exists on a microcontroller in addition to other software modules.

[0030] Also advantageous is a computer program product or a computer program having program code that can be stored on a machine-readable carrier or storage medium such as a semiconductor memory, a hard disk, or an optical memory, and is used to execute, implement, and / or control the steps of a method according to one of the above-described embodiments - especially when the program product or the program is implemented on a computer or a device. Description of the Drawings

[0031] Embodiments of the solution proposed herein are shown in the drawings and further elaborated in the following description. The drawings show:

[0032] Figure 1 A schematic diagram of a device showing the arrangement of at least two cameras of a multi-camera system for detecting a mobile carrier platform relative to each other;

[0033] Figure 2 A schematic diagram of a device showing the arrangement of a camera relative to an object outside a mobile carrier platform according to an embodiment;

[0034] Figure 3 A schematic system structure showing a method for using the arrangement of at least two cameras of a multi-camera system for detecting a mobile carrier platform relative to each other according to an embodiment and a method for detecting the arrangement of a camera relative to an object outside a mobile carrier platform;

[0035] Figure 4 A schematic system structure showing a method for using the arrangement of at least two cameras of a multi-camera system for detecting a mobile carrier platform relative to each other according to an embodiment;

[0036] Figure 5 A schematic system structure showing a method for using the arrangement of a camera relative to an object outside a mobile carrier platform according to an embodiment;

[0037] Figure 6A flowchart showing an embodiment of a method for detecting the relative arrangement of at least two cameras of a multi-camera system for detecting a mobile carrier platform with respect to each other;

[0038] Figure 7 A flowchart showing an embodiment of a method for detecting the arrangement of a camera with respect to an object outside a mobile carrier platform.

[0039] In the following description of advantageous embodiments of the present invention, the same or similar reference numerals are used for elements shown in different figures and acting similarly, where the repeated description of these elements is omitted. Detailed Description

[0040] Figure 1 A schematic diagram of a device 100 for detecting the relative arrangement of at least two cameras 103, 106 of a multi-camera system 109 for detecting a mobile carrier platform 112 with respect to each other is shown. Here, the device 100 is arranged on the mobile carrier platform 112, where the mobile carrier platform 112 is, for example, a vehicle 112. The multi-camera system 109, which at least includes a first camera 103 and a second camera 106, is arranged on the vehicle 112, where the first camera 103 and the second camera 106 are, for example, wide-angle cameras with a 170-degree field of view. At least one marker 115 with a two-dimensional marker pattern structure 118 is arranged in the immediate surroundings of the vehicle 112. Here, the marker pattern structure 118 is, for example, a pattern with six symmetrically arranged circles, which are arranged in two rows here.

[0041] Device 100 has a reading device 121, an extraction device 124, and a calculation device 127. The reading device 121 is configured to read at least a first image signal 130 and a second image signal 133, wherein the first image signal 130 represents an image of a marker pattern 118 and first camera parameters 136 of a first camera 103, and the second image signal 133 represents an image of the same marker pattern 118 and second camera parameters 139 of a second camera 106. Here, the second camera 106 is oriented or directed to a different viewing direction and position than the first camera 106, wherein the viewing areas 142 and 145 of the first camera 103 and the second camera 106 at least partially overlap, and the marker 115 with the marker pattern 118 is read as being arranged in a partial area 148 of the viewing areas 142 and 145 of the first camera 103 and the second camera 106. The extraction device 124 is configured to use the read first image signal 130 and second image signal 133 to extract first arrangement parameters 151 of the first camera 103 and / or second arrangement parameters 154 of the second camera 106, wherein the first arrangement parameters 151 and the second arrangement parameters 154 represent the position and / or orientation of the first camera 103 and the second camera 106 relative to the marker 115. Finally, the calculation device 127 is configured to use the extracted first arrangement parameters 151 and / or second arrangement parameters 154 to calculate differential arrangement parameters 157 of the second camera 106 relative to the first camera 103 in order to obtain the relative camera position of the second camera 106 relative to the first camera 103.

[0042] According to one embodiment, the vehicle 112 has a third camera 160, wherein the third camera 160 is oriented to a different viewing direction than the first camera 103 and / or the second camera 106, and the viewing areas 142, 145, and 163 of the first camera 103, the second camera 106, and / or the third camera 160 at least partially overlap. According to one embodiment, a second marker 166 is also arranged in the immediate surroundings of the vehicle 112, wherein the first marker 115 and the second marker 166 are arranged on opposite sides of the vehicle 112, and the second marker 166 is arranged in a partial area 169 of the viewing areas 142 and 163 of the first camera 103 and the third camera 160. Both markers 118 and 166 have the same marker pattern structure 118.

[0043] Now, the reading device 121 is further configured to read at least a third image signal 172 of the third camera 160. Now, the extraction device 124 is further configured to use the read third image signal 172 to extract a third arrangement parameter 175 of the third camera 160. Finally, the calculation device 127 is now configured to use the extracted third arrangement parameter 175 to calculate a differential arrangement parameter 178 of the third camera 160 relative to the second camera 106 and / or the first camera 103, so as to obtain the relative camera position of the third camera 160 relative to the second camera 106 and / or the first camera 103.

[0044] Figure 2 Schematic illustration of a device 200 for detecting the arrangement of a camera 103 relative to an object 205 outside a mobile carrier platform 112 according to an embodiment. Here, the device 200 is arranged on the mobile carrier platform 112, where the mobile carrier platform 112 is, for example, a vehicle 112. According to an embodiment, a camera 103 is arranged on the vehicle 112, which relates to the first camera 103 or the reference camera 103. According to an embodiment, a second camera 106 is also arranged on the vehicle 112. At least one object 205 is arranged in the immediate surroundings of the vehicle 112, where the object 205 is an object marker 205 having an object marker pattern structure 208. Here, the object marker pattern structure 208 corresponds to the marker pattern structure of a marker from Figure 1 the marker, where the object marker pattern structure 208 depicts, for example, a pattern having six symmetrically arranged circles. The viewing area 142 of the first camera 103 is shown in dashed lines, where, according to an embodiment, the object marker 205 is arranged in the direct viewing area 142 of the first camera 103.

[0045] The device 200 has a reading device 210, a determination device 215, and an obtaining device 220. The reading device 210 is configured to read an object image signal 225, where the object image signal 225 represents an image of the object marker pattern 208 and the camera parameters 136 of the first camera 103. Here, the reading device 210 is particularly configured to read the object image signal 225, where the axis of the object marker pattern 208 is oriented along the optical axis of the first camera 103, which has less distortion compared to another optical axis of another viewing area of the first camera 103. The determination device 215 is configured to use the read object image signal 225 and the first camera parameters 136 to determine an object arrangement parameter 230 of the first camera 103 relative to the object marker 205, where the object arrangement parameter 230 represents the position and / or orientation of the first camera 103 relative to the object marker 205, so as to obtain the camera position of the first camera 103 relative to the object marker 205. Finally, the obtaining device 220 is configured to use the data fromFigure 1 The difference arrangement parameter 157 provided by the computing device of the device and read by the obtaining device 220 and the object arrangement parameter 230 of the first camera 103 are used to obtain another object arrangement parameter 235 of the second camera 106 relative to the object marker 205, wherein the said another object arrangement parameter 235 represents the position and / or orientation of the second camera 106 relative to the object marker 205, so as to obtain the camera position of the second camera 106 relative to the object marker 105.

[0046] Figure 3 According to an embodiment, a schematic system structure 300 is shown for a method of using at least two cameras 103, 106 of a multi-camera system 109b for detecting a mobile carrier platform 112 to arrange relative to each other and a method of detecting the arrangement of a camera 103 relative to an object 205 outside the mobile carrier platform 112.

[0047] The schematic system structure 300 has a mobile carrier platform 112, wherein the shown mobile carrier platform 112 is, for example, a vehicle 112. According to an embodiment, four cameras 103, 106, 160 and 305 are arranged on the vehicle 112, wherein the cameras 103, 106, 160 and 305 are, for example, wide-angle cameras of a multi-camera system 109 (such as a surround camera system) for the vehicle 112. The viewing areas 142, 145, 163 and 310 of the cameras 103, 106, 160 and 305 are shown as dotted lines. The system structure 300 also has five markers 115, 166, 205, 315 and 320, wherein the marker 205 is the object marker 205, which is used to obtain the camera position of the first camera 103 or the reference camera 103 relative to itself. The markers 115, 166, 315 and 320 are respectively clearly visible and arranged in the overlapping viewing areas 148, 169, 325 and 330 of the respectively adjacent cameras 103 and 106, 103 and 160, 106 and 305, and 160 and 305. There is no limitation on the arrangement of the markers 115, 166, 315 and 320 relative to each other or relative to the vehicle 112. This enables a simple and time-saving implementation of the method solutions proposed herein, especially in the case of multiple cameras, or in the case of time and / or space limitations, such as in a production enterprise.

[0048] To implement a method for detecting the arrangement of a first camera 103 or a reference camera 103 relative to an object 205 outside a mobile carrier platform 112, an additional object 205 is arranged in the immediate surroundings of the vehicle 112 and serves as an object marker 205 in order to orient the reference point of the multi-camera system 109 to a pre-given and / or pre-givable reference point of the vehicle 112. Here, the arrangement of the additional object marker 205 is matched relative to the vehicle 112. This makes the method particularly useful in cases where flexibility in choosing the reference point is desired or the reference point cannot be determined in advance. Markers 115, 166, 315, and 320 are arranged in the overlapping regions 148, 169, 325, and 330 of the viewing areas 142, 145, 163, and 310 of cameras 103, 106, 160, and 305, and the object marker 205 is arranged in the immediate viewing area 142 of the reference camera 103, wherein the axis of the object marker pattern is oriented to the visual axis of the reference camera 103. If the system configuration 300 includes multiple cameras, multiple reference cameras can also be selected in order to minimize the cumulative error in the camera-marker pattern-camera-marker pattern chain.

[0049] Figure 4Schematic system structure 400 showing a method for using the arrangement of at least two cameras 103, 106 of a multi-camera system 109 for detecting a mobile carrier platform 112 relative to each other. The mobile carrier platform 112 is, for example, a vehicle 112. The system structure 400 shown here can be used with any multi-camera system 109 including at least two cameras 103, 106, which, according to one embodiment, includes four cameras 103, 106, 160 and 305. The viewing areas 142, 145, 163 and 310 of the cameras 103, 106, 160 and 305 are shown as dashed lines. The cameras 103, 106, 160 and 305 are arranged asymmetrically on the vehicle 112, where although this asymmetric arrangement is not a requirement of the method solution presented here. The system structure 400 also has four markers 115, 166, 315 and 320, where the markers 115, 166, 315 and 320 are respectively visibly arranged in the overlapping viewing areas 148, 169, 325 and 330 of two adjacent cameras 103 and 106, 103 and 160. There is no restriction on the arrangement of the markers 115, 166, 315 and 320 relative to each other or relative to the vehicle 112. Each marker 115, 166, 315 and 320 carries the same marker pattern 118, where the marker pattern 118 used here depicts a group of six symmetrically arranged circles, and the marker pattern to be used is not limited to this one marker pattern 118. A general two-dimensional marker pattern structure, such as a checkerboard pattern, can also be used. The prerequisite for using a marker pattern structure is that the geometric information of the marker pattern is known - for example, the distance between the center points of the circles in this case.

[0050] As long as the markers 115, 166, 315 and 320 are arranged in the at least partially overlapping viewing areas 148, 169, 325 and 330 of the cameras 103, 106, 160 and 305 and the marker pattern 118 can be clearly identified in the image, the system structure 400 is not restricted with respect to the position and orientation of the individual markers 115, 166, 315 and 320. This enables a person not skilled in the art to arrange the markers 115, 166, 315 and 320 around the vehicle 112 arbitrarily and simply. For maintaining good quality, each of the markers 115, 166, 315 and 320 should be matched such that it is respectively located in the middle overlapping viewing areas 148, 169, 325 and 330 of the corresponding camera images, where the influence of distortion is minimal.

[0051] Since markers 115, 166, 315, and 320 are used in a method for arranging at least two cameras 103, 106 of a multi-camera system 109 for detecting a moving carrier platform 112 relative to each other to estimate the orientation differences of cameras 103, 106, 160, and 305 relative to each other, the positions and orientations of markers 115, 166, 315, and 320 themselves can also be set individually to improve their visibility. Thus, for example, markers 115, 166, 315, and 320 can be arranged at an angle of 45 degrees to the vehicle plane or at a height of 1 meter from the ground. The positions and orientations of markers 115, 166, 315, and 320 do not change the positions and orientations of cameras 103, 106, 160, and 305 relative to each other as long as markers 115, 166, 315, and 320 are clearly visible and at least two cameras (e.g., 103 and 106) observe the same marker (e.g., 115) at the same position and orientation.

[0052] Figure 5 FIG. 500 shows a schematic system configuration for using a method for detecting the arrangement of a camera 103 relative to an object 205 outside a moving carrier platform 112 according to one embodiment. The moving carrier platform 112 is, for example, a vehicle 112. The system configuration 500 shown here can be used with any multi-camera system 109 including at least two cameras 103, 106, where, according to one embodiment, the multi-camera system 109 shown here includes four cameras 103, 106, 160, and 305. The camera 103 is hereinafter referred to as the reference camera 103. The viewing area 142 of the reference camera 103 is shown in dashed lines in Figure 5 FIG.

[0053] To implement a method for detecting the arrangement of a first camera 103 or a reference camera 103 relative to an object 205 outside a moving carrier platform 112, an object marker 205 additionally arranged in the immediate surroundings of the vehicle 112 is used as the object 205 in order to orient the reference point of the multi-camera system 109 to the reference point R2 of the vehicle 112. Here, the arrangement of the additional object marker 205 is matched relative to the vehicle 112, where the object marker 205 is arranged in the immediate viewing area 142 of the reference camera 103, and the axis of the object marker pattern 208 is oriented along the viewing axis 505 of the reference camera 103, which has less distortion compared to another viewing axis 510 of another viewing area of the reference camera 103.

[0054] When implementing this method, according to one embodiment, the reference point of the multi-camera system 109 is moved from the reference camera 103 to the reference point R2 of the vehicle 112. To orient the reference point R2 of the vehicle 112, a temporary object reference point R1 is used, where the object reference point R1 is arranged on the object marker 205. The object image data of the image recorded by the reference camera 103 representing the object marker pattern 208 is processed to determine the misalignment or offset between the image patterns or the misalignment or offset between the reference points R1 and R2. Then, the misalignment offset in the X and Y directions between the object reference point R1 and the reference point R2 of the vehicle 112 is provided to the device for detecting the arrangement of the camera 103 relative to the object 205 outside the moving carrier platform 112 of the vehicle, so as to obtain the position of the first camera 103 relative to the object marker 205. For this purpose, the change in orientation - i.e., roll, pitch, and / or yaw - between the object reference point R1 and the reference point R2 of the vehicle 112 is measured by the reference camera 103. To simplify the above steps, the reference camera 103 can be arranged along the visual axis 510 connecting the reference point R2 and the object reference point R1 of the vehicle 112, thereby eliminating the determination of the Y misalignment. If the object reference point R1 and the reference point R2 of the vehicle 112 are arranged along the same zero plane, the determination of the Z misalignment can also be eliminated.

[0055] It is proposed here that the object marker can be a marker with a marker pattern. The marker 115 used in the variant of the method for detecting the arrangement of at least two cameras (e.g., the first camera 103 and the second camera 106) of the multi-camera system proposed here can also be used in the variant of the method for detecting the arrangement of the camera relative to an object outside the moving carrier platform of the vehicle 112 proposed here.

[0056] Here, such a processing method can be set to provide reference arrangement parameters between a first reference point R1 and a second reference point R2 in the observation area of the first camera 103, and use the provided reference arrangement parameters to determine at least the camera position of the first camera 103 relative to the second reference point R2. The first reference point R1 can refer to a point of a color marker on or at an object (such as the object marker 205) or just a virtual point. In the observation area of the first camera 103, the first reference point R1 is preferably arranged on a marker such as the object marker 205. The first reference point R1 can be, for example, a point on the marker or the marker pattern of the object marker 205. The second reference point R2 is preferably arranged, for example, at the mobile carrier platform 112, on the mobile carrier platform or in the mobile carrier platform. The second reference point R2 can be, for example, a point at the center or in the middle on the axis 510, especially on the vehicle axis of the mobile carrier platform 112. The reference arrangement parameters between the first reference point R1 and the second reference point R2 can be understood as the spatial orientation of the reference points R1, R2 relative to each other, i.e., the relative positions X, Y and the distance relative to each other, for example, in a coordinate system with the second reference point R2 as the coordinate origin. The provision of the reference arrangement parameters can be set to obtain the spatial distance or relative orientation of the reference points R1, R2 relative to each other by means of a suitable distance detection device. Preferably, the relative orientation of the reference points R1, R2 is obtained by means of a rangefinder or a tape measure. Particularly preferably, the reference points R1, R2, especially the first camera 103, are arranged along the imaginary line 510. It is also advantageous that the symmetry axis 510 of the marker or the object marker 205 is oriented parallel to the symmetry axis 505 of the mobile carrier platform 112. This simplifies the obtaining of the relative orientation of the reference points R1, R2 and the obtaining of the camera position relative to the first reference point R1.

[0057] The determination of the camera position of at least the first camera 103 relative to the second reference point R2 using the provided reference arrangement parameters corresponds to a coordinate transformation from a coordinate system with the position of the first camera 103 as the coordinate origin to a coordinate system with the position of the second reference point R2 as the coordinate origin. With the embodiments described herein with respect to this method, the coordinate system can be converted or transformed into a coordinate system with an arbitrarily pre-given coordinate origin. Thus, it is not necessary to accurately know the spatial positions of the cameras 103, 106, 160, 305 at the mobile carrier platform 112.

[0058] Figure 6 A flowchart showing an embodiment of a method 600 for detecting the relative arrangement of at least two cameras of a multi-camera system for a mobile carrier platform relative to each other. Here, according to an embodiment, it can be from Figure 1The method 600 proposed herein is implemented on a device for detecting the relative arrangement of at least two cameras of a multi-camera system for a mobile carrier platform relative to each other.

[0059] In step 610 of method 600, at least a first and a second image signal are read, wherein the first image signal represents an image of a marker pattern and first camera parameters of a first camera, and the second image signal represents an image of a marker image and second camera parameters of a second camera, wherein the second camera is oriented in a different viewing direction than the first camera, wherein the viewing areas of the first and second cameras at least partially overlap, and wherein a marker with a marker pattern is read as being arranged in a partial area of the viewing areas of the first and second cameras. In a subsequent step 620, the first arrangement parameters of the first camera are extracted using the read first image signal and / or the second arrangement parameters of the second camera are extracted using the read second image signal - in particular, the first arrangement parameters of the first camera and / or the second arrangement parameters of the second camera are extracted using the read first and second image signals, wherein the first and second arrangement parameters represent the position and / or orientation of the first and second cameras relative to the marker. Finally, in step 630 of method 600, the extracted first and / or second arrangement parameters are used to calculate differential arrangement parameters of the second camera relative to the first camera in order to obtain the relative camera position of the second camera relative to the first camera.

[0060] Figure 7 FIG. shows a flowchart of an embodiment of a method 700 for detecting the arrangement of a camera relative to an object external to a mobile carrier platform according to one embodiment. Here, according to one embodiment, it can be implemented on a device for detecting the arrangement of a camera relative to an object external to a mobile carrier platform from Figure 2 The method 700 proposed herein is implemented on a device for detecting the arrangement of a camera relative to an object external to a mobile carrier platform. Before method 700 can be implemented, for example, the steps of method 600 for detecting the relative arrangement of at least two cameras of a multi-camera system for a mobile carrier platform from Figure 6 can be implemented.

[0061] In step 710 of method 700, an object image signal is read, where the object image signal represents an image of an object marker pattern and the camera parameters of a first camera. In the subsequent step 720, the read object image signal and the first camera parameters are used to determine object arrangement parameters of the first camera relative to the object marker, where the object arrangement parameters represent the position and / or orientation of the first camera relative to the object marker in order to obtain the camera position of the first camera relative to the object marker. Finally, in step 730 of method 700, a further object arrangement parameter of a second camera relative to the object marker is determined using the differential arrangement parameters and the object arrangement parameters of the first camera, where the further object arrangement parameter represents the position and / or orientation of the second camera relative to the object marker in order to obtain the camera position of the second camera relative to the object marker.

[0062] If an embodiment includes an "and / or" connection between a first feature and a second feature, this connection is to be interpreted such that the embodiment includes both the first feature and the second feature according to one embodiment, and either only the first feature or only the second feature according to another embodiment.

Claims

1. A method for detecting the arrangement of a camera (103) relative to an object (205) outside a mobile carrier platform (112), wherein, the method (700) has the following steps: reading (710) an object image signal (225), wherein the object image signal (225) represents at least one image of an object marker pattern (208) and camera parameters (136) of a first camera (103); and using the read object image signal (225) and the first camera parameters (136) to determine (720) object arrangement parameters (230) of the first camera (103) relative to the object marker (205), wherein the object arrangement parameters (230) represent the position and / or orientation of the first camera (103) relative to the object marker (205) so as to obtain the camera position of the first camera (103) relative to the object marker (205), the method has the following steps: providing reference arrangement parameters between a first reference point (R1) and a second reference point (R2) in an observation area (142) of the first camera (103), and using the provided reference arrangement parameters to determine at least the camera position of the first camera (103) relative to the second reference point (R2), wherein the first reference point (R1) is arranged on the object marker (205), and the second reference point (R2) is arranged at, on or in the mobile carrier platform (112).

2. The method according to claim 1, wherein, detecting the arrangement of at least two cameras (103, 106) of a multi-camera system (109) of a mobile carrier platform (112) relative to each other, wherein the detection has the following steps: reading (610) at least a first image signal (130) and a second image signal (133), wherein the first image signal (130) represents at least one image of a marker pattern (118) and first camera parameters (136) of a first camera (103), and the second image signal (133) represents at least one image of a marker pattern (118) and second camera parameters (139) of a second camera (106), wherein the second camera (106) is oriented in a different observation direction from the first camera (103), wherein the observation areas (142, 145) of the first camera (103) and the second camera (106) at least partially overlap, and a first marker (115) with the marker pattern (118) is read as being arranged in a partial area (148) of the observation areas (142, 145) of the first camera (103) and the second camera (106); The first arrangement parameter (151) of the first camera (103) is extracted (620) using the read first image signal (130) and / or the second arrangement parameter (154) of the second camera (106) is extracted (620) using the read second image signal (133), wherein the first arrangement parameter (151) and the second arrangement parameter (154) represent the positions and / or orientations of the first camera (103) and the second camera (106) relative to the first marker (115); The differential arrangement parameter (157) of the second camera (106) relative to the first camera (103) is calculated (630) using the extracted first arrangement parameter (151) and / or the extracted second arrangement parameter (154) so as to obtain the relative camera position of the second camera (106) relative to the first camera (103).

3. The method according to claim 2, wherein in the step of the reading (610), a third camera (160) reads at least a third image signal (172), and the third image signal represents at least one image of the marker pattern (118) of the second marker (166) and third camera parameters, wherein, the third camera (160) is oriented in an observation direction different from that of the first camera (103) and / or the second camera (106), wherein the observation regions (142, 145, 163) of the first camera (103) and / or the second camera (106) and / or the third camera (160) at least partially overlap, wherein the second marker (166) is read as being arranged in a partial region (169) of the observation regions (142, 145, 163) of the first camera (103) and the third camera (160) and / or the second camera (106) and the third camera (160), wherein in the step of the extraction (620), the third arrangement parameter (175) of the third camera (160) is extracted using the third image signal (172), and wherein in the step of the calculation (630), the differential arrangement parameter (178) of the third camera (160) relative to the second camera (106) and / or the first camera (103) is calculated using the extracted third arrangement parameter (175) so as to obtain the relative camera position of the third camera (160) relative to the second camera (106) and / or the first camera (103).

4. The method according to claim 3, wherein in the step of the reading (610), the first marker (115) and the second marker (166) are read on the same side of the mobile carrier platform (112) or on adjacent sides of the mobile carrier platform or on opposite sides of the mobile carrier platform.

5. The method according to any one of claims 2 to 4, wherein in the step of said reading (610), the first wide-angle camera (103) and / or the second wide-angle camera (106) reads the first image signal (130) and / or the second image signal (133) with a field of view of at least 130 degrees.

6. The method according to any one of the above claims 2 to 4, wherein in the step of said extraction (620), a known marker pattern structure (118) can be used.

7. The method according to any one of claims 2 to 4, the method having the following steps: using a differential arrangement parameter (157) and an object arrangement parameter (230) of the first camera (103) to obtain (730) another object arrangement parameter (235) of the second camera (106) relative to the object marker (205), wherein, the said another object arrangement parameter (235) represents the position and / or orientation of the second camera (106) relative to the object marker (205) so as to obtain the camera position of the second camera (106) relative to the object marker (205).

8. The method according to any one of claims 2 to 4, wherein in the step of said reading (710), a pattern corresponding to the marker pattern (118) is read as an object marker pattern (208).

9. The method according to any one of claims 1 to 4, wherein in the step of said reading (710), an object image signal (225) is read, wherein, the axis of the object marker pattern (208) is oriented with the visual axis (505) of the first camera (103), and compared with another visual axis (510) of another observation area of the first camera (103), the visual axis (505) has less distortion.

10. The method according to claim 5, wherein in the step of said reading (610), the first wide-angle camera (103) and / or the second wide-angle camera (106) reads the first image signal (130) and / or the second image signal (133) with a field of view of at least 170 degrees.

11. The method according to claim 6 above, wherein the marker pattern structure (118) depicts a checkerboard pattern and / or a pattern with arranged circles and / or a two-dimensional pattern.

12. A device (100; 200) for detecting the arrangement of a camera (103) relative to an object (205) outside a mobile carrier platform (112), the device being arranged to implement and / or control the steps of the method according to any one of claims 1 to 11 in corresponding units.

13. A computer program product, the computer program product comprising instructions which, when executed by a computer, cause the computer to implement the method according to any one of claims 1 to 11.

14. A machine-readable storage medium having instructions stored thereon, which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 11.

Images