Method for determining visual odometry in a driving system

Abstract

The invention relates to a method for determining visual odometry in a driving system. Furthermore, the invention relates to a computer program, a device, and a storage medium for this purpose.

Description

[0001] The invention relates to a method for determining a visual odometry in a driving system. Furthermore, the invention relates to a computer program, a device, and a storage medium for this purpose. State of the art

[0002] Visual odometry often represents an essential cornerstone in the calculation of robotics and driver assistance systems. Its relevance is particularly evident in autonomous driving, where it serves as an essential input for a variety of system functions that rely directly or indirectly on this information. In this context, the need for not only accurate but also robust visual odometry determination becomes increasingly important. It is of utmost importance that a reliable result can be guaranteed in all driving situations.

[0003] In traditional automotive or robotics applications, visual odometry is preferably calculated from a single image data stream, which can be sufficient for most applications. It is also possible to calculate visual odometry using a stereo camera, where two cameras with identical properties are arranged in parallel. When estimating visual odometry, five degrees of freedom are generally estimated. These are, in particular, the three rotation angles (yaw, pitch, and roll) and the direction of movement (translation). The length of the translation, however, is not usually part of visual odometry, as it is purely image-based and cannot be determined without additional information. The direction of the translation can also be referred to as the focus of expansion (FoE). This is specifically the point in the image from which everything is moving away.Another synonym for the FoE can be epipole. Visual odometry is determined using image correspondences or optical flow, for example.

[0004] For this purpose, the image contents of two images are preferably compared and matching points are saved as a correspondence. If the correspondences are calculated not just for specific points but for each pixel, this is preferably referred to as optical flow. Each pixel then has a flow vector that describes the displacement. With the introduction of multi-camera systems in vehicles, the possibilities for calculating visual odometry simultaneously based on multiple camera data streams are expanding. For example, parking systems sometimes have four cameras, each aimed at one side of the vehicle. In this case, visual odometry can be usefully calculated based on all four camera data streams, since the information is not redundant and the number of cameras is manageable. At the same time, not all image data streams contribute equally to improving the result.For example, images from side-view cameras are less useful at high speeds than data from front and rear cameras. On the other hand, telephoto cameras with a small aperture angle are particularly unsuitable for clearly determining the rotational degrees of freedom of visual odometry. Disclosure of the invention

[0005] The subject matter of the invention is a method having the features of claim 1, a computer program having the features of claim 13, a device having the features of claim 14, and a computer-readable storage medium having the features of claim 15. Further features and details of the invention emerge from the respective subclaims, the description, and the drawings. Features and details described in connection with the method according to the invention naturally also apply in connection with the computer program according to the invention, the device according to the invention, and the computer-readable storage medium according to the invention, and vice versa, so that with regard to the disclosure of the individual aspects of the invention, reference is or can always be made to each other.

[0006] The invention particularly relates to a method for determining a visual odometry in a driving system, e.g., a vehicle. Preferably, at least the following steps are performed, which can be performed repeatedly and / or sequentially.

[0007] Camera data can be provided in one step, wherein the camera data comprises at least image data resulting from recording by at least two cameras of the driving system during a movement, and metadata of the at least two cameras. The image data and metadata can thus be assigned to the individual cameras. Thus, image data resulting from recording by a respective camera of the driving system will also be referred to below as image data of this camera. At least two or more, e.g., at least three, at least four, or at least five, cameras can be provided.

[0008] In a further step, the provided camera data can be evaluated based on at least one weighting criterion to determine weighting information for each of the at least two cameras. The respective weighting information can include at least one or more weights for different criteria of the at least one weighting criterion.

[0009] In a further step, the visual odometry can be determined using the determined weighting information. This means, in particular, that the camera data is taken into account for the odometry according to the weighting information, meaning that different cameras can have a different influence on the determination of the odometry.

[0010] In visual odometry, a position and / or orientation of the driving system is preferably determined. It can be provided that the image data from each camera comprises only two individual frames. The movement referred to in the context of providing the camera data relates in particular to a movement of the driving system. In simplified terms, the weighting information can represent the extent to which each camera should be considered for determining the visual odometry. Weighting the cameras can be advantageous because it takes into account that, depending on the different environmental conditions, direction of travel, or speed of the driving system, different cameras may be particularly or less relevant.

[0011] Furthermore, it is advantageous if the method further comprises that the determination of the visual odometry takes place using the determined weighting information in that a degree of consideration of the image data resulting from the recording by the respective camera, i.e. the image data of this respective camera, for the determination of the visual odometry is determined on the basis of a multiplication and / or summation of several weights of the weighting information for this respective camera. The multiple weights result in particular from an evaluation based on various criteria of the at least one weighting criterion, such as the aperture angle criterion or the height criterion, which are described in detail below. The degree can refer to an extent to which the image data are used to determine, in particular calculate, the odometry.

[0012] Alternatively or additionally, the method may comprise selecting at least one of the cameras based on the weighting information determined for the selected camera in order to determine the visual odometry based on the camera data, in particular only from the at least one selected camera. Provision may be made for a camera not to be selected if the weighting information corresponds to a value of 0. This allows cameras that are not relevant or only relevant to a limited extent to be excluded from determining the odometry. As a result, computing power can be advantageously saved and valuable computing capacity can be reserved for critical driving maneuvers.

[0013] Furthermore, it is conceivable within the scope of the invention that an aperture angle of the respective camera is determined according to an (aperture angle) criterion of the at least one weighting criterion, wherein the aperture angle is preferably determined based on the metadata of the respective camera. Preferably, a weight of the weighting information for the respective camera is minimized, in particular set to 0, if the aperture angle of the respective camera falls below a predefined minimum angle. This criterion can also be referred to as the aperture angle criterion. The minimum angle can be defined by a user, or it can be determined by external requirements. In other words, the camera, or the camera data of the respective camera, is not taken into account for determining the visual odometry if the defined minimum angle is greater than the aperture angle of the respective camera.A larger camera aperture angle can mean that a larger scene section can be observed. This leads, in particular, to a more robust motion estimation. Conversely, if the aperture angle is very small, there is a risk that the motion estimation can no longer distinguish between lateral translation and yaw motion.

[0014] A further advantage within the scope of the invention can be achieved if, according to a criterion of the at least one weighting criterion, a ratio between pixels in the image data of the respective camera that represent self-moving objects and a total number of pixels in the image data of the respective camera is determined. This criterion can also be referred to as the object criterion. The ratio can be implemented using various image recognition methods known in the art, such as segmentation or machine learning models. Self-moving objects can exhibit a different displacement between individual images in the image data. Self-moving objects negatively influence the results of the estimation, particularly if it is assumed that the scene is purely static.By having a priori knowledge about the presence or absence of potentially self-moving objects, this information can be advantageously used for weighting or for determining the weighting information of the respective camera.

[0015] It is also advantageous if a position of an epipole is determined according to a criterion of the at least one weighting criterion, wherein the position of the epipole is determined based on the image data of the respective camera. This criterion can also be referred to as the epipole criterion. Preferably, a weight of the weighting information for the respective camera is minimized, in particular set to 0, if the position of the epipole lies outside a detection range of the respective camera. In other words, a respective camera, or the camera data of the respective camera, is not taken into account for determining the visual odometry if the epipole is located outside the image data.

[0016] Furthermore, within the scope of the invention, it can be provided that an exposure duration of the respective camera is determined according to a criterion of the at least one weighting criterion, wherein the exposure duration is preferably determined based on the metadata of the respective camera. This criterion can also be referred to as the exposure criterion. By taking the exposure duration into account by appropriately adjusting the weighting information of the respective camera, motion blur, a direct result of the movement of the image content in the image and the exposure duration, can also be advantageously taken into account.

[0017] According to a further possibility, it can be provided that a height of the respective camera relative to a surface of the driving system is determined according to a criterion of the at least one weighting criterion, wherein the height of the respective camera is preferably determined based on the metadata of the respective camera. This criterion can also be referred to as the height criterion. The closer the respective camera is arranged to a road surface, the less favorable the image processing in the road area can become. In a moving vehicle, a large spatial offset of the image data between two points in time and thus a geometric distortion can occur. The influence of this distortion due to the height of the camera can advantageously be reduced by corresponding weight information of the respective camera, which can be particularly important at higher speeds.

[0018] Furthermore, within the scope of the invention, it can be provided that, according to a criterion of the at least one weighting criterion, the presence of a windshield in front of the respective camera is checked in order to take into account a spatial offset of light rays caused by the windshield. The presence of the windshield in front of the respective camera is preferably checked based on the metadata of the respective camera. The spatial offset caused by the windshield leads, in particular, to a distortion of the image data. The influence of this distortion caused by the windshield can advantageously be reduced by appropriate weight information from the respective camera.

[0019] Furthermore, it can be advantageous within the scope of the invention if at least one criterion of the at least one weighting criterion is an exclusion criterion and / or a priority criterion. The exclusion criterion can determine the suitability of the respective camera for determining the visual odometry in the driving system, and the priority criterion can determine the priority of the respective camera for determining the visual odometry in the driving system. The weighting information is determined, in particular, by performing a calculation with the exclusion criterion as a multiplier and / or the priority criterion as a summand. The criteria of the aperture angle, the self-moving objects, and the epipole position can be exclusion criteria, while the criteria for exposure duration, height relative to a ground surface of the driving system, and the presence of a windshield can be priority criteria.In particular, it may be provided that the priority criteria are summed up and then multiplied by the exclusion criteria.

[0020] It may optionally be possible for rotational and / or translational weighting information to be determined based on the at least one weighting criterion. In other words, for example, the rotational and / or translational weighting information can be determined according to a criterion of the weighting criterion, such as the height criterion. It can be provided that separate information for translation and rotation is used to determine the visual odometry. In other words, information about different degrees of freedom can be determined or described independently of one another. By determining rotational and / or translational weighting information, different types of cameras, which are, for example, better suited for analyzing a rotation or a translation, can advantageously be taken into account.

[0021] According to a further possibility, the rotational weighting information can be determined if the respective camera is a camera with a wide-angle lens. The presence of a wide-angle lens can be determined based on the metadata, in particular based on the camera's aperture angle. Wide-angle cameras can be advantageous in determining the rotational degrees of freedom of visual odometry, since their large field of view allows the movement to be captured holistically.

[0022] Furthermore, within the scope of the invention, it can be provided that the translational weighting information is determined if the respective camera is a camera with a telephoto lens, wherein the presence of the telephoto lens is determined based on the metadata, in particular based on an aperture angle of the camera. In this case, it can first be detected whether an epipole is located in a field of view of the camera, wherein at least one sensor is further preferably used for the detection. Telephoto cameras, for example, have a very good angular resolution, which can enable a precise determination of the translation. One condition for the application of the method according to embodiments of the invention can be that the translation exceeds a certain threshold value. The information about the movement length is provided, for example, via additional sensors such as inertial sensors or wheel ticks.

[0023] It is possible for the method according to the invention to be used in a vehicle. The vehicle can be designed, for example, as a motor vehicle and / or passenger vehicle and / or an autonomous vehicle. The vehicle can have a vehicle device, for example, for providing an autonomous driving function and / or a driver assistance system. The vehicle device can be designed to control and / or accelerate and / or decelerate and / or steer the vehicle at least partially automatically.

[0024] The invention also relates to a computer program, in particular a computer program product, comprising instructions that, when executed by a computer, cause the computer to carry out the method according to the invention. Thus, the computer program according to the invention provides the same advantages as those described in detail with reference to a method according to the invention.

[0025] The invention also relates to a data processing device configured to carry out the method according to the invention. The device can be, for example, a computer that executes the computer program according to the invention. The computer can have at least one processor for executing the computer program. A non-volatile data memory can also be provided, in which the computer program is stored and from which the computer program can be read by the processor for execution.

[0026] The invention may also provide a computer-readable storage medium that contains the computer program according to the invention and / or includes instructions that, when executed by a computer, cause the computer to carry out the method according to the invention. The storage medium is designed, for example, as a data storage device such as a hard disk and / or a non-volatile memory and / or a memory card. The storage medium can, for example, be integrated into the computer.

[0027] Furthermore, the method according to the invention can also be implemented as a computer-implemented method.

[0028] Further advantages, features, and details of the invention will become apparent from the following description, which describes embodiments of the invention in detail with reference to the drawings. The features mentioned in the claims and in the description may be essential to the invention individually or in any combination. They show: Fig. 1 a schematic visualization of a method, a device, a storage medium and a computer program according to embodiments of the invention, Fig. 2 a schematic representation of the refraction of light through a windshield, Fig. 3 a schematic representation of a block diagram of a weighting method according to embodiments of the invention, Fig. 4 a schematic representation of a driving system with various cameras according to embodiments of the invention, Fig. 5 a schematic representation of a pure camera rotation without translation, Fig. 6 a schematic representation of a pure camera translation without rotation, Fig. 7 a schematic representation of a combined rotation and translation, Fig. 8 shows a schematic representation of a block diagram of a weighting method according to embodiments of the invention, Fig. 9 a schematic representation of a weighting of the cameras for translational and rotational parameters.

[0029] In Fig. 1, a method 100, a device 10, a storage medium 15 and a computer program 20 according to embodiments of the invention are schematically shown.

[0030] In a first method step 101 of the method 100 according to the embodiment in Fig. 1, camera data is provided, wherein the camera data comprises at least image data resulting from a recording by at least two cameras 1 of the driving system 30 during a movement. Furthermore, the camera data comprises metadata of the at least two cameras 1. In a second step 102, the provided camera data is evaluated using at least one weighting criterion in order to determine weighting information for each of the at least two cameras 1. In a third step 103, the visual odometry is determined using the determined weighting information.

[0031] Modern driving systems 30 for automated or semi-automated driving can use a dozen cameras 1 or even more. These cameras 1 sometimes have very specific tasks, can be redundant with other cameras 1, or can be better or worse suited for calculating visual odometry due to their optical properties. In addition, an estimate of visual odometry can also work better on some cameras 1 than on others, depending on the driving situation.

[0032] This finding can be further broken down into the individual degrees of freedom of visual odometry. Wide-angle cameras 2, or cameras with a wide-angle lens 2, for example, are well suited to determining the rotational degrees of freedom of visual odometry because their large field of view allows the movement to be captured holistically. At the same time, due to their large aperture angle, they cannot have a very high angular resolution. This can mean that each individual pixel covers a relatively large solid angle. In contrast, on cameras with a small aperture angle, especially cameras with a telephoto lens 3, the yaw angle and pitch angle often cannot be cleanly separated from the translation angle. In the image, the movement can appear virtually the same.At the same time, such telephoto cameras, or cameras with a telephoto lens, offer very good angular resolution, albeit only for a very limited field of view. The distinction between translation and rotation for telephoto cameras is sometimes not always possible. Nevertheless, they can be used to determine the translation very precisely if the rotation is already known. Furthermore, the translation itself should preferably not be too small.

[0033] Another aspect, in particular, is the utilization of hardware resources. Simultaneously processing a tens of camera streams may not be technically feasible and can be detrimental to the quality of the result. However, performing the calculation on a smaller, sensibly selected set of camera data streams can result in significant advantages in resource utilization and thus cost savings.

[0034] The present invention describes, in particular, a method for selecting cameras or camera image streams with the aim of calculating the most accurate image-based motion estimation (visual odometry). Furthermore, in particular, the degrees of freedom of translation and rotation of the camera movement are determined using different input data. In doing so, the respective advantages of cameras with different aperture angles are preferably utilized. In systems with a large number of cameras, for example, it is neither necessary nor expedient to process all camera image streams. Therefore, within the scope of the invention, the most optimal selection can be made based on specific criteria. Furthermore, the invention particularly takes advantage of the fact that visual odometry has five different degrees of freedom, the determination of which can place different demands on the geometric properties of a camera.The translational degrees of freedom can be separated from the rotational degrees of freedom, or the information from different cameras can be weighted higher or lower for the respective degrees of freedom.

[0035] A particular advantage of the invention is increased robustness and accuracy of motion estimation, as well as a reduction in required resources. In the context of the present invention, "visual odometry" preferably refers to an estimation of motion between two points in time (relative pose estimation).

[0036] The invention preferably relates to a method for selecting and weighting camera data, in particular image correspondences, for improved estimation of visual odometry. One possible aspect is, on the one hand, to use as few data streams as necessary, with the selection preferably being based on a list of criteria that determine the suitability of the respective camera at a particular time. Furthermore, the respective image correspondences can be weighted based on the geometry of their respective source camera, so that specific parameters (degrees of freedom) of the visual odometry can be determined as optimally as possible.

[0037] The method focuses particularly on so-called two-frame estimations, in which only two images are considered—in contrast to multi-frame estimations over a longer period of time. The basic idea is to estimate the rotational degrees of freedom based on wide-angle cameras and the translational degrees of freedom (Focus of Expansion, FoE) on cameras with high optical zoom (telephoto cameras). Since the FoE can be interpreted geometrically as a single pixel, its calculation benefits from a higher camera resolution, as is the case with telephoto cameras.

[0038] The following are examples of criteria that can be used to determine the suitability or priority of individual cameras, with the use of the criteria being explained in the respective sections.

[0039] A larger camera aperture angle means that a larger section of the scene can be observed. This can also lead to more robust motion estimation. Conversely, if the aperture angle is very small (telephoto camera), there is a particular risk that the motion estimation can no longer distinguish between lateral translation (sideways movement) and yaw (rotation on the road surface, e.g., in curves). Therefore, cameras with a large aperture angle can be advantageous over those with a small aperture angle.

[0040] Closely related to the camera's aperture angle is the position of the epipole in the image. When determining visual odometry, the position of the epipole is sometimes estimated directly. The epipole is preferably the projection of the camera center at one time onto the image taken at a different time. If the epipole is located in the image, its estimation can be more robust. In extreme cases, where the epipole is located far outside the image, it is no longer possible to estimate all of the camera's degrees of freedom. Such cases can occur, for example, with strong pitch or yaw movements. The smaller the camera's aperture angle, the faster the epipole can no longer be determined. The reason for this may be that the epipolar lines run parallel in such cases and the point of intersection can no longer be determined robustly.

[0041] Self-moving objects have a particularly negative impact on the estimation results, as the scene is generally assumed to be purely static. If one has a priori knowledge about the presence or absence of potentially self-moving objects (vehicles, pedestrians, etc.), this information can be used to prioritize the respective image data stream. Otherwise, there is a particular risk of estimating the vehicle's movement relative to the object rather than relative to the world.

[0042] The height of the camera can have an indirect influence on the estimation results. The closer the camera is to the road surface, the less favorable the image processing in the road area can be. In particular, with a moving vehicle, a very large spatial offset of the image content occurs between two points in time, and this usually results in geometric distortion, where rectangular areas in one image appear heavily deformed in the other. The distortion depends, for example, on the current speed. At low speeds, the problem may be minor, but at high speeds it can have a dramatic effect. This can make correspondence formation extremely difficult, which can have a direct impact on motion estimation. A higher installation position for the camera is therefore advantageous.

[0043] Installing the camera behind the windshield 4 may seem advantageous at first glance, as the camera is in a protected position. However, the refraction of light behind the windshield 4 can have a significant negative impact on motion estimation. As shown in Fig. 2, a spatial offset 6 of the light beam origin 5 occurs. This offset 6 can lead to a distortion of the image content. In reality, this offset 6 is not constant, particularly due to the manufacturing tolerances of the windshield 4, but rather depends on the position in the image. Although correcting windshield distortion is technically possible, it can normally be difficult to achieve with sufficient accuracy. Therefore, cameras 1 without a windshield 4 are advantageous for estimating visual odometry.

[0044] The exposure time can have a direct impact on the resulting image. Motion blur, also known as motion blur, is a direct result of the movement of the image content within the image and the exposure time. Longer exposures can result in greater motion blur. The same applies, for example, to fast-moving image content, such as during high-speed driving. At the same time, and unfortunately, contrary to this, long movements can be more favorable for motion estimation. Therefore, a large movement of the image content combined with a short exposure time is advantageous.

[0045] Based on the criteria described above, one aspect of the invention is a method for weighting the individual cameras. The calculation of the individual weights is described below. Fig. Figure 3 particularly shows that the weights for exposure time 7, height above ground 8, and windshield 9 are summed before the sum is multiplied by the weights for aperture angle 11, self-moving objects 12, and epipole position 13. From this process of calculating the final weight 14, or the weighting information, it becomes clear that the multipliers can be understood as knock-out criteria in an advantageous embodiment. If just one of them is zero, the respective camera is completely excluded in this embodiment. The reason for this is preferably that the estimation of self-motion is no longer possible if just one of the criteria is violated. In contrast, the weights for exposure time 7, height above ground 8, and windshield 9 can be soft criteria, which can have a positive or negative influence, but cannot alone lead to the failure of the estimation.

[0046] The weight w exρ for the exposure time t exp can be deviated from the parameterizable exposure time limits T min and T max depend on the exposure time. If the exposure time is less than the defined minimum, the weight is preferably 1. If it is greater than the defined maximum, it is preferably 0. In the middle interval, only a linearly decreasing weighting from 1 → 0 can occur. wexposure(texp)={1 for texp<TminTmax−texpTmax−Tminfu¨r Tmin≤texp≤Tmax0fu¨r texp> Tmax

[0047] The threshold values ​​T min and T max from the above equation can be set by the user. Possible values ​​are T min = 1ms and T max = 7ms.

[0048] The weighting of the height above ground can depend on the following user-defined parameters: - H min , which preferably corresponds to a minimum height above ground at Okm / h, - H max, which preferably corresponds to a saturation point for the height above ground at Okm / h, - S vel , which preferably corresponds to a scaling factor for the speed.

[0049] The weight w height depends preferably on the camera height h and the speed of the vehicle v. wheight(h,v)={0fu¨rh<Hmin−Svel∗v(h+Svel∗v)−HminHmax−Hminfu¨r Hmin≤h+Svel∗v≤Hmax1fu¨r h> Hmax−Svel∗v

[0050] The above equation performs a speed-dependent weighting. The lower threshold H min Preferably defines the height that results in a weight of 0 at a speed of Okm / h. With increasing speed, the upper and lower thresholds can shift upwards.

[0051] This could mean, for example, that a low camera at low speed can have the same weight as a high camera at high speed. Possible values ​​are H min = 0.5m, H max = 0.8m, S vel = 2s.

[0052] Weighting the windshield can be relatively simple. The greater the expected impact of the windshield, the lower the weight can be. wexposure(texp)={1without windshield0.5with windshield and compensation0with windshield without compensation

[0053] A special feature here may be that, if a windshield is present, the angle of attack of the windshield may also be considered. Flatter windshields can have a greater impact on distortion because the angle of refraction can be higher. This effect can be taken into account.

[0054] The weight for the opening angle is preferably determined by the interval (α min , α max ). All values ​​for the opening angle α that are smaller than α min preferably lead to a weight of 0. All values ​​greater than α max preferably lead to a weight of 1. wfov(α)={0for α<αminα−αminαmax−αminfor αmin≤α≤αmax1for α>αmax

[0055] Typical values ​​are α min = 40° and α max = 120°.

[0056] The ratio of pixels belonging to self-moving objects to the total number of pixels is referred to in the present invention as r (ratio). The weighting function w moving based on r preferably uses the following custom parameters: - R min, which in particular represents a minimum ratio of pixels with self-moving objects to the total number of pixels, from which the weighting function goes into saturation (value 1). - R max, which in particular represents a maximum ratio of pixels with self-moving objects to the total number of pixels, from which the weighting function goes to 0. wmoving(r)={1fu¨rr<RminRmax−rRmax−Rminfu¨r Rmin≤r≤Rmax0fu¨r r> Rmax

[0057] In particular, the above equation weights all values ​​of r that exceed the maximum to 0 (worst case), and all those that fall below the minimum to 1 (best case). In between, a linear weighting preferably takes place. Possible values ​​are R min = 60% and R max = 100%.

[0058] With the weight for the epipole position w epipoleIt is preferentially ensured that the epipole is located within the image. In addition, the weight is particularly higher when the epipole is closer to the image center. The weight is preferably dependent on the image width W and the image height H. Accordingly, the image center can be calculated in vector representation as follows: C=(W / 2H / 2)

[0059] The epipole in image coordinates consists in particular of the two components e x and e y . e=(exey)

[0060] The scalar distance of the epipole from the image center d is preferably defined as: d=‖C−e‖=(W / 2−ex)2+(H / 2−ey)2

[0061] The epipole position weighting function preferentially sets the weight to 0 if the epipole is outside the image. If it is inside the image, the distance to the image center preferentially results in a weight between 0 and 1. wepipole(d)={0for |W / 2−ex|>W / 2or |H / 2−ey|>H / 21−d(W / 2)2+(H / 2)2otherwise

[0062] In Fig. 5 and Fig. 6 shows a rotation R and a pure translation t of a camera 1. Combining these, one can obtain a camera movement with six degrees of freedom in 3D space (see Fig. 7). Three of these can be attributed to translation and three to rotation. If the camera movement is estimated purely from images, the length of the movement can inevitably be lost. Thus, only the direction of movement can possibly be determined. Therefore, one degree of freedom is said to be lost, leaving only five degrees of freedom – two for translation and three for rotation. Furthermore, in modern methods, the degrees of freedom can be determined largely independently of one another. This makes it possible to use different data to determine translation and rotation. One possible approach is to estimate rotation from data from wide-angle cameras 2 and translation from data from telephoto cameras 3 – but preferably under the condition that the focus of expansion (FoE) is at least close to the field of view of the respective telephoto camera 3.Here, the field of view preferably refers to the area captured by the image sensor. One could also say that the FoE should be within the image—or at least not far outside.

[0063] The reason for the condition that the FoE must be within the field of view or at least close to the field of view of the cameras 1 used to determine it has already been described in more detail above. One aspect in particular is that the FoE is determined via the intersection point of the epipolar lines. The estimation can be more robust if the intersection point is within the field of view. Therefore, one aspect of the invention in particular is to select the telephoto camera 3 in whose field of view the FoE is likely to be located. This can be one, several, or even no cameras 1 at all. However, the information as to where the FoE is located cannot be known until the actual determination has been carried out. Therefore, prior information can be used. This can be, for example, motion information from another sensor, e.g. inertial sensors, or the data from the last estimate.It may be unimportant that the position of the FoE is already known with high precision. A rough position alone may be sufficient. This process according to a possible embodiment of the invention is explained in the flowchart in a subsequent section.

[0064] Determining the visual odometry between two images may require that camera 1 has moved between the images (translation). This may be even more true for telephoto cameras 3, since a longitudinal movement of the camera (i.e., along the visual axis) can cause only a minor difference in the image if the movement is small. For this reason, a second condition for applying the method may be that the translation exceeds a certain threshold. Information about the movement length is provided, for example, by additional sensors (e.g., inertial sensors or wheel ticks).

[0065] Fig. 8 shows, in particular, a possible sequence of an embodiment of the invention. Based on an intrinsic 16 and extrinsic calibration 17, external movement data 18, and the latest results of the visual odometry, the weights for each camera can be determined. The calculation 23 of the visual odometry is thus carried out on the basis of image data 24, 25, 26 of the various cameras 1. In this context, extrinsic calibration 17 preferably refers to the alignment of a camera on a rigid body, e.g., a chassis or a robot. Intrinsic calibration 16 preferably refers to the imaging properties of a camera 1, including the aperture angle. Movement data 18 can be, for example, data from inertial sensors or data from other active or passive sensors such as radar, lidar, or ultrasound. These can be fused with the results of the visual odometry in a preprocessing step 19.In possible embodiments of the invention, it is important for weighting that there are two weights for each camera – one for estimating the translation 22 and one for the rotation 21. In this way, depending on the input data, telephoto cameras 3 can be given a higher or lower weight for determining the translation. The same applies in particular to all other cameras 1, whose weighting is generally set complementarily to that of the telephoto cameras 3. This can mean that if one type of camera 1 is given a higher weight for a parameter (e.g., translation), the other is automatically given a lower weight.

[0066] In Fig. Figure 9 shows, particularly on the left, the weighting for the translational parameters 22. In addition to the weight for the epipole position 13, a weight based on the aperture angle 11 and the movement length 27 of the underlying camera can also be included. This relationship is illustrated in the following equations. wfov,trans(α)={1for α<αminαmax−ααmax−αminfor αmin≤α≤αmax0for α>αmax wdistance,trans(t)={0fu¨rt>Tmint+Tmintmax−Tminfu¨r Tmin≤t≤Tmax1fu¨rt>Tmax wtranslation=wepipole∗wfov,trans∗wdistance,trans

[0067] The total weight for the translation estimation 22 for each camera w tiranslation can consist of the multiplication of the weights from epipole position, size of the aperture angle (field of view) and length of the translation.

[0068] In contrast to translation, estimating rotation preferably does not require any camera movement. Thus, both the weighting for the movement length and the epipole can be omitted. Fig. In Figure 9, the weighting for the rotational parameters 21 can be seen on the right. It preferably consists of only one weight based on the aperture angle 11, which is calculated differently than the weight for the translation, see the following equation: wfov,trans(α)={0for α<αminα−αminαmax−αminfor αmin≤α≤αmax1for α>αmax

[0069] A possible application example is a so-called camera belt on the car, as for example in Fig. 4. This shows a driving system 30 in which cameras with different aperture angles 28 are installed, in particular cameras with a wide-angle lens 2 and cameras with a telephoto lens 3.

[0070] Since the position of the FoE, or epipole, can play a significant role, the invention is ideal for use on robots with movable cameras. This allows the cameras 1 to always be aligned in the direction of the actual or expected movement.

[0071] One particular advantage of the invention is the dynamic weighting of cameras. This weighting, resulting in a total weight per camera, can then be used by a visual odometry estimator to select or downweight individual cameras. Furthermore, dynamic weighting with different values ​​can be applied for estimating translation and rotation. In this way, visual odometry can be determined both more robustly and more precisely, as the optical characteristics of the cameras can be optimally utilized.

[0072] The above explanation of the embodiments describes the present invention exclusively by way of examples. Of course, individual features of the embodiments can be freely combined with one another, provided they are technically feasible, without departing from the scope of the present invention.

Claims

[1] Method (100) for determining a visual odometry in a driving system (30), comprising the following steps: - Providing (101) camera data, wherein the camera data comprises at least image data resulting from a recording by at least two cameras (1) of the driving system (30) during a movement, and metadata of the at least two cameras (1), - evaluating (102) the provided camera data based on at least one weighting criterion in order to determine weighting information for each of the at least two cameras (1), - performing (103) the determination of the visual odometry using the determined weighting information. [2] Method (100) according to claim 1, characterized by , that the determination of the visual odometry is carried out using the determined weighting information in that a degree of consideration of the image data resulting from the recording by the respective camera (1) for the determination of the visual odometry is determined on the basis of a multiplication and / or summation of several weights of the weighting information for this respective camera (1), and / or that the procedure further comprises: - selecting at least one of the cameras (1) on the basis of the weighting information determined for the selected camera (1) in order to carry out the determination of the visual odometry on the basis of the camera data of the at least one selected camera (1). [3] Method (100) according to claim 1 or 2, characterized bythat an opening angle of the respective camera (1) is determined according to a criterion of the at least one weighting criterion, wherein the opening angle is preferably determined on the basis of the metadata of the respective camera (1), wherein further preferably a weight of the weighting information for the respective camera (1) is minimized, in particular set to 0, if the opening angle of the respective camera (1) falls below a predefined minimum angle. [4] Method (100) according to one of the preceding claims, characterized by that according to a criterion of the at least one weighting criterion, a ratio between pixels in the image data of the respective camera (1) which represent self-moving objects and a total number of pixels in the image data of the respective camera (1) is determined. [5] Method (100) according to one of the preceding claims, characterized byin that a position of an epipole is determined according to a criterion of the at least one weighting criterion, wherein the position of the epipole is determined on the basis of the image data of the respective camera (1), wherein preferably a weight of the weighting information for the respective camera (1) is minimized, in particular set to 0, if the position of the epipole lies outside a detection range of the respective camera (1). [6] Method (100) according to one of the preceding claims, characterized by that an exposure time of the respective camera (1) is determined according to a criterion of the at least one weighting criterion, wherein the exposure time is preferably determined on the basis of the metadata of the respective camera (1). [7] Method (100) according to one of the preceding claims, characterized bythat a height of the respective camera (1) relative to a surface of the driving system (30) is determined according to a criterion of the at least one weighting criterion, wherein the height of the respective camera (1) is preferably determined on the basis of the metadata of the respective camera (1). [8] Method (100) according to one of the preceding claims, characterized by that according to a criterion of the at least one weighting criterion, the presence of a windshield (4) in front of the respective camera (1) is checked in order to take into account a spatial offset of light rays through the windshield (4), wherein the presence of the windshield (4) in front of the respective camera (1) is preferably checked on the basis of the metadata of the respective camera (1). [9] Method (100) according to one of the preceding claims, characterized by , that at least one criterion of the at least one weighting criterion is an exclusion criterion and / or a priority criterion, wherein the exclusion criterion determines a suitability of the respective camera (1) for determining the visual odometry in the driving system (30), wherein the priority criterion determines a priority of the respective camera (1) for determining the visual odometry in the driving system (30), and wherein the weighting information is determined by performing a calculation with the exclusion criterion as a multiplier and / or the priority criterion as a summand. [10] Method (100) according to one of the preceding claims, characterized by that rotational and / or translational weighting information is determined on the basis of the at least one weighting criterion. [11] Method (100) according to claim 10, characterized bythat the rotational weighting information is determined if the respective camera (1) is a camera with a wide-angle lens (2), wherein the presence of the wide-angle lens is determined on the basis of the metadata, in particular on the basis of an aperture angle of the camera (1). [12] Method (100) according to one of claims 10 or 11, characterized by that the translational weighting information is determined if the respective camera (1) is a camera with a telephoto lens (3), wherein the presence of the telephoto lens is determined on the basis of the metadata, in particular on the basis of an aperture angle of the camera (1), wherein it is preferably first detected whether an epipole is located in a field of view of the camera (1), wherein further preferably at least one sensor is used for the detection. [13] Computer program (20) comprising instructions which, when the computer program (20) is executed by a computer (10), cause the computer (10) to carry out the method (100) according to one of the preceding claims. [14] Device (10) for data processing, which is arranged to carry out the method (100) according to one of claims 1 to 12. [15] A computer-readable storage medium (15) comprising instructions which, when executed by a computer (10), cause the computer (10) to carry out the steps of the method (100) according to any one of claims 1 to 12.

Images