Camera system for a mobile device, method for positioning a camera, and method for positioning multiple cameras - Patents.com

Abstract

The present disclosure relates to a camera system for a mobile device, the camera system comprising at least one camera freely mounted on the mobile device, at least one motion measurement unit configured to provide motion data of the camera, and a data processing circuit configured to determine a pose of the camera from the motion data.

Description

[Technical field]

[0001] SUMMARY OF THE DISCLOSURE Embodiments of the present disclosure relate to a camera system for a mobile device, a method for positioning a camera, and a method for positioning multiple cameras in a camera system. [Background technology]

[0002] In the automotive sector, vision-based driver assistance systems (ADAS) are emerging for various applications such as parking assistance, side-in warning, collision detection, etc. In the case of parking assistance surround view, for example, four cameras can be used to provide a 360-degree view of the vehicle's environment. Collision and sideline detection stereo cameras can be used that are configured to estimate the distance and speed of objects in the environment.

[0003] For autonomous driving, a vehicle may be able to estimate its own driving path and perceive its three-dimensional (3D) environment while driving. For this, the application needs to provide adequate spatial accuracy. Thus, for example, cameras need to be calibrated and rigidly mounted so that the 3D environment can be estimated from the relative position and orientation of the cameras. For stereo viewing, pre-calibration can be performed at the factory or in the workshop.

[0004] The distance between the cameras is limited to a maximum distance because rigidly mounted cameras can cause calibration issues due to camera distortion, vibration, or relative movement. Thus, rigidly mounting the cameras to the vehicle chassis may be unsuitable for autonomous driving applications because motor vibrations and chassis twists can affect the distance and orientation of the cameras.

[0005] Therefore, there is a need for an improved concept for determining camera pose. Summary of the Invention

[0006] This need may be met by the accompanying independent and dependent claims.

[0007] According to a first aspect, the present disclosure relates to a camera system for a mobile device. The camera system comprises at least one camera freely mounted on the mobile device. Further, the camera system comprises at least one motion measuring unit configured to provide motion data of the camera, and a data processing circuit configured to determine a pose of the camera from the motion data.

[0008] A mobile device may be, for example, a vehicle (eg, a car, truck, bus, boat, airplane, motorcycle, bicycle, or unmanned aerial vehicle) or a handheld device.

[0009] The camera may be freely mounted on the mobile device using a damping plate or (camera) stabilizer, which may at least partially reduce, compensate or absorb twisting and / or vibrations that may be emitted from the mobile device.

[0010] The stabilizer comprises, for example, a pivoting support (eg, a gimbal) that allows rotation of the camera about at least one axis to mount the camera in a floating position on the mobile device.

[0011] The damping plate may be at least partially made of an elastic material in order to freely (elastically) attach the camera to the mobile device.

[0012] Thus, the camera may be considered to be loosely coupled to the mobile device or to be freely or floatingly attached.

[0013] The camera can be a video / movie camera.

[0014] The movement data may for example indicate the acceleration and / or angular velocity of the camera, which may for example be measured by a gyroscope comprising the movement measuring unit.

[0015] Since the camera is freely mounted on the mobile device, the pose of the camera may change, which can be understood as the relative position and / or orientation in the environment with respect to the mobile device.

[0016] For example, the camera may undergo an angular velocity while the mobile device is moving, and the data processing unit may determine the pose of the camera, for example, from a relationship between the angular velocity and the pose of the camera.

[0017] This enables a so-called automatic calibration of the camera system with regard to the camera pose, potentially making pre-calibration or re-calibration of the camera system (e.g. in a factory or workshop) obsolete.

[0018] According to a second aspect, the present disclosure relates to a method for localizing a camera of a camera system for a mobile device, the method comprising the step of providing, by means of at least one motion measuring unit, motion data of a camera freely attached to the mobile device, the method further comprising the step of determining a pose of the camera from the motion data.

[0019] This method can be carried out, for example, using the camera system described above.

[0020] According to a third aspect, the present disclosure relates to a camera system comprising a first camera and a first motion measurement unit configured to provide first motion data of the first camera. The camera system further comprises at least one second camera and a second motion measurement unit configured to provide second motion data of the second camera, and further comprises a data processing circuit configured to determine a relative orientation of the first camera and the second camera facing each other from a correlation between the first motion data and the second motion data.

[0021] The first camera and the second camera can be video / movie cameras.

[0022] For example, the first and second motion data are indicative of an acceleration and / or an angular velocity of the first and second cameras, respectively. The first and second motion measuring units each comprise, for example, a gyroscope for measuring the angular velocity or acceleration acting on the first and second cameras, respectively.

[0023] The first and second motion data may differ from one another due to different poses of the first and second cameras, and the first and second motion data may be correlated such that, for example, a distinct difference between the first and second motion data indicates a distinct pose relative to one another.

[0024] This may allow, for example, automatic calibration of the camera system to adapt it to the changing relative pose of a first and a second camera facing each other, for example, the relative pose changing rapidly due to vibrations acting on the first and / or second camera. Alternatively and / or in addition, the relative orientation may change due to linear thermal expansion due to variations in ambient temperature.

[0025] Automated calibration can ensure adequate spatial accuracy of the camera system that may be required for applications related to autonomous vehicles.

[0026] According to a fourth aspect, the present disclosure relates to a method for localizing multiple cameras of a camera system for a mobile device. The method includes providing, using a first motion measurement unit, first motion data of a first camera of the camera system, and further including providing, using a second motion measurement unit, second motion data of at least one second camera of the camera system. Additionally, the method includes determining a relative pose of the first and second cameras toward one another through a comparison of the first and second motion data.

[0027] The method can be carried out, for example, using the camera system described above comprising a first camera and a second camera.

[0028] According to a fifth aspect, the present disclosure relates to a computer program comprising instructions which, when executed by at least one processor, cause the processor to perform one of the methods described above.

[0029] According to a sixth aspect, the present disclosure relates to a mobile device comprising one of the aforementioned camera systems.

[0030] The mobile device may be a vehicle or a handheld device, in particular, the mobile device may be an autonomous vehicle.

[0031] According to a seventh aspect, the present disclosure relates to a distributed camera system comprising a first camera configured to provide a first image sequence of an environment, a first motion measurement unit configured to provide first motion data of the first camera, and a first data processing circuit. The first data processing circuit is configured to determine a position of the object in the environment from a correlation between the first motion data and the first image sequence and to register the position of the object in a digital map of the environment. Further, the first data processing circuit is configured to provide a digital map of the environment.

[0032] The distributed camera system further comprises at least one second camera configured to provide a second sequence of images of the environment, a second motion measurement unit configured to provide second motion data of the second camera, and a second data processing circuit. The second data processing circuit is configured to determine a relative pose of the first and second cameras facing each other from a correlation between the first motion data and the second motion data. The second data processing circuit is further configured to receive a digital map of the environment and detect an object within at least one of the images of the second image sequence based on the digital map and the relative pose of the first and second cameras facing each other.

[0033] A distributed camera system can be understood as an apparatus comprising multiple camera systems, each of which includes a camera, a motion measurement unit and a data processing circuit.

[0034] In some embodiments, the first camera, the first motion measurement unit and the first data processing circuit may be mounted on the first mobile device and the second camera, and the second motion measurement unit and the second data processing circuit may be mounted on the second mobile device.

[0035] In some applications it may be desirable to track an object with multiple cameras, and therefore it is useful to identify the object in the second image sequence.

[0036] To this end, the digital map can be shared within the first data processing circuit and the second data processing circuit, and the second data processing circuit can thus estimate the position of an object in one or more images of the second image sequence by reference to the digital map and the relative pose of the first and second cameras.

[0037] Those skilled in the art having the benefit of this disclosure will understand that in some applications where the first and second cameras have different and / or opposing fields of view, this may be desirable for tracking purposes. [Brief description of the drawings]

[0038] Some examples of apparatus and / or methods are described below, by way of example only, and with reference to the accompanying drawings, in which: [Figure 1] 1 shows a schematic example of a camera system for a mobile device. [Diagram 2] 1 illustrates target tracking using a camera system. [Diagram 3] The present invention illustrates a three-dimensional map showing the motion path of a mobile device and characteristics of the mobile device's environment. [Figure 4] Characterize the environment using 3D maps. [Diagram 5] 1 shows a schematic example of a camera system comprising a first camera and a second camera. [Figure 6] 1 shows a schematic example of a camera system comprising a first camera and a second camera mounted on a mobile device. [Figure 7] 1 is a flow chart that illustrates a schematic of a camera position determination method. [Figure 8] 1 is a flow chart that illustrates a schematic of a multiple camera localization method; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0039] Various examples will now be described more fully with reference to the accompanying drawings, in which several examples are shown and in which the thicknesses of lines, layers and / or regions are exaggerated for clarity.

[0040] Thus, while the further examples are capable of various modifications and alternative forms, some specific examples thereof are shown in the drawings and will be described in detail below. However, this detailed description is not intended to limit the further examples to the specific forms described. The further examples may include all modifications, equivalents, and alternatives falling within the scope of the present disclosure. Identical or similar reference numerals refer to similar or similar elements throughout the description of the figures, which elements may be implemented in the same or modified form when compared to one another while providing the same or similar functionality.

[0041] When an element is referred to as "connected" or "coupled" to another element, these elements may be directly connected or may be coupled via one or more intervening elements. When two elements A and B are "coupled" with "or", this is understood to disclose all possible combinations, i.e., A only, B only, and A and B, unless otherwise expressly or implicitly specified. Alternative phrases for the same combination are "at least one of A and B" or "A and / or B." The same applies mutatis mutandis to combinations of two or more elements.

[0042] Terms used herein for the purpose of describing particular examples are not intended to limit further examples. Where singular forms such as "a" and "the" are used and the use of only a single element is not expressly or implicitly defined as requiring, further examples may use multiple elements to implement the same functionality. Similarly, where functionality is described later as being implemented using a plurality of elements, further examples may implement the same functionality using a single element or processing entity. It is to be further understood that the terms "comprise," "have," "comprises," "including," and / or "comprising," when used, specify the presence of stated features, integers, steps, operations, processes, acts, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, processes, acts, elements, components, and / or any groups thereof.

[0043] Unless otherwise defined, all terms (including technical and scientific terms) used herein are used in the ordinary sense of the art to which the example belongs.

[0044] For autonomous driving purposes, one or more cameras may be utilized to characterize the vehicle's environment. To this end, the camera system may be pre-calibrated to the camera pose. As they may be mounted on the vehicle, the cameras may be subject to twisting and / or vibration, which may result in calibration issues and, in some cases, inappropriate characterization of the environment for autonomous driving purposes.

[0045] Therefore, there may be a need for improved concepts for determining camera pose.

[0046] FIG. 1 shows a schematic example of a camera system 100 . The camera system 100 comprises a camera 120 and a motion measurement unit 110 configured to provide motion data of the camera 120. Furthermore, the camera system 100 comprises a data processing circuit 130 configured to determine a pose of the camera 120 from the motion data.

[0047] The camera 100 is freely (unconstrainedly) attached to the mobile device 140. As can be seen in Fig. 1, the camera 100 can be freely attached to the mobile device 140 by a stabilizing attachment 102 for a floating mount.

[0048] The stabilizing mount 102 can at least partially absorb vibrations coming from the mobile device 140. For example, the stabilizing mount 102 includes a resilient mount, such as a camera stabilizer (e.g., a gimbal) and / or a damping plate.

[0049] The motion measurement unit 110 may include an inertial measurement unit (IMU) rigidly attached to the camera 120. Those skilled in the art having the benefit of this disclosure will appreciate that the IMU may use a combination of one or more accelerometers, gyroscopes, and / or magnetometers to provide a portion of the motion data. Thus, the motion data from the IMU may be indicative of the acceleration and / or angular velocity of the camera 120. Alternatively or additionally, the motion measurement unit 110 may include a Global Positioning System sensor installed on the mobile device 140 and configured to provide at least a portion of the motion data. The GPS sensor may provide the geographic location, speed of the mobile device 140, or course of the geographic location that may contribute to the motion data.

[0050] The motion measurement unit 110 may include both an IMU and a GPS sensor for greater operational reliability than either of the IMU and GPS sensors.

[0051] The athletic measurement unit 110 is coupled to the data processing circuit 130 to communicate the athletic data.

[0052] The relationship between the pose of the camera 120 and the motion data enables the data processing circuit 130 to determine the pose of the camera 120 .

[0053] For example, the camera 120 may be displaced from a predetermined position with an angular velocity. The displacement of the camera 120 from the predetermined position may depend on the direction and the absolute value of the angular velocity. For example, the data processing circuit 130 is calibrated to determine the displacement and attitude of the camera 120 from the angular velocity included in the motion data.

[0054] This may enable automatic calibration of the camera system 100 based on the pose of the camera 120 on the mobile device 130 to restore spatial accuracy of the characterization of the environment using the camera system 100.

[0055] For automotive applications, the mobile device 140 may be a vehicle. In particular, the mobile device 140 may be an autonomous vehicle that operates based on characterization of the environment using the camera system 100.

[0056] For example, an autonomous vehicle may travel on a public road and take evasive maneuvers or output warnings based on its characterization of the environment.

[0057] To improve the accuracy of the automatic calibration, the data processing circuit 130 may further include visual data for determining the pose of the camera 120. To this end, the camera 120 may be a video / movie camera that records a series of images of the environment.

[0058] To characterize its pose within the environment, camera 120 can provide a sequence of images of the environment to processing circuitry 130. This allows, for example, data processing circuitry 130 to determine a pose from a correlation between the image sequence and motion data, as described in more detail below.

[0059] The camera 120 is coupled to a data processing circuit 130 for communication of a sequence of images.

[0060] As seen in FIG. 2, camera 120, for example, provides a sequence of subsequent images of the environment. To simplify matters, the following description of localizing the camera 120 refers to a first image 210, a second image 220, and a third image 230 of a sequence of images. In general, an image sequence may include many more images, and the concepts described below may also be applied to many more images.

[0061] Each of the images 210, 220, 230 includes an environmental object 250. Without limitation of generality and to simplify matters, the positioning of the camera 120 is described by reference to the object 250, although in further use cases the positioning of the camera 120 may include multiple objects.

[0062] The basic idea of ​​localizing the camera 120 is that the data processing circuit 130 determines the change in the viewpoint (perspective) of the camera 120 by comparing a first image 210 with a second image 220 of an image sequence.

[0063] The (absolute) pose of the camera 120 relative to the object 250 can be derived from the correlation between the motion data and the change in viewpoint on the object 250, as described in more detail below.

[0064] The data processing circuit 130, for example, determines a first patch 240-1 in the first image 210. As can be seen in Figure 2, the first patch 240-1 may correspond to a box containing a distinct feature 252, such as part of the contour (e.g., a corner or edge) of an object 250.

[0065] The first patch 240-1 can cover the "intrinsic" so-called "light differences" of the first image 210. If the images 210, 220 and 230 are monochrome, the light differences indicate, for example, a characteristic distribution of the grayscale values ​​(grayscale distribution) of the pixels contained in the first patch 240-1. The light differences can also be underlain by a distribution of colors and contrasts.

[0066] Subsequently, the data processing circuit 130 can determine a second patch 240-2 indicative of the object 252 in the second image 220 by comparing the first patch 240-1 with the second image 220.

[0067] For example, the first and second patches 240-1 and 240-2 are specified by their contours and the locations of their centers within the first and second images 210 and 220.

[0068] The data processing circuit 130 can position the second patch 240-2 in the second image 220 such that the second patch 240-2 contains (within a predetermined tolerance) the optical difference of the first patch 240-1. This ensures that the second patch 240-2 relates to the same object 252.

[0069] Due to the change in perspective, the second patch 240-2 may be shifted with respect to the first patch 240-1. The shift between the first patch 240-1 and the second patch 240-2 may represent a so-called "affine correspondence" that defines the transition between the subsequent patches 240-1 and 240-2. These may take into account rotations, displacements, warps and / or shears of the patches 240-1 and 240-2.

[0070] As can be seen in FIG. 2, the second patch 240-2 corresponds, for example, to a trapezoid / parallelogram.

[0071] The data processing circuit 130 can estimate the change in viewpoint of the camera 120 from the affine correspondence between the first patch 240-1 and the second patch 240-2. For example, the data processing circuit 130 infers the change in viewpoint, which may indicate, for example, an angle, based on the principles of homography.

[0072] Those skilled in the art having the benefit of this disclosure will understand that changes in viewpoint may enable a visually measurable pose of camera 120 towards object 252 to be estimated in accordance with the principles of visual odometry.

[0073] A visually measurable pose refers to, for example, the relative position or relative orientation of the camera 120 with respect to the object 252 with respect to other objects.

[0074] To obtain an (absolute) pose of the camera 120 towards the object 252 (e.g. indicating a coordinate or length measure), the data processing circuit 130 can provide a scale to estimate an absolute value for the pose of the camera 120 from the visually measurable pose of the camera 120.

[0075] To this end, the data processing circuit 130 can correlate the visual data with the movement data.

[0076] In some embodiments of the camera system 100, the data processing circuit 130 determines the pose of the camera 120 using a Kalman filter, whose control vector is specified by the motion data and whose measured state is determined by a relative pose calculated from the visual data.

[0077] The relationship between the motion data and the vision data may provide an observation model for the Kalman filter, which allows the relative pose of the camera 120 to be scaled to obtain the pose of the camera 120 as absolute values, e.g., in coordinates and / or angles.

[0078] Therefore, both the visual data and the motion data may have an influence in determining the pose of the camera 120. Their influence may depend on weighting factors indicating the accuracy of the motion data and the visual data, respectively, to adapt the camera system 100 to different situations (e.g., dark environments).

[0079] The above concept for determining the pose of the camera 120 may also be called "affine flow tracking."

[0080] Affine flow tracking may require motion data and time-synchronized image sequences.

[0081] In some embodiments, the data processing circuitry can be configured to determine a third patch 240-3 indicative of an object 252 in the third image 230 subsequent to the second image 220 by comparing the first patch 240-1 with the third image 230. The data processing circuitry 130 can determine the third patch 240-3, similar to the second patch 240-2, by reference to the first patch 240-1.

[0082] This may prevent the third patch 240-3 and subsequent patches from "drifting" away from the object 252 due to error propagation, and may provide sub-pixel accurate matching of subsequent patches with the object 252 for both temporal tracking and spatial mapping of subsequent patches by referencing subsequent images in the image sequence.

[0083] Subsequently, the data processing circuit 130 can determine the change in viewpoint from the shift / affine correspondence between the first patch 240-1 and the third patch 240-3 to determine the pose of the camera 120.

[0084] Because the third patch 240-3 can be thought of as being mapped onto the first patch 240-1 to determine the pose of the camera 120, this concept can be called "affine flow mapping."

[0085] In addition, the data processing circuit 130 can determine the position of the object from the correlation between the change in viewpoint and the motion data.

[0086] In contrast, the pose of the camera 120 can be considered as the origin of the coordinate system, so that the data processing circuit 130 can determine the position of the object 252 in the coordinate system from the pose of the camera 120 towards the object 252.

[0087] For example, the position of object 252 within the coordinate system represents three-dimensional coordinates.

[0088] In some embodiments of the camera system 100 , the data processing circuit 130 is further configured to determine the velocity of the object 252 by tracking the position of the object 252 .

[0089] To this end, the data processing circuitry 130 may, for example, as described above, continuously determine the position of the object 252 and derive its motion from the change in position over time. Implicitly, the data processing circuitry 130 may determine velocity from the motion kinematics.

[0090] Additionally, by tracking the position of the object 252, the camera system 100 can determine whether the object 252 is static or moving within the environment.

[0091] For example, the camera system 100 may similarly track the positions of other objects surrounding the object 252 to ascertain whether the object 252 moves relative to the other objects.

[0092] If the positions of other objects do not change over time, they can be identified as static objects that do not move in the environment.

[0093] If the object 252 moves relative to one or more static objects, the object 252 may be identified as a moving object.

[0094] Alternatively, the data processing circuit 130 can compare the velocities of several objects to distinguish between static and moving objects.

[0095] As shown in FIG. 3, the data processing circuit 130 can register the position of the object 252 within a (three-dimensional) digital map 300 of the environment, for example, by mapping one or more of the patches 240-1, 240-2 and 240-3 to the position of the object 252 within the digital map 300.

[0096] Thus, during movement of the camera 120 along the motion path 310, multiple objects can be successively registered to the digital map 300.

[0097] The patches 240-1, 240-2, 240-3 are further characterized by a surface normal 242. As a result, the data processing circuit 130 is able to characterize the surface of the environment with respect to its direction.

[0098] As can be seen from FIG. 4, the digital map 300 enables the data processing circuit 130 to determine a fourth patch 240-4 indicating the object 252 in the fourth image 440 based on the pose of the camera 120 and the digital map 300.

[0099] In some embodiments of the camera system 100, the data processing circuit 130 is configured to compare the position of the object 252 with the pose of the camera 120 to predict the position of the object 252 in the fourth image 440. As a result, the data processing circuit 130 can align the fourth patch 240-4 to the predicted position.

[0100] Further, the data processing circuit 130 may be configured to determine whether the fourth patch 240-4 matches one or more of the preceding / registered patches 240-1, 240-2 and 240-3, and optionally determine a deviation from the preceding patches 240-1, 240-2 and 240-3.

[0101] This deviation allows, for example, an (automatic) recalibration of the pose of the camera 120 .

[0102] In some cases, patch 240-5 cannot be mapped to an object and / or appropriate patch registered in digital map 300. Thus, such object or patch may be ignored in connection with characterizing the environment and / or discarded from digital map 300.

[0103] During operation of the camera 120, the data processing circuit 130 may further continually detect "new" objects and associate "new patches" 240-6 with those objects.

[0104] The digital map 300 can further be shared with an auxiliary camera that supports visual odometry for characterization of the environment. In general, the field of view from the auxiliary camera is different from the field of view of the camera 120.

[0105] Sharing the digital map 300 can provide automatic calibration between the camera 120 and the auxiliary camera by allowing the auxiliary camera to associate a different patch with the object 252 by referencing the digital map 300 and the pose of the auxiliary camera.

[0106] This may be desirable, for example, in a driver assistance system that uses multiple differently oriented cameras.

[0107] As mentioned above, some applications call for the use of multiple cameras.

[0108] In camera systems with multiple cameras, the cameras may experience motion or vibration relative to one another, for example in connection with in-vehicle applications. As a result, the relative positions of the multiple cameras may be subject to variations that may cause calibration problems. These calibration problems may result in inadequate characterization of the environment.

[0109] Therefore, an improved concept for environment characterization using multiple cameras is required to overcome such calibration problems resulting from variations in the relative positions of the cameras with respect to each other.

[0110] FIG. 5 shows a camera system 200 comprising a first camera 120-1 and a first movement measurement unit 110-1 configured to provide first movement data of the first camera 120-1. Furthermore, the camera system 200 comprises at least one second camera 120-2 and a second motion measurement unit 120-2 configured to provide second motion data of the second camera 120-2. Furthermore, the camera system 200 comprises a data processing circuit 230 configured to determine a relative orientation of the first camera 120-1 and the second camera 120-2 towards each other from a correlation between the first motion data and the second motion data.

[0111] To this end, the first and second motion measuring units 110 - 1 and 110 - 2 are coupled to the data processing circuit 230 .

[0112] In some embodiments, the data processing circuit 230 may comprise first and second processing circuits (not shown) mounted to the first and second cameras 120-1 and 120-2, respectively, which may be desirable when the first and second cameras 120-1 and 120-2 are mounted on separate mobile devices.

[0113] In some embodiments, the data processing circuitry 230 may correspond to the data processing circuitry 130 described in connection with the camera system 100 .

[0114] The relative position of the first and second cameras 120-1 and 120-2 may be related to the relative motion of the cameras 120-1 and 120-2. By itself, the relative position can be derived from the first and second motion data indicative of the relative motion.

[0115] The first and / or second movement data may, for example, be indicative of angular velocity and / or acceleration acting on cameras 120-1 and 120-2.

[0116] For example, data processing circuit 230 determines the relative positions of cameras 120-1 and 120-2 from the difference between the first and second motion data.

[0117] In this manner, data processing circuitry 230 enables automatic calibration of camera system 200 with reference to the relative positions of cameras 120-1 and 120-2 to restore spatial accuracy of camera system 200.

[0118] In some further embodiments of the camera system 200, the first camera 120-1 is configured to provide a first sequence of images of the environment and the second camera 120-2 is configured to provide a second sequence of images of the environment. In relation to these embodiments, the data processing circuit 230 may be further configured to determine a relative orientation of the first camera 120-1 and the second camera 120-2 towards each other from a correlation between the first motion data, the second motion data, the first sequence of images, and the second sequence of images.

[0119] To this end, the first and second motion data and the first and second image sequences may be time synchronized.

[0120] Each of cameras 120-1 and 120-2 may be coupled to data processing circuitry 230 to communicate the first and second image sequences to data processing circuitry 230.

[0121] The data processing circuit 230 may apply the concept of visual odometry to the first and second image sequences, as described in connection with previous embodiments of the camera system 100.

[0122] For example, the data processing circuit 230 utilizes a Kalman filter to determine the relative pose of the cameras 120-1 and 120-2.

[0123] The observation model of the Kalman filter may be derived from an affine correspondence correlation between images of the first and second image sequences and a difference between the first and second motion data.

[0124] The control vector of the Kalman filter may be specified by the first and second motion data, while the measurements of the relative pose of cameras 120-1 and 120-2 may indicate an affine correspondence.

[0125] This may enable the Kalman filter to continuously determine the relative pose of the first and second cameras 120-1 and 120-2 by referring to the first and second motion data and the first and second image sequences.

[0126] In this manner, camera system 200 can use (only) motion data to determine the relative positions of cameras 120-1 and 120-2, increasing the accuracy of the relative positions compared to previous embodiments of camera system 200.

[0127] 6, the first and second cameras are freely mounted (mounted in a floating position) on the mobile device 140. To this end, each of the cameras 120-1 and 120-2 can be coupled to the mobile device 140 using a stabilizing mount 102.

[0128] In this way, perturbations (eg, vibration and / or twist) coming from the mobile device can be dampened to improve the spatial accuracy of the first camera 120-1 and / or the second camera 120-2.

[0129] In some embodiments, the first camera 120-1 is mounted on a first mobile device (not shown) and the second camera 120-2 is mounted on a second mobile device (not shown).

[0130] Each of the first and second mobile devices may be, for example, an unmanned aerial vehicle (UAV). Because the UAVs may move relative to each other, the relative attitude of the cameras 120-1 and 120-2 may change.

[0131] The above-described concept for determining the relative pose of the first and second cameras 120-1 and 120-2 may enable the camera system 200 to (continuously) automatically calibrate the camera system 200 with respect to the relative pose.

[0132] In some embodiments of the camera system 200, the data processing circuit 230 is further configured to determine a position of the object in the environment from a correlation between the first image sequence and the first motion data. The data processing circuit 230 can further be configured to register the position of the object in a digital map of the environment and detect the object in the second image sequence using the digital map, the motion data, and the relative poses of the first and second cameras 120-1 and 1202.

[0133] This may be desirable, for example, when the first and second cameras 120-1 and 120-2 are mounted on different mobile devices or have opposing fields of view, as is the case with some driver assistance systems.

[0134] The data processing circuit 230 can, for example, determine the position of the object using the first image sequence and the first motion data and register the position of the object in the three-dimensional map 300 as described in connection with the camera system 100.

[0135] By referencing the relative poses of the first and second cameras 120-1, 120-2, the motion data, and the digital map, the data processing circuit 230 can determine whether an object is within the field of view of the second camera 120-2, which in turn enables the data processing circuit 230 to predict the position of the object in the images of the second image sequence that has a deviation from the position of the object derived from the second image sequence.

[0136] Those skilled in the art having the benefit of this disclosure will understand that this deviation may be used to auto-calibrate the first and second cameras 120-1 and 120-2 such that the deviation is reduced by the auto-calibration.

[0137] FIG. 7 illustrates generally a method 700 for positioning a camera of a camera system for a mobile device. The method includes a step 710 of providing motion data of a camera using at least one motion measurement unit, the camera being freely mounted on the mobile device, and further includes a step 720 of determining a pose of the camera from the motion data.

[0138] The method 700 may be performed, for example, by the camera system 100. To this end, at least a portion of the method 700 may be performed by the data processing circuitry 130 by executing a suitable computer program.

[0139] FIG. 8 illustrates generally a method 800 for positioning multiple cameras of a camera system for a mobile device. The method 800 includes providing 810 first motion data of a first camera of the camera system using a first motion measurement unit. The method 800 further includes providing 820 second motion data of at least one second camera of the camera system using a second motion measurement unit. Additionally, the method 800 provides for determining 830 a relative pose of the first and second cameras facing each other through a comparison of the first and second motion data.

[0140] The method 800 may be performed, for example, by the camera system 200. To this end, at least a part of the method 800 may be performed by the data processing circuitry 230 by executing a suitable computer program.

[0141] The following examples relate to further embodiments. (1) A camera system for a mobile device, comprising: At least one camera freely attached to the mobile device; at least one motion measurement unit configured to provide motion data of the camera; a data processing circuit configured to determine a pose of the camera from the motion data; Equipped with Camera system. (2) the camera is configured to provide a sequence of images of an environment; The data processing circuitry is further configured to determine a pose of the camera within the environment from a correlation between the image sequence and the motion data. The camera system according to (1). (3) The data processing circuit comprises: determining a change in the camera's viewpoint by comparing a first image in the image sequence with a second image in the image sequence that follows the first image; Correlating the motion data with the change in viewpoint to determine the pose of the camera within the environment. It is further configured as follows: The camera system according to (2). (4) The data processing circuit comprises: determining a first patch in the first image; determining a second patch indicative of the object in the second image by comparing the first patch with the second image; and determining the change in viewpoint from a shift between the first patch and the second patch; It is further structured as follows: The first patch depicts an object in the environment. (3) A camera system according to the present invention. (5) The data processing circuit comprises: determining a third patch indicative of the object in the third image subsequent to the second image by comparing the first patch with a third image; determining the change in viewpoint from a shift between the first patch and the third patch; It is further configured as follows: (4) A camera system according to the present invention. (6) The data processing circuit comprises: and determining a position of the object from a correlation between the change in viewpoint and the motion data. A camera system according to (4) or (5). (7) The data processing circuit comprises: and determining whether the object is static or moving within the environment by tracking the position of the object. The camera system according to (6). (8) The data processing circuit is further configured to determine a velocity of the object by tracking a position of the object. The camera system according to (6). (9) The data processing circuit comprises: registering the location of said object within a digital map of said environment; determining a fourth patch indicative of the object in a fourth image based on the camera pose and the digital map; It is further configured as follows: The camera system according to (6). (10) The motion measurement unit includes an inertial measurement unit (IMU), the inertial measurement unit (IMU) rigidly attached to the camera and configured to provide at least a portion of the motion data. A camera system according to any one of (1) to (9). (11) The motion measurement unit includes a global positioning system (GPS) sensor, the sensor being installed on the mobile device and configured to provide at least a portion of the motion data. A camera system according to any one of (1) to (10). (12) The camera is freely attached to the mobile device by a camera stabilizer. A camera system according to any one of (1) to (11). (13) The camera is freely attached to the mobile device by an elastic mounting. A camera system according to any one of (1) to (12). (14) a first camera; and a first motion measurement unit configured to provide first motion data of the first camera; at least one second camera; a second motion measurement unit configured to provide second motion data of the second camera; and a data processing circuit configured to determine a relative orientation of the first camera and the second camera facing each other from a correlation between the first motion data and the second motion data; Equipped with Camera system. (15) The first camera is configured to provide a first sequence of images of an environment; and the second camera is configured to provide a second sequence of images of the environment; The data processing circuitry is further configured to determine a relative orientation of the first camera and the second camera relative to one another from a correlation between the first motion data, the second motion data, the first image sequence, and the second image sequence. The camera system according to (14). (16) At least one of the first camera and the second camera is freely attached to a mobile device. A camera system according to (14) or (15). (17) The data processing circuit comprises: determining a position of an object within the environment from a correlation between the first sequence of images and the first motion data; registering the location of said object within a digital map of said environment; Detecting the object in the second image sequence using the digital map, the motion data of the first camera and the second camera, and the relative pose. It is further configured as follows: A camera system according to any one of (14) to (16). (18) The first camera is attached to a first mobile device, and the second camera is attached to a second mobile device. The camera system according to (14). (19) A method for locating a camera of a camera system for a mobile device, comprising: providing, by means of at least one motion measurement unit, motion data of said camera freely attached to said mobile device; determining a pose of the camera from the motion data; Includes method. (20) A method for locating multiple cameras of a camera system for a mobile device, comprising: providing first motion data of a first camera of the camera system using a first motion measurement unit; providing second motion data of at least one second camera of the camera system using a second motion measurement unit; determining a relative orientation of the first camera and the second camera facing each other through a comparison of the first motion data and the second motion data; Includes method. (21) A first camera configured to provide a first sequence of images of an environment; a first motion measurement unit configured to provide first motion data of the first camera; A first data processing circuit; A distributed camera system comprising: The first data processing circuit includes: determining a position of an object within the environment from a correlation between the first motion data and the first image sequence; registering the position of the object within a digital map of the environment; and Providing the digital map of the environment It is structured as follows: The distributed camera system includes: at least one second camera configured to provide a second sequence of images of the environment; a second motion measurement unit configured to provide second motion data of the second camera; and A second data processing circuit; It is equipped with The second data processing circuit includes: determining a relative orientation of the first camera and the second camera relative to one another from a correlation between the first motion data and the second motion data; receiving the digital map of the environment; and and detecting the object within at least one of the images of the second image sequence based on the digital map and the relative pose of the first and second cameras facing each other. Distributed camera system. (22) A vehicle equipped with the camera system described in (1).

[0142] Aspects and features described in connection with one or more of the embodiments and figures detailed above may also be combined with one or more of the other examples to replace similar features of the other examples or to introduce additional features to the other examples.

[0143] An example may be or relate to a computer program having program code for performing one or more of the methods described above when the computer program is run on a computer or processor. The various method steps, operations or processes described above may be performed by a programmed computer or processor. Examples may also include a program storage device, such as a digital data storage medium, that is machine, processor, or computer readable and that encodes a program of machine-executable, processor-executable, or computer-executable instructions that perform or cause to be performed some or all of the actions of the methods described above. The program storage device may include or be, for example, a digital memory, a magnetic storage medium such as a magnetic disk and a magnetic tape, a hard drive, or an optically readable digital data storage medium. Further examples may cover a computer, processor or controller programmed to perform the actions of the above-mentioned methods or a (field) programmable logic array ((F)PLA) or a (field) programmable gate array ((F)PGA) programmed to perform the actions of the above-mentioned methods.

[0144] The specification and drawings merely illustrate the principles of the present disclosure. Moreover, all examples described herein are expressly intended to be for illustrative purposes only, primarily to aid the reader in understanding the principles of the present disclosure and concepts that the inventors have contributed to further advancing the art. All statements made herein are intended to describe principles, aspects, and examples of the present disclosure, as well as specific examples thereof, and encompass equivalents thereof.

[0145] A functional block labeled "...means" for performing a particular function may refer to a circuit configured to perform the particular function. Thus, a "means for something" may be embodied as a "means configured for or adapted for something," such as a device or circuit configured for or adapted for the respective task.

[0146] The functionality of the various components illustrated in the figures, including any functional blocks labeled as “means,” “means for providing a signal,” “means for generating a signal,” etc., may be implemented in the form of dedicated hardware such as a “signal provider,” “signal processing unit,” “processor,” “controller,” etc., as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by multiple individual processors, some or all of which may be shared. However, the term "processor" or "controller" is not limited to only hardware capable of executing software, but may include digital signal processor (DSP) hardware, network processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), read only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage. Other hardware, conventional and / or custom, may also be included.

[0147] Block diagrams may, for example, illustrate high level circuit diagrams embodying the principles of the present disclosure. Similarly, flowcharts, flow diagrams, state transition diagrams, pseudocode, and the like may represent various processes, operations, or steps that are substantially represented in, for example, a computer-readable medium and executed by a computer or processor, whether or not such a computer or processor is explicitly shown. The methods disclosed in this specification or claims may be performed by an apparatus having means for performing each of the actions of those methods.

[0148] It should be understood that the disclosure of multiple acts, processes, operations, steps or functions disclosed in this specification or claims should not be construed as being in a particular order, unless expressly or implicitly stated, for example, for technical reasons. Thus, the disclosure of multiple acts or functions does not limit them to a particular order, unless such acts or functions are interchangeable for technical reasons. Furthermore, in some instances, a single action, function, process, operation, or step may include or be divided into multiple subsidiary actions, functions, processes, operations, or steps, respectively, and such subsidiary actions may be included and may be part of the disclosure of the single action, unless expressly excluded.

[0149] Additionally, the following claims are hereby incorporated into the Detailed Description, where each claim may stand on its own as a separate example. Although each claim may stand on its own as a separate example, a dependent claim may be referenced in multiple claims for a specific combination with one or more other claims, with the understanding that other examples may also include combinations with the subject matter of the dependent or independent claims. Such combinations are expressly suggested herein unless it is specified that a particular combination is not intended. Furthermore, it is intended that features of one claim be included in other independent claims even if that claim is not directly dependent on that claim.

Claims

1. 1. A camera system for a mobile device, comprising: At least one camera freely attached to the mobile device; at least one motion measurement unit configured to provide motion data of the camera; a data processing circuit configured to determine a pose of the camera from the motion data; Equipped with the camera is configured to provide a sequence of images of an environment; the data processing circuitry is further configured to determine a pose of the camera within the environment from a correlation between the image sequence and the motion data; The data processing circuit includes: determining a change in the camera's viewpoint by comparing a first image in the image sequence with a second image in the image sequence that follows the first image; and determining a pose of the camera within the environment from a correlation between the motion data and the change in viewpoint. The data processing circuit includes: determining a first patch indicative of an object in the environment in the first image; and further configured to determine a second patch indicative of the object in the second image by comparing the first patch with the second image, and to determine the change in viewpoint from a shift between the first patch and the second patch; The data processing circuit includes: determining a third patch indicative of the object in the third image subsequent to the second image by comparing the first patch with a third image; and determining the change in viewpoint from a shift between the first patch and the third patch. Camera system.

2. 1. A camera system for a mobile device, comprising: At least one camera freely attached to the mobile device; at least one motion measurement unit configured to provide motion data of the camera; a data processing circuit configured to determine a pose of the camera from the motion data; Equipped with the camera is configured to provide a sequence of images of an environment; the data processing circuitry is further configured to determine a pose of the camera within the environment from a correlation between the image sequence and the motion data; The data processing circuit includes: determining a change in the camera's viewpoint by comparing a first image in the image sequence with a second image in the image sequence that follows the first image; and determining a pose of the camera within the environment from a correlation between the motion data and the change in viewpoint. The data processing circuit includes: determining a first patch indicative of an object in the environment in the first image; and further configured to determine a second patch indicative of the object in the second image by comparing the first patch with the second image, and to determine the change in viewpoint from a shift between the first patch and the second patch; The data processing circuit includes: and further configured to determine a position of the object from a correlation between the change in viewpoint and the motion data; The data processing circuitry is adapted to register the location of the object within a digital map of the environment; and determining a fourth patch indicative of the object in a fourth image based on the camera pose and the digital map. Camera system.

3. The motion measurement unit includes an inertial measurement unit (IMU), the inertial measurement unit (IMU) rigidly attached to the camera and configured to provide at least a portion of the motion data. The camera system of claim 1 .

4. The motion measurement unit comprises a global positioning system (GPS) sensor, the sensor being mounted on the mobile device and configured to provide at least a portion of the motion data. The camera system of claim 1 .

5. The camera is freely attached to the mobile device by a camera stabilizer. The camera system of claim 1 .

6. The camera is freely attached to the mobile device by a resilient mounting. The camera system of claim 1 .

7. A first camera; a first motion measurement unit configured to provide first motion data of the first camera; at least one second camera; a second motion measurement unit configured to provide second motion data of the second camera; and a data processing circuit configured to determine a relative orientation of the first camera and the second camera facing each other from a correlation between the first motion data and the second motion data; Equipped with Camera system.

8. the first camera is configured to provide a first sequence of images of an environment; the second camera is configured to provide a second sequence of images of the environment; The data processing circuitry is further configured to determine a relative orientation of the first camera and the second camera relative to one another from a correlation between the first motion data, the second motion data, the first image sequence, and the second image sequence. The camera system of claim 7.

9. At least one of the first camera and the second camera is freely attached to a mobile device. The camera system of claim 7.

10. The data processing circuit includes: determining a position of an object within the environment from a correlation between the first sequence of images and the first motion data; registering the location of said object within a digital map of said environment; Detecting the object in the second image sequence using the digital map, the motion data of the first camera and the second camera, and the relative pose. It is further configured as follows: The camera system of claim 8.

11. The first camera is mounted on a first mobile device and the second camera is mounted on a second mobile device. The camera system of claim 7.

12. 1. A method for locating multiple cameras of a camera system for a mobile device, comprising: providing first motion data of a first camera of the camera system using a first motion measurement unit; providing second motion data of at least one second camera of the camera system using a second motion measurement unit; determining a relative orientation of the first camera and the second camera facing each other through a comparison of the first motion data and the second motion data; Includes method.

13. a first camera configured to provide a first sequence of images of an environment; a first motion measurement unit configured to provide first motion data of the first camera; A first data processing circuit; A distributed camera system comprising: The first data processing circuit includes: determining a position of an object within the environment from a correlation between the first motion data and the first image sequence; registering the position of the object within a digital map of the environment; and Providing the digital map of the environment It is structured as follows: The distributed camera system includes: at least one second camera configured to provide a second sequence of images of the environment; a second motion measurement unit configured to provide second motion data of the second camera; and A second data processing circuit; It is equipped with The second data processing circuit includes: determining a relative orientation of the first camera and the second camera relative to one another from a correlation between the first motion data and the second motion data; receiving the digital map of the environment; and and detecting the object within at least one of the images of the second image sequence based on the digital map and the relative pose of the first and second cameras facing each other. Distributed camera system.

Images