Method for determining exposure time interval of camera of camera system and camera system

Abstract

The invention relates to a method for determining an exposure time (t) of a camera (1) of a camera system (10) for preferably moving and / or for machine object recognition in a moving scene, in which a predefined minimum angular image resolution (Ra *) is followed, comprising the following steps: a) determining an effective angular image resolution (Ra, eff (t)) as a function of the exposure time (t), b) the maximum allowable exposure time (tmax) is determined as the exposure time when the effective angle image resolution (Ra, eff (t)) is the value of the minimum angle image resolution (Ra *), and c) the exposure time (t) is set to a value less than or equal to the maximum allowable exposure time (tmax). The invention also relates to a camera system (10).

Description

Technical Field

[0001] This invention relates to a method for determining the exposure time of a camera in a camera system for preferred motion and / or for a camera system for machine object recognition in a moving scene. The invention also relates to a camera system. Background Technology

[0002] Driver assistance and autonomous driving systems in vehicles include optical systems, typically camera systems, whose recorded data is digitally processed. Of particular significance in digital processing is computer-aided object recognition, which identifies other vehicles, pedestrians, and more from recorded camera images. The design of the camera systems used involves trade-offs between various conflicting design goals, including the largest possible field of view (FoV), the widest possible operating range and / or spectral range, sensitivity, and temporal resolution.

[0003] For example, modern vehicle front-facing camera systems place high demands on horizontal field of view and working range or effective distance. These requirements are so incompatible that they often cannot be met by a single camera. Therefore, camera systems typically installed in the field of driver assistance systems employ so-called multi-purpose camera systems. These multi-purpose camera systems mostly consist of an arrangement or cluster of multiple cameras with different specifications. For example, in the field of front-facing cameras, a camera system is required that, on the one hand, has a horizontal field of view of at least 100° to capture lateral movement across the vehicle's heading as early as possible, and on the other hand, must be able to resolve objects identibly at a distance. Therefore, camera clusters are used, which combine cameras with a field of view of up to 150° (so-called fisheye cameras) and 28° telephoto cameras for achieving higher resolution at greater distances.

[0004] A core issue with using telephoto cameras is the resolution limitation caused by motion blur. Motion blur occurs when the camera or the scene being recorded is not stationary at the moment of recording—more precisely, during the exposure time. Camera rotation, lateral object movement, and long exposure times have the greatest impact on motion blur. The angular image resolution Ro is used to evaluate the impact of motion blur. a It is given in dimensionless pseudo-units, pixels per degree (ppd). Angular image resolution indicates how many pixels (in pixels) can be used to represent a 1° wide scene. a The larger the value, the more detailed the image. Therefore, a 28° telephoto camera with a 2-megapixel Full HD image sensor will have the following angular image resolution:

[0005]

[0006] Based on experience, in computer-aided object recognition, a pedestrian's size on the camera recording is less than 45 pixels, and is therefore classified as "small." This size, combined with an estimated average pedestrian height of 1.7m and the typically required recognition distance of 200m, yields the necessary angular image resolution:

[0007]

[0008] Similar considerations for vehicles rather than pedestrians lead to R a,min = 93.1 ppd. Therefore, a 2-megapixel image sensor is insufficient to meet the requirements; a 4-megapixel image sensor is necessary, which is technically a high requirement. However, this resolution was initially only suitable for still cameras in static scenes.

[0009] A particular challenge is that the effective image resolution of a camera system with high angular image resolution decreases significantly when the camera or scene moves within the exposure time. For example, relative to a camera movement ω = 5° / s considered moderate in reality and an exposure time t = 10ms, the effective image resolution of a camera system with high angular image resolution R0 will decrease significantly. a At 93.1ppd, this will result in motion blur in dp:

[0010] dp=5° / s×10ms×93.1ppd=4.7px (3)

[0011] This means that the detail typically resolved to a single pixel is now estimated to be around 5 pixels. Therefore, under moderate motion, the effective angular image resolution drops to nearly one-fifth, a situation that cannot be meaningfully avoided technically by increasing the image sensor resolution. Experience also shows that real-world camera motion in moving vehicles can reach 20° / s. Such considerations necessitate taking into account performance metrics other than image sensor resolution in camera system design. Summary of the Invention

[0012] Therefore, the objective of this invention is to set the exposure time of a camera in a camera system so as to adhere to a predetermined angular image resolution in a moving camera system and / or in a moving scene. To address this objective, a method according to the invention for determining the exposure time of a camera in a moving camera system and / or for machine object recognition in a moving scene is proposed. Other preferred embodiments are described below.

[0013] A method is proposed for determining the exposure time t of a camera for a camera system used for motion and / or for machine object recognition in a moving scene, wherein a pre-given minimum angular image resolution is followed. Includes the following steps:

[0014] a) Set the effective image resolution R a,eff (t) is determined as a function of the exposure time t.

[0015] b) Set the maximum permissible exposure time t max Determined as effective image resolution R a,eff (t) Take the minimum angular image resolution R a* Exposure time under the value, and

[0016] c) Set the exposure time t to be less than or equal to the maximum permissible exposure time t max The value of .

[0017] Here, steps a) and b) are performed in any order, followed by step c). The proposed method enables adherence to a pre-defined minimum angular image resolution in moving cameras and / or moving scenes. Predefined minimum angular image resolution It arises from external requirements of the camera system, such as those based on the technical functions or tasks to be performed. Effective angular image resolution R a,eff (t) is inversely proportional to the exposure time t:

[0018]

[0019] Where K corresponds to the inverse proportionality coefficient. It can be determined empirically, modeled, or otherwise set.

[0020] According to requirements

[0021] R a,eff (t max ) = R a* (5)

[0022] By finding the corresponding solution to equation (4), the maximum permissible exposure time t can be obtained. max .

[0023] Furthermore, it is proposed that in step a), the effective image resolution R a,eff (t) is also based on the maximum tolerance motion fuzzy dp and motion quantity ω K To determine. Exercise volume ω K This is a total quantification of camera movement and / or the motion of the scene to be recorded. Since any motion can be expressed as a change in viewpoint relative to the camera, the motion quantity ω K Preferably, the angular velocity is given in degrees per second [° / s]. This preferred embodiment opens the way to probabilistic motion models, which can map real-world scene and / or camera motion more accurately than by estimation or averaging. With this preferred embodiment, equation (4) is extended to:

[0024]

[0025] The maximum tolerance for motion blur, dp, is preferably measured in pixels. The magnitude of motion ω, such as camera motion or scene motion, is determined at a specific moment by three motion ω values ​​in three linearly independent spatial directions x, y, and z. x ,ω y and ω z The superposition of Euclidean norms is given as follows:

[0026]

[0027] The effective image resolution R used to determine the overall existence a,eff The amount of motion ω (t) K As a quantified representation of the expected motion, all camera and / or scene motion ω is reduced to a single estimate. This estimate can be, for example, an average, a safety-side estimate (in the form of a maximum value), a statistically expected value, or a similar value.

[0028] In addition, the exercise volume ω is proposed. K It is modeled as a probability distribution of angular velocity relative to the camera system. Using this preferred embodiment, the real scene and / or camera motion can be mapped more accurately than, for example, by estimation or averaging. To determine what the probability distribution of the camera and / or scene motion ω is, measurement driving can be performed in advance and the measurement data evaluated, for example. In this case, the motion quantity ω is preferably used. K To estimate the proportion of the safe side. This means that it is preferable to choose the amount of exercise ω in this way. K This ensures that 99.7% of all camera and / or scene motion ω is less than or equal to the motion magnitude ω. K .

[0029] In an extended embodiment of the invention, the probability distribution is modeled as a Gaussian normal distribution. This preferred embodiment enables near-realistic modeling of camera and / or scene motion commonly seen in traffic. The Gaussian normal distribution analysis is also straightforward, as it relies solely on the mean and standard deviation. In the preferred embodiment, the mean is chosen to be zero. This is combined with the motion magnitude ω... K Preferably, 99.7% of all occurring motion ω is estimated upwards, for the amount of motion ω K Conclusion:

[0030] ω K =3σ ω (8)

[0031] Where σ ω This corresponds to the standard deviation of a Gaussian normal distribution. In a preferred embodiment, the standard deviation σ is chosen. ω = 2° / s.

[0032] Furthermore, it is proposed to perform the following steps before step c):

[0033] i) Determine the effective signal-to-noise ratio S / Deff(t) as a function of the exposure time t, and

[0034] ii) The minimum required exposure time t min The effective signal-to-noise ratio S / Deff(t) is determined by taking the pre-defined minimum required signal-to-noise ratio S / D. * Exposure time under the given value.

[0035] This preferred embodiment is advantageous because excessively short exposure times can lead to images corrupted by noise. The greater the noise corruption, the lower the robustness of subsequent computer-aided object recognition. Therefore, this preferred embodiment ensures that when determining the exposure time using the method according to the invention, situations where the selection of exposure time t might negatively impact subsequent object recognition can be identified. The dimensionless minimum required signal-to-noise ratio S / D * Preferably, the effective signal-to-noise ratio S / Deff(t) is pre-set based on the requirements of the computer-aided object recognition algorithm to be used. The effective signal-to-noise ratio S / Deff(t) as a function of exposure time t can be determined in various ways. A preferred embodiment will be exemplarily described below. Preferably, the effective signal-to-noise ratio S / Deff(t) is based on the radiant energy Q of each pixel. p (expressed in watt-seconds [Ws]):

[0036]

[0037] Preferably, the average photon energy Q0 = 2.2 eV and the quantum efficiency η are selected. q =0.9. Meanwhile, the radiant energy Q of each pixel... p It can be based on image irradiance E i The pixel pitch p (also known as pixel pitch, usually measured in μm) applicable to the corresponding image sensor is expressed as a function of exposure time t as follows:

[0038] Q p =E i p 2 t (10)

[0039] Image irradiance E i (with [W / m 2 The irradiance (E) can be measured or estimated using simplified assumptions (such as the Lambertian reflection condition). Otherwise, it depends only on known quantities, including the f-number of the camera used and the expected light intensity of the observed scene. i One possible method of determination is in As described in Volume 1 of *Handbook of Computer Vision and Applications*, 1999, E et al. Therefore, E i For example, it can be calculated using the following formula:

[0040]

[0041] Where, L e Radiation corresponding to the surface of a Lambert object, measured in watts per steradian per square meter [W / (sr m] 2 )],n f This reflects the camera's f-number. By making equations (9) and (10) equal, the effective signal-to-noise ratio, which depends on the exposure time t, is obtained:

[0042]

[0043] According to requirements

[0044] S / Deff(t min ) = S / D * (13)

[0045] By finding the corresponding solution to equation (12), the minimum required exposure time t can be obtained. min .

[0046] It is also proposed that in step c), the exposure time t be set to be greater than or equal to the minimum required exposure time t. min The value of t. Using this preferred embodiment, the exposure time t is ensured to guarantee sufficient angular image resolution, while the resulting image record has sufficient quality for robust computer-aided object recognition.

[0047] Furthermore, a camera system is proposed, comprising at least one camera, wherein, according to the invention, the camera is gimbaled for image stabilization to support longer possible exposure times, and the exposure time t of the camera is preset according to the method of the invention. Such a camera system possesses the aforementioned advantages. In particular, the gimbaled suspension reduces camera motion and stabilizes the image. With reduced camera motion, the longer exposure time results in less motion blur. Simultaneously, the longer exposure time leads to a higher signal-to-noise ratio, which positively impacts the robustness of computer-aided object recognition.

[0048] The proposed gimbal suspension structure is a motor-driven arm, preferably a gimbal, comprising at least one servo motor for performing rotational motion about the roll and pitch axes, respectively. The gimbal improves the stability of recorded images in the case of a moving camera. Using this preferred embodiment, camera motion can be further reduced, which in turn results in less motion blur over longer possible exposure times, thereby obtaining images with a higher signal-to-noise ratio (S / D) and more robust object recognition.

[0049] Furthermore, the proposed camera system includes at least one inertial measurement unit (IMU) for measuring spatial acceleration. Using the IMU, preferably arranged on the camera, the motion of the camera system and / or at least one camera can be measured. Based on these measurement data, compensating motion can be determined for the occurring motion and executed by servo motors, thereby stabilizing the image.

[0050] Furthermore, it is proposed that the intersection of the roll axis and pitch axis coincides with the camera's perspective center. As a result of this preferred embodiment, any camera rotation caused by the motor-driven arm lies within the image plane, thus avoiding parallax shift effects.

[0051] Furthermore, the camera is equipped with a telephoto lens. This preferred embodiment improves distance resolution and enables clearer and more robust object recognition at greater distances. Attached Figure Description

[0052] The present invention will be shown in more detail below with the aid of the accompanying drawings. The drawings show:

[0053] Figure 1 A graph showing the relationship between angular image resolution, signal-to-noise ratio, and exposure time, and

[0054] Figure 2 A schematic top view of the camera in the camera system according to the present invention is shown. Detailed Implementation

[0055] Figure 1 The angular image resolution R is shown. a,eff A graph illustrating the relationship between signal-to-noise ratio (SNR) S / D(t) and exposure time t. (Using...) Figure 1 The method flow according to the present invention will be described below. Based on the application scope or other external specifications, a minimum angular image resolution R to be followed is pre-defined. a* This is represented by Formula 1. Its... Figure 1 The example shown is 40 ppd. To determine the exposure time, the effective image resolution R, expressed by Equation 2, is then created. a,eff (t) is a function of the exposure time t. This function depends in particular on the maximum tolerance motion blur dp and the amount of motion ω quantized by the camera and / or the moving scene.K The effective image resolution R, expressed by Equation 2, is... a,eff (t) is inversely proportional to the exposure time t. The minimum angular image resolution R, expressed by Equation 1, is... a * and the effective image resolution R expressed by Equation 2 a,eff The intersection of (t) yields the maximum permissible exposure time t expressed by Equation 3. max In this example, it is 4.2 ms. Therefore, the exposure time t to be determined must be less than or equal to the maximum permissible exposure time t expressed by Formula 3. max To satisfy the minimum angular image resolution R expressed by Equation 1 a* Requirements.

[0056] Insufficient exposure time t results in images corrupted by noise. This corruption significantly hinders computer-aided object recognition. Therefore, based on the application scope of the camera system (typically the specific requirements of the object recognition algorithm), the minimum required signal-to-noise ratio S / D, expressed by Equation 4, is derived. * In this example, it is 30. Furthermore, the effective signal-to-noise ratio S / Deff(t), expressed by Equation 5, is determined as a function of the exposure time t. The minimum required signal-to-noise ratio S / D, expressed by Equation 4, is... * The intersection of this intersection with the effective signal-to-noise ratio S / Deff(t) expressed by Equation 5 yields the minimum required exposure time t expressed by Equation 6. min In this example, it is 1.2 ms. Therefore, the exposure time t to be determined must be greater than or equal to the minimum required exposure time t expressed by Equation 6. min To satisfy the minimum requirement of signal-to-noise ratio S / D expressed by Equation 4 * Requirements.

[0057] Finally, the exposure time t of the camera system is set to the minimum required exposure time t expressed by Equation 6. min The downward limit and the maximum permissible exposure time t expressed by Formula 3 max Within the upward-limited interval.

[0058] Figure 2A camera 1 is shown for use in the camera system 10 of the present invention. The camera 1 includes a telephoto lens 2 and an image sensor 3, and is suspended on a motor-driven arm 4. The motor-driven arm 4 has two servo motors 6 for transmitting rotational motion to the camera 1 on two axes (7a, 7b). One servo motor 6 is designed to perform rotational motion about the roll axis 7a of the camera 1, and the second servo motor is designed to perform rotational motion about the pitch axis 7b of the camera 1. The intersection of the roll axis 7a and the pitch axis 7b coincides with the perspective center 9 of the camera 1. Furthermore, the camera 1 has an inertial measurement unit 8. For mounting to an external support, the motor-driven arm 4 also has a base 5. When the camera system 10 is moved, the inertial measurement unit 8 detects the direction and magnitude of the motion. Based on the measurements, the servo motors 6 initiate corresponding compensating rotational motions about the roll axis 7a and the pitch axis 7b to maintain image stability during movement.

Claims

1. A method for determining the exposure time (t) of a camera (1) of a camera system (10) for motion and / or for machine object recognition in a motion scene, wherein, Adhere to the pre-defined minimum angular image resolution The method includes the following steps: a) The effective angular image resolution (R) a,eff (t) is determined as a function of the exposure time (t). b) Set the maximum permissible exposure time (t) max The effective angular image resolution (R) is determined to be... a,eff (t) takes the minimum angular image resolution (R) a* Exposure time under the value of ), and c) Set the exposure time (t) to be less than or equal to the maximum permissible exposure time (t). max The value of ).

2. The method according to claim 1, characterized in that, In step a), the effective angular image resolution (R) a,eff (t) also depends on the maximum tolerance motion fuzzy dp and the motion quantity (ω) K To determine.

3. The method according to claim 2, characterized in that, The amount of motion (ω) K The ω is modeled as a probability distribution of the angular velocity relative to the camera system (10).

4. The method according to claim 3, characterized in that, The probability distribution is modeled as a Gaussian normal distribution.

5. The method according to any one of the preceding claims, characterized in that, Perform the following steps before step c): i) Determine the effective signal-to-noise ratio (S / Deff(t)) as a function of exposure time (t), and ii) The minimum required exposure time (t) min The effective signal-to-noise ratio (S / Deff(t)) is determined to be the minimum required signal-to-noise ratio (S / D) set in advance. * Exposure time under the condition of ) value.

6. The method according to claim 5, characterized in that, In step c), the exposure time (t) is further set to be greater than or equal to the minimum required exposure time (t). min The value of ).

7. A camera system (10), the camera system comprising at least one camera (1), characterized in that, The camera (1) is gimbaled for image stabilization to support longer possible exposure times, and the exposure time (t) of the camera (1) is preset according to the method of any one of claims 1 to 6.

8. The camera system (10) according to claim 7, characterized in that, The universal joint suspension is implemented as a motor drive arm (4), preferably as a universal joint, and the motor drive arm includes at least one servo motor (6) for rotating about the roll axis (7a) and the pitch axis (7b) respectively.

9. The camera system (10) according to any one of claims 7 or 8, characterized in that, The camera system (10) has at least one inertial measurement unit (8) for measuring spatial acceleration.

10. The camera system (10) according to claim 8 or 9, characterized in that, The intersection of the roll axis (7a) and the pitch axis (7b) coincides with the perspective center (9) of the camera (1).

11. The camera system (10) according to any one of claims 7 to 10, characterized in that, The camera (1) is equipped with a telephoto lens (2).

Images