Detection of 3D image data

Abstract

The present application relates to the detection of three-dimensional image data. A camera (10) for detecting three-dimensional image data from a detection area (18) is provided, comprising: an illumination unit (12) for emitting emission light (16) modulated at least at a first modulation frequency; an image sensor (26) having a plurality of receiving elements (26a) for generating corresponding received signals; a plurality of demodulation units (32) for demodulating the received signals at the first modulation frequency to obtain sample values; a reference illumination unit (38) for emitting reference light (42) modulated at the first modulation frequency and directed within the camera onto the image sensor; and a control and evaluation unit (34). The control and evaluation unit is also designed to distribute reference partial measurements for functional testing over a plurality of distance measurements.

Description

Technical Field

[0001] The present invention relates to a camera for detecting three-dimensional image data from a detection area, in particular a 3D time-of-flight camera, and a method for detecting three-dimensional image data from a detection area. Background Art

[0002] This camera measures distance and, thereby, obtains depth information. The detected three-dimensional image data, which includes distance or spacing values for each pixel, is also referred to as a 3D image, range image, or depth map. Various methods are known for determining depth information. Among these, time-of-flight (TOF) or LiDAR (Light Detection and Ranging) measurement based on phase-sensing techniques will be examined in more detail here.

[0003] Here, a scene is illuminated with amplitude-modulated light. The light returning from the scene is received and demodulated with the same frequency used to modulate the transmitted light (lock-in method). Demodulation produces an amplitude measurement value that corresponds to a sampled value of the received signal. According to the Nyquist criterion, at least two sampled values are required to determine the phase of a periodic signal. Therefore, the measurement is performed at a different relative phase position between the modulated signal on the transmitting side and the demodulated signal on the receiving side. This allows the determination of the absolute phase shift between the transmitted and received signals, which is dependent on the propagation time and is proportional to the distance to objects in the scene.

[0004] Figure 5a The figure shows a conventional modulation scheme. In the upper section, S represents the emitted light, periodically modulated at the modulation frequency. The lower section shows the returned received light, labeled E. This returned received light is phase-shifted from the emitted light S, depending on the distance from the object where the emitted light returns. The lower section shows the time periods allocated to the charge storage of the corresponding camera pixels, during which the photoelectrons generated during that time period are stored. As shown in the figure, the time periods do not necessarily need to be aligned with the emitted light, but possible time shifts should be considered, as otherwise they will lead to errors in phase measurement and, consequently, distance measurement.

[0005] exist Figure 5a In the CMOS process, each pixel has two charge stores, A and B, which are switched back and forth using the modulation frequency of the emitted light. The charge is integrated over multiple modulation cycles, i.e. the accumulated charge carriers are added together before the total charge is read out from the pixel, where Figure 5a Two such modulation cycles are shown. The sought phase is derived from B / (A+B).

[0006] A disadvantage of this simple measurement is its sensitivity to extraneous or background light. Admittedly, as described above, the minimum value for determining the phase of the periodic signal can be achieved with just the two values A and B. However, to additionally account for the variable background light (assuming a constant level, the background light is constant), at least one additional sample value is missing. To account for background light and other factors, such as compensating for nonlinearities and asymmetries in the image sensor's pixels, which lead to relatively high unknown systematic measurement deviations, additional measurements can be performed. Therefore, for robust and accurate distance measurement, more than two individual measurements are required to obtain a depth value per pixel. In other words, it is advantageous if a frame of a 3D image acquisition consists of more than two partial measurements. Alternatively, the number of charge stores in a pixel can be increased to account for background light, but this does not compensate for pixel asymmetries without repeated individual measurements. Alternatively, a pixel can have only a single charge store, and the number of partial measurements can be increased accordingly to compensate for this.

[0007] The extended modulation scheme of the four sample values is displayed together Figure 5a and Figure 5b In. Use according to Figure 5a The values A and B are determined by measuring Figure 5b Another measurement is shown, in which the time period after switching back and forth between the two charge stores is offset by ¼ of the modulation frequency period. To distinguish the measured values obtained in this way, they are labeled C and D. The sought phase is now derived from the arctan ((CD) / (AB)). As already mentioned, the number of charge stores in a pixel can be exchanged in a sense for the number of partial measurements per frame. Comparable measurements are linked by a constant product of the number of charge stores and the number of partial measurements, as this product gives the number of sample values obtained in a frame, from which the modulation frequency can be reconstructed.

[0008] For example, based on Figure 5a and Figure 5b An introductory discussion of the two methods presented can be found in the technical white paper SLOA190B (2014) by Li, Larry, “Time-of-flight camera - an introduction”.

[0009] In addition to the general measurements using two or four sampling values, variants using three sampling values at 0°, 120°, and 240° are also known. In alternative image sensor structures, the measured values of the two charge stores A and B are not read individually, but rather the differential pixel of the output value AB. Furthermore, it is known to measure the sampling values, such as A, B, C, and D, again with a 180° phase offset to compensate for asymmetries in the pixels. The corresponding number of sampling values brings different advantages and disadvantages, which are meaningful for different applications. The more sampling values obtained, the higher the measurement accuracy or the smaller the depth measurement deviation. The fewer sampling values, the greater the measurement error, but the acquisition time is shortened and fewer motion artifacts are generated.

[0010] Another aspect of time-of-flight measurements using the phase method is the limited unambiguous range, since the phase shift is only unambiguous within a period of the modulation frequency. To extend the unambiguous range, and thus the effective range of the camera, measurements are usually performed sequentially with multiple different modulation frequencies. An alternative reduction in the modulation frequency can only be considered within a narrow range, as this would affect the measurement accuracy. Measurements using multiple modulation frequencies combine high depth resolution with a large depth measurement range. This increase in effective range is possible with any number of partial measurements.

[0011] The additional distance dimension can be used in a variety of applications to obtain more information about the objects in the scene detected by the camera, thereby solving various tasks. For example, in automation technology, 3D image information can be used to detect and classify objects, allowing further automated processing steps to be performed based on which objects were identified (preferably including their position and orientation). This can be used, for example, to support the control of various types of actuators on robots or conveyor belts.

[0012] For vehicles operating on public roads or in closed environments, particularly in the field of factory and logistics automation, 3D cameras should be used to record the entire environment as completely and three-dimensionally as possible, particularly for route planning. This applies to almost all conceivable vehicles, whether they have drivers such as PKWs, LKWs, work machines, and forklifts, or driverless vehicles such as AGVs (Automated Guided Vehicles), AGCs (Automated Guided Carts), AMRs (Autonomous Mobile Robots), or ground transport vehicles. The image data is used to enable autonomous navigation or support the driver in identifying obstacles, avoiding collisions, or facilitating the loading and unloading of transported goods, such as cartons, pallets, containers, or trailers.

[0013] In safety technology, 3D cameras detect people near hazardous points (e.g., machines, especially robots). If an unauthorized intrusion into the protective envelope or a fall below the safety distance to the machine is detected, a safety-oriented response is initiated to prevent an accident. Sensors used in safety technology must operate extremely reliably and therefore meet high safety requirements, such as those in the EN 13849 standard for machine safety and the IEC 61496 or EN 61496 device standard for contactless protective devices (BWS). Meeting these safety standards requires a range of measures, such as safe electronic evaluation through redundant and diverse electronics, functional monitoring, or specialized monitoring of optical components for contamination. Safety standards also specify so-called safety classes or safety claim levels, which classify achievable safety. Examples include the SIL (Safety Integrity Level) according to IEC 61508 / IEC 61511, with 1 being the lowest and 4 being the highest, or the PL (Performance Level) according to EN ISO 13849-1, ranging from a "low" to e "high."

[0014] However, in practice, the application of safety technology in industrial environments has so far been addressed with safety light grids or laser scanners. The increasing level of automation itself requires the protection of increasingly complex automation tasks and geometries, which can only be achieved to a limited extent with these relatively simple, conventional safety sensors. However, safety cameras, and even 3D cameras, are still not available on the market or have significant limitations due to range, size, cost, and similar factors. In contrast, 3D time-of-flight cameras are based on highly integrated image sensors developed for completely different applications, integrating the illumination modulation and demodulation signals as well as most of the measurement technology. Therefore, these image sensors are not specifically designed for safety applications, and missing safety functions or safety-related diagnostic options can only be addressed purely theoretically with the prohibitively large and expensive development of new chips for extensive safety technology.

[0015] To ensure the safety of conventional safety laser scanners, an internal reference target is scanned with each revolution of the laser scanner and the desired distance must be measured from this reference target. This ensures the reliability of the entire measurement chain, with the same response time as new scan data is provided. This was first described in DE 43 40 756 A1 and remains common practice today. However, due to the lack of a scanning motion, this proven principle cannot be transferred to cameras.

[0016] In principle, it's conceivable to test individual components of an image sensor individually using different stimuli or test patterns. The functionality of the entire measurement chain can then be inferred from the functionality of the individual components. However, in reality, the test pattern never completely passes through the measurement chain. Therefore, conclusions about the overall measured values are only conceivable with a very good understanding of the subregion of the image sensor, and thus tailored specifically to the specific image sensor. Changing the image sensor is therefore extremely complex, as safety must be re-enforced. However, the diagnostic level is insufficient to achieve a high safety level within the meaning of the aforementioned safety standards; at best, it only achieves a moderate safety level. This is due to the considerable execution time required to repeatedly test partial functions using different stimuli. This not only highlights the effort required in terms of computing resources and runtime, but is also prohibitively slow if the safety level demands fault detection within a short response time.

[0017] The German patent application with the as-yet-unpublished document number 102019131988.9 relates to a 3D time-of-flight camera in which the number of measurement repetitions or partial measurements can be adjusted. This increases flexibility and application possibilities, but does not address safety concerns.

[0018] EP3 525 004 B1 discloses a TOF sensor with a test transmitter. During sensor operation, the transmitter is interrupted approximately every 100 milliseconds. During this pause, a test signal is sent, and the measured virtual distance is compared with the expected value. This test cycle is too long for a high level of safety. If the transmitter were interrupted more frequently, this would affect the frame rate and, therefore, the response time, because within a sufficiently short test cycle, the time required for the actual measurement is similar to that required for the test signal.

[0019] DE 10 2010 038 566 A1 discloses another time-of-flight camera with a monitoring function. For this purpose, a reference channel with a reference light source is provided. Reference measurements are performed at predetermined time intervals, for example, after each distance measurement, or at longer time intervals. These alternatives again exhibit the disadvantages mentioned in the previous paragraph, as the frame rate is affected by the reference measurement after each distance measurement, while infrequent reference measurements do not allow for the short response times required for a high level of safety.

[0020] EP 3 091 271 A1 describes a single-photon avalanche diode (SPAD)-based optical sensor, which, in one embodiment, includes a light emitter as a reference light emitter for safety-related self-tests. Pulsed methods are preferred, and the CW method is only mentioned without further explanation. Furthermore, it is unclear at what point the self-test can be performed without adversely affecting the frame rate or the length of the test cycle.

[0021] DE 10 2007 008 806 B3 describes photoelectric monitoring with dynamic testing. For the external test, the received light from the scene is superimposed with the test light. For the internal test, additional lighting is provided that radiates modulated light of a predetermined phase into the receiving element. Again, there is no explanation of how to achieve the required time for this test without affecting the frame rate or extending the test cycle in a manner incompatible with high safety levels. Summary of the Invention

[0022] Therefore, the object of the present invention is to increase the safety of universal cameras.

[0023] This task is accomplished by a camera, particularly a 3D time-of-flight camera, and a method for detecting three-dimensional image data from a detection area. The camera operates essentially as described in the introduction. An illumination unit generates emitted light modulated at a first modulation frequency. As a precaution, this is an artificial amplitude modulation, and the emitted light has a modulation frequency selected so as not to be confused with the carrier frequency of the actual light wave. The emitted light, reflected from objects in the detection area, is superimposed with extraneous or background light and falls on an image sensor having multiple receiving elements or pixels, which then generate corresponding received signals.

[0024] Multiple demodulation units perform demodulation using signals at a first modulation frequency derived from the corresponding received signals, obtaining sample values in a lock-in manner. For distance measurement, a first number of at least two partial measurements are performed, each with a different phase between the signals at the modulation frequency used for the transmitted light and the demodulation. The partial measurements can be sequential repetitive measurements via re-exposure and / or parallel measurements in multiple charge stores of the receiving element. The phase position of the received signal is reconstructed from the multiple sample values to thereby obtain the time of flight and, ultimately, the distance. Thus, the distance measurement obtains a distance value for each of the involved receiving elements or pixels by performing the multiple partial measurements corresponding to the first number, thereby forming a frame of image acquisition.

[0025] The reference illumination unit emits reference light, also modulated with a first modulation frequency, and exposes the image sensor. In contrast to the transmitted light, the reference light is fed back to the image sensor internally, i.e., within the camera, particularly within the camera housing. Therefore, the reference light does not escape from the camera into the detection area. This creates a reference channel in which at least a portion of the receiving elements, preferably all receiving elements, or at least those corresponding to safety-relevant areas within the detection area, can be actively detected by illumination.

[0026] For this functional test, a second plurality of reference partial measurements is performed, similar to the distance measurement. These are partial measurements, but in this case, the reference lighting unit now assumes the role of the lighting unit. Based on the reference partial measurements, a reference distance value is determined that corresponds to the optical path length between the reference lighting unit and the image sensor without further modification. The reference lighting unit can simulate or emulate other reference distances. In any case, the complete camera should measure the intended reference distance for a successful functional test.

[0027] The distance measurement and functional test are controlled and evaluated by a control and evaluation unit. The demodulation unit is preferably already implemented in the pixels, which are then also referred to as ToF pixels (Time of Flight) or lock-on pixels. Other components of the control and evaluation unit, particularly those responsible for partial and reference measurements and / or phase reconstruction from multiple sampled values, can already be implemented in the pixels or on the image sensor, or integrated with the image sensor on a common chip. Alternatively, the distance can also be determined downstream of the image sensor from multiple sampled values, for example in an FPGA (Field-Programmable Gate Array).

[0028] The present invention is based on the basic idea of distributing the reference partial measurements required for the functional test over multiple distance measurements or frames. Consequently, the second number of reference partial measurements is performed over multiple distance measurements or frames. In other words, the reference partial measurements are distributed over or interspersed with the partial measurements and are performed less frequently. In other words, even if the second number of reference partial measurements is not reached within the current frame, a partial measurement is already performed for the next distance value or frame. It should be noted here, and will be explained in more detail later, that by back-dating previous reference partial measurements, the functional test can still be performed at a high repetition rate, up to the frame rate itself. However, the second number of reference partial measurements may still not be reached within a frame.

[0029] The advantage of the present invention is that it provides a reference channel that can be fully integrated into the camera, allowing the entire measurement chain for 3D image data detection to be tested during operation. The existing calculation paths for distance calculation can be used for functional testing. This not only simplifies implementation but also implicitly tests the distance calculation. By distributing the reference partial measurement over multiple distance measurements or frames, the actual camera acquisition time is largely unaffected. At the same time, functional testing can be performed within the response time. Depending on the distribution of the partial and reference partial measurements, the power consumption and thermal load of the device are affected differently. Overall, the camera can be designed for safety according to the aforementioned or similar safety standards for personal protection, machine safety, or contactless protective equipment, achieving a high safety level, such as at least SIL 2 or PL d. This opens up flexible applications in a wide range of areas, particularly in mobile applications where motion precludes reliable predictions of certain image sections.

[0030] Therefore, the control and evaluation unit is preferably designed to activate only the lighting unit during the partial measurement and only the reference lighting unit during the reference partial measurement. This means that the two lighting units do not interfere with each other, and only the partial measurement or the reference partial measurement is performed. The respective lighting units do not necessarily have to be active throughout the respective time window; instead, there should preferably be as many phases as possible without any lighting.

[0031] Preferably, the first number and / or the second number is at least three. Thus, the corresponding phase can be reconstructed from at least three samples from at least three partial measurements or at least three reference partial measurements. As discussed in the introduction, two samples are generally sufficient, but even constant background light can distort the distance value to the point of being unrecognizable. At least a third sample can account for this shift in background light. Generally, the additional samples allow for a more accurate reconstruction of the phase caused by the run time, thereby improving the distance measurement. A preferred constellation position (Konstellation) for the three samples is achieved by phase shifts of 0°, 120°, and 240° between the modulation on the transmitting side and the demodulation on the receiving side. For four samples, the phase shifts are preferably 0°, 90°, 180°, and 270°. More samples and other phase positions are possible, as well as additional partial measurements or reference partial measurements of the offset or background light.

[0032] Preferably, the first number is not equal to the second number. This ultimately means that the accuracy of distance measurement and functional testing differs. Preferably, the first number is greater than the second number, so that the distance measurement is more reliable and accurate through more partial measurements. Due to differences in distance, ambient light, and other factors, greater variability can be expected within the detection area. Furthermore, in the reference channel, the accuracy of measuring the fixed distance between the reference lighting unit and the image sensor is less important; what matters is detecting errors. Alternatively, the first number can be equal to the second number to provide more similar conditions during distance measurement and functional testing.

[0033] Preferably, the control and evaluation unit is designed to perform one or two reference partial measurements for each distance measurement. Particularly preferably, only one reference partial measurement is performed for the first number of partial measurements of a distance measurement. This minimizes the impact on the duration of the distance measurement and the frame rate. Functional testing requires multiple reference partial measurements and can therefore only be performed after multiple distance measurements or frames. To speed up this process or to keep it as up-to-date as possible, preferably by back-dating previous reference partial measurements, as will be described, two reference partial measurements can also be performed for each distance measurement. Under controlled conditions for the functional test compared to the distance measurement, with only internal access within the protective housing of the camera, it is at least conceivable to determine a reference distance value from two reference partial measurements. However, according to the present invention, there is no provision for determining the reference distance value from a reference partial measurement of a single distance measurement, and for sufficient reliability, the functional test should preferably be based on at least three reference partial measurements, as already explained.

[0034] Preferably, the control and evaluation unit is designed to perform at least one reference partial measurement with every i-th distance measurement. Particularly preferably, exactly one reference partial measurement or exactly two reference partial measurements are performed with every i-th distance measurement. The period of the reference partial measurements is denoted by i. Preferably, the case i=1 is permitted, meaning that at least one reference partial measurement is performed with each distance measurement. For i>1, slower periods with distance measurements in between are generated, with no reference partial measurements at all. Irregular distributions are also conceivable instead of periods, for example one reference partial measurement in the first frame, two reference partial measurements in the second frame, and no reference partial measurements in the third frame, as well as identical, similar, or completely different distributions in the further frames.

[0035] Preferably, the control and evaluation unit is designed to vary the first number, the second number, the phase offset of the partial measurements, the phase offset of the reference partial measurements, and / or the distribution of the reference partial measurements over the distance measurements. Thus, the camera is not fixed to a single configuration but can be switched over, depending on the situation, within a short timeframe or permanently during a subsequent, longer operating phase. This allows for adapting to different safety levels. One of the countless possible examples is switching from three partial measurements at 0°, 120°, and 240° to five partial measurements at 45°, 90°, 120°, 270°, and 350°, with the reference partial measurements also being performed in the same or completely different manner. Another example is switching from one reference partial measurement for every distance measurement to two reference partial measurements for every three distance measurements. This allows for adjustments to the response time and accuracy of the distance measurement, the reliability of the functional test, the cycle time, and ultimately the safety level. Such changes can, in particular, diversify the functional tests.

[0036] Preferably, the control and evaluation unit is designed to determine a reference distance value from at least one current reference partial measurement in the current distance measurement and at least one previous reference partial measurement in the previous distance measurement, in particular in this manner for each distance measurement. This is the already mentioned use of previous reference partial measurements by a rolling method similar to a sliding average. Since the second plurality of reference partial measurements has not been reached in a single distance measurement, a previous reference partial measurement with a currently unmeasured phase reference is added. Preferably, the earlier or previous reference partial measurement is the most recently available. This allows the functional test to be performed more frequently, since it is not necessary to wait for the completion of the second plurality of reference partial measurements. In particular, the functional test can even be performed for each distance measurement or frame, even if the reference partial measurements in question were collected over multiple frames.

[0037] The control and evaluation unit is preferably designed to use a shorter integration time during the reference partial measurement than during the partial measurement. The integration time, or exposure time, refers to the time window during which the receiving element collects photons for the corresponding sample value. Due to the well-defined conditions in the reference channel, including a short, known optical path, a relatively short integration time is sufficient. This further reduces the measurement time required for the reference partial measurement, which is already limited by the distribution of multiple distance measurements. In contrast, such short integration times are insufficient in measurement channels for potentially distant and dim objects.

[0038] Preferably, the control and evaluation unit is designed to check at least one function of the image sensor, in particular to detect defective pixels, based on only one reference partial measurement. The actual functional test of the entire measurement chain and the determination of the time of flight can only be achieved using multiple reference partial measurements. However, even based on only one reference partial measurement, supplementary functional tests can be performed to detect errors in the image sensor. For example, these can detect errors in defective pixels, groups of pixels, or rows or columns, or their readout and actuation.

[0039] Preferably, the control and evaluation unit is designed to perform additional partial measurements and / or reference partial measurements using at least a second modulation frequency. As mentioned in the introduction, the unambiguous range is expanded by using two or more modulation frequencies for measurement. For each modulation frequency, more partial measurements are required, with the number of measurement repetitions again preferably, but not necessarily, the same for each adjustment frequency. It is conceivable that the number of modulation frequencies, as well as the modulation frequency, can be adjusted depending on the desired effective range and measurement resolution. In a reference channel with freely adjustable runtime, the unambiguous range is not crucial. However, for more reliable diagnosis, it is preferred to also test using multiple modulation frequencies, preferably performing reference partial measurements at the first modulation frequency and / or at least the second modulation frequency. Since this provides greater freedom in modulation frequency, more combinations are available regarding how reference partial measurements can be interspersed with distance measurements. Advantageously, but not necessarily, the reference channel uses the same modulation frequency as the one currently used in the measurement channel. The reference light source can be operated independently with its own modulation frequency, even one that is never used in the measurement channel.

[0040] Preferably, the control and evaluation unit is designed to apply an artificial delay to the reference beam. This simulates or emulates a reference target at different distances from the physical beam path. The delay can be negative, in which case the reference target appears to be closer. This artificial delay further expands and diversifies the functional test.

[0041] Preferably, the receiving element has multiple charge stores. As explained in the introduction, multiple charge stores allow for simultaneous acquisition of multiple sample values, thus enabling multiple simultaneous partial measurements or reference partial measurements. However, if there are too many charge stores, only a small fraction of the integration time is allocated to each charge store, so this division of the limited number of incident photons has its limitations. Particularly preferably, each receiving element has exactly two charge stores, and even more preferably, these two charge stores are read out differentially. Then, instead of providing a sample value A from one charge store and a second sample value B from another charge store, the two charge stores provide a difference value AB. Even with multiple charge stores, distance measurement involves repeated partial measurements with different phase references. This facilitates obtaining more sample values at different phases. A 180° offset is also advantageous for both differential and single readout. This is because, while theoretically the same phase is measured, in practice there are asymmetries in the channels formed by the charge stores, which can be compensated for by dual, 180°, counter-measurements. Alternatively, embodiments with only one charge store per receiving element are possible.

[0042] Preferably, the reference light is guided directly onto the image sensor via at least one reflective region and / or a light guide. This leaves room for design changes within the camera. The reflective region can be combined with or integrated into other components, such as the optics, front panel, hood, housing, etc., and the light guide can be used to implement virtually any optical path or portion thereof to suit the camera's internal conditions.

[0043] The control and evaluation unit is preferably designed to vary the frame rate at which the distance measurements are repeated. The frame rate cannot be faster than that specified by the selected first number of partial measurements and the effective range. However, the effective range can be varied to shorten the partial measurements, slowing down the frame rate or reversing the slowdown. Waiting times below the technically possible frame rate reduce light emissions and save energy.

[0044] The control and evaluation unit is preferably designed to adjust the emitted light of the lighting unit. Numerous criteria are conceivable for this purpose, such as the required measuring range, certain areas of interest within the detection area, or extraneous light loads. Adjustment can be achieved by switching the lighting modules of the lighting unit on and off, or alternatively, by means of an adaptive lighting unit designed to selectively distribute the emitted light across the scene depending on location and / or time. Adjustment of the reference light is also conceivable, but since the conditions within the camera do not change, a fixed, one-time setting is generally sufficient. It is also conceivable to at least temporarily forego the functional testing of receiving elements that are not relevant to safety, for example, located outside the area of interest, where the detection area is not adequately illuminated due to the lighting unit's adjustment.

[0045] The method according to the invention can be further developed in a similar manner while exhibiting similar advantages. Such advantageous features are described by way of example and not by way of exhaustive enumeration. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Other features and advantages of the present invention will be further described below based on embodiments and with reference to the accompanying drawings. In the accompanying drawings:

[0047] Figure 1 shows a schematic block diagram of a 3D time-of-flight camera with a directly illuminated reference channel;

[0048] Figure 2 shows a top view of a receiving pixel of a 3D time-of-flight camera with two charge stores;

[0049] Figure 3 shows a schematic block diagram of a 3D time-of-flight camera with a reflective reference channel;

[0050] Figure 4a shows an exemplary distribution of measurements of various reference portions at the end of the distance measurement;

[0051] Figure 4b An exemplary distribution of individual reference partial measurements between partial measurements of distance measurements is shown;

[0052] Figure 4c shows a distribution of exemplary cyclic transformations of various reference partial measurements between different partial measurements of a distance measurement;

[0053] Figure 4d shows an exemplary irregular distribution of various reference partial measurements between different partial measurements of distance measurements;

[0054] Figure 4e Shown in a similar Figure 4aExample distribution of individual reference partial measurements at the end of the distance measurement, but now the distance measurement has four partial measurements instead of three;

[0055] Figure 4f shows an exemplary non-uniform distribution of two reference partial measurements between different partial measurements of a distance measurement;

[0056] Figure 5a A diagram showing a conventional modulation scheme with two sampled values for distance measurement; and

[0057] Figure 5b A diagram showing a conventional spreading modulation scheme with four sample values is shown. DETAILED DESCRIPTION

[0058] Figure 1 A schematic block diagram of a camera 10 is shown, which is preferably designed as a 3D time-of-flight camera. An illumination unit 12 emits emission light 16 modulated by emission optics 14 into a detection area 18. The light source can be an LED or laser in the form of an edge emitter or VCSEL. The illumination unit 12 is controllable so that the amplitude of the emission light 16 is modulated at a frequency typically in the range of 1 MHz to 1000 MHz. For example, the modulation can be sinusoidal or rectangular, but in any case, it is periodic. To reduce the mutual influence of multiple systems, artificial dithering or some type of coding (spread spectrum) can also be used. This frequency results in a limited unambiguous range for distance measurements, so a low modulation frequency is required for a camera 10 with a large effective range. Alternatively, measurements can be performed at two, three, or more modulation frequencies to combine the measurements to extend the unambiguous range.

[0059] If the emitted light 16 strikes an object 20 in the detection area 18, a portion is reflected back to the camera 10 as received light 22, where it is guided through receiving optics 24, such as a single lens or a receiving lens, to an image sensor 26. The image sensor 26 has a plurality of receiving elements or receiving pixels 26a, for example, arranged in a matrix or rows. The resolution of the image sensor 26 can range from two or a few to thousands or millions of receiving pixels 26a.

[0060] Figure 2 A top view of a portion of a receiving pixel 26a of an image sensor 26 is shown. This view is purely functional and schematic, and the specific semiconductor structure of the image sensor 26 is not the subject of this description. The receiving pixels 26a each have a photosensitive surface 28 and at least one charge storage 30, for example a capacitor. Figure 2In the exemplary embodiment, two charge stores 30 are provided. The other switching elements of receiving pixel 26a are combined in a block, very schematically and purely symbolically, as a demodulation unit 32. Receiving pixel 26a detects received light 22 in its charge store 30 during a measurement period or integration time. Demodulation unit 32 controls the time during which charge is collected in charge store 30, depending on the modulation frequency also used to modulate emitted light 16. Demodulation thus occurs according to a lock-in method.

[0061] The pixel arrangement is typically a matrix, resulting in lateral spatial resolution in the X and Y directions, supplemented by distance measurement in the Z direction to form three-dimensional image data. This type of 3D detection is preferably referred to when discussing 3D cameras, 3D time-of-flight cameras, or 3D image data. However, other pixel arrangements are generally conceivable, such as pixel rows selected in a matrix or pixel rows forming the entire image sensor of a line scan camera.

[0062] Back to Figure 1 , the charge quantity collected in the charge storage 30 of the receiving pixel 26a is read out, digitized and transmitted to the control and evaluation unit 34. The two charge storages 30 generate two sample values. As an alternative to the single readout of two sample values, a differential receiving pixel 26a can also be envisaged, which outputs the difference between the two charge storages 30 as a single sample value. In this case, the information related to the phase determination is actually equivalent, because the difference is formed in the evaluation anyway. In order to obtain additional sample values, the described partial measurement of collecting and reading out the charge storage 30 is repeated n times, preferably two to four times. In each case, the phase between the modulation frequency used for the emitted light 16 and the modulation frequency used for demodulation in the demodulation unit 32 is changed between the partial measurements.

[0063] From the plurality of sampled values, the control and evaluation unit 34 now reconstructs a phase shift based on the time of flight through the detection region 18, which can be converted into a distance value for each receiving pixel 26a. A three-dimensional image, a range image, or a depth image is generated, which is output to the interface 36. The interface 36, or alternatively one or more further connections (not shown), in turn serves to input control signals or parameterizations of the camera 10.

[0064] Therefore, the distance measurement is performed according to the indirect time-of-flight method, the principle of which has already been described in the introduction. For example, to determine the distance value, three partial measurements are performed in sequence with corresponding phase references 0°, 120° and 240°, or four partial measurements are performed in sequence with corresponding phase references 0°, 90°, 180° and 270°, wherein for the latter embodiment, the distance calculation has already been described in the introduction. Figure 5a-5b Explained.

[0065] There are many variants of this, which differ in the number of partial measurements and the corresponding phase reference between the transmitter modulation and the receiver modulation during the partial measurements. Thus, to give an arbitrary example that is not particularly relevant in practice, seven partial measurements can also be performed at 6°, 90°, 105°, 170°, 250°, 300° and 310°. In addition, for Figure 2 Alternatively to the embodiment shown with two charge stores 30, more charge stores or just one charge store are conceivable. The number of charge stores 30 allows for simultaneous acquisition of multiple sample values, so that simultaneous and sequentially performed partial measurements can be interchanged in some manner by the charge stores.

[0066] The illustrated embodiment preferably has two charge stores 30, which are sampled 180° out of phase with each other and then preferably read out differentially. Whether or not differential readout is performed, an additional partial measurement is advantageously performed in each case with a 180° offset from the other partial measurement. For three partial measurements, the phase references are preferably 0°, 180°; 120°, 300°; 240°, 60°; and for four partial measurements, the phase references are 0°, 180°; 90°, 270°; 180°, 0°; and 270°, 90°. The latter is particularly redundant but helps to compensate for hardware-related differences in the two channels formed by the charge stores 30. This results in a total of eight individual sampled values A, B, C, D, A', B', C', D', or, in the case of differential pixels, four differences in the sampled values AB, CD, A'-B', C'-D'. The phase sought is then calculated as arctan(((CD)-(C'-D')) / ((AB)-(A'-B'))) Of course, for other numbers of partial measurements or other relative phase offsets, different formulas will result, but the phase can still be reconstructed using known mathematical techniques that can be implemented with low computational resources.

[0067] In order to perform a functional test on the camera 10, in addition to the illumination unit 12, a diagnostic or reference illumination unit 38 is provided, which generates a reference light 42 via an optional reference optical device 40, with which the interior of the image sensor 26 is illuminated. Internal means that the beam path of the reference light 42 extends within the camera 10, in particular within its housing (not shown), and therefore does not enter the scene of the detection area 18 and is not affected by the scene and its surrounding conditions. Figure 1In the embodiment, the reference light 42 is directly coupled from the reference illumination unit 38 to the image sensor 26. The reference illumination unit 38 may be preferably arranged near the image sensor 26. Alternatively, for the illustration, it may be arranged between the image sensor 26 and the receiving optical device 24 while avoiding shadows. Figure 3 An alternative embodiment of a reference channel is presented.

[0068] The structure and light source of reference illumination unit 38 are similar to those described above for illumination unit 12. Due to the short internal optical path, the requirements for reference illumination unit 38 are lower. Therefore, compact and inexpensive laser diodes or LEDs can be used, as only a low light output is required, and even strong light sources must be suppressed to avoid saturation. Reference illumination unit 38 can be driven separately from image sensor 26 using a modulation signal. For example, a second drive channel of image sensor 26 (if available) can be used for this purpose, or alternatively, the application of the modulation signal to illumination unit 12 or reference illumination unit 38 can be selected via corresponding enable signals.

[0069] Therefore, in addition to the actual measurement channel, which measures distances in detection area 18, a reference channel is also established for functional testing. Similar to the distance values of the measurement channel, receiving pixels 26a measure reference distance values via the reference channel. In a complete system, the reference distance values must correspond to expected values, i.e., the internal optical paths from reference illumination unit 38 to image sensor 26. Dynamic changes in the scene in detection area 18 have no effect on these internal optical paths. It is conceivable to simulate different distances for the reference distance value to be measured by adding an artificial delay between modulation and demodulation. This, in particular, allows for diagnostic purposes for possible phase-dependent measurement errors. The corresponding expected values for the complete system are initially taught or specified based on theoretical considerations or simulations.

[0070] The control and evaluation unit 34 thus detects, based on the reference channel, when the camera 10 can no longer reliably perform its task. In safety-related applications, a safety-oriented signal is preferably output in the event of a fault, which causes the machine, vehicle, robot, or other monitored hazard to be brought to a safe state, either by slowing down, avoiding, or stopping.

[0071] Will Figure 1 and Figure 2The division of the image sensor 26 into receiving pixels 26a (each with a demodulation unit 32) and the control and evaluation unit 34 is merely a preferred embodiment. The control and evaluation functions can also be distributed in other ways. As shown, the control and evaluation unit 34 need not consist of the single component shown, but can instead consist of one or more digital computing components, such as a microprocessor, an FPGA (field programmable gate array), or an ASIC (application-specific integrated circuit). Furthermore, the illumination shown is planar, for which a diffuser, for example, is used as part of the transmission optics 14. In another embodiment, the illumination unit 12 arranges multiple individual light sources to project clearly into the detection area 18, thereby individually illuminating the receiving pixels 26a and increasing the effective range. Furthermore, unlike the illustration, the illumination unit may not be integrated into the camera 10 but may be structurally or spatially separated from it.

[0072] Figure 3 Another embodiment of the camera 10 is shown. Figure 1 In comparison, only the reference channel is changed. The optical path of the reference light 42 from the reference illumination 38 to the image sensor 26 is not direct here, but is folded once with the help of a reflective element 44. An optional reference optical device 40 can also be provided. The reflective element 44 can be a separate element, or an at least partially reflecting area of another component can be used or attached to this area, for example, attached to one of the optical devices 14, 24, the front panel, the shielding of the transmitter-receiver chip or the housing components. The optical path of the reference light 42 can also be folded or deflected multiple times. As a further alternative, it is conceivable that at least part of the optical path is in a light guide, where the reflection and light guide can be combined with each other. Ultimately, it is important that enough reference light 44 reaches the internally reproducible optical path on the image sensor 26 and illuminates the receiving pixel 26a to be tested there.

[0073] Figure 4a-4f Various exemplary approaches are shown for interleaving partial reference measurements for functional testing into partial distance measurements. Thus, the functional test or reference measurement is distributed across multiple distance measurements or frames of the camera 10. Consequently, the additional time and power consumption required for the functional test is spread over a longer time interval.

[0074] Figure 4aA first embodiment is shown, in which the distance measurement is based on three partial measurements with phase references of 0°, 120°, and 240° between modulation and demodulation, as well as three reference partial measurements that determine a reference distance value for the functional test. In this embodiment, the reference partial measurements are inserted at the end of each distance measurement or frame (image acquisition cycle), specifically, only one reference partial measurement per frame. Each inserted reference partial measurement has its own phase reference, independent of the partial measurements, which varies from frame to frame. In the first frame, a reference partial measurement is performed with a phase reference of 0°, in the second frame, a reference partial measurement with a phase reference of 120°, and in the third frame, a reference partial measurement with a phase reference of 240°. The corresponding intermediate results are stored, for example, in a preprocessing device, preferably an FPGA, or elsewhere in a memory area accessible to the control and evaluation unit 34. After the third frame, a complete reference channel data set is obtained, from which a reference distance value is generated. The illustrated cycle then repeats.

[0075] Figure 4b Another embodiment is shown, in which the reference partial measurement is now inserted between the second and third partial measurements, rather than at the end of the frame. The reference partial measurement could also be inserted between the first and second partial measurements, or set at the beginning of the frame. Furthermore, due to the computation time between frames, the start time of one frame is not necessarily equal to the end time of the previous frame.

[0076] Figure 4c Another embodiment is shown in which a reference partial measurement is inserted alternately between different partial measurements of a frame in a cyclic manner. Alternatively, the cycle can start at different points in time between other partial measurements, run in the opposite direction, and so on.

[0077] Figure 4d Another embodiment is shown, intended to demonstrate that even with a fixed number of partial measurements per frame, a fixed number of reference partial measurements per frame and per functional test, and fixed phase relationships, there are many other options for distributing the reference partial measurements within and across frames. In the illustrated embodiment, the reference partial measurements vary in an irregular manner at various points in time within the frame, and this irregular variation repeats or continues after the third frame. Furthermore, the order of the phase references for the reference partial measurements changes, now first measuring at 120°, then at 0°, and then at 240°. This can also be repeated or changed after the third frame. The order of the phase references for the partial measurements within a frame can also vary from frame to frame.

[0078] Figure 4e A further embodiment is shown, which corresponds in principle to Figure 4a, wherein exactly one reference partial measurement is performed at the end of each frame. In addition, the number of partial measurements per frame has been increased to four, so the phase references are now 0°, 90°, 180°, and 270°. The same applies to the reference partial measurements, which now have a phase reference of 0° in the first frame, 90° in the second frame, 180° in the third frame, and 270° in the fourth frame. It will be appreciated that similar to Figure 4b-4d Other embodiments are possible in which the reference partial measurements change their time points within the frame and the order of the phase references of the partial measurements and / or the reference partial measurements. Other numbers of partial measurements and reference partial measurements (e.g., five or more) and other phase references are also contemplated.

[0079] Figure 4f Another embodiment is shown, in which two reference partial measurements are now performed per frame. The example shown uses irregular time points for the reference partial measurements within the frame and a non-ascending sequence of the phase reference, with the specific illustration merely representing possible irregularities. It is also conceivable to repeat a similar cycle or generally regular time points, for example at the beginning, middle or end of the frame and / or an ascending sequence of the phase reference. Another variant, not shown, varies the number of reference partial measurements inserted into each frame, for example one reference partial measurement in the first frame, two reference partial measurements in the second frame, no reference partial measurement in the third frame and one reference partial measurement in the fourth frame, and then the cycle repeats or continues irregularly. It is also conceivable to have longer cycles in which the reference partial measurements are inserted only in the second frame, the third frame or generally the i-th frame, respectively.

[0080] Figure 4a-4f The embodiments shown in the are not complete examples, which can be combined with each other and even so represent only a few of countless possibilities. The number of partial measurements of the distance measurement and / or the number of reference partial measurements on which the functional test is based can be further varied, and the associated phase reference can also be varied. This can even be used in different ways for distance measurement and functional test, for example a distance measurement with four partial measurements at 0°, 90°, 180° and 270°, and a functional test consisting of three reference partial measurements at 0°, 120° and 240°. Differential pixels can be used. The order in which the phase references follow each other in the partial measurements or reference partial measurements can be mixed cyclically and non-cyclically. The reference measurements can be inserted at different points in time of the respective frames, and this can be the same or different from frame to frame.

[0081] Figure 4a-4fOne possible variation not shown here concerns the modulation frequencies used. In principle, the measurement channel and the reference channel are independent of each other in this respect. Thus, it is conceivable to measure distance values using a first modulation frequency, while the reference component measurement uses a completely different modulation frequency. Alternatively, a well-defined range of distance values can be measured using a first and a second modulation frequency, while the reference component measurement uses these first and second modulation frequencies, only one of them, the other, or even two or more other modulation frequencies, depending on the embodiment. This results in at least one additional degree of freedom in the modulation frequencies, which can be combined with all of the previously described variation possibilities.

[0082] Therefore, functional testing is based on reference portion measurements from different frames. However, this does not mean that functional testing can only be performed every n>1 frames after a new set of reference portion measurements has been completed. Although this is possible, it would result in a rather slow response time for functional testing, which does not necessarily meet high safety levels.

[0083] In order to be able to perform the functional test within several frames, preferably within each frame, a rolling method similar to a sliding average is preferably used. The reference distance value is determined based on the reference partial measurement from the current frame and stored reference partial measurements from the previous frame, preferably the immediately preceding frame, for other still-needed phase relationships. The expected value of the reference distance value is preferably kept sufficiently narrow so that deviations or errors that are intolerable from a safety perspective are only detectable in the current reference partial measurement.

[0084] by Figure 4a This rolling method is specifically explained using an example; it can be applied similarly to all other variants. In the first frame, a reference partial measurement is performed with a phase reference of 0°; thus, no reference distance value can yet be obtained. In the second frame, a reference partial measurement is performed with a phase reference of 120°, but the two available reference partial measurements are still insufficient to determine the reference distance value. In the third frame, a reference partial measurement is performed with a phase reference of 240°, and now that the system has stabilized for the first time within a fraction of a second, a reference distance value can be determined from the three available reference partial measurements of 0°, 120°, and 240°, even though only the reference partial measurement of 240° originates from the current frame. In the fourth frame, a reference partial measurement is again performed with a phase reference of 0°, and this reference partial measurement replaces the old reference partial measurement with a phase reference of 0° from the first frame to determine the reference distance value.

[0085] This allows for short response times for both the actual measurement and the functional test. By distributing the reference partial measurement over several frames, the distance value can be determined with barely noticeable delays. Furthermore, the rolling method allows functional tests to be performed with short response times, up to once per frame. Consequently, according to the present invention, high safety levels, such as SIL 2 or PL d, can also be achieved.

[0086] The total integration time for all partial measurements typically accounts for approximately one-third of the camera 10 cycle time. Subsequently, the raw data is calculated and the actual 3D image is computed, which is then evaluated for the application. Therefore, the cycle times listed above correspond to possible frame rates, which can still be artificially slowed down, for example, to meet external expectations for the interface or configuration of camera 10.

[0087] Compared to the integration time for the partial measurement, the integration time for the reference partial measurement can be very small. The signal strength of the reference illumination unit 38 is known and reproducible, independent of the scene in the detection area 18, and does not decay. A short integration time not only shortens the minimum possible duration required for the reference partial measurement within a frame, since the functional test should have no or at least minimal impact on the response time of the camera 10. When the demodulated signal is only present briefly, the integration time also reduces power consumption in the image sensor 26, as the modulation and demodulation performed during the integration time account for the majority of power consumption in the image sensor 26, and indeed in the entire camera 10. The time gained by the short integration time can alternatively be used for a higher level of diagnostics, testing for additional phase relationships, frequency variations, etc. However, this comes at the cost of relatively higher power consumption.

[0088] In addition to the aforementioned functional test of the entire measurement chain based on multiple reference partial measurements, it is also possible to evaluate the current reference partial measurement individually without evaluating the time-of-flight of errors that occur locally in the image, such as defective pixels, columns, or rows. Such local errors can already be identified without having to calculate multiple reference partial measurements to obtain a reference distance value.

[0089] Alternatively, distributing the reference partial measurements across multiple frames can record a complete reference channel dataset (including all required reference partial measurements) within a single frame, and then repeat this operation for each frame or for all n frames. However, this only works if the image sensor 26 and downstream raw data processing are fast enough; otherwise, this would result in a longer response time for the actual measurement or a lower frame rate. This problem is addressed by distributing the reference partial measurements across the partial measurements in different frames.

Claims

1. A camera (10) for detecting three-dimensional image data from a detection area (18), comprising: a lighting unit (12) for emitting emitted light (16), said emitted light being modulated at least at a first modulation frequency, an image sensor (26) having a plurality of receiving elements (26a) for generating corresponding receiving signals, a plurality of demodulation units (32) for demodulating the received signal at the first modulation frequency to obtain sampling values, a reference illumination unit (38) for emitting reference light (42), the reference light being modulated at the first modulation frequency and directed onto the image sensor (26) within the camera (10), and a control and evaluation unit (34) designed to drive the illumination unit (12) and / or the demodulation unit (32) for a first number of partial measurements at different phase shifts between the first modulation frequency for the emitted light (16) and the first modulation frequency for demodulation for distance measurement, and to determine a distance value from the sampled values obtained by the partial measurements of each receiving element (26a), and For a functional test, the reference illumination unit (38) and / or the demodulation unit (32) are driven for a second number of reference partial measurements at different phase offsets between the first modulation frequency for the reference light (42) and the first modulation frequency for demodulation, and a reference distance value is determined from the sampled values obtained by the reference partial measurements of each receiving element (26a), It is characterized in that The control and evaluation unit (34) is further designed to distribute the reference portion measurement for the functional test over a plurality of distance measurements.

2. The camera (10) according to claim 1, in, The control and evaluation unit (34) is designed to activate only the lighting unit during a partial measurement and only the reference lighting unit during a reference partial measurement.

3. The camera (10) according to claim 1, in, The first number and / or the second number is at least three.

4. The camera (10) according to claim 2, in, The first number and / or the second number is at least three.

5. The camera (10) according to any one of claims 1 to 4, in, The first quantity is not equal to the second quantity.

6. A camera (10) according to any one of claims 1 to 4, in, The control and evaluation unit (34) is designed to perform one reference partial measurement or two reference partial measurements per distance measurement and / or to perform at least one reference partial measurement in each i-th distance measurement.

7. A camera (10) according to any one of claims 1 to 4, in, The control and evaluation unit (34) is designed to vary the first number, the second number, the phase shift of the partial measurement, the phase shift of the reference partial measurement and / or the distribution of the reference partial measurement over the distance measurement.

8. A camera (10) according to any one of claims 1 to 4, in, The control and evaluation unit (34) is designed to determine a reference distance value from at least one current reference partial measurement of a current distance measurement and at least one previous reference partial measurement of a previous distance measurement.

9. A camera (10) according to any one of claims 1 to 4, in, The control and evaluation unit (34) is designed to use a shorter integration time during the reference partial measurement than during the partial measurement.

10. The camera (10) according to any one of claims 1 to 4, in, The control and evaluation unit (34) is designed to check at least one function of the image sensor (26) based solely on one reference partial measurement.

11. The camera (10) according to any one of claims 1 to 4, in, The control and evaluation unit (34) is designed to perform further partial measurements and / or reference partial measurements using at least a second modulation frequency.

12. The camera (10) according to any one of claims 1 to 4, in, The control and evaluation unit (34) is designed to apply an artificial delay to the reference light.

13. The camera (10) according to any one of claims 1 to 4, in, The receiving element (26a) has a plurality of charge memories (30) which are read out differentially.

14. The camera (10) according to any one of claims 1 to 4, in, The reference light (42) is guided via at least one reflective region (44) and / or a light guide directly onto the image sensor (26).

15. The camera (10) according to any one of claims 1 to 4, in, The control and evaluation unit (34) is designed to change the frame rate at which the distance measurement is repeated and / or to adjust the emitted light (16) of the lighting unit (12).

16. The camera (10) of claim 1, wherein: The camera is a 3D time-of-flight camera.

17. The camera (10) according to claim 8, wherein In this way, a reference distance value is determined for each distance measurement.

18. The camera (10) according to claim 10, wherein The control and evaluation unit (34) is designed to check at least one function of the image sensor (26) for defective pixels based solely on a reference partial measurement.

19. The camera (10) of claim 13, wherein: The receiving element (26a) has two charge storage devices (30).

20. A method for detecting three-dimensional image data from a detection area (18), wherein: emitting emission light (16), said emission light (16) being modulated with at least one first modulation frequency, receiving received light (22) from the detection region (18) and generating a plurality of received signals therefrom in a spatially resolved manner (26a), The received signal is demodulated using the first modulation frequency to obtain a sample value, For distance measurement, a first number of partial measurements are performed at different phase offsets between the first modulation frequency for the emitted light (16) and the first modulation frequency for demodulation, and corresponding distance values are determined in a spatially resolved manner from the sample values obtained by the partial measurements, emitting a reference light (42), which is modulated with the first modulation frequency and is received again without passing through the optical path of the detection region (18), and For a functional test, a second number of reference portion measurements are performed at different phase offsets between the first modulation frequency for the reference light (42) and the first modulation frequency for demodulation, and corresponding reference distance values are determined in a spatially resolved manner from the sample values obtained by the reference portion measurements, It is characterized in that The reference portion measurement used for the functional test is distributed over a plurality of distance measurements.

Images