3D Imaging System

Abstract

A three-dimensional imaging system, comprising: a. a pixel matrix (1) including a number of pixels each having a photodetector (9) capable of detecting a single photon impinging thereon; and an optical system capable of forming an image of a field of view on said pixel matrix (1), said single photon detector having within a time window a binary logic state of true if a photon is detected and a logic state of false if no photon is detected; b. a projection device (5) capable of projecting a pattern in a time window of less than 10 μsec, preferably less than 1 μsec; c. a control device for synchronizing the time window of said projection device (5) with the time window of said imaging device; and d. logic for determining, in use, the presence of consecutive pixels (11) in a true state during said time window and for calculating a depth profile corresponding to said consecutive pixels (11).

Description

[Technical field]

[0001] The present invention relates to a three-dimensional imaging system. [Background technology]

[0002] In one state-of-the-art implementation of a depth imaging system, an imaging system images structures or lines projected onto a scene.

[0003] In industrial applications, e.g., in conveyor-based systems, a fixed line is projected onto a conveyor belt and the object to be scanned passes through a curtain of lasers. A high-speed imaging system (frame rate of approximately 1 kps) monitors the distortion of the projected line and infers the depth profile of the object onto which the line is projected.

[0004] In the case of a conveyor system, the speed of scanning along the object is determined by the speed of the conveyor, which is typically slow relative to the frame rate of the imaging system.

[0005] The same type of sensor is not suitable when a laser line is scanned across a scene, for example by a one-dimensional MEMS system that projects a moving laser line onto the scene. Assuming a refresh rate of 100 Hz and a lateral resolution of 800 laser lines, the sensor system would need to image each line in 12.5 μs, which implies a frame rate of 80 kHz, which is beyond the technical capabilities of typical systems today.

[0006] A solution to this was implemented by the first generation RealSense camera designed by Intel: instead of imaging each line individually, each imaging frame sees a collection of lines projected onto the scene. For example, in a MEMS laser scanner that sweeps a scene at 10 kHz with a laser line, the line passes through the field of view 100 times every 10 ms. The power of the laser line is modulated, or pulsed, so that the illuminator projects a set of lines that are imaged together.

[0007] The drawback is that the detection speed and delay are no longer determined by the illumination but by the sensor system, which imposes a frame rate on the system.

[0008] Furthermore, sufficient optical power is required to operate the system under strong ambient light conditions, such as in an outdoor environment.

[0009] The present invention provides a sensor that is optimized to maximize the performance of laser line scanning systems by enabling imaging of lines with exposure times as short as 10 ns while minimizing optical power requirements even under ambient light conditions. Summary of the Invention

[0010] A first aspect of the present invention relates to a method for determining a depth profile of a field of view, said method comprising: - projecting at least one light pattern onto said field of view by a projection device (5), said projection being performed in a time window of less than 10 μsec, preferably less than 1 μsec; imaging the projected pattern by means of a camera sensor (4) and an optical system synchronized with the projection of the pattern during at least one observation window within the time window, the camera sensor having a pixel matrix (1), each pixel (9) having a photodetector, the pixel (9) being in a false state when no light is detected by the corresponding photodetector and in a true state when light is detected by the corresponding photodetector, thereby obtaining a first binary matrix of pixels representing the field of view; - separating the projected pattern from ambient light noise (10) on a binary matrix of pixels (1) obtained from one observation or a combination of at least two observations within the time window by considering only pixels which have at least one neighboring pixel in a true state and which are also in a true state themselves, thereby obtaining a second binary matrix of pixels representing the projected pattern; calculating a depth profile corresponding to the projected pattern based on a triangulation between the position of the projection device (5), the position of the camera (4) and the second binary matrix of pixels; scanning the projected pattern by repeating steps a to d across the field of view to determine a depth profile across the field of view; has. Each separated element of the pattern extends over at least two consecutive pixels in the binary representation. .

[0011] Preferred embodiments of the method of the invention include one of the following features or a suitable combination of two or more of the following features: - the projected pattern comprises at least one continuous line, and the step of separating the projected pattern from the disturbance noise is performed by considering only real pixels that form at least one continuous line; - Project two or more consecutive lines simultaneously; - the projected continuous lines are straight lines and are scanned sequentially over the entire field of view or over a portion of the field of view forming a region of interest; - the line is oriented in a predetermined direction, and the step of separating the line from the disturbance noise uses the predetermined direction to analyze the probability that neighboring pixels are part of the projected line; - determining the probability that a pixel is part of the projected pattern by a trained neural network; - the displacement of said projection pattern between two successive projections corresponds to less than one pixel width, thereby improving the depth resolution by interpolating the determined depth between successive line scans; - generating said projected line by moving at least one laser beam over said field of view; each of said photodetectors comprises a single photon detector;

[0012] A second aspect of the present invention relates to a depth imaging system implementing the inventive method for determining a depth profile of a field of view by: - an imaging device having a pixel matrix (1) including a number of pixels, each having a photodetector (9) capable of detecting a single photon impinging on it, and an optical system capable of forming an image of a field of view at said pixel matrix (1), said single photon detector having, in a time window, a binary logic state of true when a photon is detected and a logic state of false when no photon is detected. - A projection device capable of projecting a pattern in a time window of less than 10 μsec, preferably less than 1 μsec. a control device for synchronizing the time window of the projection device with the time frame of the imaging device; logic that, when used, determines the presence of consecutive pixels in a true state during said time frame and calculates a depth profile corresponding to said consecutive pixels.

[0013] Preferably, the projection device is configured to project the line in a predetermined direction to improve line detection by the logic.

[0014] Advantageously, the pixel matrix has a first layer including the single photon detectors and a second layer including proximity test circuitry configured such that an output of a circuit in the second layer is true only if the corresponding single photon detector and at least one adjacent single photon detector are simultaneously in a true state, thereby determining which pixels have at least one consecutive pixel in the true logical state. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram showing an ultra-high speed line scanner according to the present invention.

[0016] [Diagram 2] 2(a)-(d) are diagrams illustrating pixelable lines under line-scan spatial dithering conditions.

[0017] [Diagram 3] FIG. 3 shows the result of interpolating the positions of the line scans in FIG. 2 along line AA.

[0018] [Figure 4] FIG. 4 illustrates the results of random spatial dithering to estimate sub-pixel resolution.

[0019] [Diagram 5] FIG. 5 shows the results of estimating sub-pixel resolution with known dot shift.

[0020] [Figure 6] FIG. 6 is a diagram illustrating the line scan interleaving operation.

[0021] [Figure 7] FIG. 7 shows the projection of a different pattern rather than a simple line. [Figure 8] FIG. 8 shows the projection of a different pattern rather than a simple line.

[0022] [Figure 9] FIG. 9 is a diagram showing a two-layer image sensor. [Explanation of symbols]

[0023] 1 Image sensor (pixel matrix) 2 Laser Line 3. Deformed laser line 4 Camera (including sensor and optical system) 5 Laser Projector 6 Laser beam (Limit of line scanning) 7 Scanning Object 8 Projection Lines 9 Individual Pixels 10 Pixels that detect noise caused by ambient light 300 Time Windows for Spatial Dithering 301 Dithering spatial range in the X direction (projection device) 303 Pixelation in X 312 Actual line position 314 Hit distribution (sum of instantaneous measured positions during time scan) 315 Average position of measurement locations 322 Spatial distribution of moving dots 324 Hit Distribution Average position of 325 measurement positions 400 field of view 401 Dot Pattern 402 Dot Pattern 403 Cross Pattern 404 Circular Pattern 411 Pattern Projection Device 412 Light source 413 Imaging surface of diffraction grating etc. 414 Mirror DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] The invention described below proposes a system capable of imaging a three-dimensional scene at high speed, for which purpose it projects a light pattern, such as a line, at high frequency onto the scene and is able to reconstruct the entire scene by displacing said pattern between each shot.

[0025] As an example, imagine a single-axis vibrating MEMS projects a laser line and sweeps this laser line across the field of view (FoV) at 1 kHz to 10 kHz, this means that the complete scene or FoV is sensed in a period of 100 μs to 1 ms.

[0026] For example, if there are 1000 line pulses per vibration period, each line (T line) needs to be imaged by the imaging system within 100 ns to 1 μs per line.

[0027] A sensor system was used to provide a sensor or imaging system that can image a line in a scene T-line by T-line in the presence of ambient light.

[0028] The present invention relates to a depth imaging system having a sensor, the sensor comprising an array of pixels, each pixel being capable of detecting a single photon, for example by means of a Single Photon Avalanche Diode (SPAD).

[0029] The sensor is preferably synchronized with the pulsed laser line (or pattern) so that the sensor knows when to expect a photon incident from each projected laser line. Pixels are activated for a short observation window, preferably from 1 ns to a few hundred ns. During this observation window, the pixels of the single photon detector are triggered or change state when a photon is detected. The detected photons may be due to the energy of the laser line, but also due to ambient light present in the scene.

[0030] To localize the laser lines, a coincidence imaging method is proposed, which considers a pixel trigger as a valid trigger only if one or more neighboring pixels are also triggered during the observation window. Since the disturbance light statistically triggers all pixels, the probability that the disturbance light causes neighboring pixels to fire during the short observation window can be minimized, while the pulse light energy of each laser line can be designed to increase the probability that neighboring pixels connected along the line will fire together. In this way, the laser lines can be easily separated from the noisy binary image.

[0031] In a first embodiment shown in FIG. 2, a region of interest (ROI) is read out column-by-column or row-by-row and a pixel map is constructed to identify the position and shape of the line. Since the line is known to be continuous and piecewise continuous, this can be used as a constraint to identify which pixels are part of the line. For example, isolated pixel events are likely not part of a piecewise continuous structure. Furthermore, another constraint is that if we consider a vertical line as the projected structure, in the detected ROI map, two separate events in the same row must be adjacent (in this case the line is detected at the border of adjacent pixels or partially covers adjacent pixels, in which case some super-resolution can be applied) or if the detected events are not adjacent, only one of the pixels has an event connected to the line and the other pixel has a random event. By looking at the events in the next or previous row, we can get additional information about which events were real events. Because we are projecting a vertical line, we can expect the real events to be connected in a piecewise linear manner in the vertical direction.

[0032] A similar explanation is possible for the other direction, which projects horizontal lines.

[0033] Constraining the projected lines to a vertical or horizontal shape makes it easier to determine the region of interest that will be read out, reducing the number of pixels that are read out and analyzed for each line shot. The ROI is determined by the maximum disparity that you want to allow in your system.

[0034] Of course, any line orientation is applicable if there is no such restriction, for example this is applicable to massively parallel processing in the case of stacked systems where the pixel planes are connected in parallel to the processing layers, rather than by a bus as is typical in image sensors.

[0035] A second aspect of the invention relates to a stacked device. A first layer comprises photon detectors such as a SPAD array and some optional electronic circuitry (possibly quenching resistors or circuits) for controlling the SPAD array and the pixels. A second layer is connected in parallel to the first layer by one or more electrical connections per detector or detector group. The second layer processes the information coming from the first layer. In a first variant, coincidence imaging and line detection are performed by the logic of the second layer.

[0036] In another variation, a massively parallel connection connects the SPAD array to a neural network that calculates the location of line segments in the raw data image, the neural network being trained to distinguish between noise and line information.

[0037] The diagram shows a sequence of lines where each oscillation period of the line scanner results in a horizontal resolution of 10 lines. From T0 to T4, the laser line is moving from left to right, which can be called a right sweep line, whereas from T5 to T9, the laser line is moving from right to left, which can be called a left sweep line. In this diagram, the interleaved method produces a horizontal resolution of 10 for each period of the line scanner.

[0038] In another embodiment, the horizontal resolution of the depth image can be reduced by matching the positions of the lines in the left sweep with the positions of the lines in the right sweep, but the temporal resolution of these line samples can be improved.

[0039] With 500 right sweep lines and 500 interleaved left sweep lines, or vice versa, the total line resolution per period is 1000.

[0040] This means that 1000 lines need to be constructed every 100 μs, and therefore the ROI needs to be read out approximately every 100 ns. In combination with a SPAD pixel array or a bit-depth limited quantum image sensor, this requirement is feasible.

[0041] Alternatively, the entire pixel array or a region of interest of the array may be connected in parallel to the processing plane, for example in a stacked sensor device, where an array of pixels or sensors sensitive to optical input may be arranged on a first plane, with each pixel connected by an interconnection method to a processing layer physically arranged on a second plane of the sensor.

[0042] In this case, a processing element or neural network can be described that is connected to the active ROI or the entire pixel array and is capable of finding lines, or fragmentary lines, in a noisy image that contains both random event or pixel data and events or pixel data connected to the line of interest.

[0043] Coincidence can be applied to the ROI after reading out the individual pixels, or to the pixel region itself.

[0044] The line moves every time step with sub-pixel resolution, as shown in Figure 2. Sub-resolution allows to obtain a better estimate at a given timestamp by analyzing the triggered pixel positions of previously detected lines. This of course imposes constraints on the spatial frequency of the object to be detected.

[0045] However, by taking into account the temporal behavior of the line motion, a method for super-resolution is obtained.

[0046] [Super-resolution with special (structural) dithering to improve depth accuracy] In traditional dot pattern active stereo vision systems, the dots are stationary in space. If the scene is not moving, the dots have a fixed position on the scene and therefore in the image. Spatial quantization by sampling the scene with an imaging device with a certain spatial resolution imposes limitations on the depth estimation. The paper "Noise in Structured-Light Stereo Depth Cameras: Modeling and its Applications" by A. Chatterjee et al. (https: / / arxiv.Org / pdf / 1505.01936.pdf) describes noise in stereo systems.

number

[0047] Delta depth is linear with respect to delta disparity, which means that quantization of disparity limits the accuracy of depth estimation.

[0048] In this invention, the limitation of disparity quantization due to pixel pitch is relaxed by spatial dithering. Depth estimation can be improved by constructing it from a set of nearby related measurements at slightly different positions in the scene. The idea is similar to that of quantization and dithering with a 1-bit AD converter. If the dithering noise is white and exceeds the quantization, a better estimate can be obtained by averaging the final result. The concept of spatial dithering is similar, with the pixel being the quantizer and the moving dot being the spatial dither. If the spatial dither exceeds the pixel quantization in amplitude and the distribution of the dither is known (because the system controls the light), more accurate sub-resolution measurements can be obtained.

[0049] The invention is further based on the possibility of almost continuous, extremely fast sensing, whether of lines, dots or dot patterns.

[0050] The idea is to impose a known spatial probability on the exposure.

[0051] In the case of a single point, dot, or dot pattern, multiple points are projected in parallel. For simplicity, we focus on one point, but the method can be extended to any projected structure.

[0052] If the position of the dots (or structures) can be tracked very quickly, super-resolution can be achieved by spatial dithering.

[0053] We impose a known spatial probability distribution on the dot locations, for example in the horizontal direction. This can be a uniform probability distribution, a Gaussian distribution, or some other function, but a smooth distribution is preferred.

[0054] FIG. 4 illustrates the positional dithering method.

[0055] In Figure 4(a), the dot position is varied, or dithered, over a particular range of positions 301 during a time window 300. The probability distribution of the dot's position x during this time window 300 can be accumulated to determine a probability density function 312.

[0056] The pixel array samples and quantizes this function at discrete time instants using finite size pixels 303. This quantization produces a histogram 314. By monitoring the dithered x-positions during the period 300, an average position 315 can be calculated from the histogram 314, which is more accurate than if dots with stationary positions were observed by the quantized pixels. In this way, the accuracy of the average dot position is obtained with a higher resolution.

[0057] This principle can be extended to very simple spatial dithering or distribution, for example as shown in FIG.

[0058] A third aspect of the invention relates to a method for determining a depth profile of a field of view similar to the method according to the first aspect. However, step c (separating) is optional, since the method for determining a depth profile can also be performed with other filtering steps, as will be appreciated by those skilled in the art. A fourth aspect of the invention relates to a method for determining a depth profile of a field of view similar to the method according to the first aspect. However, step b includes imaging with two cameras, each having a pixel matrix, and step d includes triangulation based on the positions of the two cameras and the two pixel matrices. Those skilled in the art will appreciate that triangulation of data from two cameras or one camera and one light source is conceptually similar. A fifth aspect of the invention relates to a depth imaging system, preferably implementing the methods of the third and fourth aspects.

[0059] Preferably, the projection device projects simple structures, such as elliptical or circular dots, onto the scene and scans the scene in a raster, Lissajous or other pattern.

[0060] Preferably, each camera is adapted to track the position of the dot in each of the image planes and output a stream of (x,y,t) data, where x,y are the coordinates in the image plane (pixel size) and t is the timestamp of the detection. This has the advantage that such a stream can be output with a time resolution of up to 10ns. For example, Fig. 10 shows how a laser-based illumination is continuously sweeping the world in fast strokes. The Lissajous pattern is generated based on fast 2D MEMS mirrors. Two or more sensors snapshot the position of the laser dot at very high speed (up to 100MHz). Each sensor transmits the position of the laser dot, after which a simple triangulation algorithm can calculate the exact 3D shape, position, contour and movement at the aforementioned ultra-high speed.

[0061] The method and system of the present invention are advantageous for several reasons, summarized below. The first advantage is related to the power. Here, a single dot (or a set of dots) is projected at once, pooling all the energy to sample a specific location in the scene in the shortest possible time, resulting in the best immunity to ambient light with the smallest photon budget. However, the photon budget required to detect a dot depends on the detection technology used. For example, it is possible to detect a dot with as few as 12 photons (6 photons / pixel) distributed over two pixels. Furthermore, the shorter the time required to generate a return of 6 photons per pixel, the better the immunity to ambient light. Therefore, the system preferably uses short, high-power pulses.

[0062] Additionally, because power can be changed rapidly during a scan, it is also possible to create frames that dynamically allocate laser power - for example, distant or dark objects can be illuminated with greater laser intensity, while closer objects can be illuminated with lower laser power - this can be modulated within a single scan and optimized to maintain class-1 eye safe limits while optimizing system range, power consumption and scan density.

[0063] In another embodiment, in the near field region, the system can increase the pulse density and decrease the pulse intensity to obtain a finer mesh. In the far field region, the system can decrease the pulse density and increase the pulse intensity to obtain a usable signal. The pulse density-intensity product can be maintained within Class 1 eye-safe constraints as needed, or can be further reduced if power consumption concerns outweigh the need for scanning density. No changes to the scan pattern are required to achieve such scanning at different densities.

[0064] The second advantage of the present invention is related to bandwidth. Scanning depth sensing systems offer a unique advantage in how depth data is represented and transmitted. Depth data can be represented as a series of x, y, z points, as would be expected from any point cloud representation. However, the points in the point cloud are created over time along a fixed laser beam trajectory. Therefore, the points along the trajectory are typically "connected" in a particular way, influenced by the scene itself. The fact that the points or samples are connected along the trajectory can be exploited in the data representation. Instead of outputting each sample individually, the trajectory can be described with a limited set of coefficients and two points that describe the start and stop of a polynomial descriptor. This allows for "live" compression, significantly reducing the bandwidth requirements from the sensor to the system, as well as reducing memory and computation requirements for further processing.

[0065] A third advantage of the present invention is related to robustness. A system based on laser beam scanning using the sensor of the present invention provides best-in-class robustness against concurrently operating systems. With a laser beam scanning approach, depth information is obtained by triangulating pairs of single dots in the image space of each tracking dot. Thus, as additional systems are introduced into the environment, additional dots will be scanned through the scene. If the sensor used to track the dots is capable of tracking multiple dots, the additional optical information created by the additional scanning dots helps to obtain depth data as another pair of dots that can be triangulated.

[0066] The laser beam scanning system is also resistant to flood systems (active stereo, mono NIR flood, iToF). The system is able to suppress the effects of ambient light by imposing requirements on the photon statistics and behavior of the projected dot. Flood illumination is similar to ambient light in terms of statistics and behavior. That is, it produces high peak luminous flux (energy / pixel / time) and typically requires hundreds of microseconds to milliseconds to obtain a pixel-integrated signal, but has no spatial and temporal structure. For example, the system uses a combination of photon statistics, beam shape, and spatial and temporal motion of the beam to distinguish between signals from ambient / flood sources and optical signals from the laser beam scanning system itself.

[0067] Laser beam scanning systems are also resistant to dot pattern projection devices. Structured light systems using dot patterns are distinguished from dynamic laser beam scanning structured light in two ways. First, the dot pattern is static and has no moving behavior or trajectory. Second, the available optical power is distributed in parallel among the 10k or so dots, so the peak power allocated per dot is much smaller than that allocated by laser beam scanning systems. Third, structured light projection devices are designed to work with global shutter imagers with typical exposure times on the order of hundreds of μs. Thus, the energy required to see a dot is distributed within the exposure window.

[0068] The fourth advantage of the present invention is related to latency. The system employs a fully serialized sensing scheme, where data is generated continuously as the beam scans the field of view. The time required between optical sampling of the scene (projection of a dot on a particular surface) and generation of depth data corresponding to that sample is very short, since the scheme relies purely on: first, sensing of dot positions in different image spaces is on the order of 10 ns using this system of the present invention; second, the transfer of coordinates to the sensing host is N clock cycles, typically several hundred ns; and third, the filtering method using past data and the triangulation of points in space allows pipelined mathematical operations of several hundred ns.

[0069] This results in extremely low latency from optical sample to depth measurement. Of course, to obtain a complete scene sample, the laser beam still needs to scan the desired field of view at the desired density. Depending on the scanning speed of the laser beam scanning and the density required for the application, a specific data aggregation window can be considered. In contrast to frame-based sensors, there is no need to consider a frame buffer that would increase the overall sensing latency.

[0070] An interesting feature of laser beam scanning sensing technology is that it allows the selection of optimal density vs. refresh rate tradeoffs customized to the application. This is due to the sequential nature of the system where the sensed data is generated sequentially. Depending on the data aggregation window applied to the data stream, the scan dots create a sparse or dense pattern of scan lines during the window.

[0071] Algorithms requiring a fast update rate (500Hz) using sparse data can be run in parallel with algorithms requiring a normal update rate (50Hz) using dense data, using the same data stream as input. Figures 11-14 show the aggregated scan lines applying aggregation windows of 2ms, 5ms, 10ms, and 20ms to the same data stream. The LBS frequency pairs of the 2D vibrating MEMS structure used to obtain the simulated scan lines are (8174Hz, 5695Hz).

[0072] The use of SPADs in pixels is advantageous because the detectors are single photon sensitive, which has the advantage that minimal energy is required for the active projection structure, as the detectors are single photon sensitive. Another advantage is the sub-ns response time, which means that photons can be detected and encoded into a digital signal in nanoseconds.

[0073] The system can filter out false positives caused by thermal noise or ambient light impinging on the sensor area. Figure 15 (a) shows the ground truth, (b) the raw detection, and (c) the filtered detection. The first filtering step is done inside the pixel plane, which means that the filter operates in parallel for each pixel. Typically, the filter represents either spatial filtering methods, temporal filtering methods, or a combination of both. These filters can be implemented using transistor level, RTL, more complex computational architectures, or neural networks, etc.

[0074] The system relies on two basic principles of filtering in the pixel plane. The first principle is spatial: by imposing a particular shape on the projected dot, we can constrain the pixel to flag a valid detection only if the imposed kernel is detected within a specified time. For example, we can impose a 2×1 kernel, which means that the projected dot must span at least 2×1 pixels. If we shoot a light pulse of length tp, we impose a constraint on the pixel that a cluster of 2×1 pixels must fire within tp in order for a detected photon to be considered as coming from an active projection. This principle is called synchrony.

[0075] The second principle is spatio-temporal. If the repetition frequency of the pulses is higher than the displacement speed of the dot, the dot trajectory can be assumed to be continuous. For example, if a 1 ns pulse is repeated every 40 ns, a time window of 41 ns can capture two pulses. By imposing that a pixel (or pixel cluster) must observe at least two pulses in a 41 ns window, a significant portion of the detected disturbance photons can be removed. This is because the photon statistics of the disturbance light are unlikely to produce two consecutive detections within a 42 ns window. This principle is called persistence. Combining both simultaneity and persistence builds a powerful filtering strategy that is very efficient in terms of hardware implementation.

[0076] After filtering projected within pixels, the fastest way to output the array data is to compress it, for example by projecting it onto the axis of the imager. To support multi-dot scanning and reduce aliasing after projection, the imager is first divided into tiles, for example 16x16 tiles of 64x64 pixels each. Each tile can output its projection to digital logic, typically located on the periphery of the device. The projection operation creates two vectors (created for NxN tiles), one each for row and column projections.

[0077] It is important to note that creating vectors based on projection data is a step that moves data from asynchronous regions inside the pixel array to synchronous regions on the periphery. Intra-pixel filtering can also be implemented synchronously, but this is not required. Thus, vectorizing the projections discretizes the data in time. For example, every 10 ns, a new vector is created based on the projection events that were input during that 10 ns.

[0078] When these vectors are plotted over time, as shown in Figure 16, a new image is formed that begins to show the trajectories of the projected dots or structures (d,e). This "image" may be processed and filtered again to create a filtered projection-time data set (d',e') for each projection. These directly produce x,y data (x,y,t) for each time step.

[0079] Further embodiments and related disclosures are described in the parent applications International Patent Application No. PCT / IB2021 / 054688 "Pixel sensor system", International Patent Application No. PCT / EP2021 / 087594 "Pixel sensor switched by neighbouring gates", International Patent Application No. PCT / IB2022 / 000323 "Pixel array with dynamic lateral and temporal resolution", and International Patent Application No. PCT / IB2022 / 058609 "Persistent filtering in SPD arrays", the disclosures of which are incorporated by reference.

[0080] As mentioned above, the displacement of the projection pattern between two successive projections corresponds to less than one pixel width, and the depth resolution is improved by interpolation of the determined depth between successive line scans, however, this should not be understood as being limited to line scans, as said interpolation can work equally well when using dot scans.

Claims

1. In a method for determining the depth profile of a field of view (7), a. projecting at least one light pattern onto the field of view by a projection device (5), the projection being performed within a time window of less than 10 μsec, preferably less than 1 μsec; b. synchronizing with the projection of the pattern in at least one observation window within the time window, and imaging the projected pattern by a camera sensor (4) and an optical system, the camera sensor having a pixel matrix (1), each pixel (9) having a photodetector, the pixel (9) being in a false state when no light is detected by the corresponding photodetector and in a true state when light is detected by the corresponding photodetector, thereby obtaining a binary matrix of first pixels representing the field of view; c. processing the binary matrix of the first pixels by a logic circuit and applying a filter for distinguishing the projected pattern from ambient light noise to create a filtered data set representing the projected pattern; d. calculating a depth profile corresponding to the projected pattern based on triangulation between the position of the projection device (5), the position of the camera (4), and the filtered data set; characterized by comprising a method for determining the depth profile of a field of view.

2. In the method for determining the depth profile of a field of view according to Claim 1, the projected pattern has at least one continuous line, and the step of separating the projected pattern from the ambient light noise is performed by considering only the true pixels forming at least one continuous line. A method for determining the depth profile of a field of view.

3. In the method for determining the depth profile of a field of view according to Claim 2, two or more continuous lines are projected simultaneously. A method for determining the depth profile of a field of view.

4. In the method for determining the depth profile of a field of view according to Claim 2, the projected continuous line is a straight line (8) and is sequentially scanned across the entire field of view or across a portion of the field of view forming a region of interest. A method for determining the depth profile of a field of view.

5. In the method for determining the depth profile of a field of view according to Claim 4, the straight line is directed in a predetermined direction. The step of separating a straight line from the external light noise uses the predetermined direction to analyze the probability that a neighboring pixel is part of the straight line (8) on which it is projected. Method for determining the depth profile of a field of view. According to claim 6, in the method for determining the depth profile of a field of view according to claim 5, When applying the filter, the probability that a pixel (9) is part of the projected pattern is determined by a trained neural network. Method for determining the depth profile of a field of view. According to claim 7 In the method for determining the depth profile of a field of view according to claim 1, The displacement of the projected pattern between two consecutive projections corresponds to less than one pixel width. By interpolating the determined depth between consecutive line scans, the depth resolution is improved. Method for determining the depth profile of a field of view. According to claim 8 In the method for determining the depth profile of a field of view according to claim 2, The projected line is generated by moving at least one laser beam over the field of view. Method for determining the depth profile of a field of view. According to claim 9 In the method for determining the depth profile of a field of view according to claim 1, Each of the photodetectors has a single photon detector (9). Method for determining the depth profile of a field of view. According to claim 10, in the method for determining the depth profile of a field of view according to claim 1, e. Further includes a step of scanning the projected pattern by repeating steps a to d over the entire field of view to determine the depth profile of the entire field of view. In step c. On the pixel matrix (1) obtained from one observation within the time window or a combination of at least two observations, only pixels where at least one neighboring pixel and itself are in the true state are considered. By doing so, the projected pattern is separated from the external light noise on the binary matrix of the first pixel, and thus a binary matrix of the second pixel representing the projected pattern is obtained as the filtered dataset as the binary matrix of the second pixel. Each separated element of the pattern extends over at least two consecutive pixels in the binary representation. Method for determining the depth profile of a field of view. According to claim 11 In a depth imaging system that implements the method for determining the depth profile of a field of view according to any one of claims 1 to 10, a. A pixel matrix (1) including a plurality of pixels each having a single photon detector (9) capable of detecting colliding single photons, and an optical system capable of forming an image of a field of view in the pixel matrix (1), wherein the single photon detector has a true binary logic state when a photon is detected and a false logic state when a photon is not detected within a one-hour frame, and an imaging device (4); b. A projection device (5) capable of projecting a pattern in a time window of less than 10 μsec, preferably less than 1 μsec; c. A control device for synchronizing the time window of the projection device (5) and the time frame of the imaging device; d. Logic that, during use, determines the presence of consecutive pixels in a true state within the time frame and calculates a depth profile corresponding to the consecutive pixels; A depth imaging system having the above.

12. In the depth imaging system according to Claim 11, the projection device (5) is configured to project the line (8) in a predetermined direction in order to improve line detection by the logic. A depth imaging system.

13. In the depth imaging system according to Claim 11, the pixel matrix (1) has a first layer including the single photon detector and a second layer including a neighborhood test circuit, the neighborhood test circuit is configured such that the output of the circuit of the second layer becomes true only when the corresponding single photon detector and at least one adjacent single photon detector are simultaneously in a true state, thereby determining which pixels have at least one consecutive pixel in the true logic state. A depth imaging system.

Images