An imaging system for capturing scenes containing one or more objects with transparent surfaces.

Abstract

An imaging system for imaging a scene containing one or more objects comprises: a light source configured to illuminate the scene by projecting light rays through an array of points in space, and configured to illuminate the scene with different illumination patterns by projecting light rays through different groups of points in the array for each time step in a series of time steps; a detector including an array of pixel elements, configured to acquire an image at each time step by detecting light projected onto the array of pixel elements toward the scene and reflected toward the detector from one or more surfaces of the objects; and an image processor configured to process the acquired image to determine, for each pixel element, the light signal incident on the pixel element over the course of a series of time steps, and to determine, based on knowledge of the light signal incident on the pixel element at each time step and the illumination pattern projected at each time step, that the object contains a first surface that is at least partially transparent, and that at least a portion of the light incident on the first surface is reflected toward the detector and another portion is reflected from a second surface at a greater depth in the scene.

Description

[Technical Field]

[0001] The embodiments described herein relate to an imaging system for imaging a scene that includes one or more objects. [Background technology]

[0002] Three-dimensional surface imaging (3D surface imaging) is a rapidly growing field of technology. As used herein, the term "3D surface imaging" can be understood as the process of generating a 3D representation of an object's surface by capturing all spatial information in three dimensions—in other words, by incorporating depth information in addition to the two-dimensional spatial information present in conventional images or photographs. This 3D representation can be visually displayed, for example, as a "3D image" on a screen.

[0003] Several different techniques may be used to acquire the data necessary to generate a 3D image of the surface of an object. These techniques include, but are not limited to, structured light illumination, time-of-flight imaging, holographic techniques, stereo systems (both active and passive), and laser line triangulation. In each case, the data may be acquired in the form of a "point cloud," in which intensity values ​​are recorded for different points in three-dimensional space, and each point in the point cloud has its own set of {x,y,z} coordinates and associated intensity values ​​I.

[0004] One challenge in this field is acquiring 3D images of objects with transparent surfaces. This is partly because transparent surfaces reflect only a fraction of the light projected onto them, resulting in a reduced signal-to-noise ratio in attempts to reconstruct the object from the captured image. The problem is further complicated by the need to accurately distinguish between signals reflected from the upper surface of the object and signals located deeper inside the object, which can also reflect light backward through its upper surface. These signals can vary in magnitude depending on the properties of the surface, and the signal from any one surface is determined by its specific levels of reflectivity, transmittance, and absorptivity. For example, when imaging a translucent soap bottle against a black, highly absorptive background, the camera will see a different mixture of signals than when imaging a clear wine glass placed on a white, highly reflective surface.

[0005] The ability to accurately identify transparent surfaces when constructing 3D images of objects would be advantageous in numerous applications. One particular application lies in the field of automated warehouses, where robotic agents are used to identify and collect goods from shelves or other storage locations. These agents may use cameras to map the contours of objects they intend to lift and transport within the warehouse. Such cameras, however, may fail to correctly identify the (transparent) plastic packaging that such goods often come in, meaning the agent may be unable to retrieve the item or may damage it during collection. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Liu, Y. et al., "3D Imaging, Analysis and Applications", Springer Cham, September 11 / 12, 2020, ISBN: 978-3-030-44070-1 [Non-Patent Document 2] Lawson, CL, RJ Hanson, "Solving Least-Squares Problems", Upper Saddle River, NJ: Prentice Hall, 1974, Chapter 23, p. 161. [Non-Patent Document 3] Mallat SG, Zhang, Z., "Matching pursuits with time-frequency dictionaries", IEEE Transactions on Signal Processing, Vol. 41(12), 1993, pp. 3397-3415. [Non-Patent Document 4] Machidon, AL, Pejovic, V., "Deep learning for compressive sensing: a ubiquitous systems perspective", Artificial Intelligence Review, vol. 56, no. 4, April 2023, pp. 3619-3658. [Overview of the project] [Problems that the invention aims to solve]

[0007] Therefore, it is desirable to provide enhanced means for identifying the presence of transparent surfaces when constructing a 3D image of an object. [Means for solving the problem]

[0008] According to a first aspect of the present invention, an imaging system is provided for imaging a scene including one or more objects, and this system is A light source positioned to illuminate a scene by projecting light rays through an array of points in space, wherein, for each time step in a series of time steps, the light source is configured to illuminate the scene in different lighting patterns by projecting light rays through different groups of points in the array, A detector comprising an array of pixel elements, configured to capture an image at each time step by detecting light projected toward the scene and reflected toward the detector from one or more surfaces of an object onto the array of pixel elements, It is an image processor, The captured image is processed to determine the light signal incident on each pixel element during a series of time steps, Based on knowledge of the light signals incident on the pixel elements at each time step and the illumination patterns projected at each time step, it is determined that the object includes a first surface that is at least partially transparent, and that at least a portion of the light incident on the first surface is reflected toward the detector, and another portion is reflected from a second surface at a greater depth in the scene. An image processor and It is equipped with.

[0009] The image processor may be configured to determine, based on the light signal incident on the pixel element at each time step and the illumination pattern at each time step, that the light signal includes a first portion of light reflected from a first surface and a second portion of light reflected from a second surface.

[0010] The first and second portions of the light may travel along the same path between the object and the detector.

[0011] The image processor, based on the light signal incident on the pixel element at each time step and the illumination pattern at each time step, A first location in the array of illumination points, where light reflected by a first surface is projected through that location onto a pixel element. It may be further configured to determine

[0012] The image processor, based on the light signal incident on the pixel element at each time step and the illumination pattern projected at each time step, A second location within the array of illumination points, through which light that has passed through the first surface and is reflected from the second surface toward the pixel element is projected. It may be further configured to determine.

[0013] The image processor may be configured to determine the distance between the detector and the first surface and / or the second surface based on the first location or the second location.

[0014] The illumination pattern may include a series of line patterns, one or more of which are projected onto the scene. Each line pattern may include a plurality of parallel lines. Each line may be formed by a light ray passing through a respective row or column of the array of illumination points. The line patterns may be selected such that each line is projected onto the scene once over a series of time steps.

[0015] The image processor may be configured to determine that the scene includes a first surface that is at least partially transparent by identifying the presence of two or more peaks in the optical signal incident on the pixel element during the process of projecting the line pattern onto the scene.

[0016] The image processor may be configured to identify the region of the array of illumination points through which the light reflected by the first surface and incident on the pixel element has passed, based on the peaks in the optical signal incident on the pixel element during the process of illuminating the scene with a plurality of line patterns.

[0017] The illumination pattern may further include a series of coded patterns from which the global location within the array of illumination points through which the light ray incident on each pixel of the detector passes can be determined.

[0018] The coded pattern may include a Gray code pattern.

[0019] The image processor may be configured to identify a region of an array of illumination points through which light reflected by a first surface and incident on a pixel element has passed, based on (i) peaks in the light signal incident on the pixel element during the process of illuminating the scene with multiple line patterns, and / or (ii) signals observed at the detector's pixels during the process of illuminating the scene with an encoded pattern.

[0020] The system may be configured to generate illumination patterns by defining illumination sequences for each region of a point in an array of illumination points, the illumination sequences specifying variations in the amplitude of light projected through each region over a series of time steps. The illumination sequences may differ for each region. Each region of a point may contain each line of points in an array of illumination points.

[0021] For each pixel element, the image processor may be configured to model the light signal incident on the pixel element over a series of time steps as a function of contributions from regions within the array of illumination points. The image processor may be configured to determine the number N of regions whose illumination sequences, when combined, become the light signal seen at the detector. If N ≥ 2, the image processor may be configured to determine that the object includes a first surface that is at least partially transparent and a second surface.

[0022] Each lighting sequence may be encoded using one or more bits, which define the relative amplitude of light projected from a region of points toward the scene at each time step.

[0023] One or more bits may specify the frequency modulation applied to light projected from a region of points over a series of time steps.

[0024] Each lighting sequence may be encoded as a sequence of bits, where each bit is associated with a time step, and the value of each bit defines the relative amplitude of light projected from a point region toward the scene at each time step.

[0025] For each region, the illumination sequence may define one or more time steps in which light is projected from the region with a first amplitude, and one or more time steps in which the amplitude of the light projected from the region is reduced to or zero compared to the first amplitude.

[0026] The illumination sequence may be defined such that for each region of the point region, the number of time steps in the sequence in which light is projected at a first amplitude is the same.

[0027] Each region of a point may include each line of points in the array of illumination points. The number of bit changes between illumination sequences for each consecutive pair of lines may be the same.

[0028] A region may include a column or row of points in an array of illumination points. The bit sequence for each region may be defined such that, for any one region, the correlation between the bit sequence for that region and the bit sequences for other regions within its parallax window is reduced compared to the correlation between the bit sequence for that region and the bit sequences for other regions outside the parallax window. The parallax window may define the maximum number of consecutive regions in an array of illumination points, and the light from those regions is detectable by a single pixel of the detector according to the geometric arrangement of the system.

[0029] The light source may include a projector having an array of projector elements, and each lighting pattern is generated by activating one or more of the projector elements.

[0030] According to a second aspect of the present invention, a robotic device is provided which is configured to manipulate a physical object, and which is configured to identify an object having a transparent surface by using a system according to a first aspect of the present invention.

[0031] According to a third aspect of the present invention, a warehouse is provided that includes one or more robotic devices according to a second aspect of the present invention.

[0032] According to a fourth aspect of the present invention, a computer-readable medium is provided which, when executed by a computer, includes computer-executable instructions that cause the computer to operate the system according to the first aspect of the present invention.

[0033] According to a fifth aspect of the present invention, a computer-readable medium is provided which, when executed by a computer, includes computer-executable instructions for operating a robotic device according to a second aspect of the present invention.

[0034] Next, an embodiment of the present invention will be described as an example, with reference to the attached drawings. [Brief explanation of the drawing]

[0035] [Figure 1] This figure shows an example of an imaging system according to one embodiment. [Figure 2A] This figure shows an example of a light source for use in a system according to one embodiment. [Figure 2B] This figure shows an example of a light source for use in a system according to one embodiment. [Figure 3] This figure shows an example of how light rays emitted by a light source at a first time point are reflected from the object toward the detector, and an example of how light rays emitted by the light source in Figure 3A are reflected from the object toward the detector at a second time point, in an imaging system according to one embodiment. [Figure 4] This figure shows an example of how a temporal sequence of a lighting pattern may be configured in one embodiment. [Figure 5A] This figure shows an example scene that includes two objects being imaged. [Figure 5B] This figure shows the scene in Figure 5A where a specific row of camera pixels is highlighted. [Figure 6] This figure shows an image captured by a camera when different line patterns are projected onto an object in the field of view and reflected from the object, according to one embodiment. [Figure 7] This figure shows a table of intensity values ​​measured by camera pixels at each of a series of time steps according to one embodiment. [Figure 8] This figure shows the intensity values ​​from Figure 7 plotted on a graph. [Figure 9] Figure 8 shows an example of a Gray code pattern used when collecting the data shown in Figure 8. [Figure 10] This figure shows a plot illustrating columns within an array of illumination points, where, based on the captured image when a series of line patterns and Gray code patterns are projected onto a scene, it is determined that light incident on different camera pixels is then emitted from them. [Figure 11] This figure shows the results of combining data from the line pattern and Gray code pattern in Figure 10. [Figure 12] This figure shows a recreation of the scene shown in Figure 5A. [Figure 13] This figure shows a 3D reconstruction of an object in the scene of Figure 5A, obtained using an imaging system according to one embodiment. [Figure 14] This figure shows an example of how multiple lighting sequences can be defined by using a code matrix, according to one embodiment. [Figure 15] This figure shows exemplary images captured by a camera when using these different lighting patterns to illuminate the same field of view as shown in Figure 6. [Figure 16A]This figure shows an example of how different regions within an array of illumination points and associated illumination sequences may contribute to the signal detected by the camera pixels, according to one embodiment. [Figure 16B] This figure shows an example of how different regions within an array of illumination points and associated illumination sequences may contribute to the signal detected by the camera pixels, according to one embodiment. [Figure 17] This figure shows a series of images captured by a camera when different lighting patterns are used to illuminate the same field of view. [Figure 18] This figure shows the intensity value of a single pixel in the image sequence shown in Figure 17. [Figure 19] This figure shows a plot illustrating the rows within an array of illumination points, which are determined to be emitted from light incident on different camera pixels when the scene is illuminated using the illumination sequence shown in Figure 14. [Figure 20] Figure 14 shows a further plot illustrating the columns in the array of illumination points, which are determined to be emitted from light incident on different camera pixels when the scene is illuminated using the illumination sequence shown in Figure 14. [Figure 21] This figure shows the cross-correlation matrix of codes in the matrix in Figure 14. [Figure 22] This figure shows an enlarged portion of the cross-correlation matrix in Figure 21. [Figure 23] This figure shows another example of how multiple lighting sequences can be defined by using a code matrix, according to one embodiment. [Figure 24] This figure shows the cross-correlation matrix of the codes in the matrix in Figure 23. [Figure 25] This figure shows a plot illustrating the rows in an array of illumination points, which are determined to be emitted from light incident on different camera pixels when the scene is illuminated using the lighting sequence shown in Figure 23. [Figure 26] This figure shows an example of how an imaging system according to one embodiment is employed by a robotic device in a factory or warehouse. [Modes for carrying out the invention]

[0036] Figure 1 shows an example of an imaging system 100 according to one embodiment. The system includes a light source 101 and one or more detectors 103, the detectors 103 being used to capture images of a scene including one or more objects 105, 107 placed on a surface 109. The detectors are mounted at an angle to the light source. An image processor 104 is used to process the images captured by the detectors.

[0037] The light source 101 may be one of several different types of light sources that can project light rays through an array of illumination points in space and can generate different illumination patterns by projecting light through different groups of points in the array at different times. For example, the light source 101 may include a spatial light modulator, an array of LEDs, or a moving laser line or spot.

[0038] Figures 2A and 2B illustrate two examples of such light sources. In Figure 2A, the light source 201 is a projector containing multiple individually addressable pixel elements 203. The layout of the pixel elements in the projector defines an array of illumination points. Different groups of pixel elements 203 may be activated at different times to generate an illumination pattern. Figure 2B illustrates an alternative type of light source, including a point source such as a laser 205 configured to project light onto a mirror 207. The mirror 207 can be rapidly rotated, so that the laser beam sweeps a plane 209 when directed towards the scene. Here, as the mirror rotates, different rays from the laser form an array of illumination points at points in the plane 209 through which each ray passes with a specific direction. The laser may be time-synchronized with the mirror and modulated to generate a desired pattern. It will be understood that such an array of illumination points may be formed in other ways depending on the light source used.

[0039] In this embodiment, the light source 101 includes a projector having an array of individually addressable pixel elements, the pixel elements being actuated to form different patterns of illumination. Individual areas of the projector elements, such as columns or rows, may be actuated to project one or more shapes (e.g., lines) of light 111a, 111b, ..., 111n onto an object in the field of view.

[0040] The detector 105 includes an array of pixel elements that can detect light projected from the light source 101 and reflected from objects in the scene. In this embodiment, the detector includes a camera such as a CCD sensor, a CMOS sensor camera, or an event-based vision sensor (EVS). Different pixels of the detector will detect light rays 113a, 113b, ..., 113n reflected from different points on objects in the field of view, enabling the detector to construct a 2D image of the scene.

[0041] Generally, for each point on an object within the field of view, two corresponding positions can be defined: (i) the position in the array of illumination points of the origin from which the light incident on that point on the object is emitted, and (ii) the camera pixel coordinates, i.e., the pixel position in the camera from which the light reflected by that point on the object is captured. Using a suitable algorithm, and taking into account the relative positions of the camera and projector (these relative positions are easily determined using standard calibration measurements), the image captured by the camera can be processed to determine the corresponding coordinates in the array of illumination points for each camera pixel. For a given camera pixel p, once it is determined that pixel p receives light from, for example, a point on a projector g, the estimated position E of the point on the object with coordinates {x,y,z} may be derived by using a known triangulation method similar to that used in stereo vision, taking into account lens parameters, the distance between the camera and the projector, etc. Such methods are described, for example, in "3D Imaging, Analysis and Applications, Chapter 3.4" (Liu, Y. et al., Springer Cham, September 11 / 12, 2020, ISBN 978-3-030-44070-1). Similar methods can be used when the light source is a point light source, such as in Figure 2B.

[0042] The image above is complex when the object contains one or more transparent surfaces. Here, the signal detected by each camera pixel may be the sum of signals reflected from different depths within the object. For example, when looking down at the object shown in Figure 1, the camera pixels may detect a small portion of the light reflected from the object's transparent upper surface, as well as light that has passed through the transparent upper surface and been reflected from the object's bottom surface or the material on which the object rests. To recognize the presence of transparent surfaces in the object (and to accurately determine the depth of each surface), it is necessary to "unmix" the signals received by the same camera pixels from these different surfaces by determining where in the array of illumination points these different rays originate.

[0043] Embodiments described herein use a temporal sequence of light patterns to illuminate an object and process captured images with knowledge of the illumination patterns to enable the system to recognize that the object includes one or more transparent surfaces. Each light pattern can be programmed using a projector. As the light pattern projected onto the object changes over time, by monitoring the intensity received by each camera pixel, it is possible to identify cases where the signal incident on a particular camera pixel actually consists of two or more distinct signals that travel along the same path between the object and the camera pixel, but originate from different points in the array of illumination points and are reflected, respectively, from a first transparent surface of the object and a second surface at a greater depth in the scene.

[0044] The above principle can be further understood by referring to Figures 3A and 3B. In this example, it is assumed that the light source is a projector, and different rows of projectors 101 are operated over time, and an image of the object 303 is captured by the camera 103 at each individual time step.

[0045] Figure 3A shows the system at a first time point t1. Here, rays 301 from the first row of projector elements are emitted toward the object 303. Upon reaching the upper surface 305 of the object, a small portion of the projected light is reflected backward toward the camera, as shown by ray 307. Meanwhile, most of the light from the first row of projector elements passes through the object 303 and is reflected from the second surface 309. From here, the light returns through the object 303 toward the camera 103, as shown by ray 311. As a result of the geometric arrangement of the system, rays 307 and 311 reflected from the upper and lower surfaces of the object 303 are incident on different pixels on the camera 103.

[0046] Referring to Figure 3B, this shows the same arrangement as in Figure 3A at a later time point t2. Here, a second row of projector elements is activated, instead of the same first row of projector elements as in time step t1 shown in Figure 3A. The second row of projector elements emits rays 313 toward the object. For illustrative purposes, rays 301 emitted by the first row of projector elements in the earlier time step in Figure 3A are shown as dashed lines in Figure 3B, but this is simply for comparison, and it should be understood that rays 301 do not actually exist in this later time step since the first row of projector elements is no longer active. As previously mentioned, when rays 313 reach the upper surface 305 of the object, a small portion of the light is reflected, as shown by rays 315. On the other hand, most of the light from the second row of projector elements passes through the object 303 and is reflected from the second surface 309. From here, the light, as shown by ray 317, passes through the object and returns, traveling towards the camera.

[0047] Importantly, the point on the camera onto which ray 317 enters in Figure 3B coincides with that of ray 307 at an earlier time step; that is, ray 317, as it leaves the object, travels along the same path as ray 307, which was projected by the first row of projector elements at t1, and is detected by the same camera pixel as ray 307. Since we know which projector element was active at each time step, it is possible to determine the projector element from which the two rays 307 and 317 are produced, even though they are detected by the same pixel on the camera (and appear to originate from the same point in the captured scene from the camera's viewpoint). The information thus acquired can then be used to determine the depths of the two surfaces measured from the camera and then to construct a 3D image of the object 303.

[0048] Therefore, by illuminating a field of view with different patterns of light at different time points and measuring the intensity of light incident on each camera pixel at those different time points, it is possible to determine the point in the array of illumination points through which each light ray incident on the same camera pixel passes.

[0049] Figure 4 shows an example of how the temporal sequence of a lighting pattern may be constructed in one embodiment. Here, the lighting pattern includes a series of line patterns, which are generated by activating different rows of projector light elements. The rows of projector elements can be divided into a series of m groups, each group containing a number of rows K. For example, group m=1 may contain rows 1 through K, group m=2 may contain rows K+1 through 2K, group m=3 may contain rows 2K+1 through 3K, and so on. If the projector has a total number of rows N and the number of rows per group is K, then the number of groups m is N / K.

[0050] In the example shown in Figure 4, the value K is chosen as 16. For illustrative purposes, Figure 4 shows only the first four groups of columns, but it should be understood that a projector may contain many more columns across its entire width. For example, the total number of columns N may be approximately 1968. Assuming that the number of columns K in each group is 16, this equals the total number of groups m = 123.

[0051] At each time step, a single column of projector elements is activated within each group of the column. Figure 4A shows the illuminated column at the first time step t1 of the sequence. Here, the first column of each group of the column is illuminated. Figure 4B shows the illuminated column at the second time step t2 of the sequence. Here, the second column of each group of the column is illuminated. Figure 4C shows the illuminated column at the third time step t3 of the sequence. Here, the third column of each group of the column is illuminated. Figure 4D shows the illuminated column at the fourth time step t4 of the sequence. Here, the fourth column of each group of the column is illuminated.

[0052] Since each group has 16 columns, the illuminated lines remain spaced 16 columns apart at each time step. Furthermore, once the 16th column of each group is activated, the total number of patterns in the sequence will equal the pitch (column spacing) K, meaning that each column of projector elements has been activated once over a series of time steps. (In some embodiments, the lines of illumination may be widened so that two or more adjacent columns of projector pixels are illuminated at once, and it will be understood that the columns of elements activated in subsequent time steps begin with the first column that was not illuminated in the previous time step. For example, columns 1 and 2 may be activated at time t1, columns 3 and 4 at time t2, and columns 5 and 6 at time t3. Doing so may help reduce the overall acquisition time, but at the cost of requiring a higher minimum height for the object to resolve its upper and lower surfaces.)

[0053] In some embodiments, the line pitch K may be predefined using knowledge of the operating range and / or maximum transparent object thickness in the scene. For example, the line pitch is defined as the first and second surfaces of the object being

number

[0054] Figure 5A shows an exemplary scene including two objects to be imaged, a bottle and a lid, both of which are at least partially transparent. Figure 5B shows the same field of view with a specific row of camera pixels (row A) highlighted. Here, the bottle has a first edge detected by pixel element 206 of row A on the camera and a second edge detected by pixel element 448 of that row. Figure 5B also shows the location of camera pixel element 501 in row A, between the two edges of the bottle.

[0055] Referring to Figure 6, this shows the images captured by the camera when different line patterns are projected onto objects in the field of view and reflected from them. (For illustrative purposes, the images shown in Figure 6 are taken from a smaller area of ​​the field of view, indicated by the dashed rectangle in Figure 5). Each image can be seen as containing a series of bright lines corresponding to the area in the scene where light projected from the projector enters the object and is reflected towards the detector. The rest of the scene, which does not receive light from the projector, remains dark. As a result of the different height profiles of the bottles and lids, the line patterns seen by the camera are distorted, and the lines no longer appear to be uniform straight lines, but rather curved and / or appear to be characterized by discontinuities that coincide with the edges of the objects.

[0056] As shown in Figure 6, the image detected by the camera changes over time as the projector periodically repeats different lighting patterns (here, the sequence of line patterns includes a total of 16 steps, but for brevity, only images from the first time step t1 and each even-numbered time step are shown in the figure). By activating different rows of projector elements at each time step, vertical lines appear to traverse the field of view.

[0057] Returning to camera pixel 501 shown in Figure 5B, it is possible to measure the intensity of light received by that camera pixel during the course of the image sequence. Figure 7 shows the intensity measured at the camera pixel at each of the 16 time steps in the line pattern sequence, with the intensity normalized by the time step with the highest intensity value in the image sequence. Figure 8 shows the normalized intensity measurements plotted on a graph. Here, time points t6 and t 14 Two distinct peaks in intensity can be clearly observed, and these peaks indicate frames in the temporal sequence of the image reflected from one of the object surfaces to the camera pixel p. A threshold may be applied to distinguish the true peaks of the signal from background noise.

[0058] The presence of two peaks in the signal shown in Figure 7 helps indicate that the object contains at least one transparent surface. Since each row of the projector is illuminated only once by the sequence of line patterns, and the camera pixels see two peaks in the course of that signal, the camera pixels are detecting light projected from two distinct rows in the projector array. From the camera's viewpoint, both of those signals travel along the same path from the object to the camera. The geometric arrangement of the system means that it can be inferred that the light in one of those signals is reflected from the surface below it, from which the other signal is reflected. The upper surface would then be at least partially transparent.

[0059] By identifying the positions of the two peaks in their temporal sequence and knowing which projector row was active when those images were captured, it is possible to determine which projector row the light ray incident on camera pixel p originates from. At this stage, the camera does not know which group of row m the light ray originates from, so it is not possible to determine the global position of those rows. In other words, as shown in Figure 4, it is not known whether a row is from the first set of 16 rows, the second set of 16 rows, the third set of 16 rows, etc. Instead, what can be determined at this stage is which row numbers 1-16 (out of an unspecified set of rows up to that point) were illuminating the scene when the reflection was detected at camera pixel p, and in this case, these are row numbers 6 and 14.

[0060] To obtain absolute column numbers, an additional set of coded patterns may be projected onto the field of view to enable global localization of column positions. The coded patterns may include, for example, Gray codes, i.e., a set of binary codes having the property that only one bit changes between subsequent codes (this makes the code more robust to, for example, lens blurring). The Gray codes may be projected such that, for example, projector columns 0 and 1 project Gray code index 0, projector columns 2 and 3 project Gray code index 1, and more generally projector columns y, y+1 project Gray code index y / Z, where Z is the number of repetitions of each code (Z=2 in the above example). For a projector with a width of 1020 pixels, using the same code index for every two subsequent projector columns would require generating 510 different Gray codes. At a minimum, this would require a Gray code consisting of T=9 bits, and 2 9 = 512 different codes. The first image projects the first bit of all codes, the second image projects the second bit of all codes, and so on. The number of columns Z in which each code is repeated is a tunable variable (more repetitions mean fewer Gray code patterns), but generally, it is preferable to have a fairly large number of Gray codes so that the number of columns with the same Gray code is smaller than the pitch K of the lines used for each pattern, and there is a sufficient surplus margin to cover a thick transparent object.

[0061] Figure 9 shows examples of 10 Gray codes that can be used in conjunction with the 16 line patterns used to capture the data shown in the graph of Figure 8. If gc(r,c,t) is the value of the camera pixel at position (r,c) at time t (equal to image number t) in a set of images containing the Gray code image, then the Gray code can be used to identify the projector row through the following steps. 1. For each camera pixel, find the minimum and maximum pixel intensities, a(r,c) and b(r,c), across the entire image. 2. Camera pixel thresholds

number

[0062]

number

[0063] 4. Convert the pixel observation value to its Gray code number by calculating the following:

[0064]

number

[0065] 5. Use a lookup table that returns the code index i when a Gray code number n is entered. 6. Convert the code into a projector sequence by multiplying the code by Z.

[0066] The algorithm described above assumes that the Gray code is designed so that codes consisting of only zero bits and only one bit are excluded from the projected signal. It will be understood that other methods are available for decoding the Gray code and extracting the underlying column numbers, as described in "3D Imaging, Analysis and Applications" (Liu, Y. et al., Springer Cham, September 11 / 12, 2020, ISBN 978-3-030-44070-1), which includes an up-to-date overview of state-of-the-art methods.

[0067] For transparent objects, a mixture of two Gray codes is returned to the camera. In practice, it has been observed that the algorithm described above can recover the stronger of the two mixed Gray codes returned to the camera. This corresponds to the return signal from one of the surfaces observed by the camera (the strongest one).

[0068] When processing both the image captured when illuminating the scene with the line pattern shown in Figure 6 and the image captured when illuminating the scene with the Gray code shown in Figure 9, two sets of results are obtained for the projector sequence. To show the projector sequence returned from the Gray code to the camera pixels using the above algorithm, gp(r,c) is used, and to show the projector sequence returned when illuminating the scene with the line pattern, mp(r,c,k) is used. i ) can be used. Here, k i This indicates the position of the i-th peak (i.e., frame number) seen in the camera pixels when the scene is illuminated with a line pattern (in the example shown in Figure 8, k1=6 and k2=14). As mentioned above, K represents the pitch of the lines in the line pattern.

[0069] Figure 10 shows the values ​​gp(r,c) and mp(r,c,k) acquired for each camera pixel in row A of Figure 5B. i The plot of ) is shown. As mentioned above, the data collected from the line pattern does not allow for the determination of all column numbers, but can instead be used to determine which of columns 1 through 16 was active at that time in a particular set of columns. For this reason, the values ​​mp(r,c,k) shown for each camera pixel in Figure 10 are i ) are all between 1 and 16.

[0070] The data signals from line patterns and Gray code patterns are analyzed by calculating the following for each peak k observed in a particular camera pixel. t The final projector column value for fp(r,c,k i ) can be combined to obtain.

[0071]

number

[0072] Here, o is the offset used to ensure consistency of data from line patterns and Gray code patterns, and mod is the modulo function.

[0073] Figure 11 shows the results of combining data from line patterns and Gray code patterns. By analyzing the distribution of points in the graph, it is possible to identify the presence of two surfaces corresponding to the top (transparent) surface of the bottle and the (background) surface on which the bottle is placed. By extending this analysis to each row of camera pixels, it is possible to construct a point cloud representing the three-dimensional shape of the object, including the surface profile of the (top) transparent surface of that object. (As explained above, for a given camera pixel, if the pixel receives light from a point in the projector array, the coordinates {x,y,z} of the point on the object from which the light is reflected can be determined using known triangulation methods. In this case, once it is determined which column in the projector array the camera pixel is receiving light from, the row number can be determined from the geometric arrangement of the system, so the actual point in the projector array from which the light is being emitted can be easily inferred.) Figure 13 shows an exemplary 3D reconstruction of the bottle and lid obtained using this method. For ease of comparison, the 3D reconstruction is shown alongside Figure 12, which reproduces the scene shown in Figure 5B.

[0074] The embodiments described above utilize a combination of line patterns and Gray code patterns to recognize and reconstruct the (upper) transparent surfaces of objects in a scene. It will be noted that the use of Gray code is optional and arises from the decision to use multiple lighting lines in each of the line patterns. In some embodiments, a single row of projectors may be illuminated at each time step in the sequence of line patterns, thereby allowing the absolute row number to be immediately retrieved without the need to illuminate the scene with the Gray code pattern as well. Limiting the number of lighting lines to a single line per pattern, however, comes at the cost of significantly longer collection times. In many applications, it is therefore preferable to use a method that combines multi-line and Gray code.

[0075] In the embodiments described above, the illumination pattern is defined by rows within an array of illumination points, but this is merely illustrative. In other embodiments, the pattern may be defined using rows within an array of illumination points, or diagonals within that array of points. Furthermore, the lines do not necessarily have to be straight lines; they may be curved.

[0076] It will be further understood that the line patterns and Gray codes described above are examples of a group of illumination patterns that can be used to identify and model the transparent surfaces of objects in a scene. Next, a set of further embodiments will be described in which the problem of identifying a point in an array of illumination points, from which a particular camera pixel detects light, can be considered a compressed sensing problem. In this embodiment, the scene is again illuminated with a temporal sequence of illumination patterns, each pattern being different from the others. Here, the illumination patterns are generated by defining separate illumination sequences for different regions within the array of illumination points. For each region, the illumination sequence defines how the amplitude of the light projected from each region toward the scene changes at each time step.

[0077] FIG. 14 shows an example of how an illumination sequence can be defined by using a code matrix M. In this example, each region of points within the array of illumination points is considered to be a column in that array, although as explained above this is not at all essential and in other embodiments the regions may be formed as rows within the array of illumination points or indeed as any one of several different shapes formed from the points in that array.

[0078] Each region (column) is assigned a binary code, and it can be seen that the binary codes in FIG. 14 extend vertically in the code matrix. In this embodiment, the code includes a sequence of bits that define whether a particular projector column is illuminated for each time step. For each column of the matrix, the white elements indicate the time steps in the sequence at which the respective projector column is switched on, and the black elements indicate the time steps in the sequence at which the respective column is switched off. Thus, each column of projectors may be illuminated more than once over a series of time steps.

[0079] FIG. 15 shows exemplary images captured by a camera when using these different illumination patterns to illuminate the same field of view as shown in FIGS. 6 and 9. There are a total of 37 patterns, but only 12 are shown in FIG. 15 for the sake of brevity. The images are normalized pixel by pixel according to the maximum value for that particular pixel across all images.

[0080] As described above, each single camera pixel receives one measurement s of intensity for each time step. i These measurements can be combined into a single vector S = {s1, s2, …, s I}, where I is the total number of images captured and is equal to the number of time steps in the sequence. The signal vector S is a linear mixture of the projected codes plus ambient light A, which can be assumed to be constant during exposure. The linear mixture is a vector R = {r1,r2,…,r} containing the relative responses of each individual code that is projected. P It can be expressed as}.

[0081] Subtracting the response from ambient light, the vector R becomes very sparse, meaning most of its elements are zero. A reasonable estimate of ambient light is:

number

[0082]

number

[0083] After subtracting the estimated ambient light, S' = S - A', the observational model can be formed as follows:

[0084]

number

[0085] (Here, since it's not possible to "subtract" light from the scene, each non-zero element of R is positive.)

[0086] The number of non-zero elements in R depends on the surface being imaged. If the object is not transparent, there is simply a single return signal from the object's (opaque) surface. This means that R contains only zero elements, with the exception of one non-zero element representing the return signal from the projector code returned from the surface. If the object is transparent, there are return signals for two of the projected codes. This means that R contains only zero elements, with the exception of two non-zero elements: one from the object's upper (transparent) surface and one from a second surface below that transparent surface (the same applies if there is a reflection from another object in the scene on the top surface of the opaque surface). When observing a transparent object and a reflection simultaneously, three of the elements in R will be non-zero.

[0087] The above principle can be further understood by referring to Figures 16A and 16B. Figure 16A shows an example in which a scene is illuminated by light from three rows on a projector. Each projector row has its illumination sequence encoded as a set of bits, which defines the time steps in which the projector row is on. For row 1, the illumination sequence may be written as 100011001, indicating that row 1 is switched on at time steps 1, 5, 6, and 9. Projector row 2 has the illumination sequence 010100110, meaning it is switched on at time steps 2, 4, 7, and 8. Projector row 3 has the illumination sequence 10100101, meaning it is switched on at time steps 1, 3, 6, and 8. From this, we can see that each row is switched on a total of four times during the acquisition process.

[0088] The upper line in Figure 16A shows the signal detected by the camera pixels over time. Here, the signal detected by the camera is directly mapped to the illumination sequence of projector row 1, with no contribution from projector row 2 or 3. Following this, the vector R can be determined to be R = {1, 0, 0}. Since R has only one non-zero element, it can be inferred that the object being imaged is opaque, as the light detected by the camera pixels comes simply from a single row in the projector array.

[0089] Referring to Figure 16B, the codes for each of the projector rows 1, 2, and 3 are the same as in Figure 16A. However, here the signal detected by the camera pixel is a linear combination of the codes from rows 1 and 3, and therefore R={1,0,1}. Since the camera pixel receives light projected from two separate rows of projectors, it can be inferred that the light from one of those projector rows is reflected from a first surface of the object that is at least partially transparent, and the light from the other one of the projector rows is reflected from a second surface beneath its transparent surface.

[0090] Therefore, in order to recover the surface depth, it is necessary to determine the value of R in Equation 5 above.

[0091] Since I ≪ P, this linear system is in an extremely underdetermined state, but since R is sparse, it is possible to use the non-negative least squares method to solve this problem and obtain an estimate of R (this estimate will be denoted as R' below). More precisely, it is possible to solve the following:

[0092]

number

[0093] In reality, due to noise and signal blurring, R is not perfectly sparse, but very small r iIt is assumed to be zero, which is still sufficient to obtain the estimate R'.

[0094] To achieve the same sparsity, other optimization goals can be conceived, such as the following:

[0095]

number

[0096] Here, λ is a suitable parameter for balancing goodness of fit and sparsity. However, in practice, the method presented in Equation 6 above works well with the appropriate algorithm. Several suitable such algorithms are known in the art. One example is the Lawson-Hanson algorithm, which performs iterative optimization of a function, as described in "Solving Least-Squares Problems" (Lawson, CL and RJ Hanson, Upper Saddle River, NJ: Prentice Hall, 1974, Chapter 23, p. 161). Another method is the Orthogonal Matching Pursuit, which performs the same process (see "Matching pursuits with time-frequency dictionaries" (Mallat SG and Zhang, Z., IEEE Transactions on Signal Processing, Vol. 41(12), 1993, pp. 3397-3415)). In summary, these algorithms first find the code r that best matches the received signal S. i It works by finding the residual D and determining which subsequent code can be used to explain it. This process is repeated by adding more code in continuing iterations until the residual becomes small enough. Naturally, to speed up the process, for example, S i By using the largest index of r i It is possible to optimize the exploration.

[0097] After the optimization loop has converged (e.g., after the residual D has become sufficiently small), elements r within R' that are below the selected threshold i can be further removed by ignoring them. The threshold can be set either absolutely or relative to the maximum element of R', and the relative threshold is the preferred choice. Further, when the camera pixels have a single return signal, adjacent codes are easily included as secondary candidates. These can be removed by post-filtering R' to save only the largest of the subsequent non-zero indices r i thereof. By itself, the Lawson-Hanson method (and similar methods) only determines the closest integer code index that can explain the signal S'. To increase the 3D accuracy, it is necessary to recover the code index with sub-integer precision. In practice, this means refining the detected integer code index r

[0098] (and thus the projector sequence) to a projector sequence / code index r' that has a position greater than an integer. This can be done by calculating the correlation score i for J adjacent codes (generally 1 to 2), where -J < j < J, fitting a parabola to CS(j), and using the maximum point of the fitted parabola as the sub-integer code index. i This is [Number]

[0099] ​The optimization problem can be further reduced by considering the relative parallax window between the camera and the projector. The parallax window defines the number of consecutive columns in the projector array whose light can be detected by the same camera pixel. The geometric arrangement of the light source and detector imposes a limit on this number. Therefore, codes assigned to projector columns outside that parallax window (i.e., located more than a certain number of columns away from the column in question) can be excluded from consideration. In most relevant setups, this can exclude up to 90% of the codes, setting their contribution to the camera pixels to zero.

[0100] To determine the contribution of each code to the signal detected by camera pixels, a deep learning-based method may be employed. An overview of applicable methods is provided in the journal publication "Deep learning for compressive sensing: a ubiquitous systems perspective" (Machidon, AL and Pejovic, V., Artificial Intelligence Review, vol. 56, no. 4, pp. 3619-3658, April 2023). These methods may be trained on datasets generated by performing a linear combination of M codes, where a limited number of codes (typically 2-4) are mixed together at different amplitudes and used as input, and their corresponding code indices are used as ground truth to train the network.

[0101] The camera / projector system blurs the projected code. This can be compensated for by pre-blurring the matrix M using the expected point spread function before the R' estimation, and / or by performing a local search for adjacent codes once the first-order code is identified in an iterative optimization loop.

[0102] Figures 17 and 18 show exemplary images that further support the method described above in relation to Figures 14 to 16. Figure 17 shows exemplary images captured by a camera when different lighting patterns are used to illuminate the same field of view. Figure 18 shows the intensity value of a single pixel in the image sequence (in the top line), and the two subsequent rows show the codes recovered by dissociating that system, in this case the codes corresponding to columns 171 and 164 on the projector.

[0103] By determining the codes contributing to the detected signals and the corresponding regions (columns) within the array of illumination points, it is possible to generate a plot similar to that shown in Figure 11. Figure 19 provides an example showing the two code indices found per camera pixel for the same image row shown in Figure 11 using the Lawson-Hanson algorithm. The results of the two methods are found to be comparable, with Figure 11 containing more data points and more noise, while Figure 19 contains fewer data points and much less noise. Since this method can only recover integer indices, the results shown in Figure 19 have a staircase appearance. By using subcode detection methods based on correlation and parabolic fitting, it is possible to achieve a more continuous and smoother line, as shown in Figure 20.

[0104] In general, the methods that yield the results shown in Figures 11 and 19 both share the common goal of providing multiple projector indices per camera pixel due to multiple, mixed return signals from the scene, enabling the reconstruction of transparent objects. The method shown in Figure 11 provides more data points and is computationally very efficient. At the same time, it requires that the maximum object thickness (to determine the pitch K) be predefined and has a slight tendency towards noise closer to the true signal. In contrast, the second, code-based method, described with respect to Figures 14 to 20, does not require the pre-definition of the maximum object thickness and is more applicable in situations with less controllability. Noise points also tend to be located further away from the true signal, making them easier to remove with filtering. The code-based method, however, is computationally more expensive and may generate fewer data points.

[0105] It will be understood that the results obtained using the code-based methods shown in Figures 14 to 20 can be improved by controlling several parameters. In the embodiments described herein, each code has a constant amplitude (i.e., the number of "on" bits is the same for all codes, and each column of the projector is illuminated for the same total time during the acquisition process). The number of bit changes in the code for adjacent columns may also be kept constant, somewhat similar to Gray codes. These constraints are by no means essential, but can improve the convergence speed of the algorithm.

[0106] Furthermore, performance can be improved by reducing the correlation of codes assigned to columns within the system's disparity window. As is well known in the art, the disparity window refers to the code range that can be observed by a single camera pixel due to the geometric arrangement of the imaging setup. For a given camera pixel, an object at zero distance from the camera observes code index c0, and an object at infinity observes code index c ∞ Observe c0 and c ∞This varies depending on which pixel is selected. For example, if prior knowledge about the operating range is available (e.g., all objects will be within distances d1 and d2), the minimum and maximum code indices per camera pixel may be further reduced, thus further restricting the effective disparity window W.

[0107] If W is the parallax window of the projector row p', then the correlation

number

[0108] The improvements brought about by controlling the above parameters can be understood by referring to Figures 21 to 25. More specifically, Figure 21 shows the cross-correlation matrix for the codes shown in Figure 14 (and used to obtain the results shown in Figure 19). Here, we can see that the intensity along the diagonal is constant, which is a result of the fact that each code has the same number of "on" bits. Also in Figure 14, we can see that in columns located near the column in question and within the residual window, the correlation between codes is reduced in detail, and by reducing this correlation, it is possible to minimize interference between columns whose light can be detected by the same camera pixel. Figure 22 shows an enlarged portion of the matrix in Figure 21, where we can see that the width C of the main lobe is equal to that of the three columns. It is also preferable that C be kept as small as possible, but in practice, a trade-off must be found between W, C, I, and X to balance performance and time. In the example case, W=100, C=3, I=37, X≦1.0, where I is the number of images in the sequence.

[0109] For comparison, Figure 23 shows an alternative code matrix, the only constraint applicable here is that a single bit changes between consecutive codes (the codes for pairs of consecutive columns are the same in this example, but this does not affect the results). Figure 24 shows the cross-correlation matrix of the code matrix shown in Figure 23. In contrast to the cross-correlation matrix shown in Figure 21, the intensity along the diagonal changes quite drastically, which is due to the changing number of "on" bits per code, meaning that signal normalization is required. Also, there is no clear band around the main diagonal where interference is minimized, which is due to the lack of an attempt to design the codes so that the correlation between codes within the disparity window is minimized. Figure 25 shows the results obtained when the code matrix is ​​used to identify the column into which the light is incident on each camera pixel. Here, the presence of the secondary surface can still be resolved, but the reconstruction is not as successful as in Figure 19.

[0110] With respect to Figures 14 to 25, the embodiments described above rely on the region (column) being switched "on" and "off" at different time steps. However, in practice, the intensity of the light source may still have a finite amplitude at each time step. That is, the system may be configured such that for time steps in a sequence where the column is "off," light is still emitted from that region of the array of illumination points, but with a reduced amplitude compared to time steps where the column is "on." In extreme cases, this reduced amplitude may be zero, but generally, "off" does not necessarily mean that there is no signal at all.

[0111] Furthermore, in some embodiments, the illumination sequence for each region (e.g., a column) may be defined by applying a specific frequency modulation to the amplitude of the light emitted from that region. That is, rather than each region having a binary sequence of "on" and "off" periods throughout the acquisition process, the amplitude of intensity may change more continuously throughout the acquisition process. For example, the intensity of the light emitted from each region may change sinusoidally as the image frame progresses, and the frequency of the modulation may differ for each region. In this case, each illumination sequence may still be encoded using one or more bits, but here the bits encode the frequency modulation applied to that region. The signal detected at each camera pixel throughout the image acquisition process is then the sum of frequencies associated with regions in the array of illumination points from which the light incident on the camera pixel is emitted. In a similar manner to that described above, the signal detected by the camera can be unmixed to recover the contributing frequencies and the regions (columns) from which the light detected at that pixel is emitted.

[0112] Figure 26 shows an example in which an imaging system according to one embodiment is employed by a robotic device in a factory or warehouse. The robotic device 2600 may be one of several such devices operating in the factory or warehouse.

[0113] The robotic device 2600 includes one or more arms 2601 configured to manipulate an object 2603. The robotic arms 2601 may be used, for example, to lift the object 2603 from a shelf 2605 on which the object is placed. To do so, the robotic device needs to determine the object's position in space in order to properly engage with it. However, the robotic device's ability to do so may be impaired if the object 2603 has a transparent surface 2607, which can be difficult for the robotic device to properly detect, for example if the object is made of plastic or has transparent packaging. If the robotic device cannot detect the transparent surface, the robotic arm may be lowered too far, penetrating the transparent surface 2607 and damaging the object 2603. To enable the robotic device to properly map the object's position in space, the robotic device may utilize the imaging system described herein to detect whether the object has one or more transparent surfaces and adjust its position accordingly. In the example shown in Figure 26, the robot device itself includes an imaging system 2609 according to the embodiments described herein, which is used to map the surface of an object before interacting with it. However, it should be understood that the imaging system does not have to be part of the robot device itself, but can be housed separately from the robot device and used to provide the robot device with information about an object before the robot device interacts with the object 2003.

[0114] The imaging system may also be used to inspect the surface of a transparent object. In this case, the captured 3D data of the transparent surface may be compared to a CAD model to look for defects, for example, or a local surface characterization method (e.g., an edge detection filter) may be used to detect imperfections on the surface. If a defect is detected, a robotic device similar to that in Figure 26 may be used to remove the object, or the error may be logged in a suitable database for subsequent processing and handling of the part.

[0115] It will be understood that the implementations of the subject matter and operations described herein may be realized in digital electronic circuits, or in computer software, firmware, or hardware, or in one or more combinations thereof, including the structures disclosed herein and their structural equivalents. Implementations of the subject matter described herein may be realized using one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a computer storage medium for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or additionally, program instructions may be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals generated to encode information in preparation for transmission to a receiver device suitable for execution by a data processing device. The computer storage medium may be a computer-readable storage device, a computer-readable storage board, a random or serial access memory array or device, or one or more combinations thereof, or may be included therein. Furthermore, the computer storage medium may not be a propagating signal itself, but may be the source or destination of computer program instructions encoded on artificially generated propagating signals. Computer storage media may also consist of one or more separate physical components or media (for example, multiple CDs, disks, or other storage devices), or may be contained within them.

[0116] While several embodiments have been described, these embodiments are presented merely as examples and are not intended to limit the scope of the invention. In fact, the novel methods, devices, and systems described herein may be embodied in various forms, and furthermore, various omissions, substitutions, and modifications in the forms of methods and systems described herein may be made without departing from the spirit of the invention. The appended claims and their equivalents shall encompass forms or modifications that fall within the scope and spirit of the invention. [Explanation of symbols]

[0117] 100 Imaging Systems 101 Light source 103 Detector 105 Object 107 Object 109 Surface 111 light 113 Ray of light 201 Light source 203 pixel elements 205 Laser 207 Mirror 209 plane 301 Ray of light 303 Object 305 Upper surface 307 Ray of light 309 Second surface 311 Ray of light 313 Ray of light 315 Ray of light

Claims

1. An imaging system for capturing a scene containing one or more objects, A light source configured to illuminate the scene by projecting light rays through an array of points in space, wherein for each time step in a series of time steps, the light source is configured to illuminate the scene in different lighting patterns by projecting light rays through different groups of points in the array, A detector comprising an array of pixel elements, configured to capture an image at each time step by detecting light projected onto the array of pixel elements toward the scene and reflected toward the detector from one or more surfaces of the object; It is an image processor, The captured image is processed to determine, for each pixel element, the light signal incident on the pixel element during the course of the series of time steps, Based on knowledge of the light signal incident on the pixel element at each time step and the illumination pattern projected at each time step, it is determined that the object includes a first surface which is at least partially transparent, and that at least a portion of the light incident on the first surface is reflected toward the detector, and another portion is reflected from a second surface at a greater depth in the scene. An image processor and A system that includes these features.

2. The system according to claim 1, wherein the image processor is configured to determine, based on the optical signal incident on the pixel element at each time step and the illumination pattern projected at each time step, that the optical signal includes a first portion of light reflected from the first surface and a second portion of light reflected from the second surface.

3. The system according to claim 2, wherein the first portion of the light and the second portion of the light travel along the same path between the object and the detector.

4. The image processor, based on the optical signal incident on the pixel element at each time step and the illumination pattern projected at each time step, A first location in the array of illumination points, the first location where light reflected by the first surface and incident on the pixel element is projected through that location. The system according to any one of claims 1 to 3, further configured to determine

5. The image processor, based on the optical signal incident on the pixel element at each time step and the illumination pattern projected at each time step, A second location in the array of illumination points, wherein light that has passed through the first surface and reflected from the second surface toward the pixel element is projected through that location. The system according to claim 4, further configured to determine

6. The system according to claim 4 or 5, wherein the image processor is configured to determine the distance between the detector and the first surface and / or the second surface based on the first or second location.

7. The system according to any one of claims 1 to 6, wherein the lighting pattern includes a series of line patterns in which one or more lines are projected onto the scene.

8. The system according to claim 7, wherein each line pattern includes a plurality of parallel lines.

9. The system according to claim 7 or 8, wherein each line is formed by a ray of light passing through each row or column of the array of illumination points.

10. The system according to any one of claims 7 to 9, wherein the line pattern is selected such that each line is projected onto the scene once over the series of time steps.

11. The system according to any one of claims 7 to 10, wherein the image processor is configured to determine that the object includes a first surface that is at least partially transparent by identifying the presence of two or more peaks in the light signal incident on the pixel elements during the process of projecting the line pattern onto the scene.

12. The system according to claim 11, wherein the image processor is configured to identify a region of the array of illumination points through which light reflected by the first surface and incident on the pixel element has passed, based on the peak in the light signal incident on the pixel element in the process of illuminating the scene with the line pattern.

13. The system according to any one of claims 10 to 12, wherein the illumination pattern further comprises a series of coded patterns, and the global location in the array of illumination points through which light rays incident on each pixel of the detector pass is determined from the coded patterns.

14. The system according to claim 13, wherein the coded pattern includes a Gray code pattern.

15. The system according to claim 13 or 14, wherein the image processor is configured to identify a region of the array of illumination points through which light reflected by the first surface and incident on the pixel element has passed, based on (i) a peak in the light signal incident on the pixel element in the process of illuminating the scene with the line pattern, and (ii) a signal observed in the pixels of the detector throughout the process of illuminating the scene with the coded pattern.

16. The system according to any one of claims 1 to 6, wherein the light source is configured to generate the illumination pattern by defining an illumination sequence for each region of the points in the array of illumination points, the illumination sequence specifies a variation in the amplitude of light projected through each region over a series of time steps, and the illumination sequence is different for each region.

17. The system according to claim 16, wherein each region of the points includes each line of points in the array of illumination points.

18. For each pixel element, the image processor is configured to model the light signal incident on the pixel element over a series of time steps as a function of the contribution of the illumination point from the region in the array, The system according to claim 16 or 17, wherein the image processor is configured to determine the number of regions N such that the illumination sequences of those regions, when combined, become the optical signal seen by the detector.

19. The system according to claim 18, wherein the image processor is configured to determine, when N ≥ 2, that the object includes a first surface and a second surface which are at least partially transparent.

20. The system according to any one of claims 16 to 19, wherein each lighting sequence is encoded using one or more bits, the one or more bits defining the relative amplitude of light projected from the region of a point toward the scene at each time step.

21. The system according to claim 20, wherein one or more bits specify a frequency modulation applied to the light projected from the region of the point over a series of time steps.

22. The system according to claim 20, wherein each lighting sequence is encoded as a sequence of bits, each bit is associated with a respective time step, and the value of each bit defines the relative amplitude of light projected from the region of a point toward the scene at each respective time step.

23. The system according to claim 22, wherein for each region, the illumination sequence defines one or more time steps in which light is projected from the region with a first amplitude, and one or more time steps in which the amplitude of the light projected from the region is reduced to or zero compared to the first amplitude.

24. The system according to claim 23, wherein the illumination sequence is defined such that the number of time steps in the sequence in which light is projected with the first amplitude is the same for each region of the region of the point.

25. The system according to claim 23 or 24, wherein each region of a point includes each line of points in the array of illumination points, and the number of bit changes between illumination sequences is the same for each consecutive pair of lines.

26. The system according to any one of claims 22 to 25, wherein the region includes a column or row of points in the array of illumination points, and the sequence of bits for each region is defined such that, for any one region, the correlation between the sequence of bits for that region and the sequence of bits for other regions within the parallax window of that region is reduced compared to the correlation between the sequence of bits for that region and the sequence of bits for other regions outside the parallax window, the parallax window defines the maximum number of consecutive regions in the array of illumination points, and light from the consecutive regions is detectable by a single pixel of the detector according to the geometric arrangement of the system.

27. The system according to any one of claims 1 to 26, wherein the light source includes a projector having an array of projector elements, and each lighting pattern is generated by activating one or more of the projector elements.

28. A robotic device configured to manipulate a physical object, wherein the robotic device is configured to identify the object as having a transparent surface by using the system described in any one of claims 1 to 27 for imaging the object.

29. A warehouse comprising one or more robotic devices which are robotic devices according to claim 28.

30. A computer-readable medium containing computer execution instructions that, when executed by a computer, cause the computer to operate the system described in any one of claims 1 to 27 or the robotic device described in claim 28.

Images