Projection equipment
By using an independently addressable array of light-emitting modules and a multi-camera system, the problems of large size and limited scanning speed of existing structured light 3D imaging systems are solved, enabling faster and more efficient 3D imaging, which is suitable for small devices.
Patent Information
- Application Number
- CN202580027393.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-02-27
- Filing Date
- 2025-02-27
- Publication Date
- 2026-11-17
AI Technical Summary
Existing structured light 3D imaging systems are typically large and complex, and are inefficient when projecting sparse patterns, with limited scanning speeds, making it difficult to achieve efficient imaging in small devices.
The system employs an array of independently addressable light-emitting modules, with each module projecting a different light pattern. Combined with at least two cameras and a processing unit, it generates a depth map by detecting light pattern distortion, allowing the system to cover the entire scene without scanning.
It achieves faster and more efficient 3D imaging, and the system is more compact, capable of efficiently projecting dense or sparse patterns in various scenarios, making it suitable for small devices such as endoscopes and micro drones.
Smart Images

Figure CN123125459A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the imaging of 3D objects. Background Technology
[0002] Structured light 3D cameras typically project structured light patterns, such as vertical stripes and dots, onto a scene. The camera (or a pair of cameras) detects distortions in the projected pattern to determine the distance, depth, and surface information of objects in the scene. Such 3D imaging systems have a wide range of applications, including robotics vision, industrial inspection, gaming, and mapping. Smaller 3D imaging systems also exist, such as dental scanners and optical profilometers.
[0003] To image the entire scene, numerous patterns need to be projected. For example, a typical structured light projector illuminates a spatial light modulator and projects the resulting image. This type of system is usually large and complex, has a fixed focal length, and suffers from losses for sinusoidal patterns, especially sparse ones. Laser triangulation systems scan the entire scene with a single light pattern (e.g., a single line), thereby acquiring numerous images to form a 3D distribution map. Other systems, such as those using micromirror interferometer projectors, are more efficient but limited to projecting sinusoidal patterns. Summary of the Invention
[0004] According to a first aspect, the present invention provides a system for 3D imaging using structured light, comprising: an independently addressable array of light-emitting modules, wherein each light-emitting module is arranged to project light with a different pattern; one or more cameras arranged to detect the projected light from each light-emitting module; and a processing unit arranged to generate a depth map based on the projected light detected by the one or more cameras.
[0005] According to a second aspect, the present invention provides an apparatus for 3D imaging using structured light, comprising: An array of independently addressable light-emitting modules arranged on a substrate, wherein each light-emitting module is arranged to project light with a different pattern.
[0006] Therefore, it can be seen that, according to the present invention, an independently addressable array of light-emitting modules is provided, wherein each light-emitting module is capable of projecting a different light pattern. For example, different light patterns can be different dot patterns, stripe patterns, or any other desired patterns. Those skilled in the art will understand that projecting different light patterns from the same device allows for 3D imaging of a scene without the need for back-and-forth scanning to cover the entire scene. Since each light-emitting module is independently addressable, the system can quickly change the projected light pattern by changing which light-emitting module is turned on or off. Therefore, compared to existing devices, the present invention allows structured light to be projected onto a scene faster and more efficiently because the device is not limited by the scanning speed of the projected pattern.
[0007] Furthermore, different light-emitting modules can project different light patterns simultaneously, enabling the projection of complex composite patterns with the same speed and efficiency as simple patterns. The array of light-emitting modules also allows for a more compact projector size compared to other projectors, such as spatial light modulators or laser grating scanners, because the projected pattern is encoded into the hardware of the light-emitting module array. Sparse patterns can also be projected more efficiently than existing devices such as spatial light modulators. Patterns projected from different light-emitting modules can be spatially separated or partially or completely superimposed on each other. This allows the system to be used in a variety of situations, whether the projected pattern required for accurate imaging of a scene is dense or sparse, or whether different sinusoidal intensity distributions are needed.
[0008] A camera is positioned to detect projected light reflected from objects present in the scene to be imaged. A processing unit is positioned to form a depth map using the detected projected light, for example, by comparing the light patterns detected by one or more cameras with what the light patterns would look like without objects present, such as when light is normally projected onto a flat surface. The light patterns will be distorted due to interaction with any objects in the scene being imaged, and the processing unit can use these distortions to determine the shape and depth of objects in the scene. By projecting different light patterns, different features can be selected from the scene and a depth map can be formed.
[0009] While a single camera can be used to image the projected pattern, it is preferable to deploy at least two cameras to detect the projected pattern. This can aid in imaging—for example, by using projected patterns from different perspectives, it is possible to resolve blurry areas. For instance, once the projector is installed, no calibration is required because the projected pattern and the resulting depth map can be inferred from the correlation between groups of images detected by each camera; for example, the projector is capable of self-calibration.
[0010] The array of light-emitting modules can be arranged to emit different patterns sequentially (e.g., from different light-emitting modules). This allows any ambiguity in the scene to be resolved.
[0011] In one set of embodiments, each light-emitting module is arranged to emit patterns with different spatial frequencies. For example, each light-emitting module may be arranged to project a series of lines or dots with different spacing. For example, a pattern with stripes between 10 and 100 vertical stripes can be used, with different patterns projected and detected sequentially. Preferably, the system is arranged such that each part of the scene is illuminated by multiple different light patterns. For example, to cover the scene with the vertical lines and resolve any ambiguity, four patterns with different spatial frequencies may be projected.
[0012] The light-emitting module array can be a 2D array, in which each light-emitting module is arranged on a plane. This arrangement allows the light-emitting modules to be easily manufactured with a precise, known arrangement, for example, using photolithography.
[0013] In one set of embodiments, each light-emitting module includes at least one light source and an optical element. Each light source can be arranged to direct light to a corresponding optical element, wherein the optical element generates a light pattern from the light. In one set of such embodiments, the array of light-emitting modules includes an array of light sources and a corresponding array of optical elements. Each light source or group of light sources may have a corresponding optical element. For example, the array of light sources and the array of optical elements may be mapped to each other.
[0014] The light source may include LEDs, lasers (e.g., edge-emitting lasers), or any other suitable light source for projection onto the scene, as will be understood by those skilled in the art. Preferably, the light source includes a vertical-cavity surface-emitting laser (VCSEL). VCSELs are highly efficient and emit light with a narrow emission angle and narrow spectral width, which allows for the projection of sharp, collimated patterns. Furthermore, VCSELs can be modulated at high speeds, which is useful when the light-emitting modules are sequentially and rapidly switched on and off to quickly project a series of patterns (e.g., patterns with different spatial frequencies).
[0015] Furthermore, the cost of an integrated VCSEL array can be lower than the cost of assembling individual light sources on different integrated circuits. This integrated VCSEL array, for example, involves arranging multiple VCSELs on the same substrate, such as within an integrated circuit. Additionally, the VCSEL array can be smaller than other light source arrangements, for example, smaller compared to an LED array.
[0016] The optical elements in each light-emitting module can be arranged above the light source, for example, above the light source along the projection direction. For instance, the light source can be arranged in a 2D array, and the optical elements can be arranged in a corresponding 2D array, positioned above the light source in a plane parallel to and spaced apart from the light source. During the fabrication of the light-emitting module array, the optical elements can be stacked on top of the respective light sources, allowing for precise alignment between each light source and its corresponding optical element.
[0017] The optical element may include a lens, wherein the lens is arranged to focus light from one or more corresponding light sources from each light-emitting module.
[0018] In one set of embodiments, at least one (preferably each) optical element is a diffractive optical element. The diffractive optical elements of different light-emitting modules can be arranged to project different patterns. Appropriate diffractive optical elements can be selected to project specific patterns, for example, by using a micro-structured surface relief profile to shape the light emitted from the light source into a desired distribution via diffraction. For example, diffractive optical elements can be fabricated to project specific patterns suitable for a given application, such as a sparse dot matrix for facial recognition. A variety of different patterns, such as patterns with different spatial frequencies, can be used to image complex scenes, while only a small number of patterns can be used to image simpler scenes or for lower resolution imaging.
[0019] In one set of embodiments, the diffractive optical element includes multiple phase retardation levels. For example, a single pattern projected from a single diffractive optical element can consist of multiple sub-patterns with different phase shifts. This can enable the depth map formed by the processing unit to have higher resolution, and / or enable further customization of the projected pattern.
[0020] The one or more diffractive optical elements can act as beam splitters encoded as phase masks. For example, in a simple diffractive optical element, the light source is split into two dummy source points. Interference between the dummy source points produces a diffraction pattern with sinusoidal intensity variations and spatial frequencies proportional to the spacing between the dummy source points. Multiple dummy source points can be created using a single diffractive optical element or multiple adjacent (e.g., stacked) diffractive optical sub-elements, where the resulting projection pattern can be determined by the interference of all combined optical paths from the dummy source points to the object on which the light is projected.
[0021] In one set of embodiments, the projected light from each light-emitting module is collimated in a first direction and diverged in a second direction, wherein the first and second directions are perpendicular. For example, each light-emitting module can project a line pattern. Line patterns are advantageous because fewer different patterns, such as patterns with different spatial frequencies, are needed to illuminate the entire scene.
[0022] One or more light sources in each light-emitting module can be aligned with optical elements to achieve optimal light transmission through the optical elements. For example, one or more light sources can be arranged on a substrate using photolithography to form an array of light-emitting modules. Using photolithography to fabricate light-emitting modules allows for precise positioning of one or more light sources relative to their respective optical elements. For example, in the case where one or more light sources include one or more VCSELs, the VCSELs can be positioned at intervals between 50 µm and 100 µm. This can further allow more VCSELs to be positioned in a single device, thereby increasing the intensity of emitted light and / or increasing the variety of projectable patterns.
[0023] Optical elements, such as diffractive optical elements, can be integrated into each light-emitting module. For example, the light source can be integrated into a light source array in the first layer, and the corresponding optical elements can be integrated into an optical element array in the second layer, wherein the second layer is arranged above the first layer to form a stack providing the array of light-emitting modules. This arrangement maximizes the amount of light transmitted through each corresponding optical element.
[0024] Optical elements may include lenses and / or mirrors, and / or any combination of diffractive optical elements. For example, each optical element may comprise several diffractive optical elements arranged in series. This allows the pattern of light emitted from the optical elements to be selected and / or optimized more precisely.
[0025] Those skilled in the art will understand that although the light-emitting modules may be spaced apart from each other (e.g. on a common substrate), at the working distance of the device used for 3D imaging, such spacing will generally have a negligible difference in the detection of the projected pattern.
[0026] In one set of embodiments, each light-emitting module includes multiple master emitters, wherein each master emitter is arranged to project a light pattern at a different lateral position compared to another master emitter (e.g., an adjacent master emitter). Each master emitter may include at least one light source, such as at least one VCSEL. However, there may only be one optical element for each light-emitting module, i.e., the master emitters of the light-emitting module may share an optical element. Therefore, while each light-emitting module may include multiple master emitters capable of changing the lateral position of the projected pattern, the structure of the projected pattern from each light-emitting module may be fixed, for example, the spatial frequency of the projected pattern may be fixed. This allows the light projected by the master emitters in the array of light-emitting modules to cover the entire scene over a period of time, enabling every object in a given scene to be imaged, for example, generating a high-resolution 3D map.
[0027] Multiple main emitters can be arranged in a 2D planar array, such that they are arranged in the same plane. Each main emitter within the light-emitting module can be spaced apart from adjacent main emitters, for example, the main emitters can be equally spaced. Each main emitter can be spaced at different distances from other main emitters, and / or can be arranged in a grid or other form. Therefore, each main emitter can illuminate the corresponding optical element of the light-emitting module at a different known angle δ. The distance between the light source and the corresponding optical element... f In the case of, according to the relation The pattern moves laterally (e.g., along a direction perpendicular to the direction of light projection) by a distance Δ. Therefore, the position of the pattern projected from each main emitter within the light-emitting module is different. Thus, enabling each main emitter within the light-emitting module to be independently addressable allows for the projection of light patterns with different lateral positions.
[0028] In one example, each light-emitting module includes three main emitters, which can be spaced apart to project light at three lateral positions, such as 0 degrees, 120 degrees, and 240 degrees. By illuminating different main emitters, the light pattern can be moved laterally and thus cover the entire scene. Furthermore, one or more cameras can detect the lateral movement of the detected light, allowing the processing unit to better estimate the relative distances of the light traveling to different parts of the image, thereby enabling higher accuracy in forming a depth map of the imaged scene. For example, a single pattern with three different lateral positions can be used to image the scene to fully illuminate it, and / or because the reflectivity of objects in the scene may be unknown. These three different patterns can be used to calculate the depth corresponding to each camera pixel.
[0029] As described above, according to some embodiments, light can be projected at three different lateral positions for each projected light pattern. Therefore, when a total of four light-emitting modules are present (e.g., each emitting a pattern with a different spatial frequency), twelve different patterns can be used to image a single scene. The arrangement according to the invention advantageously facilitates rapid changes to the projected pattern for efficient imaging.
[0030] While using two projection patterns (each with different spatial frequencies) for imaging may be sufficient for 3D imaging under ideal, low-noise conditions, this may not be enough in noisy environments. Furthermore, moving each pattern laterally less than three times (e.g., less than three main emitters per light-emitting module) may not cover the entire scene, potentially making it difficult to obtain reliable 3D images. However, in some cases, moving more than three laterally (i.e., denser sampling) may degrade device performance.
[0031] Therefore, in some embodiments, the independently addressable light-emitting module array is arranged to emit patterns having at least three, preferably four or five spatial frequencies, for example, three lateral positions for each spatial frequency. For example, the light-emitting module array may include three light-emitting modules, wherein each light-emitting module includes three independently addressable main emitters and diffractive optical elements, wherein each diffractive optical element projects light with a different spatial frequency.
[0032] Furthermore, simultaneously projecting patterns with different lateral positions allows for the projection of more complex aperiodic patterns with the same efficiency as simpler patterns, such as sine waves. The processor can be programmed with known patterns that can be projected for different combinations of activated main transmitters.
[0033] In one set of embodiments, each master emitter includes an elongated array of secondary emitters. The elongated array of secondary emitters of each master emitter can be driven by the same control signal. Each secondary emitter can include a light source, such as a VCSEL. Multiple secondary emitters can increase the light intensity emitted from each master emitter, thereby enabling the projection of a sharper pattern, for example, when projecting a pattern toward a far-field object, or when projecting a pattern in a bright environment (e.g., outdoors). Furthermore, the elongated arrangement of the secondary emitters can project a sharp vertical array, for example, making it easier to illuminate the entire scene with projected light, because the projected pattern can be moved laterally in only one direction to illuminate the entire scene.
[0034] In one set of embodiments, the secondary emitters in each elongated array are incoherent with each other. This can reduce speckle contrast in the projected light. For example, for arrays including n The primary emitter of the secondary emitters can reduce speckle noise by a factor of √n. This improves the resolution of the projected pattern, enabling imaging of small or fast-moving objects in the scene. A motion diffuser can be provided, which can additionally or alternatively reduce speckle noise in the projected pattern.
[0035] According to the present invention, scene imaging can be performed without the need for scanning. This would be useful in small devices or environments where scanning is difficult to perform, such as endoscopes, microrobots, or micro-drones.
[0036] Furthermore, because a series of patterns can be projected rapidly, embodiments of the present invention can obtain high-resolution images at a distance very close to the object being imaged. For example, the present invention can be applied to fingerprint scanners with a working distance of approximately 10 mm to 50 mm, or even to endoscopes in which tissue is imaged very close to the imaging device.
[0037] The applicant has recognized that this is novel and inventive in itself, and therefore, from another perspective, the present invention includes an endoscope for 3D imaging of in vivo tissues of a human or animal subject using structured light, comprising: an endoscope lumen; an array of independently addressable light-emitting modules arranged in the lumen to project light onto the tissue; and one or more cameras arranged in the lumen to detect projected light from each light-emitting module, wherein each light-emitting module is arranged to project light with a different pattern; and wherein the cameras are arranged to detect light reflected by the tissue.
[0038] Typically, in 3D imaging using structured light, the only light projected onto the scene is the structured light itself. However, the applicant recognizes that projecting flat-field light with a specific spectrum or wavelength can be useful in certain situations. For example, this is useful when illuminating a specific wavelength of light makes features that would otherwise be invisible visible. One such case is when the device is used for imaging blood vessels; using light of a specific wavelength as a flat-field source allows for imaging of blood oxygenation and tissue structure. Therefore, this would be useful when an independently addressable array of light-emitting modules forms part of an endoscope.
[0039] Therefore, in one set of embodiments, the system or device includes a flat-field light source arranged to project light with a different spectrum compared to the light projected by the independently addressable light-emitting module array. The flat-field light source may be arranged on the same substrate as the independently addressable light-emitting module array.
[0040] In some embodiments, the flat-field light source is arranged to project light with multiple different spectra at different times. For example, the flat-field light source may include at least two light sources arranged to project different spectra, or the flat-field light source may include an adjustable filter. For example, the flat-field light source may include one or more LED emitters, such as those projecting red, green, blue, and / or near-infrared light.
[0041] To obtain a 3D image of a scene, an independently addressable light source (IOLS) array typically projects light during a series of exposures, each at a different spatial frequency and / or lateral position. A flat-field light source can be controlled to project light before, after, and / or between exposures of the IOLS array; that is, the flat-field light source can be controlled to project light at different times compared to the IOLS array. The flat-field light source can project different spectra during each projection. For example, to accurately obtain a 3D image, 12 to 15 exposures of structured light from the IOLS array may be required. Therefore, by using a flat-field light source to project different spectra (preferably before, after, or between different exposures), the camera can obtain a multispectral image, which can be mapped by a processor onto a 3D point cloud generated by imaging the structured light. Thus, this arrangement enables the generation of multispectral 3D images without significantly increasing the size or cost of the device (e.g., due to the simplicity of the light source required by the flat-field light source) and without requiring any significant additional time to perform multispectral measurements.
[0042] Optionally, the flat-field light source can project flat-field light simultaneously with the structured light projected by the independently addressable light-emitting module array. For example, the camera may include a filter, and / or the flat-field light source and the independently addressable light-emitting module array may include narrowband sources, allowing the camera and processor to distinguish between the structured light and the flat-field light. This can further reduce the time required to generate multispectral images.
[0043] The applicant has recognized that this arrangement is novel and inventive in itself, and therefore, from another perspective, the present invention provides an apparatus for 3D imaging using structured light, the apparatus comprising: an independently addressable array of light-emitting modules, wherein each light-emitting module is arranged to project light with a different pattern; a second light source arranged to project light of a different spectrum compared to the array of light-emitting modules; and one or more cameras arranged to detect the projected light from each light-emitting module.
[0044] The second light source can be part of an independently addressable array of light-emitting modules. However, in one set of embodiments, the second light source is separate. This is particularly suitable when a subset of the second light sources is arranged to project flat-field light according to an embodiment. For example, the array of light-emitting modules can project structured light of a specific wavelength, while the second light source (e.g., one or more emitters such as VCSELs or LEDs) can be arranged to project flat-field light of different wavelengths. For example, no diffractive optical element array may be arranged above the second light source.
[0045] In one set of embodiments, the independently addressable light-emitting module array includes a diced wafer stack, i.e., manufactured by etching multiple wafers, aligning and stacking the wafers, and then dicing the wafer stack into multiple individual devices (i.e., multiple individual light-emitting module arrays). This means that only a set of wafers needs to be aligned, rather than many individual components, which greatly reduces the time and cost of assembling multiple light-emitting modules.
[0046] The light-emitting module array may include an emitter wafer and an optical wafer. The emitter wafer includes at least one light source (e.g., multiple main emitters), and the optical wafer includes at least one optical element (e.g., a diffractive optical element). The optical wafer may include a diffractive optical element etched on its top side and spacers etched on its bottom side to separate the diffractive optical element from the emitter array.
[0047] In another example, the light-emitting module array comprises three stacked wafers, such as an emitter wafer including at least one light source, a spacer wafer, and an optical wafer including at least one optical element. In this arrangement, the optical wafer does not need to have the spacer etched on its bottom side; instead, the optical element can be etched on both sides of the optical wafer.
[0048] In one set of exemplary embodiments, a large emitter wafer is etched with a plurality of light sources, and a large optical wafer is etched with a plurality of optical elements. Larger emitter wafers are stacked below and aligned with the optical wafers such that each light source is aligned with at least one optical element. The large stacked wafers are then diced into individual diced wafer stacks, each diced wafer stack comprising an array of light-emitting modules.
[0049] The applicant has further recognized that integrating components for structured light imaging with an array of light-emitting modules for projecting structured light is useful, for example, to eliminate the need to align the camera with the light-emitting modules; and to enable imaging in environments with very limited space, such as in endoscopic inspection applications. Therefore, in another aspect, the present invention includes an apparatus for 3D imaging using structured light, comprising: an independently addressable array of light-emitting modules disposed on a common substrate, wherein each light-emitting module is arranged to project light with a different pattern; and one or more cameras arranged to detect the projected light from each light-emitting module. The light-emitting module array and the cameras are integrated in a common housing.
[0050] In one set of embodiments, the camera includes one or more image sensors arranged on a common substrate with an array of independently addressable light-emitting modules. For example, the image sensors may be arranged on the same substrate as the main emitter array. The image sensors may be arranged adjacent to the emitter array, for example, two image sensors may be arranged on opposite sides of the emitter array.
[0051] In another set of embodiments, the camera includes one or more image sensors on a first substrate, and an independently addressable array of light-emitting modules is arranged on a second substrate above the one or more image sensors. For example, the device may include one image sensor, with the independently addressable array of light-emitting modules covering a portion of the image sensor. This can divide the image sensor into two virtual image sensors, a design that is advantageous because reading data from a single image sensor is simpler than reading from two sensors, thus simplifying synchronization and interface connections between the image sensors, light-emitting modules, and processor. This can further reduce the size and cost of the device, as fewer wires and a smaller connector area are required. This arrangement advantageously allows the device to be fabricated as a diced wafer stack placed on top of the image sensors.
[0052] The device may also include one or more imaging lenses disposed on a lens substrate above the one or more image sensors, wherein each imaging lens is arranged to focus light onto a corresponding image sensor. Thin imaging lenses (e.g., refractive, diffractive, or metasurfaces) can be fabricated on a single substrate. These can provide good image resolution, particularly for monochromatic light. For example, in embodiments where each light-emitting module includes optical elements, the one or more imaging lenses may be disposed on the same lens substrate as the optical element array; for example, two imaging lenses may be disposed on opposite sides of the optical element array.
[0053] In one set of exemplary embodiments, the imaging lens includes refractive surfaces on both sides of the lens substrate, while optical elements (e.g., diffractive optical elements) are etched or stamped on one side of the lens substrate. Thus, in one set of embodiments in which each light-emitting module includes at least one light source (e.g., a main emitter array) and optical elements, one or more image sensors and light sources are arranged on a common substrate, the optical element array and imaging lens are arranged on the lens substrate, and the lens substrate is arranged on top of the common substrate.
[0054] In some embodiments, multiple devices are manufactured using fan-out wafer-level packaging. This allows multiple devices to be manufactured on the same substrate at once and then diced into individual devices. For example, image sensors, emitter arrays, and / or additional light sources (e.g., flat-field light sources) to be used in multiple devices are arranged on an interposer wafer. A large optical wafer can be etched with many optical elements and stacked and bonded on top of the interposer wafer. The interposer wafer and the optical wafer can then be diced into individual devices to efficiently produce many devices at once.
[0055] In some embodiments, the camera includes at least two image sensors on a first substrate with a space between them, and an independently addressable light-emitting module array disposed on a second substrate within the space between the image sensors. For example, a chip including the independently addressable light-emitting module array can be positioned between the image sensors using fan-out wafer-level packaging technology. A lens substrate can be further disposed on the first and second substrates. This arrangement allows the device to be manufactured as a diced wafer stack without the independently addressable light-emitting module array covering the image sensors.
[0056] In one set of embodiments, the system further includes at least one micromechanical actuator arranged to move at least one of the optical elements relative to a corresponding light source.
[0057] For example, the actuator can be arranged to move an optical element, and / or the actuator can be arranged to move a light source. The actuator can be attached to the optical element and / or the light source. Different light-emitting modules can have their own actuators, or the optical element / light source array can be moved by a common actuator. The actuator can introduce a small lateral movement (e.g., a movement between about 1 µm and 10 µm) between the light source and the optical element.
[0058] Lateral movement introduced by actuators can shift the position of the projected pattern. For example, by moving each light source relative to its corresponding optics, the angle between the light source and the optics changes, thus changing the lateral position of the projected pattern. This arrangement can be used to introduce additional lateral movement when each light-emitting module includes multiple master emitters, thereby activating different master emitters to change the lateral position of the projected light. For example, the actuators can enable the lateral position of the light to be moved by a different amount than the possible displacement allowed by the arrangement of the master emitters. This is useful when many different lateral positions are needed to completely cover the scene, and therefore positioning the master emitter array for each desired lateral position of the light to be projected may be impractical.
[0059] The actuator can be arranged to have a continuous range of motion, but in one set of embodiments, the actuator is arranged to move in discrete steps. For example, the actuator can be arranged to move between certain predetermined positions corresponding to the position of each light source relative to each diffractive optical element (which corresponds to certain lateral positions, such as 0 degrees, 120 degrees, and 240 degrees).
[0060] The actuator may include a piezoelectric actuator or an electrostatic actuator. The actuator may be controlled by a processing unit.
[0061] In an exemplary embodiment, a light source array (e.g., a VCSEL) is integrated onto a common substrate, which is spring-connected to a piezoelectric actuator (e.g., a stack of piezoelectric materials). When a voltage from 0 V to 100 V is applied to the piezoelectric actuator, the entire light source array will move approximately 1 µm to 10 µm relative to the corresponding array of optical elements (e.g., a diffractive optical element array). The voltage applied to the piezoelectric actuator can depend on the desired lateral movement. For example, a voltage of 100 V can cause the light source array to move approximately 10 µm. With a 1 mm distance between the light source array and the optical element array, the pattern is laterally moved by a distance of 10 mrad, which is sufficient to produce a 240-degree phase shift in a periodic line pattern with a spatial frequency of approximately 1 line / degree.
[0062] In one set of embodiments, the actuators are arranged to move each light source relative to its corresponding optical element along one dimension. For example, each light source may move along a straight line (e.g., using a linear actuator). However, this is not necessary—the actuators (or another actuator) may be arranged to move each light source along two dimensions, such as moving each light source along a direction parallel to a plane of the light-emitting module array.
[0063] The applicant has recognized that, in addition to employing actuators to provide the lateral movement described above, actuators can also be used to provide another mechanism for generating different patterns. Therefore, from another perspective, the present invention provides a system for 3D imaging using structured light, the system comprising: an array of optical elements; at least one light source arranged to direct light to at least one optical element in the array of optical elements; an actuator arranged to provide movement of the light source and the array of optical elements relative to each other, such that light with different patterns is projected from the array of optical elements when the light source and the array of optical elements are in different relative positions; one or more cameras arranged to detect the projected light from the array of optical elements; and a processing unit arranged to generate a depth map based on the projected light detected by the one or more cameras.
[0064] In another aspect, the present invention provides an apparatus for 3D imaging using structured light, the apparatus comprising: an array of optical elements; at least one light source arranged to direct light to at least one optical element in the array of optical elements; and an actuator arranged to provide movement of the light source and the array of optical elements relative to each other, such that light with different patterns is projected from the array of optical elements when the light source and the array of optical elements are in different relative positions.
[0065] Therefore, it can be seen that, according to these aspects of the invention, the actuator is arranged to move the light source relative to the optical element array to project different light patterns. For example, the actuator may be connected to either the light source or the optical element array. Those skilled in the art will understand that, since the light source can move relative to the optical element array, multiple different patterns can be projected by moving the light source to different relative positions, for example, by illuminating different optical elements in the optical element array that produce different patterns. Therefore, a single light source can be used to project multiple different patterns.
[0066] Embodiments of these aspects of the invention enable the rapid and efficient projection of a series of different light patterns for 3D imaging. For example, different light patterns can be different dot patterns, stripe patterns, or any other desired pattern. Projecting different light patterns from the same device allows the scene to be imaged in 3D without the need for back-and-forth scanning to cover the entire scene. Therefore, embodiments of the invention are simpler, smaller, faster, and more energy-efficient compared to existing projectors based on scanning mirrors or laser triangulation.
[0067] As mentioned earlier, the camera can detect projected light reflected from objects present in the scene being imaged, and can analyze it to determine distortions in different patterns to form a depth map.
[0068] In one set of embodiments, the optical element array includes a diffractive optical element array. Each diffractive optical element in the array can be arranged to project a different pattern. Each diffractive optical element can be selected to project a specific pattern, for example, by using a microstructured surface undulation profile to shape light transmitted from a light source into a desired distribution through diffraction. For example, each diffractive optical element can be fabricated to project a specific pattern suitable for a given application, such as a sparse dot matrix for facial recognition, and / or to project a pattern with a specific spatial frequency.
[0069] In one set of embodiments, the actuator is arranged to move the light source and the optical element array relative to each other to project different patterns by illuminating different optical elements at different times. Alternatively, the actuator is arranged to move the light source and the optical element array such that the light source illuminates the same optical element, but from different angles at different times, for example, by moving the light source by about 0.5 µm to 3 µm to illuminate the same optical element. By illuminating the optical element from different angles, the lateral position of the pattern projected by the optical element can be moved. The advantages of doing so have been summarized above in the preceding aspects of the invention.
[0070] In one set of embodiments, the actuator includes a micromechanical actuator, such as a piezoelectric actuator. The actuator can be controlled in response to feedback from the processing unit. Of course, more than one actuator can be provided, for example, providing a push-pull arrangement and / or providing movement in more than one direction.
[0071] In one set of embodiments, the actuator operates at discrete steps, for example by positioning the light source and array at one of a plurality of predetermined relative positions. For example, these positions may correspond to the projection of a desired pattern from a particular optical element. These positions may also additionally or optionally vary the lateral position of the projected pattern, for example, 0 degrees, 120 degrees, or 240 degrees.
[0072] The light source may include LEDs, lasers (e.g., edge-emitting lasers), or any other light source suitable for projection onto the scene, as will be understood by those skilled in the art. In one set of embodiments, the light source includes a vertical-cavity surface-emitting laser (VCSEL). In another set of embodiments, the light source includes an emitter array, such as a VCSEL array.
[0073] Features of any aspect or embodiment described herein may be applied, where appropriate, to any other aspect or embodiment described herein. When different embodiments or different groups of embodiments are referred to, it should be understood that these are not necessarily different but may overlap. Attached Figure Description
[0074] Some preferred embodiments of the invention will now be described by way of example only with reference to the accompanying drawings, in which: Figure 1 This is a perspective view of a first device for 3D imaging using structured light according to the present invention; Figure 2a Is using Figure 1 A schematic diagram of the first pattern of light projected by the device shown; Figure 2b Is using Figure 1 A schematic diagram of the second pattern of light projected by the device shown; Figure 3aIs using Figure 1 A schematic diagram of the first lateral position of the light projected by the device shown; Figure 3b Is using Figure 1 A schematic diagram of the second lateral position of the light projected by the device shown; Figure 3c Is using Figure 1 A schematic diagram showing the third lateral position of the light projected by the device. Figure 4 yes Figure 1 A schematic diagram of an alternative layout for the light source of the device shown in Figure 2; Figure 5 It is a block diagram of a system for operating a device to perform 3D imaging; Figure 6a This is a three-dimensional cross-sectional view of a second embodiment of a device for 3D imaging using structured light; Figure 6b yes Figure 6a A 3D view of the equipment in the middle; Figure 7 It is used for operation Figure 6a and Figure 6b A block diagram of the system of devices in the diagram; Figure 8 It is possible to be with Figure 1 The device shown or Figure 6a and Figure 6b A schematic diagram of a 3D imaging system used in conjunction with the devices shown. Figure 9 This is a perspective view of a device for 3D imaging using structured light according to a third embodiment of the present invention; Figure 10 This is a schematic diagram of a device for 3D imaging using structured light according to a fourth embodiment of the present invention; and Figure 11a This is a perspective view of a device for 3D imaging using structured light according to a fifth embodiment of the present invention; Figure 11b yes Figure 11a Another perspective view of the device shown; Figure 12 It includes Figure 9 , Figure 10 or Figure 11a A three-dimensional view of the endoscope of the device shown. Detailed Implementation
[0075] Figure 1A 3D imaging device 100 is shown, comprising an array of emitter groups 106a to 106d positioned on the surface of a substrate 102. An array of diffractive optical elements 104a to 104d is arranged in a layer that, in use, sits above the substrate layer to which the emitter groups 106a to 106d are mounted, such that light emitted from each emitter group passes through a corresponding diffractive optical element in the array of diffractive optical elements 104a to 104d.
[0076] Each emitter group 106a to 106d is independently addressable, meaning that a single emitter group can be turned on or off independently of the other emitter groups. Each diffractive optical element diffracts the light from the corresponding emitter group 106a to 106d, such that different light patterns are projected from each diffractive optical element; more specifically, each diffractive optical element projects light with different spatial frequencies, i.e., different periodic light patterns.
[0077] Each transmitter group 106a, 106b, 106c, 106d consists of a subarray of three elongated main transmitters 108a, 108b, 108c, which are spaced apart from each other by a distance x. The main transmitters in each transmitter group 106a to 106d may have different spacing.
[0078] Each main transmitter 108a, 108b, 108c consists of a column of seven secondary transmitters 110. The secondary transmitters 110 constituting each main transmitter 108a, 108b, 108c are controlled using the same control signal, enabling them to effectively function as line light sources. Figure 1 As shown, the main emitter 108b in the emitter group 106b is turned on, so that light is projected along path 103 onto the corresponding diffractive optical element 104b.
[0079] Each secondary emitter 110 is an individual vertical-cavity surface-emitting laser (VCSEL). VCSELs are considered a good choice because they are highly efficient, smaller than other existing coherent light sources, and emit light with a narrow emission angle and narrow spectral width, which allows for the projection of sharp, collimated patterns. Furthermore, VCSELs can be modulated at high speeds, which is useful when the light-emitting modules are sequentially and rapidly turned on and off to quickly project a series of patterns (e.g., patterns with different spatial frequencies).
[0080] Figure 1 The illustrated device 100 includes four emitter groups 106a to 106d, each having four different corresponding diffractive optical elements 104a to 104d. Therefore, the device 100 is capable of projecting light with four different spatial frequencies. Figure 2a and Figure 2bThe diagram shows two different patterns 109 and 111 projected by two different emitter groups 106b and 106c and corresponding diffractive optical elements 104b and 104c. For example, Figure 2a The pattern 109 shown is... Figure 2b The pattern 111 shown has different spatial frequencies.
[0081] The secondary emitters 110 constituting each primary emitter 108a, 108b, 108c are mutually incoherent. This incoherence between the secondary emitters helps to reduce the speckle contrast of the projected pattern. Within the primary emitters, the secondary emitters are spaced approximately 10 µm to 100 µm apart.
[0082] Each master transmitter 108a, 108b, 108c within each transmitter group 106a to 106d is independently addressable, and typically during operation of device 100, when transmitter groups 106a to 106d are activated, only one of their master transmitters 108a to 108c is activated at any given time. This allows the position of the light pattern emitted from each transmitter group 106a to 106d to be moved by activating different master transmitters 108a, 108b, 108c.
[0083] Because each master emitter 108a, 108b, 108c is spaced a distance x from its adjacent master emitter, each master emitter illuminates the corresponding diffractive optical elements 104a to 104d from a different angle. This causes a lateral shift in the projected light pattern depending on which master emitter 108a, 108b, 108c is illuminated, as shown in the reference... Figure 3a , Figure 3b and Figure 3c As shown. The distance x determines the relative lateral position of the light emitted from the main emitter in each emitter group. For example, the distance x can be between about 1 µm and 10 µm, which allows the projected light pattern to produce meaningful phase shifts of 0 degrees, 120 degrees, and 240 degrees.
[0084] Figure 3a A schematic diagram shows a first light pattern 113a projected from a first main emitter 108a in emitter group 106b using device 100. The light passes through corresponding diffractive optical elements 104b of emitter group 106b. For reference, a stationary reference point 115 is shown. Figure 3a As shown, the stationary reference point 115 is located to the left of the dark stripe of the first light pattern 113a.
[0085] Figure 3b A schematic diagram is shown showing a second light pattern 113b projected using the second main emitter 108b in emitter group 106b. The second main emitter 108b is spaced apart from the first main emitter 108a by a distance x, therefore the second light pattern 113b is relative to... Figure 3a The first light pattern 113a shown is shifted. Since the first main emitter 108a and the second main emitter 108b are both within the same emitter group 106b, and both project light through the same diffractive optical element 104b, they project patterns with the same spatial frequency but shifted laterally relative to each other. This is evident from the fact that the stationary reference point 115 is positioned at the center of the dark stripe of the second light pattern 113b, therefore the second light pattern 113b shifts to the right relative to the first light pattern 113a.
[0086] Figure 3c A schematic diagram shows a third light pattern 113c projected from the third main transmitter 108c in transmitter group 106b. The third main transmitter 108c is spaced apart from the second main transmitter 108b by a distance x, therefore the third light pattern is relative to... Figure 3b The second light pattern 113b shown in the diagram moves. Therefore, the projected light pattern moves slightly in space depending on which of the main emitters 108a to 108c initially emitted the light from.
[0087] By addressing different main transmitters 108a to 108c, the projected pattern can be moved to three different lateral positions, so that the entire scene is covered by the projected pattern.
[0088] This operation is repeated for each emitter group 106b to 106d, where each emitting module projects a different pattern. By illuminating the scene four times, each time with a different pattern, the ambiguity caused by imaging with a single pattern can be resolved, thus enabling accurate imaging of the entire scene.
[0089] For reference Figure 1 Each emitter group 106a, 106b, 106c, 106d is aligned with its corresponding associated diffractive optical elements 104a, 104b, 104c, 104d. As shown with reference to FIG2, each diffractive optical element is manufactured to project a vertical array of lines with a specific spatial frequency. However, any other desired pattern can be selected—it does not necessarily have to be a regular pattern. The selected pattern will generally be chosen based on the scene to be imaged by device 100. For example, for forming a depth map involving facial recognition, projecting a sparse dot matrix is preferable; however, for complex scenes imaging many objects at different depths, a more complex pattern is preferable.
[0090] Therefore, as will be understood, device 100 is capable of emitting twelve different light patterns because there are four emitter groups 106a to 106d, each emitter group having a corresponding diffractive optical element 104a to 104d to project four different light patterns (with four different spatial frequencies), and each master emitter 108a to 108c of a given group can assign one of three different phases to the pattern.
[0091] Each VCSEL emits light at the same wavelength, for example, about 940 nm. The typical size of each diffractive optical element ranges from 200 µm to 500 µm, while the resolution of the diffractive optical element pattern should be on the order of wavelength or smaller. The spacing between diffractive optical elements should be as small as possible, but not so small that light from one VCSEL can enter an adjacent diffractive optical element.
[0092] Those skilled in the art will understand that device 100 may include more or fewer than the four emitter groups 106a to 106d shown, for example, only two emitter groups and two corresponding diffractive optical elements and projecting only two different patterns. Each main emitter may also consist of more or fewer secondary emitters, for example, only a single secondary emitter may constitute each main emitter, such as when it is desirable to project many different light patterns while keeping the area of the emitter group small.
[0093] Figure 4 Show Figure 1 An alternative arrangement to the secondary transmitters 112 shown in Figure 2, wherein the secondary transmitters 112 constituting each primary transmitter 108a, 108b, 108c are offset from each other. This would be useful when the required spacing between primary transmitters is less than the minimum distance between secondary transmitters allowed by the hardware (e.g., the minimum distance between VCSELs).
[0094] The main transmitter can be quickly turned on and off, allowing the scene to be imaged rapidly, for example by a camera or a pair of cameras.
[0095] The device 100 can be used to image a given scene by sequentially projecting a series of patterns from each emitter group. Different main emitters can be activated at different times throughout the measurement period to project patterns with different lateral positions to achieve a complete image of the scene.
[0096] Figure 5 The system is schematically illustrated for operating a system that performs 3D imaging using structured light. Processor 118 is connected to two cameras (schematically shown as a block) 114 and... Figure 1The illustrated device 100 has a group 106 of transmitters. A processor 118 is capable of controlling which master transmitters 108 in each transmitter group 106 are activated and when they are activated to emit desired sequences of different patterns. The processor 118 is programmed to cause the device 100 to project a specific pattern with a specific spatial frequency at a given time. Similarly, the processor 118 can control the lateral position of the projected pattern because it is capable of independently controlling each master transmitter 108a to 108c. Therefore, the processor 118 is programmed to project a specific sequence of light patterns over a given area.
[0097] The camera 114 feeds back the detected pattern reflected by the object being imaged to the processor 118. The processor 118 then processes the image detected by the camera 114 to form a depth map, i.e., to image the scene in 3D. It is possible that the processor 118 can also detect the presence of any features that have not yet been adequately resolved, such as any areas missed by the projected pattern, and thus extend the imaging sequence, for example, by projecting more patterns, which makes it possible to image and resolve the missed features and / or areas.
[0098] Figure 6a A schematic diagram of a second device for 3D imaging using structured light according to the present invention is shown. The second device 200 includes a substrate 222, wherein an array of light sources 221, such as VCSELs, is integrated on the surface of the substrate 222. Each light source in the array of light sources 221 is independently addressable, that is, each light source can be turned on and off independently of the other light sources in the array.
[0099] An array of diffractive optical elements 224 is arranged in layers parallel to and spaced apart from the substrate 222. There may not be a one-to-one correspondence between the diffractive optical elements 224 and the light source 221; for example, there may be far more diffractive optical elements 224 than light sources 221. Each diffractive optical element 224 is designed to project a different light pattern. For example, each diffractive optical element 224 may project a vertical line pattern with a different spatial frequency.
[0100] The substrate 222 is connected to an actuator 220, such as a piezoelectric actuator. The actuator 220 causes the substrate 222 to move laterally in a first direction 223 in a plane parallel to and spaced apart from the array of diffractive optical elements 224. Alternatively, a second actuator may be used to move the substrate 222 in an orthogonal second direction.
[0101] Actuator 220 is capable of moving substrate 222 to align different light sources 221 with different diffractive optical elements 224 to project different patterns. Furthermore, actuator 220 is capable of moving each light source a small amount relative to a specific diffractive optical element to change the lateral position of the projected pattern. Those skilled in the art will understand that, theoretically, only a single light source 221 is needed on substrate 222 to sequentially project a series of different patterns and lateral positions by moving the light source to different diffractive optical elements in the array and to different positions relative to a given diffractive optical element. However, it is preferable to have multiple light sources 221, for example, such as... Figure 6a As shown, by combining the movement of the light source 221 achieved by moving the substrate 222 with the actuator 220 and the turning on and off of the light source 221, a sequence of different light patterns can be projected more quickly.
[0102] Figure 6b yes Figure 6a The diagram shows a stereoscopic cross-sectional view of a second device for 3D imaging. The array of light sources 221 is shown as a linear array of five light sources mounted on a substrate. The substrate is mounted on an actuator 220, which in this case is a stack of piezoelectric material connected to a voltage source (not shown). When a voltage between 0 V and 100 V is applied to the stack of piezoelectric material, the substrate will move 0 µm to 10 µm relative to the array of diffractive optical elements 225 along a first direction 223. This 0 µm to 10 µm movement is used to change the lateral position of the light projected from each light source 221. Each light source 221 can be turned on independently of the other light sources 221, thus the second device 200 is capable of projecting five different patterns, or a superposition of patterns, at any given time, wherein each pattern can be moved to many different lateral positions.
[0103] In addition, actuator 220 can additionally induce high-frequency, low-amplitude vibrations in the array of light sources 222 (e.g., vibrations at a rate much higher than the camera frame rate) to reduce the speckle contrast of the detection pattern without blurring the resulting 3D image.
[0104] Figure 7 Schematic illustration of the operation Figure 6a and Figure 6b The second system of the illustrated device. A processor 218 is connected to an array of cameras 214, actuators 220, and light sources 221. The processor 218 is capable of controlling the actuators 220 to move the substrate 222 to different positions relative to the array of diffractive optical elements. The processor 218 is also capable of controlling which light source in the array of light sources 221 is turned on.
[0105] For example, processor 218 is programmed to specify which actuator position 220 corresponds to which corresponding diffractive optical element, which is aligned with the light source, and what pattern (e.g., the spatial frequency of lines) the diffractive optical element projects. Processor 218 is programmed to cause device 200 to project a specific pattern, such as a pattern with a specific spatial frequency. Furthermore, processor 218 can control the lateral position of the projected pattern because it can control actuator 220 to move a specific light source 221 to different positions relative to a given diffractive optical element. Therefore, processor 218 is also programmed to control the lateral position in the same manner as in the first embodiment.
[0106] The processor 218 may also perform the analysis of the image fed back from the camera 214 to form a depth map, i.e., to image the scene in 3D, using the same method as described above.
[0107] Figure 8 For use Figure 1 The device 100 shown or Figure 6a This is a schematic diagram of a system for 3D imaging using device 200. Devices 100 and 200 project a periodic line pattern 224 having a given spatial frequency. The periodic pattern 224 is projected onto an object, such as a sphere 226, and the projected pattern is distorted through interaction with the surface of the sphere 226. The distorted pattern 224' is detected by cameras 114 and 214 and processed by a processing unit to form a depth map.
[0108] Figure 9 A perspective view of a third device 300 for 3D imaging using structured light according to the present invention, comprising an array of emitter groups 308 between a first image sensor 314a and a second image sensor 314b, the entire array of emitter groups 308 being positioned on a common substrate 334. This embodiment is provided in a very compact package, making it suitable for applications where small size is critical, such as placement within an endoscope.
[0109] Lens substrate 332 is positioned above common substrate 334 and includes: a first lens 328a above a first image sensor 314a; a second lens 328b above a second image sensor 314b; and a diffractive optical element array 304 disposed above the array of emitter groups 308. The first lens 328a and the second lens 328b are molded onto lens substrate 332, and the diffractive optical element array 304 is etched (or stamped / imprinted) into lens substrate. This means that the diffractive optical element array 304 and the first lens 328a and the second lens 328b can be manufactured as a single lens substrate 332. The array of emitter groups 308 is mounted such that light emitted from each emitter group passes through a corresponding diffractive optical element (not shown separately) in the diffractive optical element array 304.
[0110] The array of transmitter group 308 shown includes five transmitter groups, and the diffractive optical element array 304 includes five diffractive optical elements (not shown separately), wherein each transmitter group is independently addressable, and each diffractive optical element has a different spatial frequency. Each transmitter group projects light through a corresponding diffractive optical element, thus the device 300 of the third embodiment can project light with different spatial frequencies, as shown in the reference. Figure 1 As stated above.
[0111] Each group of transmitters in the array of transmitter group 308 is similar to the reference. Figure 4 The described transmitter group, i.e., each transmitter group includes three offset VCSEL transmitter lines. Each offset line is independently addressable, and therefore can project light independently of the other transmitters. Thus, each transmitter group can project light in three phases to shift the light pattern projected by device 300 to three different locations, as shown in the reference. Figure 1 Therefore, device 300 can project light with five different spatial frequencies, each projected at three different locations. Those skilled in the art will understand that the number of transmitter groups in the array of transmitter groups 308 and the number of offset lines within each transmitter group can be selected according to the requirements of a given application.
[0112] Device 300 also includes housing 336 and electrical connector 330, which connects the first image sensor 314a and the second image sensor 314b, as well as the array of transmitter group 308, to an external processor and controller (e.g., as shown in reference). Figure 7 (as described).
[0113] Light from the array of emitter group 308 is projected onto the scene, and the projected light is then detected by the first image sensor 314a and the second image sensor 314b. The first lens 328a and the second lens 328b focus the incident light onto their respective first image sensors 314a and 314b. Thus, device 300 provides the projection and detection of structured light in a single package. This is useful in applications where it is difficult to accommodate and align a separate camera, such as in endoscopic examinations. The two image sensors 314a, 314b are connected to a processor, for example, as shown in reference... Figure 5 The operation is performed as described. This arrangement means that the emitter group 308 can be easily aligned with its corresponding diffractive optical elements in the diffractive optical element array 304 by aligning the common substrate 334 and the lens substrate 332 (i.e., each emitter group does not need to be aligned with each diffractive optical element individually).
[0114] Figure 10 This is a schematic diagram of a fourth device 400 for 3D imaging using structured light according to the present invention. Device 400 includes a single large image sensor 414 and an array of emitter groups 408. An array of diffractive optical elements 404 is positioned above the array of emitter groups 408, such that each emitter group 408 has a corresponding diffractive optical element 404. In this embodiment, the image sensor 414 is positioned directly below the array of emitter groups 408 and is positioned on a different substrate. Therefore, the array of emitter groups 408 blocks the center of the image sensor 414 from detecting light, dividing the image sensor 414 into two parts, one on each side of the emitter group 408. These two sensor parts act as two virtual cameras, for example, as shown in reference... Figure 9 The operation is performed as follows. The array of image sensor 414 and emitter group 408 is connected to the processor to control the operation of the array of emitter group 408 and determine which part of image sensor 414 is detecting light.
[0115] This arrangement means that device 400 requires only one image sensor, which further reduces size and manufacturing cost compared to a third device 300. The array of emitter groups 408 is arranged vertically above the image sensor to avoid obstructing a large portion of the image sensor. This approach is also particularly suitable for wafer fabrication processes to produce the emitters and diffractive optics by dicing stacked wafer sandwich layers to create individual modules placed above the corresponding image sensors.
[0116] Figure 11a and Figure 11bThis is a perspective cross-sectional view of a fifth device 500 for 3D imaging using structured light according to the present invention. The fifth device 500 includes an array of emitter groups 508 located between a first image sensor 514a and a second image sensor 514b, all positioned on a common substrate 534. Two flat-field light sources 538a and 538b are arranged on either side of the array of emitter groups 508, diagonally opposite the first image sensor 514a and the second image sensor 514b.
[0117] Equipment 500 is similar to... Figure 9 The third embodiment described operates as follows. However, in addition to projecting a structured light pattern using the array of emitter group 508, device 500 also uses one or both of the first flat-field light source 538a and the second flat-field light source 538b to project flat-field light. The flat-field light sources 538a and 538b are operated by the processor to project light before, after, and between exposures to the array of emitter group 508 when imaging an object. This enables image sensors 514a and 514b to acquire multispectral images of the object. For example, the first flat-field light source 538a and the second flat-field light source 538b can project light of different wavelengths, allowing device 500 to quickly switch between projecting different wavelengths of light.
[0118] This can help in acquiring 3D images of objects whose features are only visible under certain lighting conditions, such as blood vessels. This can be used, for example, when device 500 is mounted at the end of an endoscope, as shown in reference [reference needed]. Figure 12 As described above. By incorporating flat-field light sources 538a, 538b, image sensors 514a, 514b, and transmitter group 508 into device 500, a single device can be used for multispectral three-dimensional imaging in applications where it is inconvenient to introduce too many devices (e.g., in endoscopy).
[0119] Figure 12 Showing includes Figure 11a The diagram shows a three-dimensional view of the endoscope 600 of the device 500. Those skilled in the art will understand that device 500 can also be... Figure 9 The device 300 shown or Figure 10 The device shown is 400.
[0120] Endoscope 600 includes a flexible housing 640, which is inserted into human or animal tissue to be imaged. A distal portion 642 extends from the flexible housing 640, and device 500 is mounted at the end of the distal portion 642. Associated wiring, such as wiring for powering and transmitting data from device 500, extends from device 500 through the flexible housing 640.
[0121] Therefore, the endoscope 600 enables the device 500 to perform 3D imaging inside the human or animal body by using the device 500 to project and detect structured light and optional flat field light.
[0122] Those skilled in the art will understand that the invention has been described by way of one or more specific embodiments, but is not limited to these embodiments. Many variations and modifications are possible within the scope of the appended claims.
Claims
1. A device for three-dimensional imaging using structured light, comprising: An array of independently addressable light-emitting modules arranged on a common substrate, wherein each light-emitting module is arranged to project light with a different pattern; and One or more cameras are arranged to detect projected light from each light-emitting module. The light-emitting module array and the camera are integrated in a common housing.
2. The device according to claim 1, wherein, The one or more cameras are arranged on the common substrate.
3. The device according to claim 1, wherein, The one or more cameras are arranged on different camera substrates, and the independently addressable light-emitting module array is disposed above the camera substrate.
4. The device according to any preceding claim, comprising one or more imaging lenses disposed on a lens substrate above the one or more cameras, wherein, Each imaging lens is arranged to focus light onto the corresponding camera.
5. The device according to any of the preceding claims, comprising a flat-field light source arranged to project light with a different spectrum compared to the light projected by the independently addressable light-emitting module array.
6. The device according to claim 5, wherein, The flat-field light source is arranged to project light at different times compared to the independently addressable light-emitting module array.
7. The device according to claim 5 or 6, wherein, The flat-field light source is arranged to project light with at least two different spectra.
8. The device according to any of the preceding claims, wherein, Each light-emitting module includes at least one light source and optical element.
9. The device of claim 8 further comprises at least one micromechanical actuator arranged to move at least one of the optical elements relative to a corresponding light source.
10. The device according to claim 8 or 9, wherein, At least one of the optical elements is a diffractive optical element.
11. The device according to any of the preceding claims, wherein, Each light-emitting module is arranged to emit patterns with different spatial frequencies.
12. The device according to any of the preceding claims, wherein, The projected light from each light-emitting module is collimated in a first direction and diverged in a second direction, wherein the first direction is perpendicular to the second direction.
13. The device according to any of the preceding claims, wherein, Each light-emitting module includes multiple main emitters, wherein each main emitter is arranged to project a light pattern at a different lateral position compared to another of the multiple main emitters.
14. The device according to claim 13, wherein, The multiple main transmitters are arranged in the same plane.
15. The device according to claim 13 or 14, wherein, Each primary transmitter includes an elongated array of secondary transmitters.
16. The device according to claim 15, wherein, The secondary transmitters in each elongated array are mutually incoherent.
17. A system for three-dimensional imaging using structured light, comprising the device according to any of the preceding claims, and A processing unit is arranged to generate a depth map based on the projected light detected by the one or more cameras.
18. An endoscope for three-dimensional imaging of internal tissues of human or animal subjects using structured light, comprising: Endoscopic lumen; An independently addressable light-emitting module array is arranged in the lumen to project light onto the tissue; as well as One or more cameras, arranged within the cavity, are used to detect projected light from each light-emitting module. Each light-emitting module is arranged to project light with a different pattern; and The camera is configured to detect light reflected by the tissue.
19. A device for three-dimensional imaging using structured light, comprising: An independently addressable array of light-emitting modules, wherein each light-emitting module is arranged to project light with a different pattern; A second light source is arranged to project light of a different spectrum compared to the light-emitting module array; One or more cameras are arranged to detect projected light from each light-emitting module.
20. The system according to claim 19, wherein, The second light source is arranged to project flat field light.
21. A system for three-dimensional imaging using structured light, comprising: Optical element array; At least one light source, said at least one light source being arranged to direct light to at least one optical element in the optical element array; An actuator is arranged to provide movement of the light source and the optical element array relative to each other, such that light with different patterns is projected from the optical element array when the light source and the optical element array are in different relative positions; One or more cameras are arranged to detect projected light from the optical element array; as well as A processing unit is arranged to generate a depth map based on the projected light detected by the one or more cameras.
22. The system according to claim 21, wherein, The optical element array includes a diffractive optical element array.
23. The system according to claim 21 or 22, wherein, The actuator is arranged to move the light source and the array of optical elements relative to each other to project different patterns by illuminating different optical elements at different times.
24. The system according to any one of claims 21 to 23, wherein, The actuator includes a micromechanical actuator.
25. A device for three-dimensional imaging using structured light, comprising: Optical element array; At least one light source, said at least one light source being arranged to direct light to at least one optical element in the optical element array; as well as An actuator is arranged to provide movement of the light source and the optical element array relative to each other, such that light with different patterns is projected from the optical element array when the light source and the optical element array are in different relative positions.
