Acquisition of 3D image data using a light-section method

Description

[0001] The invention relates to a camera and a method for recording 3D image data using a light section method according to the preamble of claims 1 and 15, respectively.

[0002] In a light-section method, a light pattern is projected into the detection area, and the height profile of the objects is deduced from the distortions using a triangulation method. Laser triangulation using a line light source is particularly common. The detection is typically performed in relative motion to combine the respective two-dimensional height profiles into 3D image data. In some cases, the level is also measured, thus capturing a brightness image or texture.

[0003] Figure 8illustrates the well-known Scheimpflug arrangement, in which a laser line can be recorded sharply over the entire measuring range in order to acquire high-quality 3D image data. In this case, the camera 100 with its lens 102 is constructed such that its image plane 104, the lens plane 106, and the sharpness or focal plane 108 intersect in a common straight line. However, due to the Scheimpflug arrangement and the chief ray angle of the lens 102, the light hits the image sensor of the camera 100 at an angle. The effect of the Scheimpflug arrangement affects all pixels equally, whereas the chief ray angle has individual effects for each pixel, namely at 0° for a central pixel and then increasing towards the edges of the image sensor. The Scheimpflug arrangement thus reduces the light yield, since not all of the light reaches the respective pixel due to the oblique incidence of the light. In addition, cameras constructed in a Scheimpflug arrangement are quite large.

[0004] Figure 9 illustrates a conventional solution for increasing the light sensitivity of an image sensor. Each of the Figure 9a-c shows an exemplary pixel 110 with its light-sensitive surface 112 and a microlens 114 arranged in front. With perpendicular light incidence as in Figure 9a The light is thereby focused onto the light-sensitive surface 112. In the case of oblique incidence of light as in Figure 9b However, there is a lateral shift, which leads to a partial or complete miss of the light-sensitive surface 112. To compensate, as in Figure 9cThe microlens 114 is offset δx relative to the light-sensitive surface 112. However, the offset δx must match the angle of incidence. This requires a very expensive, custom design, which must be repeated for each sensor variant. The light that misses the light-sensitive surface 112 not only reduces the light sensitivity of the image sensor but also creates stray light, which further degrades image capture.

[0005] Another negative effect of oblique light incidence is multiple reflections in a glass cover. This doesn't occur with perpendicular light incidence; there is only a harmless back reflection in the direction of the light source, which does not reach the image sensor. Anti-reflective coatings also exhibit an angle dependence.

[0006] In the paper Howard, James W.: "Formulas for the coma and astigmatism of wedge prisms used in converging light," Applied Optics 24.23 (1985): 4265-4268, various applications of wedge prisms are discussed. Among other things, they can be used to tilt the image plane or correct its tilt. However, this effect is always accompanied by aberrations. They can exhibit either no coma or no astigmatism, but never prevent both. Their use in laser triangulation, which is not addressed in the paper anyway, would therefore introduce additional measurement errors.

[0007] The paper by Reshef, Orad, et al., "An optic to replace space and its application towards ultra-thin imaging systems," Nature communications 12.1 (2021): 1-8, deals, without any connection, with laser triangulation using so-called metal lenses and space plates. Metal lenses are extremely thin optical elements with a lens effect that feature special nanostructures to influence beam paths. Space plates, in turn, use similar technologies to target the area between the lenses. Like conventional lenses, multiple metal lenses must initially maintain a certain distance from each other. The space plates are intended to reduce this distance in order to further reduce the depth of an optic. A special process for producing such nanostructures is nanoimprinting.

[0008] DE 10 2021 119 423 A1 discloses a triangulation probe that uses an optical metaelement in its reception path to compensate for various unfavorable triangulation effects. Such a triangulation probe measures only scalar or one-dimensionally; this should not be confused with a generic camera for capturing 3D image data using a light-section method or laser triangulation. Camera-specific issues such as a Scheimpflug arrangement and optimization in both axes of an image sensor do not arise at all in DE 10 2021 119 423 A1. In DE 10 2021 122 418 A1, the displacement of the received light spot in the near field is specifically limited by an optical metaelement.

[0009] US 2019 / 0170314 A1 presents a LARP (Laser Activated Remote Phosphor) device. Various embodiments of the device include a wedge prism, a metal lens, and a Scheimpflug arrangement, but these are not closely related. The document has nothing to do with laser triangulation.

[0010] US 10 746 529 B1 discloses an optical device for measuring an offset with an image sensor and a receiving lens in a Scheimpflug arrangement using a light-section method. To suppress reflections, the cover glass is arranged offset and at an angle to the image sensor.

[0011] It is therefore an object of the invention to improve the acquisition of 3D image data using a light section method.

[0012] This object is achieved by a camera and a method for recording 3D image data using a light-section method according to claim 1 and 15, respectively. The term light-section method generally refers to the acquisition of 3D image data from distortions of a projected light pattern by the object contours using triangulation, and in particular laser triangulation, in which a light line is projected as the light pattern. A light source, in particular a laser, projects the light pattern or the light line in a focal plane. An image sensor with a plurality of light-receiving elements or pixels, which are typically arranged in a matrix, records the light pattern on the objects struck by it. A receiving optics with an objective plane is arranged upstream of the image sensor. A control and evaluation unit evaluates the recorded profile of the light pattern to determine the 3D image data. The image plane is tilted relative to the focal plane.

[0013] The invention is based on the basic idea of ​​providing at least one optical metaelement in the receiving optics. This compensates for oblique light incidence on the light receiving elements. The oblique light incidence results in particular, as discussed in the introduction, from the tilt between the image plane and focal plane and / or a respective chief ray angle. Depending on the embodiment, the metaelement acts on the remitted light beam in various ways and replaces and / or supplements the receiving optics. For this purpose, the metaelement has a metasurface and / or a metamaterial, i.e. nanostructures that specifically form certain wavefronts of the remitted light beam. With a metasurface, such nanostructures are provided on the surface; a metamaterial achieves corresponding properties through the nanostructure of a layer system or a solid body.

[0014] The invention has the advantage that smaller angles of incidence on the image sensor can be realized. This improves light sensitivity and, depending on the embodiment, either mitigates the disruptive side effects of a Scheimpflug arrangement or even achieves the advantages of a Scheimpflug arrangement without an actual Scheimpflug arrangement, which in turn enables more compact designs and simplifies manufacturing. In some embodiments, the height resolution can be linearized. The optical metaelement facilitates variant development, since for different designs or different focal planes, only a different metaelement needs to be used, or the metaelement can even be adapted to the other design or can also fulfill its function in the other design.The optical metaelement can perform the tasks of the receiving optics with minor lens aberrations or compensate for lens aberrations of a lens still used in the receiving optics. The multiple reflections in a glass cover mentioned above can be avoided thanks to the optical metaelement.

[0015] The metaelement preferably creates a prism effect. This changes the angle of incidence of the incoming light before it hits the image sensor. An oblique incidence of light onto the image sensor is thus reduced to a perpendicular incidence, or at least an oblique angle of incidence is reduced toward a perpendicular incidence.

[0016] The metaelement preferably exhibits a different optical effect across the light-receiving elements. Individual compensation for each light-receiving element can be incorporated into the metaelement, but this would not be feasible using conventional methods. It is conceivable to compensate specifically for groups of light-receiving elements, such as rows or columns, in the metaelement in a uniform manner within each group.

[0017] The receiving optics preferably comprise a microlens array arranged upstream of the image sensor. This allows the effects of a microlens array for focusing received light onto the light-sensitive areas of the image sensor and the optical metaelement provided according to the invention to be combined.

[0018] The microlenses of the microlens array are preferably offset from the light-receiving elements. The offset is traditionally used to compensate for the oblique incidence of light, which is achieved according to the invention by the optical metaelement. Therefore, a standard microlens array can be used without offset, which has advantages in terms of flexible application possibilities in different camera variants and cost. Nevertheless, this embodiment remains possible with a combination of offsetting the microlenses and using a metaelement.

[0019] The metaelement favors vertical light incidence into the respective microlens. This even completely compensates for oblique light incidence.

[0020] The metaelement preferably also functions as a microlens array. In this embodiment, a microlens array can be dispensed with, yet a comparable optical focusing effect on the respective light-sensitive area of ​​a light-receiving element can be achieved. In addition to compensating for the oblique incidence of light, the metaelement itself ensures appropriate focusing on the light-sensitive areas of the image sensor.

[0021] In other conceivable combinations, the metaelement only partially replaces the function of the microlens array, so that the microlens array and metaelement complement each other. It is also conceivable that two or more metaelements share the required optical functionality.

[0022] The receiving optics preferably comprise a receiving lens. In such embodiments, there is therefore still a conventional receiving lens or a conventional receiving objective.

[0023] The image sensor is preferably tilted relative to the receiving lens in a Scheimpflug arrangement. The image plane, an objective plane of the receiving lens, and the focal plane thus intersect in a common straight line. The optical metaelement ensures that, despite the tilt of the image sensor, the received light hits the light receiving elements as perpendicularly as possible.

[0024] The image sensor and the receiving lens are preferably arranged such that the image plane and the objective plane are parallel. This alternative avoids the Scheimpflug arrangement in a physical perspective. However, the optical metaelement emulates the advantageous optical effects of the Scheimpflug arrangement.

[0025] The metaelement is preferably designed to focus light from the focal plane onto the image sensor. The optical metaelement thus assumes the function of the receiving lens, preferably alone or alternatively in combination with the receiving lens or as part of a receiving optics system. As already mentioned, there can be multiple metaelements, for example, one metaelement for the function of the receiving lens and another metaelement to compensate for oblique light incidence, or a single metaelement can perform both.

[0026] The metaelement is preferably an active metaelement. This means a metaelement whose optical properties are adaptable. Thus, it can be adapted for a specific focal plane or tilt, as well as for specific cameras or recording situations. This eliminates the need to produce multiple metaelements, for example, for camera variants or specific applications; or, at least to a certain extent, camera variants can be dispensed with altogether by adapting them using an active metaelement.

[0027] The receiving optics preferably comprise a space plate. A German technical term for this has not yet been established. A space plate is also a metamaterial, but is not primarily designed to alter the wavefront according to a tilt or lens effect. Rather, it is intended to create a longer light path in a smaller physical space in order to be able to arrange optical elements, particularly metal lenses, closer together. In particular, it is conceivable to provide a front and / or back of the space plate with a metasurface in order to combine the effects of a space plate with those of the optical meta element. A space plate enables a particularly compact structure in embodiments in which the optical meta element also assumes a dual function as a receiving lens. For further details on space plates, please refer to the work by Reshef et al. mentioned in the introduction.

[0028] The metaelement is preferably designed for multifocal image acquisition. This allows for sharp capture of different focal planes at different tilts. This is useful for capturing multiple regions of interest (ROIs), particularly in the case of multiple laser lines or when using image sensor lines for special purposes, such as some edge lines as color lines.

[0029] The metaelement preferably exhibits anamorphic properties, i.e., different focus properties along the two axes of the image sensor. This would require additional lenses with conventional optical means, which would require additional space and cost. The metaelement can fulfill such requirements if necessary.

[0030] The method according to the invention can be further developed in a similar manner and thereby exhibits similar advantages. Such advantageous features are described by way of example, but not exhaustively, in the subclaims following the independent claims.

[0031] The invention will be explained in more detail below with regard to further features and advantages, using exemplary embodiments and with reference to the accompanying drawings. The figures of the drawing show: Fig. 1 shows an overview of a camera for 3D image capture using a light section method or laser triangulation and its detection range; Fig. 2 shows a schematic representation of an embodiment of the camera in a Scheimpflug arrangement to compensate for oblique light incidence using an optical metaelement; Fig. 3 shows a schematic representation of an embodiment of the camera without a physical Scheimpflug arrangement, but with a comparable optical effect using an optical metaelement; Fig. 4 shows a schematic representation of an embodiment of the camera in which the optical metaelement additionally takes on the function of the receiving lens; Fig. 5 shows a schematic representation of an embodiment of the camera with an additional space plate; Fig. 6 shows a schematic representation of an embodiment of the camera with multiple projected illumination lines and recording areas; Fig.Fig. 7 is a schematic representation of a multifocal embodiment of the camera; Fig. 8 is a schematic representation of the conventional Scheimpflug arrangement; and Fig. 9 is a schematic representation of the conventional use of microlenses, in particular offset microlenses, in front of an image sensor.

[0032] Figure 1shows an overview of a camera 10 for 3D image acquisition using a light section method or by means of laser triangulation in an exemplary recording situation. The principle of laser triangulation is known per se and will therefore only be introduced briefly. An illumination unit 12 generates a fan of light 14 in a focal plane and ultimately projects a line of light onto the object 16 to be recorded. The three-dimensional contour of the object 16 distorts the line of light. In a recording from camera 10, the shape of the line of light is evaluated in order to infer the three-dimensional contour based on knowledge of the geometric arrangement and properties of camera 10 and illumination unit 12. In this case, only a two-dimensional height profile is created for each recording, corresponding to the section of the object under the line of light. By means of relative movement, for example of a conveyor belt 18, the line of light is gradually guided over the object 16.This results in the 3D image data 20 being created by successively combining the two-dimensional height profiles.

[0033] Figure 2shows a schematic representation of an embodiment of the camera 10 in a Scheimpflug arrangement. For the sake of simplicity, only the receiving optics of the camera 10 (which will be explained later), an image sensor 22 with the light receiving elements or pixels (not shown individually), and a control and evaluation unit 24 in which the triangulation evaluation takes place are shown. The control and evaluation unit 24 can be an internal processing unit, an external processing unit, or a combination of both. Examples of an internal processing unit are digital computing components such as a microprocessor or a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), an ASIC (Application-Specific Integrated Circuit), a KI processor, an NPU (Neural Processing Unit), a GPU (Graphics Processing Unit), or the like.An external computing unit can be a controller or a computer of any type, including notebooks, smartphones, tablets, as well as a local network, an edge device, or a cloud.

[0034] In this embodiment, the camera 10 is constructed in a Scheimpflug arrangement. A single receiving lens 26 is shown as the receiving lens. Scheimpflug arrangement means, as already explained in the introduction, Figure 8 explains that the image plane of the image sensor 22, the objective plane of the receiving lens 26, and the focal plane 28 intersect each other in a common straight line. As a necessary, but not sufficient, condition for this, the focal plane 28 is tilted relative to the image plane.

[0035] To compensate for the oblique incidence of light onto the image sensor 22, an optical metaelement 30 is provided in front of the image sensor 22. In addition, a microlens array 32 can optionally be used, preferably between the optical metaelement 30 and the image sensor 22. The microlenses can be arranged as in Figure 9a a standard component with centered microlenses or as in Figure 9c be equipped with an offset, the latter being no longer necessary due to the optical metaelement 30, but nevertheless possible, for example by the offset and the optical metaelement 30 dividing the required angle compensation between themselves.

[0036] The optical metaelement 30 comprises a metamaterial or a metasurface, the latter preferably created by nanoimprinting. Conventional optical components such as lenses, prisms, waveplates, or holograms rely on the propagation of light over distances much greater than the wavelength of the light fan 14 to form wavefronts. In this way, significant changes in the amplitude, phase, or polarization of light waves are gradually accumulated along the beam path. The optical metaelement 30, on the other hand, comprises structures that can be considered miniature anisotropic light scatterers or resonators, or optical antennas. These structures have dimensions and spacing in the nanometer range, much smaller than the wavelength of the light fan 14.Thus, the optical metaelement 30, following the Huygens principle, shapes optical wavefronts into any desired shape with subwavelength resolution. The nanostructures introduce spatial variations in the optical response of the light scatterers. This allows the simulation of optical effects, particularly those of lenses or prisms. A special feature is the high flexibility of adapting the nanostructures to achieve a desired output wavefront and thus a wide variety of optical effects. Depending on the wavelength range, materials with suitable transmission properties are used, for example, titanium dioxide, silicon nitride, or gallium phosphide in the visible spectral range and aluminum nitride in the ultraviolet spectral range, as well as chalcogenide alloys in the mid- and silicon in the long-wave infrared range.

[0037] The optical metaelement 30 in the camera 10 is expected to compensate for the angle of incidence of the received light onto the image sensor 22, in particular individually for each light-receiving element. This is comparable to the effect of wedge prisms, but in a metamaterial, this effect can be achieved without aberrations. The simulated wedge prisms can set a specific deflection angle for each individual light-receiving element and thus ensure perpendicular light incidence onto the light-receiving element. In this case, the oblique light incidence due to the Scheimpflug arrangement and / or a chief ray angle dependent on the location on the image sensor can be compensated. Given this specific optical effect, the required nanostructure can be calculated and generated. With conventional optical elements, such compensation in the structure size of individual pixels would be impossible.The optical metaelement 30 can also be easily exchanged to accommodate a different camera variant or application situation. The required optical effects can be realized in a single optical metaelement 30 as shown or, alternatively, distributed across two or more optical metaelements.

[0038] Figure 3shows a schematic representation of an embodiment of the camera 10 without a physical Scheimpflug arrangement, but with a comparable optical effect by means of an optical metaelement 30. The optical effect of the Scheimpflug arrangement is a variation of the image distance, i.e. the distance between the image-side main plane and the image sensor 22 across its light-receiving elements. This serves to compensate for the effects of an inclined object plane and thus a varying object distance. A sharp image with a varying object distance can be achieved by a changed focus position across the image sensor 22 instead of by the physical Scheimpflug arrangement. The optical metaelement 30 is designed accordingly in this embodiment: It generates a focus position for each light-receiving element that corresponds to a physical inclination of the image sensor 22 in a Scheimpflug arrangement.A variation of the focus position by the optical metaelement 30 is possible individually for each light-receiving element, but preferably only via the height direction, i.e. the same direction in which the focal plane 28 is tilted to the image plane of the image sensor 22. This can be referred to as an emulated Scheimpflug arrangement using the optical metaelement 30. In addition, it is conceivable to compensate for the chief ray angle for each light-receiving element in the optical metaelement 30, corresponding to the optical effect of offset microlenses. In this regard, reference is made to the explanations for the embodiment according to . Figure 2 The functions of an emulated Scheimpflug arrangement and a compensation of the chief ray angle can be distributed over two or more optical metaelements or implemented in a single optical metaelement 30.

[0039] Figure 4shows a schematic representation of another embodiment of the camera 10. In contrast to Figure 3 The optical metaelement 30 additionally takes over the function of the receiving lens 26 or the receiving lens and is accordingly arranged at a greater distance from the image sensor 22, approximately in the position of the replaced receiving lens 26. The optical metaelement 30 thus receives additional focusing properties as a metal lens, like the replaced receiving lens 26. Superimposed on this focusing is the Figure 3described function of an emulated Scheimpflug arrangement, particularly in combination with compensation of the chief ray angle. Again, the various optical functions can be implemented in a single optical metaelement 30 or distributed across at least two optical metaelements, here in particular with a position of another optical metaelement close to the image sensor 22. As an additional optical function of the same optical metaelement 30 or another optical metaelement, the height resolution can be linearized, as described in the documents DE 10 2021 119 423 A1 and DE 10 2021 122 418 A1 cited in the introduction, now transferred from the conventional one-dimensional probe to the camera 10.

[0040] Figure 5 shows a schematic representation of an embodiment of the camera 10. In Figure 4The optical metaelement 30, which replaces the receiving lens 26, must be arranged at a distance from the image sensor 22. A space plate 34 is now added for a more compact design. As briefly explained in the introduction, a space plate 34 is a special optical metaelement that can replace a longer light path through the air in a very small space. Thanks to the space plate 34, it is therefore again possible to arrange the optical metaelement 30 very close to the image sensor 22. The entire receiving optics can thus be realized with a minimal installation depth as a type of sandwich of image sensor 22, optional microlens array 32, space plate 34, and optical metaelement 30, particularly in a monolithic design. A space plate can also be used in the other embodiments, but does not achieve the same compactness there due to the continued presence of the classic receiving lens 26.

[0041] Figure 6shows a schematic representation of an embodiment of the camera 10 in an arrangement that can be referred to as a reverse Scheimpflug arrangement. This is particularly suitable for the case of multiple recording areas; in the example shown, these are two laser lines from two illumination units 12a-b and RGB areas 36. The two laser lines are preferably offset from one another in a conveying direction to enable multiple measurements. In this case, the focal plane is not the plane of the light fan 14a-b, which is no longer clear anyway, but a ground plane of the objects, such as the surface of the conveyor belt 18. The arrangement shown is particularly suitable for flat objects, for example when testing wood.

[0042] Figure 7 shows a schematic representation of a multifocal embodiment of the camera 10. This is based on the Figure 3explained embodiment and can be transferred to the other embodiments. The optical metaelement 30 supports multiple focus positions, so that multiple regions of interest can be captured with different triangulation parameters or focal planes 28a-b. This allows, for example, line-by-line color recording in addition to laser triangulation. The reverse Scheimpflug arrangement according to Figure 6 for different object distances or object heights and not just flat objects.

[0043] As a further variant without a separate figure, it is conceivable to give the receiving optics anamorphic properties via the optical metaelement 30, i.e. different focus properties in the two axes of the image sensor 22. This is particularly true for the embodiment according to Figure 4Suitable for this purpose. Anamorphic properties, for example, allow an image to be compressed or stretched in width while maintaining the same height. With conventional lenses, this would only be possible with considerable effort, requiring additional lenses and the additional space required.

Claims

1. A camera (10) for recording 3D image data (20) using a light sectioning process that has an illumination unit (12) to project a light pattern (14) in a focal plane (28), an image sensor (22) having a plurality of light reception elements in an image plane, a reception optics (26) upstream of the image sensor (22) and having an objective plane, and a control and evaluation unit (24) that is configured to generate the 3D image data (20) by evaluating the light pattern (14) in a recording of the image sensor (22), wherein the image plane is tilted with respect to the focal plane (28), characterized in that the reception optics (26) has at least one metaelement (30) that compensates an oblique light incidence on the light reception elements to implement smaller angles of incidence on the image sensor (22) and thus to alleviate disturbing side effects of a Scheimpflug arrangement or toe achieve the advantages of a Scheimpflug arrangement without an actual Scheimpflug arrangement.

2. A camera (10) in accordance with claim 1, wherein the metaelement (30) generates a prismatic effect.

3. A camera (10) in accordance with claim 1 or claim 2, wherein the metaelement (30) has a different optical effect over the light reception elements.

4. A camera (10) in accordance with any one of the preceding claims, wherein the reception optics (26) has a microlens array (32) upstream of the image sensor (22).

5. A camera (10) in accordance with claim 4, wherein the microlenses of the microlens array (32) have an offset with respect to the light reception elements.

6. A camera (10) in accordance with claim 4 or claim 5, wherein the metaelement (30) provides a perpendicular light incidence in the respective microlens.

7. A camera (10) in accordance with any one of the preceding claims, wherein the metaelement (30) additionally has the function of a microlens field.

8. A camera (10) in accordance with any one of the preceding claims, wherein the reception optics has a reception lens (26).

9. A camera (10) in accordance with claim 8, wherein the image sensor (22) is tilted with respect to the reception lens (26) in a Scheimpflug arrangement.

10. A camera (10) in accordance with claim 8 or claim 9, wherein the image sensor (22) and the reception lens (26) are arranged with respect to one another such that the image plane and the objective plane are in parallel.

11. A camera (10) in accordance with any one of the preceding claims, wherein the metaelement (30) is configured to bundle light from the focal plane (28) on the image sensor (22).

12. A camera (10) in accordance with any one of the preceding claims, wherein the metaelement (30) is an active metaelement that is adaptable in its optical properties.

13. A camera (10) in accordance with any one of the preceding claims, wherein the reception optics (26) has a spaceplate (34).

14. A camera (10) in accordance with any one of the preceding claims, wherein the metaelement (30) is configured for a multifocal image recording; and / or wherein the metaelement (30) has anamorphic properties.

15. A method of recording 3D image data (20) in accordance with the principle of the light sectioning process, in which a light pattern (14) is projected in a focal plane (28), an image sensor (22) having a plurality of light reception elements in an image plane records the light pattern (14) by a reception optics (26) upstream of the image sensor (22) and having an objective plane, and the recorded light pattern (14) is evaluated to generate the 3D image data (20), wherein the image plane is tilted with respect to the focal plane (28), characterized in that the reception optics (26) has at least one metaelement (30) that compensates an oblique light incidence on the light reception elements to implement smaller angles of incidence on the image sensor (22) and thus to alleviate disturbing side effects of a Scheimpflug arrangement or toe achieve the advantages of a Scheimpflug arrangement without an actual Scheimpflug arrangemen.t