Method and apparatus for acquiring and relating multispectral 2D image data to 3D image data from optical triangulation

Abstract

To provide methods and arrangements for obtaining multispectral 2D image data and associating the data with 3D image data from light triangulation.SOLUTION: A method comprises: obtaining 601 3D image data as first sensor positions (SP1s) of first images (IM1s) corresponding to locations of intensity peaks of a reflected first light beam from an object as part of the light triangulation; obtaining 602 two or more second images (IM2s) by means of multispectral two or more second light beams; selecting 603 respective second sensor positions (SP2s) in the respective IM2s for the respective SP1s; and associating 604 intensity values of the selected SP2s with the SP1s; so that multispectral 2D data corresponding to the intensity values in the SP2s from the reflected multispectral second light beams become associated with the 3D data corresponding to the SP1s.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] SUMMARY OF THE INVENTION Embodiments herein relate to methods and arrangements for acquiring and correlating multispectral 2D image data with 3D image data from light triangulation. [Background technology]

[0002] Industrial vision cameras and systems for factory and logistics automation are often based on three-dimensional (3D) machine vision, in which 3D images of a scene and / or object are captured. By 3D image, we refer to an image that includes information about pixels in only two dimensions (2D), e.g., intensity and / or color, as in traditional images, but also “height” or “depth” information. That is, each pixel of the image may include such information that is associated with the pixel’s position in the image and maps to the position of what is imaged, e.g., the object. Processing may then be applied to extract information about the object’s properties, i.e., its 3D characteristics, from the 3D image and convert it, e.g., into various 3D image formats. Such information about height is sometimes referred to as range data, where range data may thus correspond to data from height measurements of the object being imaged, or, in other words, from range or distance measurements of the object. Alternatively or additionally, the pixel may include information about material properties, e.g., regarding the scattering of light within the imaged area or the reflection of specific wavelengths of light, etc.

[0003] Thus, the pixel value may relate to, for example, the intensity, and / or range data, and / or material properties of the pixel.

[0004] For example, line scan image data occurs when image data for an image is scanned or provided one line at a time by a camera having a sensor configured to detect and provide image data one pixel line at a time.

[0005] A special case of line scan imaging is image data provided by so-called "sheets of light," or light planes, triangulation. Lasers are often preferred, but other light sources can also be used, for example light sources that can provide light that remains focused and does not diverge significantly, i.e., "structured," light, such as that provided by lasers or light-emitting diodes (LEDs).

[0006] 3D machine vision systems are often based on such optical triangulation. In such systems, a light source illuminates an object with structured light corresponding to a particular light pattern, such as a light plane, which produces a light or laser line on the object and along which 3D properties of the object corresponding to the object's contours are captured. By scanning the object with such a line, i.e., performing a line scan, with movement of the line and / or object, 3D properties of the entire object can be captured corresponding to multiple contours.

[0007] A 3D machine vision system or device that is based on optical triangulation, and that uses, for example, a sheet of light for optical triangulation, may be referred to as a system or device for 3D imaging based on light or a sheet of light, triangulation, or simply laser triangulation when laser light is used.

[0008] Typically, to create a 3D image based on optical triangulation, reflected light from an object to be imaged is captured by a camera's image sensor, and intensity peaks are detected in the image data. The peaks occur at positions corresponding to positions on the imaged object where incident light, e.g., corresponding to a laser line, is reflected from the object. The positions of the detected peaks in the image are then mapped to positions on the object from which the light producing the peaks was reflected.

[0009] Optical or laser triangulation systems, i.e., 3D imaging systems based on optical triangulation, typically project light or laser rays onto an object to create a height curve from the object's surface. By moving the object relative to the associated cameras and light sources, information about the height curve from different parts of the object is captured by images, which can then be combined and used with knowledge of the system's relevant geometry to create a three-dimensional representation of the object, i.e., provide 3D image data. This technique can be described as capturing an image of light, typically a laser line, as it is projected onto and reflected by the object, and then extracting the position of the reflected light within the image. This is successfully accomplished, for example, by using conventional peak-finding algorithms to identify the location of intensity peaks within the image frame. Although not required, the imaging system is typically set up so that intensity peaks can be searched for row by row of sensors, and the position within the row is mapped to height or depth.

[0010] In many applications, it is desirable to obtain not only 3D image data, such as the height and depth of the object, but also traditional 2D image data of the object, for example, to provide texture to a 3D model of the object formed from the 3D image data so that the model can look and / or be an even better representation of the real-world object. Additionally or alternatively, it may be important to obtain certain 2D information from the surface of the object, for example, for quality assurance reasons. It may be particularly important to obtain such information for the same locations as the 3D image data, i.e., so that the 2D image data is for and / or aligned with 3D locations that correspond to actual locations on the object from which both the 3D image data and the 2D image data were captured. In the case of optical triangulation, it is therefore important to obtain 2D image data associated with intensity peak locations. Using 2D image data associated with 3D image data locations, it may be possible, for example, to analyze the 2D image surface of the 3D model of the object to identify, for example, text, scratches, marks, and / or color variations, and, if they are also associated with height variations, etc., to be able to identify where these are located on the actual object.

[0011] Grayscale 2D image data from reflectance and intensity can be obtained, for example, using the same light, e.g., laser, as used in optical triangulation to provide 3D image data. However, in the case of lasers, due to the special properties of laser light, so-called speckle noise often occurs, which then appears in the 2D image. It can also be seen that the 2D information that can be obtained in this way from the surface of an object is quite limited. Summary of the Invention [Means for solving the problem]

[0012] In view of the above, the object is to provide one or more improvements or alternatives to the prior art, such as providing an improved method of how 2D image data can be related to 3D image data resulting from optical triangulation.

[0013] According to a first aspect of embodiments herein, this object is achieved by a method for associating multispectral 2D image data with 3D image data generated from optical triangulation performed by an imaging system for 3D imaging of an object, the imaging system comprising a first light source for illuminating the object with a first light, a camera with an image sensor, and one or more second light sources for illuminating the object with two or more second lights, the second lights being multispectral by virtue of having different light wavelengths. The optical triangulation includes illuminating successive portions of the object with the first light and detecting, by the image sensor, the first light reflected from each portion in each first image (IM1).

[0014] The method includes acquiring the 3D data as a first sensor position (SP1) of IM1 generated by the camera and image sensor, where SP1 corresponds to the position of a first light intensity peak reflected from an object as part of the optical triangulation.

[0015] The method further includes acquiring two or more second images (IM2), each generated by the camera and image sensor and imaging the object during illumination by the two or more second lights, where each IM2 is either a respective one of the IM1 and associated with it, or generated between two of the IM1 and associated with any of them.

[0016] The method further includes selecting, within each IM2, for and against each SP1 of each IM1 with which the respective IM2 is associated, a respective second sensor location (SP2) at which the second light reflected from the object has a higher intensity than the first light reflected from the object.

[0017] Further, the method includes associating the intensity values ​​of the selected SP2 with the SP1 for which the selected SP2 was selected, respectively, such that the multispectral 2D data corresponding to the intensity values ​​in SP2 from the reflected multispectral second light is associated with the 3D data corresponding to the SP1 for which the SP2 was selected.

[0018] According to a second aspect of an embodiment of the present specification, this object is achieved by one or more devices for associating multispectral 2D image data with 3D image data generated from optical triangulation performed by an imaging system for 3D imaging of an object. The imaging system comprises a first light source for illuminating the object with a first light, a camera with an image sensor, and one or more second light sources for illuminating the object with two or more second lights, the second lights being multispectral by virtue of having different light wavelengths. The optical triangulation includes illuminating successive portions of the object with the first light and detecting, by an image sensor, the first light reflected from each portion in each first image (IM1). The one or more devices are configured to:

[0019] acquiring the 3D data as a first sensor position (SP1) of IM1 generated by the camera and image sensor, SP1 corresponding to a position of a first light intensity peak reflected from an object as part of the optical triangulation;

[0020] acquiring two or more second images (IM2), each generated by the camera and image sensor and imaging the object while illuminated by the two or more second lights, each IM2 being either a respective one of the IM1 and associated with it, or generated between two of the IM1 and associated with any one of them;

[0021] selecting, within each IM2, for and against each SP1 of each IM1 with which the respective IM2 is associated, a respective second sensor position (SP2) at which the second light reflected from the object has a higher intensity than the first light reflected from the object.

[0022] Associating the intensity values ​​of the selected SP2 with the SP1 for which the selected SP2 was selected, respectively, so that the multispectral 2D data corresponding to the intensity values ​​in SP2 from the reflected multispectral second light becomes associated with the 3D data corresponding to the SP1 for which the SP2 was selected.

[0023] According to a third aspect of embodiments herein, this object is achieved by one or more computer programs comprising instructions which, when executed by one or more processors, cause said one or more devices according to the second aspect to perform the method according to the first aspect.

[0024] According to a fourth aspect of embodiments of the present specification, this object is achieved by one or more carriers comprising one or more computer programs according to the third aspect, the one or more carriers being one or more of the following: an electronic signal, an optical signal, a radio signal, or a computer-readable storage medium.

[0025] Embodiments herein provide multispectral 2D image data that is correlated with and aligned at the sensor position or pixel level with the 3D image data from optical triangulation using one and the same camera, which may be a conventional, typically monochrome camera, used in conventional 3D imaging based on optical triangulation. Furthermore, the 2D image data will also share a focal point with the 3D image data; for example, when a Scheimpflug is utilized by the imaging system to provide a focal point where the illumination of the first light typically exists in the optical plane, and thus a focal point on the object from which the 3D image data is generated, the multispectral 2D image data will also benefit from this focal point. This is not limited to acquiring 2D image data from illumination solely by the first light used to acquire the 3D image data.

[0026] As a result, 3D imaging of an object based on optical triangulation is enabled, with focused and well-aligned multispectral 2D image data of, for example, textured surface morphology, such as the full color of the 3D object resulting from the 3D imaging. Furthermore, a wide variety of 2D image data can be acquired using different types of light sources and / or illumination, including from different combinations of second light of several different wavelengths and / or illumination from different positions and directions.

[0027] The 2D image data can be the same image (IM1) from which the associated 3D image data is obtained, acquired from IM2, and / or the 2D image data can be from IM2 that is separate from IM1 but generated between IM1, so that the 2D image data is related to and associated with the 3D image data. In the latter case, IM2 is generated within the time period between two consecutive IM1, along with the 3D image data from optical triangulation. While the embodiment where IM1 = IM2 allows for less data processing and facilitates implementation with existing systems, the embodiment using IM2 that is separate from IM1 allows for greater freedom in selecting SP2 and exposure period to use for the second light, and how illumination of the second light can be provided relative to the first light, such as a laser, used by optical triangulation.

[0028] Additionally, embodiments herein enable improved detectability of object surface details, such as defects, thanks to multispectral 2D image data combined and aligned with 3D image data from optical triangulation.

[0029] Example embodiments of the present invention will now be described in more detail with reference to the accompanying schematic drawings, which are briefly described below. [Brief explanation of the drawings]

[0030] [Figure 1] FIG. 1 shows a schematic diagram of an example of a prior art imaging system in which embodiments herein may also be used and / or based. [Figure 2A] FIG. 1 is a diagram illustrating a first simplified example of an imaging system that may be configured to perform optical triangulation and embodiments of the prior application. [Figure 2B] 1 is a flow chart for outlining the method disclosed in the earlier application. [Figure 3A] FIG. 2C is a diagram illustrating a schematic situation for one main group of embodiments disclosed in the prior application and described with respect to FIG. 2B. [Figure 3B] FIG. 2C is a diagram illustrating a schematic situation for one main group of embodiments disclosed in the prior application and described with respect to FIG. 2B. [Figure 4A] FIG. 2 shows a schematic diagram of an example of a first of the main groups. [Figure 4B] FIG. 10 is an example diagram of the second of the main groups. [Figure 4C] FIG. 10 is an example diagram of the second of the main groups. [Figure 5A] FIG. 1 illustrates a simplified example of an imaging system that may be configured to perform optical triangulation and embodiments herein. [Figure 5B] 5B is a diagram of a schematic example of a secondary light source as in FIG. 5A implemented as a single lighting unit comprising multiple secondary light sources. [Figure 5C] FIG. 10 is a diagram illustrating schematically that the second light may be non-overlapping in wavelength. [Figure 5D] FIG. 10 is a diagram illustrating schematically that the second light may partially overlap in wavelength. [Figure 6] 1 is a flow chart for outlining a method according to an embodiment of the present disclosure. [Figure 7A] 3A, but in the case of multispectral second light, and for a method according to an embodiment of the present specification as described with respect to FIG. 6, a diagram schematically illustrating the situation for corresponding main groups. [Figure 7B] 3B, but in the case of multispectral second light, and for a method according to an embodiment of the present specification as described with respect to FIG. 6, a diagram schematically illustrating the situation for corresponding main groups. [Figure 8A] 10A-10C show schematic diagrams of an example with different sequences of the second light. [Figure 8B] 10A-10C show schematic diagrams of an example with different sequences of the second light. [Figure 8C]10A-10C show schematic diagrams of an example with different sequences of the second light. [Figure 8D] 10A-10C show schematic diagrams of an example with different sequences of the second light. [Figure 8E] 10A-10C show schematic diagrams of an example with different sequences of the second light. [Figure 9A] 1A-1C are diagrams of exemplary images and results from real-world and practical applications of embodiments herein. [Figure 9B] 1A-1C are diagrams of exemplary images and results from real-world and practical applications of embodiments herein. [Figure 9C] 1A-1C are diagrams of exemplary images and results from real-world and practical applications of embodiments herein. [Figure 9D] 1A-1C are diagrams of exemplary images and results from real-world and practical applications of embodiments herein. [Figure 10] FIG. 1 is a schematic block diagram illustrating an embodiment of a device for carrying out embodiments herein. [Figure 11] 7 is a schematic diagram illustrating some embodiments of a computer program and its carrier for causing a device to perform the methods and operations described with respect to FIG. 6. DETAILED DESCRIPTION OF THE INVENTION

[0031] The embodiments herein are exemplary embodiments. It should be noted that these embodiments are not necessarily mutually exclusive. Elements from one embodiment may be implicitly assumed to be present in other embodiments, and it will be clear to one skilled in the art how those elements may be used in other exemplary embodiments.

[0032] When an optical triangulation light source, typically a laser, is used to acquire reflectance images, i.e., images with 2D image data of an object, drawbacks include speckle noise in the image and the inability to gather alternative surface information other than that possible from the light used in optical triangulation.

[0033] On the other hand, if an image of the object is acquired separately from the optical triangulation, obtaining a desirable alignment with the 3D image of the measured object as resulting from the optical triangulation is extremely difficult, if possible, in all applications.

[0034] A "multi-scan" approach may be used, where, for example, one or more separate rows of the imager, i.e., image sensor, may be used to collect 2D image data, but this still does not have any good alignment with the 3D image data. Furthermore, it is difficult to create good focus for both the optical triangulation for which the majority of the imager is used and the separate 2D image data readout. Optical triangulation setups typically have a maximum focus only around the light plane used for optical triangulation, typically the laser plane, achieved using Scheimpflug focusing, as well as the ability and desire to use a large aperture opening to allow more light to reach the imager.

[0035] It would be desirable to have a method for acquiring 2D image data that is not limited to using the light used in optical triangulation as such, but at the same time is capable of obtaining 2D image data that is useful and well aligned with the 3D image data from optical triangulation.

[0036] The applicant's previous application, EP22154020.6, relates to a solution to the above. The present application and the embodiments herein are based on the solution of said previous application, but are specifically directed to embodiments involving multispectral 2D image data, i.e., how additional light with different wavelengths can be used to create 2D image data, e.g., full-color image data, together with associated 3D image data. Some of the information disclosed in said previous application is repeated below to (re)introduce some basics and to facilitate understanding of the embodiments herein. However, some reference numbers and nomenclature have been changed to better suit the context of the present application.

[0037] FIG. 1 schematically illustrates an example of a prior art imaging system 100 for 3D imaging based on optical triangulation, on which embodiments herein may be based. The imaging system 100 may alternatively be referred to as, for example, an imaging system for 3D machine vision based on optical triangulation to capture information about 3D properties of an object. The imaging system 100 is shown in the figure in a normal operating state, i.e., typically after a calibration has been performed, and thus the system is calibrated. The imaging system 100 is configured to perform optical triangulation, in which structured light, i.e., a light plane, is used, here in the form of an optical triangulation sheet. The imaging system 100 further comprises a light source 110, such as a laser, for illuminating the object to be imaged with structured light, typically a specific light pattern. In this example and figure, there is structured light 111 in the form of a light plane. The light may be laser light, but need not be; alternatively, it may be light from one or more light-emitting diodes (LEDs), for example. Another example of structured light is a light edge, i.e., the edge of an illuminated area, which is similar to but different from a conventional light plane, but has a similar effect. The generated light and illumination are typically provided through one or more lenses, for example, to focus the light. Furthermore, the camera is typically configured and arranged to have a focal plane co-located, or in other words, aligned, with the structured light 111, typically the light plane, based on the so-called Scheimpflug principle or Scheimpflug focusing. In this way, object reflections occurring at the light plane are focused at the image sensor. In the illustrated example, the object being imaged is exemplified by a first object 120 in the form of a car and a second object 121 in the form of a gear structure. The object being imaged may be referred to as a measurement object. When the structured light 111 is incident on the object, this corresponds to a projection of the structured light 111 on the object, which can be seen when the structured light 111 intersects with the object. For example, in the illustrated example, structured light 111 in the form of a light plane produces light rays 112 on a first measurement object 120 .The structured light 111 is reflected by the object, more specifically, by the portion of the object at the intersection, i.e., in the illustrated example, in ray 112. The imaging system 100 further includes a camera 130 having an image sensor (not shown in FIG. 1 ). The camera and image sensor are positioned relative to the light source 110 and the object to be imaged such that the structured light 111, when reflected by the object, is incident on the image sensor. The image sensor is a device, typically implemented as a chip, for converting incident light into image data. The portion of the object that, by reflection, causes the incident light to be incident on the image sensor can thereby be captured by the camera 130 and image sensor, and corresponding image data can be produced and provided for further use. For example, in the illustrated example, the structured light 111 is reflected in ray 112 on a portion of the car roof of the first object 120 toward the camera 130 and image sensor, which can thereby produce and provide image data along with information about the portion of the car roof. According to the principles of optical triangulation, using knowledge of the geometry of the imaging system 100, e.g., knowledge of how image sensor coordinates relate to world coordinates, such as coordinates of a coordinate system 123, such as Cartesian coordinates, associated with the object being imaged, the image data can be converted into information about 3D properties, e.g., in the form of a 3D shape or contour, of the object being imaged in a suitable format. The information about the 3D properties may comprise data representing the 3D properties in any suitable format.

[0038] By moving light source 110 and / or the object to be imaged, such as first object 120 or second object 121, so that multiple portions of the object are illuminated, and in practice typically by scanning the object with structured light 111, causing reflected light to be detected in the image by the image sensor, image data describing a more complete 3D shape of each object can be produced, corresponding to multiple successive contours of each object, such as the illustrated contour images 141-1 through 141-N of first object 120. Each contour image shows the outline of first object 120 from which structured light 111 was reflected when the image sensor of camera unit 130 detected the light that produced the contour image. As shown in the figure, a movable object support structure 122, such as a conveyor belt or the like, can be used to move and scan the object through structured light 111, while light source 110 and camera unit 130 are typically stationary. Alternatively, the structured light 111 can be moved over the object so that all parts of the object, or at least all parts facing the light source 110, are illuminated and the camera receives light reflected from all parts of the object that it is desired to image.

[0039] As can be appreciated from the above, for example, during imaging of first object 120, each image corresponding to an image frame provided by camera 130 and its image sensor may correspond to or be used to provide one of contour images 141-1 through 141-N. The location of the outline of first object shown in one of contour images 141-1 through 141-N is typically determined based on identifying intensity peaks in the image data captured by the image sensor and locating these intensity peaks. Imaging system 100 and conventional peak-finding algorithms are typically configured to search for intensity peaks for each pixel column within each image frame. If the sensor coordinates are u, v, where, for example, u corresponds to the pixel position along a row in the image sensor as shown in the figure and v corresponds to the pixel position along a column, then for each position there are u image frames searched for peak positions along v, and the identified peaks in the image frames may result in one ``clean'' contour image as shown in the figure, and the entire image frames and contour images may be used to create a 3D image of the first object 120.

[0040] 2A schematically illustrates a simplified first example of an imaging system 200 that can be configured to perform optical triangulation and embodiments of the prior application. The imaging system 200 is based on optical triangulation to capture information about 2D and 3D characteristics of one or more objects, sometimes referred to as measurement objects. The illustrated system corresponds to a basic configuration and comprises: a first light source 210 for illuminating one or more measurement objects 220, here exemplified by object 220 in the figure, with a first light 211, typically laser light, as part of optical triangulation for 3D imaging of the object; a second light source 250 for illuminating object 220 with a second light 251 for 2D imaging of the object; and a camera 230 with an image sensor 231 positioned to detect the first light reflected from object 220 as part of the optical triangulation for 3D imaging and the second light reflected from object 220 for the 2D imaging.

[0041] The camera 230, image sensor 231, and first light source 210 are configured and arranged relative to one another for optical triangulation and thus may be as or based on those used in conventional optical triangulation. For purposes of optical triangulation, i.e., with respect to how the first light source 210, camera 230, image sensor 231 are arranged relative to one another and the measurement object 220, etc., and further configured to perform as described below, system 200 may correspond to system 100 in FIG. 1. Image sensor 231 should, of course, also be sensitive to the second light being used.

[0042] Thus, object 220 may correspond to first object 120 and is shown as being located at least partially within field of view 232 of camera 230. First light source 210 is configured to illuminate measurement object 220 with first light 211, which is the light used for optical triangulation, and thus typically with structured light such as a particular light pattern, e.g., a sheet of light or a plane of light, provided, for example, by a laser. First light 211 is reflected by object 220, and the reflected first light is captured by camera 230 and image sensor 231 as part of optical triangulation. Another example of structured light that may be used as first light is a light edge, i.e., the edge of an area or part where there is illumination.

[0043] The object 220 may be illuminated in this manner, and images may be captured as in conventional optical triangulation. As such, the optical triangulation, as in the prior art, may involve movement of the first light source 210 and / or the object 220 relative to one another, so that at different successive times, different successive portions of the object 220 are illuminated by the first light source 210 and the first light 211, and the reflected first light 211 from the object 220 is detected by the image sensor 231. As in conventional optical triangulation, the camera 230 and the first light source 210 are typically, although not necessarily, fixed relative to one another, and the object 220 moves relative to them. Through the detection by the image sensor 231, each image frame is associated with the respective time at which the image frame was detected, i.e., captured, and with the respective portion of the measurement object 220 from which the image sensor 231 detected the reflected first light 211 at the respective time.

[0044] Camera 230 may be a prior art camera and may correspond, for example, to camera 130 in system 100 of FIG. 1 , and image sensor 231 may be the same or a similar image sensor described above with respect to FIG. 1 . It may be desirable to transfer, e.g., transmit, image frames provided by camera 230 and image sensor 231 and / or information derived from the image frames to a computing device 233, e.g., a computer or the like, for further processing outside of camera 230. Such further processing may additionally or alternatively be performed by a separate computing unit or device (not shown), i.e., separate from image processor 231 but still included within, e.g., integrated with, camera 230 or a unit comprising camera 230. Computing device 233, e.g., a computer or other device (not shown), may be configured to control devices involved in optical triangulation and / or involved in embodiments herein, such that optical triangulation and / or other operations related to embodiments herein are performed.

[0045] As in conventional 3D imaging with optical triangulation, the first light source 210 and the camera 230 are typically positioned at predetermined, fixed locations with a known relationship to each other for optical triangulation. The imaging system 200 also includes a second light source 250, which may also be at a fixed location relative to the camera 230 and the first light source 210, but the exact location and relationship of the camera 230 and the image sensor 231 is not utilized in embodiments herein as in optical triangulation, and therefore the second light source may be positioned more freely for the purpose of providing secondary light illumination.

[0046] FIG. 2B is a flowchart outlining the method disclosed in the prior application. The following operations, which may form the method, are for acquiring 2D image data and associating it with 3D image data. The 3D image data is based on a first intensity peak position resulting from optical triangulation performed by an imaging system, exemplified below by imaging system 200. The first intensity peak position may therefore be the intensity peak position determined in conventional optical triangulation, as will be further described and illustrated below with respect to embodiments herein and in separate figures and examples. Thus, an imaging system capable of performing this method includes a first light source for illuminating an object with a first light and an image sensor for detecting the first light reflected from the object. Hereinafter, the first light source and the first light are exemplified by first light source 210 and first light 211, and the object and image sensor are exemplified by object 220 and image sensor 231.

[0047] As in conventional optical triangulation, optical triangulation comprises:

[0048] Illuminating a first portion of an object 220 with a first light 211 and detecting first light reflected from the first portion by an image sensor 231 during a first exposure period (EXP1-1) to produce a first image (IM1-1) having a first intensity peak occurring at a first sensor position (SP1-1).

[0049] Using the first light 211 to illuminate another second portion of the object 220 adjacent to the first portion, and detecting the first light reflected from the second portion by the image sensor 231 during another first exposure period (EXP1-2), to generate a further first image (IM1-2) having an intensity peak occurring at a further first sensor position (SP1-2).

[0050] Thus, the first image (IM1) corresponds to the consecutive image frame with the first intensity peak at SP1 and is part of the optical triangulation.

[0051] The exposure periods, images, and sensor positions SP1-1 and SP1-2 are further described and illustrated below in separate figures and examples.

[0052] Furthermore, an imaging system such as imaging system 200 further comprises one or more additional second light sources for illuminating object 220 with one or more second lights different from the first light. In the following, the second light sources and second lights are exemplified by second light source 250 and second light 251.

[0053] The following methods and / or operations illustrated in FIG. 2B may be performed by one or more devices, such as a device, i.e., a camera 230 and / or a computing device 233, or by the imaging system 205 and / or a suitable device therein or connected thereto.

[0054] It should be noted that the following actions may be taken in any suitable order and / or may overlap in whole or in part, when this is possible and suitable.

[0055] Operation 201 The object 220 is illuminated using the one or more second lights 251, and the image sensor 231 detects the reflected second light from the object 220 during one or more second exposure periods (EXP2), resulting in one or more second images (IM2), respectively.

[0056] Illumination should be provided such that during any first exposure period (EXP1), any reflected secondary light will result in a lower intensity at the first sensor location (SP1) than the intensity from the reflected primary light. This ensures low secondary light interference of the primary light intensity peaks. This is further explained and illustrated below with respect to Figures 4A-4C.

[0057] The images and exposure periods are described and discussed below with respect to Figures 3A-3B for the two main groups, respectively.

[0058] Operation 202 For each first sensor position (SP1) in the first image, a respective second sensor position (SP2) in the one or more second images (IM2) is selected. The illumination using one or more second lights 251 should be provided such that, during the one or more second exposure periods (EXP2), any reflected second light from the object 220 results in a higher intensity at the selected second sensor position (SP2) than the reflected first light 211. This ensures that the reflected second light is not obscured and / or undesirably interfered with by the reflected first light at the second sensor position (SP2). This can also be achieved, when needed, by suitable selection of SP2 relative to SP1, as described below. See further description and examples with respect to Figures 4A-4C below.

[0059] The second sensor position (SP2) may be selected using a predefined or predetermined relationship with the first sensor position (SP1) resulting from the optical triangulation. For example, it may be predetermined that each SP2 should be selected using a predefined or predetermined relationship with each SP1, such as being at a predefined or predetermined distance from each SP1 and / or in a predefined or predetermined direction. In practice, the preferred direction is typically along the image sensor column, i.e., in the present example, a direction on the image sensor that maps to "height" along v in sensor coordinates, which maps to z in real-world coordinates in the coordinate system used in the present example. More generally, the direction along which SP1 is searched may be the sensor direction along which one intensity peak from the optical triangulation is expected, corresponding to the object point where the first light is directly reflected by the object toward the camera. Because the light distribution in this direction is given by the structured first light, such as corresponding to a light plane or laser plane, and how it projects as a line on the object, how the light intensity decreases in this direction is known or, if needed, can be easily found. Thus, when both the first light and the second light are captured in the same image, as in the case of the first main group described below, a predetermined distance, e.g., a distance in pixels, can be found in advance at which the second light intensity is higher than the first light intensity. The selection of the second sensor position SP2 and the sensor position SP, including what differs between the two main groups, are further described and illustrated below with respect to Figures 4A-4C.

[0060] operation 203 2D image data from each second sensor position (SP2) in the one or more second images (IM2), i.e., the second sensor positions selected in operation 202, is acquired, and the acquired image data is associated with the first sensor position (SP1) for which each second sensor position (SP2) was selected. As a result, the acquired 2D image data becomes associated with the 3D image data, since the 3D image data from the optical triangulation is based on or corresponds to the first intensity peak position at the first sensor position (SP1).

[0061] As used herein, "2D image data" may refer to image data at a location or pixel of an image sensor, e.g., image sensor 231, resulting from exposure to light, or at a corresponding location in an image generated by the image sensor. The 2D image data typically corresponds to light intensity and / or color, e.g., in the form of intensity values ​​that indicate or specify the intensity of the captured light. In embodiments herein, the 2D image data for a location or pixel comprises information about a corresponding location on an object that reflected the second light, i.e., the 2D image data for a location corresponds to image data that has information about how the corresponding location on the imaged object reflects the second light.

[0062] As used herein, "3D image data based on intensity peak locations resulting from optical triangulation" may be understood as data comprising at least depth and / or height information for corresponding locations on the measurement object that reflected the first light that resulted in the intensity peak locations. For example, in the contour images 141-1 through 141-N shown with reference to FIG. 1, the identified intensity peak locations are shown at sensor coordinates u, v (as SP1), and the intensity peak location at v for each location u comprises information about the depth of the imaged object 120, more specifically, depth information about the location on the object 120 that reflected the light 112 that resulted in the intensity peak location at v. In other words, the sensor coordinates, which may be at sub-pixel resolution, of the intensity peak locations correspond to, and thereby comprise, the 3D image data. This may be expressed as each first sensor location, such as that identified by its coordinates, corresponding to, and thus including, the 3D image data.

[0063] In practice, the 3D image data relating to the position on the object that reflected the first light may be the 3D position of the intensity peak position in u, v, t that maps to a position in x, y, z on the object, or it may be the corresponding 3D position in coordinates x', y', z' of a 3D model of the object, the model provided from calculations based on u, v, t and known operating conditions and relationships used in optical triangulation.

[0064] Thus, the method and operation relate to providing 2D image data associated with 3D image data from optical triangulation. The 2D image data relates to the same position on the measurement object from which the 3D image data originates, because the 2D image data is acquired relative to, and preferably together with, the acquisition of the image frame with the first intensity peak (SP1) used for optical triangulation, i.e., the first image (IM1), as described above and further described and illustrated below, and by using the same camera and image sensor, but is not limited to acquiring the 2D image data from illumination by the same light source as that used to acquire the 3D image data. As a result, a wide variety of 2D image data associated with the 3D image data can be acquired based on using different types of light and / or illumination, including, for example, illumination from different directions and positions.

[0065] The second light 251, or, for example, if there are several second light sources, the multiple second lights may be advantageously diffused and / or thus provided by one or more light-emitting diodes (LEDs) on which the second light source 250 may be based. At least one of the second light sources may provide diffuse second light. Diffused light contrasts with the first light 211, such as laser light, which is typically highly directional. Diffused light allows for laser speckle-free reflectivity. Another advantage of diffuse second light is a simpler system setup, where it is easier to achieve suitable intensities at SP2 and also at SP1. Diffused light also more closely resembles normally occurring illumination and may therefore be better for capturing desired 2D image data about the measurement object. Some embodiments may also be easier to implement using diffuse second light.

[0066] 3A-3B schematically illustrate the situation for two main groups of the embodiment disclosed in the previous application and described above with respect to FIG. 2B. In short, FIG. 3A is for a first main group in which the second image (IM2) is the same as the first image (IM1), and therefore the second exposure period (EXP2) is also the same as the first exposure period (EXP1). FIG. 3B is for a second main group in which the second image (IM2) is separate from the first image (IM1), thereby also having its own separate exposure period (EXP2) located between the first exposure periods of the first images IM1-1 and IM1-2, here EXP1-1 and EXP1-2. The second image IM2-1 and its second exposure period EXP2-1 are thus generated in this group between IM1-1 and IM1-2, i.e., between the images used for optical triangulation.

[0067] FIG. 3A for the first main subgroup more specifically and schematically illustrates a second image 342a-1, which corresponds to IM2-1 and is, in this example, the same image as the first image 341a-1 corresponding to IM1-1. The second image 342a-1 corresponds to EXP2-1 and is generated using a second exposure period 362a-1, which in this group is the same as EXP1-1 and the first exposure period 361a-1. Also shown is a further first image 341a-2, corresponding to IM1-2, along with a first exposure period 361a-2, corresponding to EXP2. The illustrated images IM1-1 and IM1-2 therefore correspond to images used for optical triangulation. The first image IM1-1 341a-1 is associated with capture time t1, and the second image 341a-2 is associated with capture time t2. The difference T between t2 and t1 is 3D , i.e., T 3D= t2 - t1 is a time period 363a corresponding to the time period between subsequent images as part of optical triangulation for providing 3D image data, typically corresponding to the scan rate. The shown length in time of the exposure period is merely to show that, as in conventional 3D imaging by optical triangulation, there is also an extension in time involved, and what is shown is approximate and does not represent any actual time period T between subsequent image frames used in optical triangulation. 3D Note that the desired relationship between λ and λ is not shown. The exposure period, and the time period between subsequent images, may be as in conventional optical triangulation, although in some cases it may be important to adjust the exposure period to achieve suitable detection of both the reflected first light 211 and the reflected second light 251 in the same image.

[0068] For the first main group, each second sensor position (SP2) is selected using the respective difference (d) in sensor coordinates to each first sensor position (SP1) within the first image 341a-1, i.e., within IM1-1, as in operation 202 of the method in FIG. 2B.

[0069] 3B, which relates to the second main group, more specifically and schematically illustrates a second image 342b-1, which corresponds to IM2-1 and, in this example, is a separate image from the first image 341b-1, which corresponds to IM1-1. The second image 342b-1 is generated using a second exposure period 362b-1, which corresponds to EXP2-1 and is therefore a new and / or separate exposure period, and as such is not used for optical triangulation, but relates to an image in the time period between such images, here, the first exposure period 361b-1, which corresponds to EXP1-1, and the first further exposure period 361b-2, which corresponds to EXP1-2. The second exposure period 362b-1 is the exposure period for the second image 342b-1, which corresponds to IM2-1. It should be understood that the figure illustrates an example in which the one or more second exposure periods (EXP2) and one or more second images (IM2) are unique. However, in practice there will typically be a further subsequent IM2 with a second exposure period, such as IM2-1 with EXP2-1 followed by IM2-2 with EXP2-2.

[0070] Shown are first images IM1-1, IM1-2, and first exposure periods EXP1-1, EXP1-2, and time period T 3D may be like their counterparts in FIG. 3A and / or as in conventional optical triangulation, the first exposure period for the first image is typically of the same length.

[0071] In both Figures 3A-3B, and in both main groups, the first images IM1-1 and IM1-2 correspond to the images used in and for capturing the first light and thus the optical triangulation described above with respect to Figure 2B, and the second image IM2-1 is for capturing the reflected second light 251 and thereby the 2D image data.

[0072] Compared to the first main group, the difference is that in the second main group, the first light 211 can be, and advantageously is, prevented from illuminating the object 220 during one or more second exposure periods (EXP2), for example by being switched off, attenuated, or directed elsewhere, such as during 362b-1. This is possible thanks to the second image IM2 being separate from the first image IM1, i.e., a separate second image for capturing the reflected second light 251. This can be used to eliminate the risk of light disturbance from the first light, for example a laser, in IM2, and also thereby to allow more freedom in selecting SP2.

[0073] FIG. 4A shows a schematic example of a first one of the main groups, and FIGS. 4B-4C show an example of a second one of the main groups.

[0074] 4A, which relates to the first main group, illustrates light intensity along image sensor coordinate v, corresponding to reflected light from an object, e.g., object 220, such as that included in one of the first images (IM1), detected by image sensor 231 during one of the first exposure periods (EXP1). As noted above, in embodiments according to the first main group, IM2 = IM1 and EXP2 = EXP1. Contributions from reflected first light 211 and reflected second light 251 are shown separated in the figure for illustrative reasons. The contribution from reflected first light, e.g., first light 211 reflected from object 220 and detected by image sensor 231, corresponds to first light intensity distribution 413a in the figure. The contribution from reflected second light, i.e., second light 251 reflected from object 220 and detected by image sensor 231, corresponds to second light intensity distribution 453a in the figure. In other words, the first light intensity distribution 413a is an example of the reflected first light 211, i.e., the reflected first light, and the second light intensity distribution 453a is an example of the reflected second light 251, i.e., the reflected second light. In practice, the light intensity distributions 413a and 453a are superimposed corresponding to adding the distributions shown, thereby forming a single light intensity distribution corresponding to that sensed by the image sensor.

[0075] The first light intensity distribution 413a is typically fairly narrow, with an intensity peak position 471a corresponding to SP1, as is the case when the illumination is structured light, e.g., a light plane, producing a light beam such as a laser line on the object. The second light intensity distribution 453a is typically at a substantially constant level, as is the case when the second light is provided as diffuse light illuminating the object.

[0076] With reference to operation 202, and in connection with FIG. 3A, it was described above that each second sensor position SP2 is selected in the first image IM1 using a respective difference (d) in sensor coordinates to a respective first sensor position (SP1). Second sensor position 473a corresponding to SP2 is shown in FIG. 4A. The difference d is shown in the figure as difference 475a.

[0077] Further, it was mentioned above that under operation 202, the illumination using one or more second lights 251 should be provided such that during the one or more second exposure periods (EXP2), any reflected second light from the object, such as by second light intensity distribution 453a, results in a higher intensity at the selected second sensor location (SP2) than the reflected first light 211. An example of how this may look when IM2 = IM1 and EXP2 = EXP1, as in the case of the first main group, is shown in FIG. 4A, where second light intensity distribution 453a exceeds first light intensity distribution 413a at second sensor location 473a, i.e., at SP2. Thus, the difference in light intensity contributions from the first and second lights at SP2, i.e., I DIFF (SP2), which is shown in the figure as light intensity difference 479a.

[0078] In contrast, the second light intensity distribution 453a is less than the first light intensity distribution 413a at the first sensor location 471a, i.e., at SP1. This is also consistent with what was shown above in connection with FIG. 3A and operation 201, namely, that illumination may be provided such that during the first exposure period (EXP1), any reflected second light should produce a lower intensity at the first sensor location (SP1) than the intensity from the reflected first light. Thus, the difference in light intensity from the first and second light at SP1, i.e., I DIFF (SP1), which is shown in the figure as light intensity difference 477a.

[0079] From the figure, it can be seen why SP2 should be selected using the difference d from SP1 for the first major group, i.e., when the first light and the second light are detected in the same image, i.e., IM1 and IM2 are the same. The reason is that the second light should be dominant in SP2, and preferably should be as little influenced by the first light as possible, but when IM2 = IM1, it may be difficult to completely avoid this, and both the reflected first light and the second light are imaged simultaneously. Of course, SP2 should also be located close enough to SP1 so that the 2D image data in SP2 can be considered related to the 3D image data based on SP1.

[0080] Similarly, the reflected first light should advantageously dominate at SP1 so that the intensity peak, and therefore the optical triangulation, is not adversely affected. The intensity peak position should be identifiable, e.g., conventionally, regardless of the presence of any second light. The first light intensity peak should preferably be as little affected as possible by the second light.

[0081] In practice, one skilled in the art can easily provide a second light with a suitable intensity level and / or light distribution, such as those illustrated in the figures. For example, in the case of diffuse light and even a distribution of the second light, for a given SP2, an increased I DIFF (SP1) is the reduced I DIFF(SP2). It is then a matter of finding a suitable balance between these and / or suitably selecting SP2. In most practical situations, for example, when there is even a second light distribution, this is not a problem, because the second light intensity level can typically be kept well below the first light peak intensity level, which also drops rapidly when moving away from SP1, without any problems, i.e., the peak is very narrow. If required for a particular application, a person skilled in the art will be able to select a suitable SP2 and a second light intensity level for SP2, for example, through routine testing and experimentation.

[0082] 4B-4C show light intensities along image sensor coordinate v corresponding to reflected light from an object, e.g., object 220, detected by image sensor 231 for the second main group. FIG. 4B shows detected reflected first light 211 contained within EXP1 and, e.g., IM1, and FIG. 4C shows detected reflected second light 251 contained within EXP2 and, e.g., IM2, which, in the second main group, is distinct from IM1 as described above. FIG. 4B shows first light intensity distribution 413b, which corresponds to and may be identical to first light intensity distribution 413a. FIG. 4C shows second light intensity distribution 453b, which corresponds to and may be identical to second light intensity distribution 453a. In this situation, although not shown in the figure, in the case of a further second image IM2, such as IM2-2, further figures such as Figure 4C may be provided with further second light intensity distributions resulting from detection of second light reflected into further EXP2, such as EXP2-2, which gives rise to said further IM2, i.e., IM2-2.

[0083] A first sensor position 471b, corresponding to SP1, is shown in Figure 4B. A second sensor position 473b, corresponding to SP2, is shown in Figure 4C.

[0084] 3B, it was mentioned above that first light 211 can be, and advantageously is, prevented from illuminating object 220 during one or more exposure periods (EXP2), e.g., second exposure period 362b-1. Thereby, no first light can be captured within IM2, which is clearly the case in FIG. 4C. Similarly, as also mentioned above, second light 251 can be prevented from illuminating object 220 during first exposure period 361b-1, i.e., during EXP1-1, and therefore may not be captured within IM1, as can be seen in FIG. 4B. As can be seen from comparing FIGS. 4B-4C with FIG. 4A, this means that the reflected first light and second light do not interfere with each other, regardless of where SP2 is selected relative to SP1. Therefore, since only the reflected first light or the second light is simultaneously present in the image, it is not necessary to place SP2 at a distance d from SP1 as in the case of FIG. 4A. Instead, SP2 can be, for example, at the same sensor position as SP1, as shown in FIGS. 4B-4C, making it extremely easy to select SP2. In other words, each second sensor position SP2 in the one or more second images (IM2), for example, 342b-1, can be selected at the same respective first sensor position SP1, for example, 471b, as the intensity peak in the first image (IM1), for example, 340-1. Of course, the distance at sensor position v corresponding to d is typically acceptable in these embodiments as well.

[0085] Note that because optical triangulation is typically performed with continuous movement of the measurement object and / or camera / light source relative to each other, IM2 at a time other than IM1, e.g., IM2 at time t2 that is later than IM1 at time t2, means that there has been a change in object position that nevertheless maps to one and the same sensor position. In other words, SP2, even at the same sensor position as SP1, may still map, with a certain offset, to the actual object position on the measurement object that reflected the light occurring at intensity peak position SP1. However, in principle, in all conventional and practical optical triangulation systems, the difference in object position between two consecutive images used in and for optical triangulation, such as between IM1-1 and IM1-2, will typically be less than any d as described with respect to FIG. 4A . However, still at the same sensor position, e.g., for the same u, v, to make the 2D image data in SP2 as related as possible to SP1, IM2 may be provided as soon as possible after IM1, minimizing the distance caused by any movement between t1 and t2. For example, the one or more second exposure periods (EXP2) may be adjacent to the first exposure period (EXP1), e.g., before or after EXP1 as close as possible. In the case where EXP2 is several exposure periods, these may be consecutive and follow each other as soon as possible and as close as possible to EXP1.

[0086] Of course, SP2 may also be selected with some offset in the sensor position to compensate for movements and changes in the measurement object position between t1 and t2, but this will typically not be required. Compared to the second main group, in embodiments according to the first main group, a larger offset may need to be accommodated between the object position that maps to SP1 and the object position that maps to SP2.

[0087] A solution to completely remove any offset would be to ensure that there is no relative motion between object 220 and image sensor 231 between IM1 and IM2, by temporarily stopping the relative motion used in optical triangulation, e.g., stopping at t1 and starting again at t2. However, this is typically undesirable for practical reasons and may also cause a reduction in optical triangulation system throughput if the implementation relies on adapting an existing optical triangulation system.

[0088] 5A schematically illustrates a simplified example of an imaging system 500 that may be configured to perform optical triangulation and embodiments herein. Accordingly, imaging system 500 is for optical triangulation-based 3D imaging and for capturing information about both 2D and 3D properties of one or more objects. Such an object is illustrated in the figure by object 520, shown within field of view 532 of camera 530 of imaging system 500. The illustrated system corresponds to a basic configuration and comprises:

[0089] A first light source 510 for illuminating an object 520 corresponding to the measurement object with a first light 511 used in optical triangulation, and therefore with structured light such as a light plane or a laser plane.

[0090] A second light source or sources for illuminating the object 520 with two or more second lights 551, illustrated in the figure by second lights 551-1 and 551-2, which lights are for 2D imaging of the object 520.

[0091] The camera 530 with an image sensor 531 is positioned to detect a first light reflected from the object 520 as part of the 3D imaging by optical triangulation, and to detect a second light reflected from the object 520 for the 2D imaging.

[0092] The camera 530, image sensor 531, and first light source 510 are configured and arranged relative to one another for optical triangulation, and thus may be as in conventional optical triangulation. System 500 may correspond to system 100 in FIG. 1 or system 200 in FIG. 2A for purposes of optical triangulation, i.e., with respect to how the first light source 510, camera 530, and image sensor 531 are arranged relative to one another and to the object 520, etc., but further configured to perform according to embodiments herein. Image sensor 531 should, of course, also be sensitive to the second light being used. This will typically be true for all conventional image sensors used for optical triangulation, at least with respect to human-visible light, but there is often also sensitivity to light outside the visible spectrum, at least with respect to adjacent wavelengths, e.g., infrared light. If needed, conventional 3D imaging systems based on optical triangulation can be adapted by switching to a camera and / or image sensor that is sensitive to the second light used in embodiments herein in addition to the first light used for optical triangulation.

[0093] Imaging system 500 may correspond to imaging system 200 described above for performing the operations and methods of the prior application, with the difference being that the one or more light sources are configured to provide illumination with at least two second lights having different optical wavelength contents, i.e., multispectral contents. Imaging system 500 must also operate and / or be configured differently to perform operations according to embodiments herein from those described above with reference to imaging system 200. Therefore, to avoid repetition of information, the following will primarily focus on the imaging system 200 and its corresponding components and their differences compared to those already described above. Therefore, generally, anything not described differently below can be assumed to be as described above for imaging system 200 and its corresponding components.

[0094] Just as in imaging system 200, it may be desirable to transfer, e.g., transmit, image frames provided by camera 530 and image sensor 531, and / or information derived from the image frames, to a computing device 533, such as a computer or the like, which may correspond to computing device 233, and thus may be part of imaging system 500 or external thereto, for further processing outside of camera 530. Such further processing may additionally, e.g., be performed in a distributed manner, or alternatively, by several remote and / or separate computing units or devices (not shown), e.g., computer portions of a remote server and / or computer cloud. The further processing may involve performing one or more operations of embodiments herein, such as those described below.

[0095] In some embodiments, not shown but similar to imaging system 200, computing device 533 is separate from image processor 531 but still included within, e.g., integrated with, camera 530 or a unit comprising camera 530. Computing device 533, or other similar devices as mentioned (not shown), may be configured to control devices involved in optical triangulation and / or embodiments herein, such that operations related to embodiments herein, including, for example, both 2D and 3D imaging, are performed. This may include computing device 533 as shown, or a similar device, being configured to control illumination by secondary lights, for example, by controlling one or more light sources, such as second light source 550, that provides illumination by the secondary lights. This may involve controlling when, which secondary lights are switched on / off, for how long, the illumination duration for each secondary light, etc.

[0096] While the second light source 550 may be in a fixed position relative to the camera 530 and the first light source 510, the exact location and relationship between the camera 530 and the image sensor 531 is not utilized as in the case of optical triangulation, and therefore the second light source may be more freely positioned for purposes of providing second light and illumination in accordance with various embodiments herein. However, in embodiments herein, it is typically advantageous to keep light sources providing second light of different wavelengths together such that the illumination of the second light with different wavelengths comes from the same or substantially the same location with the same or substantially the same illumination direction. This is because it is typically important that the multispectral second light all illuminate the same object position. This is facilitated by illumination of the second light from the same location and in the same direction, because it reduces the risk that some second light will not successfully illuminate the same object position as other second lights. For this reason, it may also be advantageous to position the second light source at one or more locations close to the first light source and / or such that the majority of the illumination is provided in a direction close to the illumination direction of the first light. Such a situation is illustrated in the figure.

[0097] If the second light is provided as diffuse illumination, which is typically desired as described above, then the exact direction becomes less relevant, so the same or approximately the same location should suffice. Diffuse illumination with multispectral second light from approximately the same location and in a similar manner can be achieved, for example, by use of a single lighting unit as light source 550, as will now be described.

[0098] It should be noted that the same position and / or direction here relates to multispectral light of different wavelengths, e.g., second light, in one and the same lighting unit, which may then be an additional one or more further such lighting units or other second lights at different positions in the imaging system and / or with different lighting directions, in a similar manner as disclosed in the earlier application with respect to several second light sources.

[0099] 5B is a schematic example of a second light source 550 implemented as a single lighting unit 552 comprising multiple second light sources, here two or more (sub)light sources 550-1, 550-2 for providing second light 551-1, 551-2, respectively. It is also shown how such a unit can comprise one or more further second light sources for providing further second light in a similar manner, exemplified in the figure by a third second light source 550-3 for providing a third second light 551-3.

[0100] Each secondary light from the secondary light sources may pass through a common lens and / or diffuser of the lighting unit so that the illumination provided by the secondary lights emerging from the lighting units 552 is more similar to one another except for differences in wavelength.

[0101] A first example of an alternative to that shown in FIG. 5B is multiple but separately provided second light sources, such as in separate lighting units, arranged side by side and / or grouped together within imaging system 500 to provide second light.

[0102] 5B, there is a single second light source that provides multispectral second light illumination via a device having one or more optical filters, e.g., electrically and / or mechanically controlled filters, that pass second light of different wavelengths. Thus, the second light source may provide all wavelengths, e.g., ranges, from which respective filters then select one or more of the wavelengths, typically respective subranges, as respective second light.

[0103] 5C generally illustrates that the second light, or two or more of the second lights 551-1, 551-2, etc., can be non-overlapping in wavelength, e.g., having non-overlapping wavelength ranges. Non-overlapping wavelengths are preferable in many applications and can be achieved simply by using different light sources, e.g., based on different types of LEDs.

[0104] FIG. 5D illustrates alternative forms of second lights 551-1, 551-2, generally indicated in the figure by second lights 551-1′ and 551-2′, and schematically shows that the second lights, or two or more of them, may partially overlap as an alternative to or in addition to having non-overlapping wavelengths. Partial overlap means that there are some, but not all, wavelengths of the second lights that are also present in the other of the second lights. When this is the case, it is preferable that this be reciprocal, but this need not be the case, as would be the case if one of the second lights is formed by a subrange of wavelengths of another of the second lights (not shown). While this may be important in some applications, it is generally more useful to have no overlap or only partial overlap, thereby resulting in a greater separation in wavelength content between the second lights.

[0105] 6 is a flow chart to schematically illustrate a method involving multispectral second light according to an embodiment herein, which may be considered a special case of the method described above with respect to FIG. 2B and as disclosed in the earlier application, or based at least on the same principles as disclosed therein.

[0106] The following operations, which may form a method, are for associating 2D image data with 3D image data generated from optical triangulation performed by an imaging system, exemplified herein by imaging system 500, for 3D imaging of an object, for example, object 520. Thus, the imaging system comprises a first light source, exemplified by first light source 510, for illuminating the object with a first light, exemplified by first light 511, and a camera, exemplified by camera 530, having an image sensor, exemplified by image sensor 531. The imaging system also comprises one or more second light sources, exemplified by second light source 550, for illuminating object 520 with two or more second lights, exemplified by second light 551, that are multispectral by being different from each other by comprising different light wavelengths. Optical triangulation as such may be as in conventional 3D imaging based on optical triangulation, and thus involves illuminating different successive portions of object 520 with first light 511 and detecting reflected first light from each portion by image sensor 531 in respective first images (IM1). IM1 may correspond to IM1 as described above with respect to the method of FIG. 2B, which are further described and illustrated below with respect to FIGS. 7-8 with respect to embodiments herein.

[0107] 6 may be performed by one or more devices, such as camera 530 and / or computing device 533, or by imaging system 500 and / or suitable devices therein or connected thereto. Methods and devices for performing the operations thereof are further described below.

[0108] It should be noted that the following actions may be performed in any suitable order and / or may overlap in whole or in part, where this is possible and suitable.

[0109] Operation 601 The 3D data is acquired as a first sensor position (SP1) of IM1 generated by camera 530 and image sensor 531. SP1 corresponds to the position of a first light intensity peak reflected from object 520 as part of the optical triangulation. SP1 here may correspond to SP1 as described above with respect to the method of FIG. 2B and is further described and illustrated below with respect to FIGS. 7-8 with respect to embodiments herein.

[0110] operation 602 Two or more second images (IM2) are acquired by the camera 530 and the image sensor 531. Each IM2 captures an image of an object while illuminated by the two or more second lights 551. Each IM2 is associated with a respective one of the IM1s. Each IM2 can either be and be associated with a respective one of the IM1s, or be generated between two of the IM1s and be associated with any one of them. Thus, IM2 can be IM1; in other words, IM2 and IM1 can be the same image, and / or IM2 can be separate from IM1, more specifically, generated between IM1s, preferably between consecutive IM1s. In the former case where IM2 = IM1, the IM1 associated with IM2 should be the same image as IM2. In the latter case, the IM1 associated with IM2 should be the IM1 generated before or after the respective IM2, typically the IM1 generated closest in time before or after IM2.

[0111] A single IM2 here may correspond to IM2 as described above with respect to the method of Figure 2B. IM2 as a group, such as in a sequence, according to embodiments herein is further described and illustrated below with respect to Figures 7-8.

[0112] operation 603 Within each IM2, a respective second sensor location (SP2) is selected for and with respect to each SP1 of each IM1 with which each IM2 is associated. Each SP2 is a location within each IM2 at which the second light reflected from object 520 has a higher intensity than the first light reflected from object 520.

[0113] SP2 may correspond to and / or be correspondingly selected as SP2 as described above with respect to the method of Figure 2B. SP2 according to embodiments herein is further described and illustrated below with respect to Figures 7-8.

[0114] operation 604 The intensity values ​​of the selected SP2 are respectively associated with the SP1 for which the selected SP2 was selected, such that the multispectral 2D data corresponding to the intensity values ​​in SP2 from the reflected multispectral second light become associated with the 3D data corresponding to the SP1 for which the SP2 was selected.

[0115] There will therefore be SP1 corresponding to 3D data of two or more IM1s mapping two successive portions on the object according to optical triangulation, and SP1 is associated with SP2 with 2D image data corresponding to intensity values ​​from detected second light reflected from the object, which intensity values ​​result from said illumination with second light of different wavelengths and therefore correspond to multispectral 2D image data.

[0116] From the above, it can be further seen that if IM2 is generated between IM1 and associated with the same IM1, each SP1 in this image will be associated with as many intensity values ​​as there are second lights of different wavelengths. Therefore, there will be multispectral 2D image data with the same resolution as the 3D image data. However, if each IM2 is a respective IM1, e.g., IM2 = IM1 for all images, e.g., each of the consecutive IM1s is also IM2, then the resolution of the 2D image data will be lower because the "time dimension" in laser triangulation maps to the third dimension of the 3D image data, or in the example shown herein, the "y dimension." For example, if there are two different second lights and each IM2 is a respective IM1, then two IM1s, one for each second light, are required to form the multispectral image data. Therefore, the resolution of the 2D image data is half that of the 3D image data. In practice, it is typically the case that the 2D image data resolution is lower or equal to the 3D data resolution, but the reverse is unlikely to be important.

[0117] To obtain combined image data with 3D image data and 2D image data of the same resolution, images with image data can be formed “for each light,” for example, one with 3D image data and one for each second light with 2D image data. In the example of two second lights and 2D image data with half the resolution compared to the 3D image data, the 2D image data can then be upsampled by a factor of two. The resulting image data with the same resolution can then be easily combined into a single combined image, where each pixel or voxel corresponding to an object point according to the 3D image data, and thus according to the light triangulation, is associated with two intensity values, one for each second light, and thus with multispectral 2D image data. A practical example of this principle is described with respect to FIGS. 9A-9D . Generally, as should be understood, the principles underlying the embodiments herein can be used regardless of which data has a lower or higher resolution than the other data, by utilizing upsampling and / or downsampling to reach the same resolution for each image data and then combining them. Of course, relevant information can also be extracted from 2D image data associated with 3D image data without first ensuring that all image data is of the same resolution.

[0118] operation 605 Referring to the above description, in some embodiments, the 3D image data and the 2D image data are provided at the same resolution.

[0119] If the resolutions of the 3D image data and the 2D image data are different after performing such operations, as is the case in some embodiments herein, providing the image data to the same resolution may be achieved by up-resampling and / or down-resampling of the 3D image data and / or the 2D image data, i.e. by suitable up-resampling and / or down-resampling, which may be achieved in a variety of ways as will be appreciated by those skilled in the art.

[0120] operation 606 The 3D image data and the 2D image data at the same resolution may be combined into a combined image, such that each 3D image data point of the image is associated with multispectral 2D image data corresponding to intensity values ​​resulting from the reflected two or more second lights having different wavelengths.

[0121] As explained above and elsewhere herein, differences in resolution between 3D image data and 2D image data are typically the result when IM1 and IM2 for different second lights are generated at different rates. For example, if there is a sequence of three consecutive IM2s with different second lights, e.g., second lights corresponding to the colors "red," "green," and "blue" (R, G, B), generated during the same time as three IM1s with the first light, or in other words, if each IM2 is a respective IM1, the rate of "IM2 per color" will be 1 / 3 of the rate of IM1, and therefore the second image data for each color of R, G, B will be of 1 / 3 the resolution of the 3D image data. The reduced resolution is in the dimension corresponding to time by laser triangulation, which in this example is the dimension corresponding to y. (In other dimensions, such as z and x in the examples herein, the resolution is the same, as can be seen from the fact that each SP2 is selected for each SP1, and therefore with a 1:1 relationship between them.) By upsampling the respective second image data corresponding to R, G, B by a factor of 3 in said dimensions, the 3D image data and the 2D image data are of the same resolution. An alternative in this example would be to downsample the 3D image data by a factor of 3, but this is typically not preferred because 3D information is lost when the resolution is reduced in this way.

[0122] As already indicated above, in some embodiments, the second light comprises one or more of red light (R), green light (G), and blue light (B). Using all three, full-color 2D image data can be achieved. As used herein, a typical definition of R is light having one or more wavelengths in the range of 620-750 nm, with a typical wavelength of 625 nm, which may be provided by a red LED. As used herein, a typical definition of G is light having one or more wavelengths in the range of 495-570 nm, with a typical wavelength of 525 nm, which may be provided by a green LED. As used herein, a typical definition of B is light having one or more wavelengths in the range of 450-495 nm, with a typical wavelength of 460 nm, which may be provided by a blue LED. In some embodiments, the second light comprises R, G, and B, and the difference in illumination duration of R, G, and B can be used during imaging for white balancing. That is, the on and / or off times for each second light can be separately controlled and, for example, set to suitable on durations that may differ, so as to be able to directly achieve a desired white balance in the resulting 2D image data, as will be further described and illustrated below with respect to FIG.

[0123] In some embodiments, the second light comprises second light corresponding to infrared light (IR) or near-infrared light (NIR). As used herein, NIR is defined as light within the wavelength range of 850-1000 nm, and IR is defined as light having a longer wavelength, i.e., any wavelength or range of wavelengths above 1000 nm.

[0124] It should be noted that although some light, including, for example, ultraviolet light, is not explicitly mentioned above, the principles herein are of course applicable to such light, or generally different light, according to conventional definitions.

[0125] In some embodiments, illumination by the two or more second lights is provided sequentially according to a sequence of the two or more second lights, and the second images are generated in a corresponding sequence. The sequence may be a specific sequence that may be specific to the application. The sequence may be repeated one or more times, with each iteration involving the generation of additional IM2s and one or more associated IM1s. In other words, during the iteration, additional IM1s are generated according to optical triangulation, along with the above-mentioned IM2s, i.e., IM2s corresponding to or generated between IM1s, etc. This may continue until the entire object is covered, e.g., scanned, and 3D image data is generated along with associated 2D image data for the entire object. Thus, according to the sequence iteration, there will be one or more involved IM1s and different IM1s in each sequence iteration. Therefore, since IM1s relate to 3D imaging of different object parts, each iteration will be mapped to different parts of the object. For example, during scanning of an object with a first light as part of the optical triangulation, the sequence may be repeated in this manner, and 3D image data may be captured according to the optical triangulation and SP1 during the scan, while a multispectral 2D image is also captured at SP2 relative to SP1.

[0126] In some embodiments, one or more of the second lights occur more frequently in the sequence than one or more other of the second lights. That is, one or more of the second lights may occur more than once before all illumination by the second lights and the corresponding generation of IM2 occur, and / or before the sequence is repeated. For example, when a sequence of second lights comprises R, G, and B, G may occur more frequently in the sequence than R and B, e.g., twice as frequently. Thus, the sequence may be or include RGBG or the like. It is known that a greater number of occurrences of G improves the perceived sharpness of color images. In this way, the resulting 2D image data may be perceived as sharper in an image with the combined 2D image data. Of course, there may be other reasons for generating some second lights more frequently than others.

[0127] Examples of different sequences are further described and illustrated below with respect to FIGS.

[0128] In some embodiments, the object is organic. That is, the object being imaged may be organic, and / or the method may be applied in applications where the object is an organic object. For organic objects that often differ in 3D appearance but also in 2D, e.g., with 2D surface variations, details of both and in combination are often important. Thus, embodiments herein may be particularly important for application in applications involving organic objects, e.g., applications where the object is one of the following types and / or kinds: plants, trees, logs, lumber, meat, vegetables, bread, food, and waste.

[0129] For example, embodiments herein may be beneficially applied in the following application areas: - Agriculture, where the first light can be a green laser to get good data from green leaves, and through 3D image data it can be possible to measure for example the size and height of the plant, and at the same time, by using a second light with R, G and NIR it can be possible to get information about the condition and health of the plant from 2D image data, which can be used to optimize the use of nutrients and water for the plant. - Food classification, for example to allow food products to be controlled and classified by acquiring them using 3D image data and to allow foreign objects to be detected by using 2D image data and / or for quality assurance reasons, where it may be desirable to apply different classifications based on the 2D data. - Waste classification, for example by using 3D image data to capture food waste / leftovers, to enable them to be managed and classified, and to do so in a way that differentiates according to material, color and shape, where multispectral 2D image data can provide extremely useful information. - Timber inspection, where information from 3D image data is important, but also multispectral 2D image data, for example, to be able to better distinguish between heartwood and sapwood (at the surface).

[0130] Thus, in some embodiments, the object is one of the following types and / or kinds: plants, trees, logs, timber, meat, vegetables, food, waste.

[0131] 7A-7B are similar to the main groups described above with respect to FIGS. 3A-3B and in the prior application, respectively, but here schematically illustrate the situation for the corresponding two main exemplary groups of embodiments herein in the case of multispectral second light and in the case of methods according to embodiments herein as described above with respect to FIG. 6. The figures are presented in a similar format to FIGS. 3A-3B to facilitate comparison and understanding of the differences. Cases involving multispectral second light, such as those in embodiments herein, may be considered extended or special cases. Because there are more images involved in these examples, the exposure periods shown in FIGS. 3A-3B have been removed in FIGS. 7A-7B to conserve space, although exposure periods would, of course, exist in practice.

[0132] FIG. 7A is for a first main exemplary group of embodiments in which IM2 is the same as IM1, i.e., the same as the image used for optical triangulation, and therefore also has the same exposure period of the image sensor.

[0133] FIG. 7B is for a second main exemplary group of embodiments in which IM2 is instead separate from IM1 and is between IM1, i.e., between the images used for optical triangulation, and therefore IM2 also has a separate exposure period.

[0134] In both figures, the period T between successive IM1 3D 763 indicates the time period between successive IM1 portions of the optical triangulation and corresponds thereto and therefore relates to the provision of 3D image data. 3D 763 corresponds to the scan rate typically used in optical triangulation.

[0135] Furthermore, for ease of understanding, the type of light involved for each image is indicated in the diagram and the type of image data each light is used to generate. Thus, a first light is designated as "Light 1 for 3D" because it is used in optical triangulation to generate the 3D image data, and two or more second lights are designated as "Light 2 for 2D" because they are used to generate the 2D image data. A first of the second lights is referred to as "Light 2-1 for 2D," and a second of the second lights is referred to as "Light 2-2 for 2D" because the two or more second lights differ in wavelength content, as explained above.

[0136] Furthermore, in both figures, the period T SEQ 765 is shown, the duration of which indicates the sequence as described above, i.e., a sequence of IM2s, each IM2 being exposed to one of the second lights until all of the two or more second lights have been used according to a sequence which may then be, and typically is, repeated.

[0137] Both Figures 7A and 7B show T SEQ Although an example is shown in which there are two IM2s per 765, and therefore in the sequence, this principle can be easily extended to three or more IM2s, as explained above and further illustrated below.

[0138] FIG. 7A, relating to the first main exemplary group of embodiments herein, shows a T with two IM2s that are also IM1s, namely, IM2-1 742a-1, which is the same as IM1-1 741a-1, and IM2-2 742a-2, which is the same as IM1-2 741a-2. SEQ 765a more specifically and schematically. In this case, T SEQ 765a is T 3D 763a. For more than two second lights and / or more than two IM2s in the sequence, TSEQ Of course, T 3D This will increase further compared to 763a.

[0139] Furthermore, the figure schematically illustrates respective SP1 and SP2 in each image, i.e., SP1-1 771a-1 and SP2-1 772a-1 in IM1-1 741a-1 being the same as IM2-1 742a-1, and SP1-2 771a-2 and SP2-2 772a-2 in IM1-2 741a-2 being the same as IM2-2 742a-2. As shown schematically in the figure and according to the above description, since IM2=IM1 here, the reflected second light captured in SP2 dominates the reflected first light captured, but the first light should dominate the second light in SP1, i.e., as in the method and example described with respect to FIG. 2B, SP2 involves an offset in sensor coordinates to SP1. In other words, and with respect to embodiments herein generally, the captured reflected second light from the object should have a higher intensity at SP2 than the reflected first light from the object. Because SP1 results from optical triangulation, it is implied that the first light is dominant there, and therefore the captured reflected second light from the object has a lower intensity at SP1 than the reflected first light from the object, or at least that the optical intensity of the second light is sufficiently low at SP1 so as not to adversely affect the detectability of the first light at SP1.

[0140] The figure also shows two further IM1 and IM2 belonging to a repetition of the sequence, simply to show that what is shown is typically repeated, e.g. the entire object is 3D scanned by optical triangulation and multispectral 2D image data is captured in connection with this and with respect to the 3D image data.

[0141] FIG. 7B, relating to the second main exemplary group of embodiments herein, shows a T image comprising two IM2s that are distinct from IM1, namely IM2-1 742b-1 and IM2-2 742b-2, located between a first image IM1-1 741b-1 and a further first image IM1-2 741b-2. SEQ 765b more specifically and schematically. In this case, T SEQ 765b is T 3D It can be seen that IM2 is the same as 763b, but is shown with an offset in the figure. When IM2 is separate from IM1 and is generated between IM1, IM2 is typically, but need not be, evenly spaced between IM1, i.e., so that the image frame rate with which the image sensor operates can be the same for all images, i.e., the same for IM1 and IM2.

[0142] In the figure, SP2 is illustrated schematically as being the same as SP1, because in this case, as already mentioned above, SP2 can be chosen more freely with respect to SP1 when / if the first light is not present in IM2, i.e. when / if the first light is not illuminating the object when IM2 is captured, by, for example, keeping the first light "on" only during the exposure of IM1 and off during the exposure of IM2.

[0143] SP1-1 771b-1 is shown in IM1-1 741b-1. SP2-1 772b-1, which has the same sensor location as SP1-1, is shown in IM2-1 742b-1, and SP2-2 772b-2, which has the same sensor location as SP1-1, is shown in IM2-2 742b-2.

[0144] The figure also shows the T SEQTwo further images, IM1-2 and IM2-1 portions of a repeat of the sequence are also shown simply to illustrate that the sequence shown by is typically repeated, e.g., such that the entire object is 3D scanned by optical triangulation and multispectral 2D image data is captured in connection therewith and with respect to the 3D image data.

[0145] Note that the embodiments herein may be implemented as a combination between the first and second main groups of embodiments, since there are two or more second lights, and therefore two or more IM2s, in each sequence, but this is not shown in the figures. That is, for example, in a sequence of two or more IM2s and one or more associated IM1s, one or more of the IM2s may be the same as one or more IM1s of the sequence, respectively, and one or more other IM2s of the sequence may be distinct from and generated between the IM1s. Thus, in a corresponding example such as that of FIG. 7B, it may be possible to have only one distinct IM2, e.g., IM2-1 = IM1-1, and IM2-2 may be said to be distinct from and between IM1-1 and then the next further IM1-2 portion of the repetition of the sequence.

[0146] 8A-8E schematically illustrate examples involving different sequences of the second light. These examples are shown for embodiments corresponding to the first main exemplary group, i.e., where IM2 = IM1, but it will be understood based on the above that the corresponding sequences are also applicable to other second main exemplary groups and to combinations between two main exemplary groups.

[0147] FIG. 8A corresponds to the example in FIG. 7A, but with three second lights R, G, B (red, green, blue) as described above, and a period T SEQ865a. Therefore, the sequence can be described as R, G, B. There is IM2-1 842a-1, which is the same as IM1-1 841a-1; IM2-2 842a-2, which is the same as IM1-2 841a-2; and IM2-3 842a-3, which is the same as IM1-3 841a-3. Therefore, the following occurs: During the exposure of image sensor 531 that results in the generation of IM2-1, the second light illuminating object 520 is red light; During the exposure of image sensor 531 that results in the generation of IM2-2, the second light illuminating object 520 is green light; During the exposure of image sensor 531 that results in the generation of IM2-3, the second light illuminating object 520 is blue light.

[0148] 8B is another schematic example of a sequence as described above, but in a more comprehensive view that simply shows the second light type for each image frame of the sequence. Thus, in this example, the sequence is R, G, B, IR, and therefore four IM2 (and four IM1, since IM2=IM1 as described above). The sequence has a sequence period T SEQ 865b.

[0149] FIG. 8C is yet another schematic example of a sequence as described above, shown in the same type of general diagram as in FIG. 8B. Again, a second light type is shown for each image frame of the sequence. Thus, in this example, the sequence is R, G, NIR, and therefore involves three IM2s in the sequence (and three IM1s, since IM2=IM1 as described above). The sequence has a sequence period T SEQ 865c.

[0150] FIG. 8D is yet another schematic example of a sequence as described above, shown in a comprehensive diagram as in FIGS. 8B-8C. Again, the type of secondary light for each image frame of the sequence is shown. Thus, in this example, the sequence is R, G, B, G, and thus, as noted above, green light occurs more frequently than red and blue. In this example, there are only three different secondary lights, but the sequence contains four IM2s (and four IM1s, since IM2=IM1 here as noted above). The sequence is repeated for a sequence period T SEQ 865d.

[0151] 8E corresponds to the example of FIG. 8A , except that the “on time,” i.e., illumination duration, of each second light is now different for each IM2. Thus, the on duration (and thus the corresponding off duration) of each second light during the exposure period that results in IM2 (and IM1, since IM2=IM1 in this example) is different in some embodiments, e.g., for white balancing purposes as described above. In these embodiments, the on / off duration of each second light may typically be controlled by imaging system 500, e.g., by its computing device 533, so that a preferred duration of second light illumination is achieved during the exposure period. Thus, in FIG. 8E , the following is shown: IM2-1 842e-1, which is the same as IM1-1 841e-1, and which is the exposure period EXP; IM2-2 842e-2, which is the same as IM1-2 841e-2, and which is the exposure period EXP; and IM2-3 842e-3 is the same as IM1-3 841e-3, which is the exposure period EXP. Note that the exposure period EXP as shown in this example is typically, but not necessarily, the same for all images, at least for the first group, where IM2=IM1.

[0152] During the exposure period EXP of the image sensor 531 that results in the generation of IM2-1 842e-1, the second light illuminating the object 520 is red light (R) during a red light-on period 866e-1 that may be less than EXP. During the exposure period EXP of the image sensor 531 that results in the generation of IM2-2 842e-2, the second light illuminating the object 520 is green light (G) during a green light-on period 866e-2 that may be less than EXP. During the exposure period EXP of the image sensor 531 that results in the generation of IM2-3 842e-3, the second light illuminating the object 520 is blue light (B) during a blue light-on period 866e-3 that may be less than EXP.

[0153] Note that for a second main exemplary group with IM2 separate from IM1, effective illumination by the second light can instead be achieved by controlling the exposure period EXP of IM2, and then each second light can have the same on-duration covering the exposure period. This may not be possible, or at least less suitable, for the first main exemplary group, such as in the example of FIG. 8E where IM2 = IM1, because it would mean that different exposure periods are used for the first light and the light triangulation, and therefore different exposures to the first light within different IM1s, which is typically undesirable. However, the same exposure period of the first light may also be achieved in that case by controlling the on / off duration of the first light so that it is still "on" for one and the same time period during each exposure period.

[0154] The above is just one example, and the principle is of course applicable to any type of second light of different wavelengths, not just RGB.

[0155] From what has been described above, it can be understood that providing a second light can be a matter of controlling which second light is switched on / off, for how long, and in what sequence during imaging, for example, by controlling the lighting unit and / or light source that provides the second light. In practice, this can correspond to "flashing" the second light, using a different second light in each "flashing" according to the sequence used, and the sequence can then be repeated, for example, as long as the scanning of the object continues. The first light, for example, a laser, can be switched on permanently during this time and / or according to the optical triangulation performed by the system. It should be noted that the optical triangulation as such does not need to be affected by the second light, at least in principle. However, in some embodiments, the first light, for example, a laser, can also be "flashed," i.e., switched off, if / when the first light should not be present in one or more of the second images. Also, in some embodiments, there are several single second light sources and / or lighting units for providing illumination of the second light from several different directions, e.g., as disclosed in the earlier application. In this way, similar and / or corresponding advantages to those described in the earlier application by illumination from different directions and / or in different ways may also be obtained using multispectral 2D image data associated with 3D image data as in the embodiments herein. Also, it should be noted that the second light, e.g., in the single lighting unit, may provide illumination in various ways, e.g., as disclosed in the earlier application with respect to providing second light.

[0156] 9A-9D show example images and results from a real-world application of embodiments herein. The image-generating embodiment is based on the first main example group, where IM2 = IM1, and the multispectral second light sequence is NIR, R, G, and B, i.e., four different second lights, and therefore, according to the above description, the respective 2D image data resolution is ¼ of the 3D image data resolution. The test object being imaged was formed from a piece of wood onto which several candies of different colors were placed to form a composite test object for imaging with many different colors, structures, and materials. The object was 3D imaged by optical triangulation, here via scanning through the first and second lights. The scanning involved the generation of over 5,000 IM1s, each with an SP1 within it, thus producing a first image, which produces a 3D contour according to the optical triangulation. An IM1 of approximately 5120 is used in the examples shown in the figures, and therefore corresponds to 5120 contour images of the object and therefore, in these examples, the corresponding 3D image data resolution in the dimension corresponding to the scanning direction, in the y' direction.

[0157] 9A shows a composite image of 3D image data in the form of 3D image points, corresponding to pixels, and therefore voxels, in 3D, with associated multispectral 2D image data, according to embodiments herein. Since true 3D cannot be shown here, what is shown is a 2D view of the 3D image points "from above," as the object is shown in the x'-y' plane. The associated 2D image data is clearly visible as a texture on the object formed by the 3D image points.

[0158] 9B is an enlargement of a portion of the image shown in FIG. 9A, where it can be seen that in the y' direction, for every four lines of 3D image data, i.e., for every four lines of 3D image points, there is a different 2D image data. Thus, as expected, due to the sequences used with four IM2s and four different second lights, respectively. The 2D image data resolution is therefore 1 / 4 of the 3D image data resolution in the y' direction, thus 5120 / 4=1280.

[0159] FIG. 9C shows 2D image data separated into four images, each with a resolution of 1280 in the y′ direction. It can be clearly seen that each image shown captured different 2D information about the object through its respective secondary light illumination. As explained above, the 2D image data for each image comprises intensity values ​​resulting from the detected reflected secondary light from the object, and the secondary light illuminating the object during exposure results in IM2 from which the 2D image data was obtained (from the SP2 position). Thus, while each 2D image data may be viewed as a monochrome, e.g., grayscale, image, it will be appreciated that if each is represented by a separate monochrome color, e.g., some suitable color scale for RGB and NIR, a full-color image with a mixture of all colors can be achieved. Of course, it is also possible, and sometimes sufficient, and / or desirable, to use and / or view 2D image data in isolation and not mixed or combined with each other. For example, as in FIG. 9C, to view 2D image data associated with 3D image data in separate images, e.g., side-by-side for comparison. In other cases, measurements may be performed on the 2D image data associated with the 3D image data without the need to actually view the 2D image data.

[0160] In either case, it can be appreciated that it is often desirable to have all image data at one and the same resolution. In this way, line effects and related distortions, such as those seen in FIG. 9B, can be avoided. Thus, as already mentioned above, the respective 2D image data can be provided as one and the same resolution and, for example, converted to one and the same resolution. If the 2D image data is at a lower resolution, it may be preferable to convert the 2D image data, for example in the form of an image as in FIG. 9C, to the same resolution as the 3D image data. The conversion can be performed by resampling, thus, in this example, upsampling.

[0161] Figure 9D shows a combined image of the 3D and 2D image data after the latter has first been upsampled to the same resolution as the 3D image data. The result is multispectral 2D image data represented by four intensity values ​​per 3D image data point and one intensity value for each secondary light used. Thus, a full-color image and texture on an object in 3D can be achieved, as shown in Figure 9D. Note that although the image is in color, patent drawings and figures are typically reproduced in black and white and / or grayscale, so the image in Figure 9D may nevertheless appear grayscale in the drawing.

[0162] FIG. 10 is a schematic block diagram illustrating one or more devices 1000, i.e., embodiments of device 1000, which may correspond to the devices already described above for performing embodiments herein, such as for performing the methods and / or operations described above with respect to FIG. 6 . Device 1000 may correspond, for example, to computing device 533 and / or camera 530, or suitable devices included in and / or forming imaging system 500, or one or more computing devices remote and separate from the imaging system involved. For example, it is a computing device that operates on output from imaging system 500. Thus, device 1000 performing the method may do so at some later opportunity than when imaging system 500 is operated and provides IM1, IM2, etc. Thus, the method may be performed by a device that is remote in time and space from the imaging system, and thus the imaging. Output from the imaging system as required by the method may be uploaded, for example, to a server or computer cloud comprising computing devices configured to perform the method. However, a device configured to perform the method as part of and / or in association with the imaging system and imaging involved is typically preferred, as this allows for faster execution and less image data needing to be temporarily stored and / or transferred over longer distances. It may therefore be preferred for device 1000 to be camera 530 and / or computing device 533, or in other words, for camera 530 and / or computing device 533 to be configured to perform the method.

[0163] The schematic block diagram is intended to illustrate an embodiment of how device 1000 may be configured to perform the methods and operations described above with respect to Figure 6. Thus, device 1000 is for associating multispectral 2D image data with 3D image data generated from optical triangulation performed by an imaging system, such as imaging system 500, for 3D imaging of an object.

[0164] The device 1000 may comprise a processing module 1001, such as a processing means, e.g., one or more processing circuits such as a processor, one or more hardware modules including circuits, and / or one or more software modules for performing the methods and / or operations.

[0165] The device 1000 may further comprise a memory 1002, which may comprise, include, store, or otherwise comprise a computer program 1003. The computer program 1003 comprises "instructions" or "code" executable, directly or indirectly, by the device 900, respectively, to perform the methods and / or operations. The memory 1002 may comprise one or more memory units and may further be arranged to store data, such as configurations, data, and / or values ​​involved in or for performing the functions and operations of the embodiments herein.

[0166] Further, each device 1000 may comprise, as an illustrative hardware module, processing circuitry 1004 involved in processing and, e.g., encoding, data, and may comprise or correspond to one or more processors or processing circuits. Processing module 1001 may comprise, e.g., be "embodied in" or "realized by," such processing circuitry 1004. In these embodiments, memory 1002 may comprise computer programs 1003 respectively executable by processing circuitry 1004, such that each device 1000 is operable or configured to perform the methods and / or operations thereof.

[0167] Typically, device 1000, e.g., processing module 1001, comprises an input / output (I / O) module 1005 configured to participate in, e.g., by executing, any communication to and / or from other units and / or devices, such as transmitting information to and / or receiving information from other devices. I / O module 1005 may be exemplified by an obtaining module, e.g., a receiving module, and / or a providing module, e.g., a transmitting module, when applicable.

[0168] Additionally, in some embodiments, device 1000, e.g., processing module 901, comprises one or more of a selection module, an association module, a selection module, a combination module as illustrated hardware and / or software modules for performing operations of embodiments herein, which may be implemented fully or partially by processing circuitry 1004.

[0169] Therefore, the result is as follows:

[0170] The device 1000, and / or the processing module 1001, and / or the processing circuit 1004, and / or the I / O module 1005, and / or the acquisition module are operable or configured to acquire the 3D data as the SP1 of the IM1 generated by the camera and image sensor.

[0171] The device 1000, and / or the processing module 1001, and / or the processing circuit 1004, and / or the I / O module 1005, and / or the acquisition module are each operable or configured to acquire the two or more IM2s generated by the camera and image sensor and imaging an object during illumination by the two or more second lights, each IM2 being either a respective one of the IM1s and associated therewith, or generated between two of the IM1s and associated with one of them.

[0172] The device 1000, and / or the processing module 1001, and / or the processing circuit 1004, and / or the I / O module 1005, and / or the selection module are operable or configured to select, within each IM2, for and against each SP1 of each IM1 with which the respective IM2 is associated, a respective SP2 for which the second light reflected from the object has a higher intensity than the first light reflected from the object.

[0173] The device 1000, and / or the processing module 1001, and / or the processing circuit 1004, and / or the I / O module 1005, and / or the association module are operable or configured to associate the intensity values ​​of the selected SP2 with the SP1 for which the selected SP2 was selected, respectively, so that the multispectral 2D data corresponding to the intensity values ​​in SP2 from the reflected multispectral second light becomes associated with the 3D data corresponding to the SP1 for which SP2 was selected.

[0174] The device 1000, and / or the processing module 1001, and / or the processing circuit 1004, and / or the I / O module 1005, and / or the providing module may be operable or configured to provide 3D image data and 2D image data at the same resolution.

[0175] The device 1000, and / or the processing module 1001, and / or the processing circuit 1004, and / or the I / O module 1005, and / or the synthesis module may be operable or configured to synthesize 3D image data and 2D image data at the same resolution into the synthesized image, such that each 3D image data point of the image is associated with multispectral 2D image data corresponding to intensity values ​​resulting from the reflected two or more second lights having different wavelengths.

[0176] FIG. 11 is a schematic diagram illustrating some embodiments of a computer program 1003 and its carrier for causing the device 1000 to perform the methods and operations described above.

[0177] The computer program 1003, when executed by the processing circuit 1004 and / or the processing module 1001, comprises instructions that cause the device 1000 to perform as described above. In some embodiments, one or more carriers, such as a computer program product, or more specifically, a data carrier, are provided that comprise the computer program. Each carrier may be one of an electronic signal, an optical signal, a radio signal, and a computer-readable storage medium, such as the computer-readable storage medium 1101 as shown schematically in the figure. Thus, the computer program 1003 may be stored on the computer-readable storage medium 1101. The carrier may exclude a transitory propagating signal, and the data carrier may correspondingly be referred to as a non-transitory data carrier. Non-limiting examples of data carriers that are computer-readable storage media are memory cards or memory sticks, disk storage media, or mass storage devices typically based on hard drives or solid-state drives (SSDs). The computer-readable storage medium 1101 may be used to store data accessible over a computer network 1102, e.g., the Internet or a local area network (LAN). The computer program 1003 may further be provided as a pure computer program or may be contained within one or more files. The one or more files may be stored on the computer-readable storage medium 1101 and available, for example, through download, such as via a server, over the computer network 1102 as shown in the figure. The server may be a web or file transfer protocol (FTP) server, or the like. The one or more files may be executable files for direct or indirect download to and execution on the device, for example, upon execution by the processing circuit 1004 to cause the device to perform as described above.One or more files may also, or alternatively, be for intermediate download and compilation, with the same or other processor causing the device 1000 to perform as described above to make the file executable before further download and execution.

[0178] It should be noted that any of the processing modules and circuits described above may be implemented as software and / or hardware modules, e.g., in existing hardware and / or as an application specific integrated circuit (ASIC), field programmable gate array (FPGA), etc. It should also be noted that any of the hardware modules and / or circuits described above may be included in, e.g., a single ASIC or FPGA, or may be distributed among several separate hardware components, whether individually packaged or assembled into a system on a chip (SoC).

[0179] Those skilled in the art will also appreciate that the modules and circuits described herein may refer to hardware modules, software modules, analog and digital circuits, and / or combinations of one or more processors configured with software and / or firmware, e.g., stored in memory, that when executed by one or more processors, may configure and / or cause a device, sensor, etc. to perform the methods and operations described above.

[0180] Identification by any identifier herein may be implicit or explicit. Identification may be unique in a certain context, for example, for a certain computer program or program provider.

[0181] As used herein, the term "memory" may refer to data memory for storing digital information, typically a hard disk, magnetic storage device, media, portable computer diskette or disk, flash memory, random access memory (RAM), etc. Additionally, memory may be the internal register memory of a processor.

[0182] It should also be noted that any enumerated terminology, such as first device, second device, first surface, second surface, etc., should be considered as such to be open-ended, and the terminology, as such, does not imply a hierarchical relationship. In the absence of any explicit information to the contrary, the enumerated naming should be considered merely a way of achieving distinct names.

[0183] As used herein, the phrase "configured to" may mean that a processing circuit is configured or adapted to perform one or more of the operations described herein using software or hardware configurations.

[0184] As used herein, the term "number" or "value" may refer to any kind of number, such as a binary number, a real number, an imaginary number, or a rational number. Additionally, a "number" or "value" may be one or more characters, such as a letter or a string of letters. Additionally, a "number" or "value" may be represented by a string of bits.

[0185] As used herein, the phrases "may" and "in some embodiments" are typically used to indicate that the described features can be combined with any of the other embodiments disclosed herein.

[0186] In the drawings, features that may only be present in some embodiments are typically depicted using dotted or dashed lines.

[0187] When the word "comprise" or "comprising" is used, the word should be taken to be open-ended, i.e., to mean "consist at least of."

[0188] The embodiments herein are not limited to the embodiments described above. Various alternatives, modifications, and equivalents may be used. Therefore, the above embodiments should not be construed as limiting the scope of the present disclosure, which is defined by the appended claims. [Explanation of symbols]

[0189] 100 Prior art imaging system, imaging system, system 110 Light source 111 Structured Light 112 Rays, Light 120 first object, first measurement object, object 121 Second Object 122 Movable object support structure 123 Coordinate System 130 Cameras, camera units 141-1~141-N Contour image 200 Imaging system, system 205 Imaging System 210, 510 First light source 211 First light, reflected first light, detected first light 220, 520 Measuring object, object 230 Camera 231 Image sensors, image processors 232 Field of View 233 Computing Devices 250 Second Light Source 251 Secondary light, reflected secondary light, detected secondary light 340-1, 340-1a, 340-1b First image 340-2, 340-2a, 340-2b Second image 340-3, 340-3a, 340-3b Third Image 341a-1, 341b-1 First image 341a-2 Further first images 342a-1, 342b-1 Second image 361a-1, 361a-2, 361b-1 First exposure period 361b-2 First further exposure period 362a-1, 362b-1 Second exposure period 363a time period 413a First light intensity distribution, light intensity distribution 413b First light intensity distribution 453a Second light intensity distribution, light intensity distribution 453b Second light intensity distribution 471a First sensor position, intensity peak position 471b First sensor position 473a, 473b Second sensor position 475a Difference 477a Light intensity difference, I DIFF (SP1) 479a Light intensity difference, I DIFF (SP2) 500 Imaging system, system, device 511 First Light 530 Cameras, Devices 531 Image sensors, image processors 532 Field of view 533 Computing Devices, Devices 550 Second Light Source, Light Source 550-1, 550-2 (sub) light source 550-3 Third Second Light Source 551 Second Light, Multispectral Second Light 551-1, 551-2, 551-1', 551-2' Second Light 551-3 Third Second Light 552 Lighting Unit 741 First Image "IM1", IM1 741a-1, 841a-1, 841e-1 IM1-1 741a-2, 841a-2, 841e-2 IM1-2 741b-1 First image IM1-1, IM1-1 741b-2 Further first image IM1-2 742 Second image "IM2", IM2 742a-1, 742b-1, 842a-1, 842e-1 IM2-1 742a-2, 742b-2, 842a-2, 842e-2 IM2-2 763 Period T 3D , time period T 3D 763a, 763b T 3D 765 Period T SEQ , T SEQ 765a, 765b T SEQ 771 First sensor position "SP1", SP1 771a-1, 771b-1 SP1-1 771a-2 SP1-2 772 Second sensor position "SP2", SP2 772a-1, 772b-1 SP2-1 772a-2, 772b-2 SP2-2 841a-3, 841e-3 IM1-3 842a-3, 842e-3 IM2-3 865a Period T SEQ 865b, 865c, 865d Sequence period T SEQ 866e-1 Red Light On Period 866e-2 Green light on period 866e-3 Blue light on period 1000 devices 1001 Processing Module 1002 memory 1003 Computer Programs 1004 Processing circuit, processor 1005 Input / Output (I / O) Module, I / O Module 1101 Computer-readable storage medium 1102 Computer Networks

Claims

1. 1. A method for associating multispectral 2D image data with 3D image data generated from optical triangulation performed by an imaging system (500) for 3D imaging of an object (520), the imaging system (500) comprising a first light source (510) for illuminating the object (520) with a first light (511), a camera (530) with an image sensor (531), and one or more second light sources (550) for illuminating the object (520) with two or more second lights (551), the second lights (551) being multispectral by virtue of having different light wavelengths, the second lights (551) differing from one another by virtue of having different light wavelengths, the optical triangulation comprising illuminating different successive portions of the object (520) with the first lights (511) and detecting reflected first light from each portion by the image sensor (531) in each first image "IM1" (741), the method comprising: acquiring (601) 3D data as a first sensor position "SP1" (771) of IM1 (741) generated by the camera (530) and image sensor (531), the SP1 corresponding to the position of a first light intensity peak reflected from the object (520) as part of the optical triangulation; a step (602) of acquiring two or more second images "IM2" (742) generated by the camera (530) and the image sensor (531), the two or more second images "IM2" (742) being images of the object while illuminated by the two or more second lights (551); a step (602) of acquiring two or more second images "IM2" (742), the two or more second images "IM2" being images of the object while illuminated by the two or more second lights (551), the step (602) wherein each IM2 of the two or more IM2 is either a respective IM1 of the IM1 (741) and is associated therewith, or is generated between two of the IM1 (741) and is associated therewith; selecting (603) within each IM2 (742) a respective second sensor position "SP2" (772) for and with respect to each SP1 (771) of each IM1 (741) with which said respective IM2 (742) is associated, where said reflected second light (551) from said object (520) has a higher intensity than said reflected first light from said object (520); a step (604) of associating the intensity values ​​of the selected SP2 (772) with the SP1 (771), respectively, wherein the SP2 (772) is selected for the SP1 (771) such that multispectral 2D data corresponding to the intensity values ​​in the SP2 (772) from the reflected multispectral second light (551) are associated with the 3D data corresponding to the SP1 (771), and the SP2 (772) is selected for the SP1 (771); A method comprising:

2. The method comprises: providing (605) the 3D image data and the 2D image data at the same resolution; combining (606) the 3D image data and the 2D image data at the same resolution into a combined image, whereby each 3D image data point of the image is associated with multispectral 2D image data corresponding to intensity values ​​resulting from the reflected two or more second lights having different wavelengths; 10. The method of claim 1, further comprising:

3. 3. The method of claim 2, wherein if the resolutions of the 3D image data and the 2D image data are different, the providing at the same resolution is achieved by up-resampling and / or down-resampling the 3D image data and / or the 2D image data.

4. The method of claim 1 , wherein the second light comprises one or more of red light “R”, green light “G”, and blue light “B”.

5. The method of claim 4 , wherein the second light comprises R, G, and B, and differences in illumination durations by R, G, and B are used during the imaging for white balancing.

6. The method of claim 1 , wherein the second light comprises second light corresponding to infrared (IR) light or near infrared (NIR) light.

7. The method of claim 1 , wherein the illumination by the two or more second lights is provided sequentially according to a sequence of the two or more second lights, and the second images are generated in a corresponding sequence.

8. The method of claim 7 , wherein one or more of the second lights occur more frequently in the sequence than one or more other of the second lights.

9. 9. The method of claim 8, wherein the second sequence of lights comprises red light "R", green light "G", and blue light "B", with G occurring more frequently in the sequence than R and B.

10. The method of claim 1 , wherein the object is organic.

11. The method of claim 10, wherein the object is or corresponds to one of the following types and / or kinds of object: plants, trees, logs, timber, meat, vegetables, food, waste.

12. One or more devices (500, 530, 533, 1000) for associating multispectral 2D image data with 3D image data, said 3D image data being generated from optical triangulation performed by an imaging system (500) for 3D imaging of an object (520), said imaging system (500) comprising a first light source (510) for illuminating a measurement object (520) with a first light (511), a camera (530) with an image sensor (531), and detecting the first light (211, 511) reflected from said measurement object (220, 520). and one or more second light sources (550) for illuminating the object (520) with two or more second lights (551), the second lights (551) being different from one another and therefore multispectral by having different light wavelengths for the light triangulation, the optical triangulation comprising illuminating first different successive portions of the measurement object (520) with the first light (511) and detecting reflected first light from each portion by the image sensor (531) in a respective first image "IM1" (741), the one or more devices acquiring (601) 3D data as a first sensor position "SP1" (771) of IM1 (741) generated by the camera (530) and image sensor (531), the SP1 corresponding to the position of a first light intensity peak reflected from the object (520) as part of the optical triangulation; a step (602) of acquiring two or more second images "IM2" (742) generated by the camera (530) and the image sensor (531), the two or more second images "IM2" (742) being images of the object while illuminated by the two or more second lights (551); acquiring (602) two or more second images "IM2" (742), the two or more second images "IM2" (742) being images of the object while illuminated by the two or more second lights (551), the acquiring (602) being such that each IM2 of the two or more IM2 is either a respective IM1 of the IM1 (741) and is associated therewith, or is generated between two of the IM1 (741) and is associated therewith; selecting (603) within each IM2 (742) a respective second sensor position "SP2" (772) for and with respect to each SP1 (771) of each IM1 (741) with which said respective IM2 (742) is associated, where said reflected second light (551) from said object (520) has a higher intensity than said reflected first light from said object (520); Associating (604) the intensity values ​​of the selected SP2 (772) with the SP1 (771), respectively, wherein the SP2 (772) is selected for the SP1 (771) such that multispectral 2D data corresponding to the intensity values ​​in the SP2 (772) from the reflected multispectral second light (551) is associated with the 3D data corresponding to the SP1 (771), and the SP2 (772) is selected for the SP1 (771); One or more devices (500, 530, 533, 1000) configured to:

13. One or more computer programs (1003) comprising instructions that, when executed by one or more processors (1004), cause one or more devices according to claim 12 to perform the method according to any one of claims 1 to 11.

14. One or more carriers comprising one or more computer programs (1003) according to claim 13, said one or more carriers being one or more of the following: an electronic signal, an optical signal, a radio signal, or a computer-readable storage medium (1101).

Images