Method and apparatus for obtaining and associating multispectral 2D image data with 3D image data from light triangulation

Abstract

A method and apparatus for associating multispectral 2D image data with 3D image data resulting from light triangulation performed by an imaging system for 3D imaging of an object. The imaging system includes one or more second light sources for illuminating the object with two or more second lights, the two or more second lights being multispectral. The 3D data is obtained as a first sensor position "SP1" of a first image "IM1". Two or more second images' IM2 'are obtained and the object is imaged during irradiation by the two or more second light, respectively. In a respective IM2 (542), a respective second sensor position "SP2" is selected for and in combination with a respective SP1. The intensity value of each of the selected SP2 is associated with the SP1 for which the SP2 has been selected, whereby multispectral 2D data corresponding to the intensity value at the SP2 becomes associated with 3D data.

Description

Technical Field

[0001] Embodiments herein relate to a method and apparatus for obtaining multispectral 2D image data and associating the multispectral 2D image data with 3D image data from light triangulation. Background Art

[0002] Industrial vision cameras and systems for factory and logistics automation are often based on three-dimensional (3D) machine vision, in which 3D images of scenes and / or objects are captured. By 3D images, it is meant images that also include "height" or "depth" information and do not include, or at least not only include, information about pixels that are only two-dimensional (2D) in traditional images, such as intensity and / or color. That is, each pixel of the image may include such information that is associated with the position of the pixel in the image and mapped to the position of the thing (e.g., object) that has been imaged. Processing may then be applied to extract information about the characteristics of the object (i.e., the 3D characteristics of the object) from the 3D image, and, for example, convert into various 3D image formats. This information about height may be referred to as range data, where the range data may therefore correspond to data from a height measurement of the object being imaged (or in other words, from a range or distance measurement of the object). Alternatively or in addition, a pixel may include information about, for example, material properties (such as, related to the scattering of light in the imaging area or the reflection of light of a particular wavelength).

[0003] Thus, a pixel value may, for example, be related to intensity and / or range data and / or material properties of the pixel.

[0004] Line scan image data results when image data for an image is scanned or provided one line at a time, for example, by a camera having a sensor configured to sense and provide image data one line of pixels at a time.

[0005] A special case of line scan images are image data provided by a so-called "sheet of light" or light plane triangulation. Lasers are often preferred, but other light sources can be used, such as those that provide light that remains focused and does not spread too much (that is, "structured" light, such as provided by lasers or light emitting diodes (LEDs)).

[0006] 3D machine vision systems are often based on such light triangulation. In such systems, there is a light source that illuminates an object with structured light corresponding to a specific light pattern, such as a light plane that causes a light or laser line on the object, and along this line, the 3D properties of the object corresponding to the contour of the object are captured. By scanning the object with such a line, i.e. performing a line scan involving movement of the line and / or the object, the 3D properties of the entire object corresponding to multiple contours can be captured.

[0007] A 3D machine vision system or device based on light triangulation and, for example, using a light sheet for light triangulation may be referred to as a 3D imaging system or device based on light or light sheet triangulation (or simply laser triangulation when using lasers).

[0008] Typically, to generate a 3D image based on light triangulation, reflected light from the object to be imaged is captured by the image sensor of the camera, and intensity peaks are detected in the image data. The peaks occur at locations corresponding to where the incident light is on the imaged object (e.g., corresponding to a laser line reflected from the object). The locations of the detected peaks in the image will be mapped to the locations on the object where the light causing the peaks is reflected.

[0009] An optical or laser triangulation system (i.e., a 3D imaging system based on optical triangulation) typically projects a light or laser line onto an object to create a height curve from the surface of the object. By moving the object relative to the camera and light source involved, information about the height curve from different parts of the object can be captured through images, which are then combined and used with the knowledge of the relevant geometry of the system to produce a three-dimensional representation of the object, i.e., 3D image data is provided. This technique can be described as capturing an image of the light when light (usually a laser line) is projected onto an object and reflected by the object, and then extracting the position of the reflected light line in the image. This is typically achieved by identifying the position of the intensity peak in the image frame, for example using a conventional peak finding algorithm. Typically, but not necessarily, the imaging system is set up so that the intensity peak can be searched for in each column of the sensor, and the position within the column is mapped to height or depth.

[0010] In many applications, it is desirable to obtain not only 3D image data (such as the height and depth of an object) but also conventional 2D image data of the object, for example in order to provide texture to a 3D model of the object formed from the 3D image data, thereby enabling the model to appear to be and / or be an even better representation of a real-world object. Additionally or alternatively, for example for quality assurance reasons, it is of interest to obtain certain 2D information from the surface of the object. It may be particularly of interest to obtain such information in the same position as the 3D image data, i.e. so that the 2D image data is used for and / or aligned with a 3D position corresponding to the actual position on the object at which both the 3D image data and the 2D image data are captured. In the case of light triangulation, it is therefore of interest to obtain 2D image data associated with the positions of intensity peaks. With the 2D image data associated with the positions of the 3D image data, for example, the 2D image surface of a 3D model of the object can be analyzed to identify, for example, text, scratches, markings and / or color variations, etc., and if these are also associated with height variations, etc., it can be identified where they are located on the actual object.

[0011] For example, using the same light (e.g., laser) as used in optical triangulation to provide 3D image data, grayscale 2D image data from reflectance and intensity can be obtained. However, in the case of lasers, due to the special properties of lasers, so-called speckle noise often occurs, which then occurs in the 2D image. Furthermore, it can be appreciated that the 2D information that can be obtained from the surface of an object in this way is very limited. Summary of the invention

[0012] In view of the above, it is an object to provide one or more improvements or alternatives to the prior art, such as providing improved ways of how 2D image data can be associated with 3D image data obtained by optical triangulation.

[0013] According to a first aspect of embodiments herein, an object is achieved by a method for associating multispectral 2D image data with 3D image data resulting from light triangulation performed by an imaging system for 3D imaging of an object. The imaging system comprises a first light source for illuminating the object with a first light, a camera with an image sensor and one or more second light sources for illuminating the object with two or more second lights, which differ from each other by including different light wavelengths, the two or more second lights being multispectral. The light triangulation comprises illuminating successive portions of the object with the first light and sensing, by the image sensor, reflected first lights from the respective portions in respective first images (IM1).

[0014] The method comprises obtaining the 3D data as a first sensor position (SP1) of IM1 generated by the camera and image sensor, wherein SP1 corresponds to a location of an intensity peak of reflected first light from an object as part of the light triangulation.

[0015] In addition, the method includes: obtaining two or more second images (IM2) generated by the camera and the image sensor, and imaging the object during illumination by the two or more second lights, respectively. The corresponding IM2 is a corresponding IM1 among the IM1s and is associated with a corresponding IM1 among the IM1s, or is generated between two IM1s among the IM1s and is associated with any IM1 among them.

[0016] The method further comprises selecting, in the respective IM2, for and in conjunction with the respective SP1 of the respective IM1 to which the respective IM2 is associated, a respective second sensor position (SP2) at which the reflected second light from the object has a higher intensity than the reflected first light from the object.

[0017] Furthermore, the method includes associating the intensity values ​​of the selected SP2 with the SP1 for which SP2 is selected, respectively, whereby the multispectral 2D data corresponding to the intensity value at SP2 from the reflected multispectral second light becomes associated with the 3D data corresponding to the SP1 for which SP2 is selected.

[0018] According to a second aspect of embodiments herein, the object is achieved by one or more devices for associating multispectral 2D image data with 3D image data generated from light triangulation performed by an imaging system for 3D imaging of an object. The imaging system comprises a first light source for illuminating the object with a first light, a camera with an image sensor, and one or more second light sources for illuminating the object with two or more second lights, which differ from each other by including different light wavelengths, the two or more second lights being multispectral. The light triangulation comprises illuminating successive portions of the object with the first light and sensing, by the image sensor, reflected first lights from the respective portions in respective first images (IM1). The one or more devices are configured to:

[0019] The 3D data is obtained as a first sensor position (SP1) of IM1 generated by the camera and the image sensor, wherein SP1 corresponds to a location of a peak intensity of a reflected first light from an object as part of the light triangulation. Additionally, the method comprises:

[0020] Two or more second images (IM2) generated by the camera and the image sensor are obtained, and an object is imaged during illumination by the two or more second lights, respectively. A corresponding IM2 is a corresponding IM1 among the IM1s and is associated with a corresponding IM1 among the IM1s, or is generated between two IM1s among the IM1s and is associated with any IM1 among them.

[0021] In the respective IM2, for and in conjunction with the respective SP1 of the respective IM1 to which the respective IM2 is associated, a respective second sensor position (SP2) is selected where the reflected second light from the object has a higher intensity than the reflected first light from the object.

[0022] The intensity values ​​of the selected SP2 are respectively associated with the SP1 for which SP2 is selected, so that the multispectral 2D data corresponding to the intensity value at SP2 from the reflected multispectral second light becomes associated with the 3D data corresponding to the SP1 for which SP2 is selected.

[0023] According to a third aspect of the embodiments herein, the object is achieved by one or more computer programs, which include instructions that, when executed by one or more processors, cause the one or more devices according to the second aspect to perform a method according to the first aspect.

[0024] According to a fourth aspect of the embodiments herein, the object is achieved by one or more carriers, which include the one or more computer programs according to the third aspect, wherein the one or more carriers are one or more of the following: an electronic signal, an optical signal, a radio signal or a computer-readable storage medium.

[0025] Embodiments herein provide multispectral 2D image data that is associated and aligned at the sensor location or pixel level with 3D image data from light triangulation using the same camera, which can typically be a conventional monochrome camera used in conventional 3D imaging based on light triangulation. Additionally, the 2D image data will also share focus with the 3D image data, and when, for example, Scheimpflug is used by the imaging system to typically give a focus in the light plane where there is illumination of the first light and thus a focus on the object generating the 3D image data, the multispectral 2D image data will also benefit from such focus. This is not limited to obtaining 2D image data from illumination of the first light used only to obtain the 3D image data.

[0026] As a result, 3D imaging of an object can be achieved based on light triangulation, with focused and well-aligned multispectral 2D image data, for example in the form of a textured (e.g., full-color) surface of a 3D object obtained by 3D imaging. In addition, various 2D image data can be obtained from different combinations of second lights of several different wavelengths, and / or using different kinds of light sources and / or illumination (including illumination from different positions and directions).

[0027] The 2D image data is obtained from IM2, which may be the same image (IM1) from which the associated 3D image data is obtained. And / or the 2D image data may come from IM2, which is separate from IM1 but is generated between IM1, so that the 2D image data is thereby related to and associated with the 3D image data. In the latter case, IM2 is generated in the time period between two consecutive IM1s with 3D image data from optical triangulation. The embodiment with IM1=IM2 can achieve less data processing and facilitate implementation using existing systems, while the embodiment with IM2 separate from IM1 can achieve greater freedom in selecting SP2 and the exposure time period for the second light and how the illumination of the second light can be provided in combination with the first light (such as, laser) used by optical triangulation.

[0028] Furthermore, embodiments herein can improve the detectability of object surface details, such as defects, due to the multispectral 2D image data combined and aligned with the 3D image data from light triangulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Examples of embodiments herein are described in more detail with reference to the schematic drawings that are briefly described below.

[0030] FIG. 1 schematically illustrates an example of a prior art imaging system on which embodiments herein can also be used and / or based.

[0031] Figure 2A A simplified first example of an imaging system that can be configured to perform optical triangulation and embodiments of the earlier application is schematically illustrated.

[0032] Figure 2B is a flow chart for schematically illustrating the method disclosed in the earlier application.

[0033] Figure 3A -B schematically illustrates the method disclosed in the earlier application and combined with Figure 2B Two main groups of scenarios of the embodiments are described.

[0034] Figure 4A - C schematically illustrates an example associated with a first main group of the main groups, and Figure 4B -C is an example associated with the second primary group among the primary groups.

[0035] Figure 5A A simplified example of an imaging system that can be configured to perform optical triangulation and embodiments herein is schematically illustrated.

[0036] Figure 5B is like Figure 5A 2 is a schematic example of a second light source implemented as a single illumination unit including a plurality of second light sources.

[0037] Figure 5C It is schematically illustrated that the second light may be non-overlapping in wavelength.

[0038] Figure 5D It is schematically illustrated that the second lights may partially overlap in wavelength.

[0039] Figure 6 is a flow chart for schematically illustrating a method according to embodiments herein.

[0040] Fig. 7A -B schematically illustrate the combination of Figure 3A-B corresponds to the scenario discussed in the two main groups, but in the case of multi-spectral second light and combined with Figure 6 The method according to the embodiments herein is described.

[0041] Fig. 8A -E schematically illustrates examples of different sequences with the second light.

[0042] Fig. 9A -D shows exemplary images and results from real and practical applications of the embodiments herein.

[0043] Fig.10 is a schematic block diagram for illustrating embodiments of apparatus for performing embodiments herein.

[0044] Fig.11 is a diagram for combining with a device for performing Figure 6 Schematic diagrams of some embodiments related to computer programs and carriers of the discussed methods and actions. DETAILED DESCRIPTION

[0045] The embodiments herein are exemplary embodiments. It should be noted that these embodiments are not necessarily mutually exclusive. Components from one embodiment may be assumed to exist in another embodiment by default, and it will be apparent to those skilled in the art how those components may be used in other exemplary embodiments.

[0046] When an optical triangulation light source (typically a laser) is used to obtain a reflectance image (i.e., an image having 2D image data of an object), disadvantages include speckle noise in the image and the inability to glean alternative surface information beyond that obtainable from the light used in the optical triangulation.

[0047] On the other hand, if an image of the object is obtained separately from the optical triangulation, it is difficult, if not impossible, to obtain the desired alignment with the 3D image of the measurement object obtained by optical triangulation in all applications.

[0048] A "multi-scan" scheme may be used, where for example one or more separate rows of imagers (i.e., the image sensors used) can be used to collect 2D image data, but this still does not have any good alignment with the 3D image data. Additionally, it is difficult to create good focus for both the optical triangulation that most imagers are used for and the separate 2D image data readout. An optical triangulation device will typically have maximum focus only around the plane of light (usually laser light) used for optical triangulation, using Scheimpflug focusing and the ability and desire to use a large aperture opening to allow more light to reach the imager to achieve this.

[0049] It would be desirable to obtain 2D image data in a manner that is not limited to the use of light thus used in optical triangulation, and at the same time is able to obtain useful 2D image data that is well aligned with the 3D image data from the optical triangulation.

[0050] The applicant's earlier application EP22154020.6 relates to solutions for the above content. The present application and the embodiments here are based on the solutions of the earlier application, but specifically target embodiments with multispectral 2D image data, that is, how additional light with different wavelengths can be used to generate 2D image data with associated 3D image data, such as full-color image data. Part of the information disclosed in the earlier application is repeated below to (re)introduce some basic knowledge and promote understanding of the embodiments here. However, some reference numerals and nomenclature have been changed to better fit the context of the present application.

[0051] FIG. 1 schematically illustrates an example of a prior art imaging system 100 for 3D imaging based on light triangulation, on which the embodiments herein may be based. The imaging system 100 may alternatively be named, for example, an imaging system for 3D machine vision based on light triangulation for capturing information about 3D properties of an object. The imaging system 100 is shown in the figures in a scenario of normal operation, that is, typically after calibration has been performed and the system is thus calibrated. The imaging system 100 is configured to perform light triangulation, here in the form of light sheet triangulation, that is, light triangulation in which structured light as a light plane is used. The imaging system 100 further comprises: a light source 110, such as a laser, for illuminating the object to be imaged with structured light, typically a specific light pattern. In the examples and figures, there is a structured light 111 in the form of a light plane. The light may be a laser, but need not be a laser, and it may alternatively be, for example, light from one or more light emitting diodes (LEDs). Another example of a structured light similar to a conventional light plane but not a conventional light plane but having a similar effect is a light edge, that is, an edge of an area with illumination. The generated light and the illumination are usually provided through one or more lenses, for example, in order to focus the light. In addition, the camera is usually configured and arranged so that, based on the so-called Scheimpflug principle or Scheimpflug focusing, it will have a focal plane that is co-located with the structured light 111 (in other words, aligned with the structured light 111), which is usually a light plane. In this way, the reflection of the object occurring in the light plane will be focused in the image sensor. In the example shown, the object subjected to imaging is exemplified by a first object 120 in the form of a car and a second object 121 in the form of a gear structure. The object being imaged can be referred to as a measurement object. When the structured light 111 is incident on the object, this corresponds to projecting the structured light 111 on the object, which can be regarded as the intersection of the structured light 111 and the object. For example, in the example shown, the structured light 111 in the form of the light plane causes a light ray 112 on the first measurement object 120. The structured light 111 is reflected by the object, more specifically, by the part of the object at the intersection (that is, in the example shown, at the light ray 112). The imaging system 100 also includes a camera 130, which includes an image sensor (not shown in FIG. 1 ). The camera and the image sensor are arranged in conjunction with the light source 110 and the object to be imaged so that when the structured light 111 is reflected by the object, the structured light 111 becomes incident on the image sensor. The image sensor is a device, usually implemented as a chip, for converting incident light into image data. The portion of the object that causes the incident light on the image sensor by reflection can thereby be captured by the camera 130 and the image sensor, and corresponding image data can be generated and provided for further use.For example, in the example shown, light 112, structured light 111 on a portion of the roof of the first object 120 will be reflected towards the camera 130 and image sensor, which can thereby generate and provide image data having information about the portion of the roof. According to the principles of light triangulation, using knowledge of the geometry of the imaging system 100 relative to the object being imaged, such as how the image sensor coordinates relate to world coordinates (such as coordinates of a coordinate system 123, such as Cartesian coordinates), the image data can be converted into information about 3D properties of the object being imaged, such as in the form of a 3D shape or contour, in a suitable format. The information about the 3D properties may include data describing the 3D properties in any suitable format.

[0052] By moving the light source 110 and / or the object to be imaged (such as the first object 120 or the second object 121) so that multiple parts of the object are illuminated and cause reflected light to be sensed by the image sensor in the image, in fact, typically by scanning the object with the structured light 111, image data describing a more complete 3D shape of the corresponding object can be generated, for example corresponding to a plurality of successive outlines of the corresponding object, such as the outline images 140-1-140-N of the first object 120 shown. Each outline image shows the shape of the first object 120 from which the structured light 111 is reflected when the image sensor 130 of the camera unit senses the light that causes the outline image. As indicated in the figure, a movable object support structure 122 (such as a conveyor belt or the like) can be used to move the object through the structured light 111, and thereby scan the object, the light source 110 and the camera unit 130 are typically stationary. Alternatively, the structured light 111 can be moved over the object so that all parts of the object, or at least all parts facing the light source 110, are illuminated, and the camera receives light reflected from all parts of the object that it is desired to image.

[0053] As will be appreciated from the above, for example, the corresponding images corresponding to the image frames provided by the camera 130 and its image sensor during the imaging of the first object 120 may correspond to any one of the contour images 140-1-140-N, or be used to provide any one of the contour images 140-1-140-N. Each position of the outline of the first object shown in any of the contour images 140-1-140-N is determined, typically based on the identification of intensity peaks in the image data captured by the image sensor and based on finding the positions of these intensity peaks. The imaging system 100 and conventional peak finding algorithms are typically configured to search for intensity peaks for each pixel column in each image frame. If the sensor coordinates are u, v and for example as shown in the accompanying drawings u corresponds to a pixel position along a row in the image sensor and v corresponds to a pixel position along a column, then for each position u of the image frame, the peak position is searched along v, and the identified peaks in the image frame may result in one such "clean" contour image, as shown in the accompanying drawings, and all of the image frames and contour images can be used to create a 3D image of the first object 120.

[0054] Figure 2A A simplified first example of an imaging system 200 that can be configured to perform optical triangulation and embodiments of the earlier application is schematically illustrated. The imaging system 200 is based on optical triangulation for capturing information about 2D and 3D properties of one or more objects, which may be referred to as measurement objects. The system shown corresponds to a basic configuration and comprises: a first light source 210 for illuminating the one or more objects, here illustrated in the figure by object 220, with a first light 211 (typically a laser) as part of optical triangulation for 3D imaging of the measurement object 220. A second light source 250 for illuminating the object 220 with a second light 251 for 2D imaging of the measurement object 220. A camera 230, having an image sensor 231, is arranged for sensing reflected first light from the object 220 as part of the optical triangulation for 3D imaging and is arranged for sensing reflected second light from the object 220 for the 2D imaging.

[0055] The camera 230, the image sensor 231 and the first light source 210 are configured and arranged relative to each other for light triangulation and may therefore be such a configuration or be based on such a configuration used in conventional light triangulation. For light triangulation purposes, the system 200 may correspond to the system 100 in FIG. 1 , i.e. with respect to the first light source 210, the camera 230, the image sensor 231, how they are arranged relative to each other and the measurement object 220, etc., but the system 200 is additionally configured to perform as described below. The image sensor 231 should of course also be sensitive to the second light being used.

[0056] Thus: Object 220 may correspond to first object 120 and is shown as being at least partially within field of view 232 of camera 230. First light source 210 is configured to illuminate measurement object 220 with first light 211, which is light used for light triangulation and thus is typically structured light, such as a specific light pattern, e.g. a light sheet or light plane provided by, e.g., a laser. First light 211 is reflected by object 220 and the reflected first light is captured by camera 230 and image sensor 231 as part of light triangulation. Another example of structured light that can be used as first light is a light edge, i.e. an edge with an illuminated area or portion.

[0057] The object 220 may thus be illuminated and an image may be captured, as in conventional optical triangulation. As such, the optical triangulation may thus be as in the prior art and involve movement of the first light source 210 and / or the object 220 relative to each other, such that at different successive moments, different successive portions of the object 220 are illuminated by the first light source 210 and the first light 211, and the reflected first light 211 from the object 220 is sensed by the image sensor 231. As in conventional optical triangulation, typically but not necessarily, the camera 230 and the first light source 210 are fixed relative to each other, and the object 220 moves relative to these. By the sensing performed by the image sensor 231, the respective image frames are associated with the respective moments when the image frames are sensed (i.e., captured), and are associated with the respective portions of the measured object 220 from which the image sensor 231 sensed the reflected first light 211 at the respective moments.

[0058] The camera 230 may be a prior art camera, such as corresponding to the camera 130 in the system 100 of FIG. 1 , and the image sensor 231 may be the same or similar image sensor discussed above in connection with FIG. 1 . For further processing outside of the camera 330 , the image frames provided by the camera 230 and the image sensor 231 and / or information derived therefrom may be desirable to be transferred (e.g., transmitted) to, for example, a computing device 233 , such as a computer or the like. Such further processing may additionally or alternatively be performed by a separate computing unit or device (not shown), i.e., separate from the image processor 231 , but still included in the camera 230 (e.g., integrated with the camera 230 ) or included in a unit including the camera 230 . The computing device 233 (e.g., a computer) or another device (not shown) may be configured to control devices involved in optical triangulation and / or involved in the embodiments herein, so that optical triangulation and / or other actions related to the embodiments herein are performed.

[0059] As in conventional 3D imaging performed by optical triangulation, the first light source 110 and the camera 230 are typically arranged at predetermined fixed positions and have a known relationship to each other for optical triangulation. The imaging system 200 also includes a second light source 250, which may also be located at a fixed position relative to the camera 230 and the first light source 210, but the exact position and relationship relative to the camera 230 and the image sensor 231 is not used in the embodiments here as in optical triangulation, and the second light source can therefore be arranged more freely for the purpose of providing illumination by the second light.

[0060] Figure 2B is a flow chart for schematically illustrating the method disclosed in the earlier application. The following actions that may form the method are used to obtain 2D image data and associate the 2D image data with 3D image data. The 3D image data is based on a first intensity peak position obtained by light triangulation performed by an imaging system, which is exemplified below by imaging system 200. The first intensity peak position may therefore be an intensity peak position as determined in conventional light triangulation. They are explained and illustrated in conjunction with the following embodiments herein and in separate figures and examples. An imaging system that may perform the method therefore includes: a first light source for illuminating an object with a first light; and an image sensor for sensing reflected first light from an object. In the following, the first light source and the first light are exemplified by first light source 210 and first light 211, and the object and the image sensor are exemplified by object 220 and image sensor 231.

[0061] As in traditional optical triangulation, optical triangulation includes:

[0062] A first portion of object 220 is illuminated with first light 211 and reflected first light from the first portion is sensed by image sensor 231 during a first exposure period (EXP1-1), resulting in a first image (IM1-1) with a first intensity peak occurring at a first sensor position (SP1-1).

[0063] Another second portion of the object 220 adjacent to the first portion is illuminated by the first light 211, and the reflected first light from the second portion is sensed by the image sensor 231 during another first exposure time period (EXP1-2), resulting in another first image (IM1-2) with an intensity peak occurring at another first sensor position (SP1-2).

[0064] The first image (IM1) thus corresponds to consecutive image frames, has a first intensity peak at SP1, and is part of the light triangulation.

[0065] In the following, and in separate figures and examples, the exposure time periods, images, sensor positions SP1 - 1 and SP1 - 2 are explained and illustrated.

[0066] Furthermore, an imaging system such as imaging system 200 further comprises one or more further second light sources for illuminating object 220 with one or more second lights different from the first light. In the following, the second light source and the second light are exemplified by second light source 250 and second light 251.

[0067] The following Figure 2B The methods and / or actions indicated in may be performed by a device (i.e., one or more devices, such as camera 230 and / or computing device 233), or by imaging system 205 and / or a suitable device thereof or a suitable device connected to it.

[0068] It should be noted that the following actions may be performed in any suitable order, and / or performed with full or partial overlap in time when possible and appropriate.

[0069] Action 201

[0070] The object 220 is illuminated with the one or more second lights 251, and reflected second lights from the object 220 are sensed by the image sensor 231 during one or more second exposure time periods (EXP2), resulting in one or more second images (IM2), respectively.

[0071] The illumination should be provided so that during any first exposure period (EXP1), any reflected second light will result in a lower intensity compared to the intensity of the reflected first light from the first sensor position (SP1). This ensures that the second light interference with the first light intensity peak is low. Figure 4A -C, further explains and illustrates this point.

[0072] The following combination Figure 3A -B explains and discusses the images and exposure time periods separately for the two main groups.

[0073] Action 202

[0074] For a corresponding first sensor position (SP1) in the first image, a corresponding second sensor position (SP2) is selected in the one or more second images (IM2). The illumination with the one or more second lights 251 should be provided so that during the one or more second exposure time periods (EXP2), any reflected second light from the object 220 will result in a higher intensity compared to the reflected first light 211 at the selected second sensor position (SP2). This ensures that the reflected second light is not obscured by the reflected first light at the second sensor position (SP2) and / or interfered in an undesirable manner. When necessary, this can also be achieved by appropriately selecting SP2 relative to SP1 as explained below. In combination Figure 4A -C, see below for further explanation and examples.

[0075] The second sensor position (SP2) may be selected with a predefined or predetermined relationship to the first sensor position (SP1) obtained by light triangulation. For example, it may be predetermined that the respective SP2 will be selected with a predefined or predetermined relationship to the respective SP1, such as being at a predefined or predetermined distance relative to the respective SP1 and / or at a predefined or predetermined direction relative to the respective SP1. In practice, a suitable direction is typically a direction on the image sensor mapped to "height", typically along a column of the image sensor, that is, in the example here, along v in sensor coordinates mapped to z in real world coordinates in the coordinate system used in the example here. More generally, the direction in which SP1 is searched may be a sensor direction along which an intensity peak from the light triangulation is expected and which corresponds to an object point at which the first light is reflected by the object directly towards the camera. Since the light distribution in this direction is given by the structured first light, such as corresponding to the light or laser plane and how it is projected as a line on the object, it is known or can be easily found (if necessary) how the light intensity decreases in this direction. Thereby, a distance, for example in pixels, can be found in advance, which can be used as the predetermined distance, where the second light intensity will be higher than the first light intensity in case both the first light and the second light are captured in the same image, as in the case of the first main group mentioned below. Figure 4A -C further explains and illustrates the sensor positions SP including the selection of a second sensor position SP2 and the difference between the two main groups.

[0076] Action 203

[0077] 2D image data is obtained from a corresponding second sensor position (SP2) in the one or more second images (IM2) (i.e., the second sensor position selected in action 202), and the obtained image data is associated with the first sensor position (SP1) for which the corresponding second sensor position (SP2) was selected. As a result, the obtained 2D image data becomes associated with the 3D image data because the 3D image data from the light triangulation is based on or corresponds to the first intensity peak position of the first sensor position (SP1).

[0078] As used herein, "2D image data" may refer to a position or pixel of an image sensor (e.g., image sensor 231) obtained by exposure, or image data of a corresponding position in an image generated by the image sensor. 2D image data typically corresponds to light intensity and / or color, and, for example, has the form of an intensity value that indicates or identifies the intensity of the captured light. For the embodiments herein, the 2D image data of a certain position or pixel includes information about the corresponding position on the object that reflects the second light, that is, the 2D image data of a certain position corresponds to image data having information about how the corresponding position of the imaged object reflects the second light.

[0079] As used herein, "3D image data based on intensity peak positions obtained by light triangulation" may be understood as data that includes at least depth and / or height information about a corresponding position on the measurement object of the reflected first light that caused the intensity peak position. For example, for the contour images 141-1...141-N shown in conjunction with FIG. 1, the identified intensity peak positions (as SP1) are shown in terms of sensor coordinates u, v, and the intensity peak position at v for each position u includes information about the depth of the imaging object 120, more specifically, depth information about the position on the object 120 of the reflected light 112 that caused the intensity peak position at v. In other words, the sensor coordinates of the intensity peak position (which may have a sub-pixel resolution) correspond to 3D image data and thus include 3D image data. This can be expressed as, such as, the corresponding first sensor position identified by its coordinates corresponds to 3D image data and thus contains 3D image data.

[0080] In practice, the 3D image data about the position on the object reflecting the first light may be the 3D position of the intensity peak position of u,v,t mapped to the position x,y,z on the object, or it may be the corresponding 3D position of the coordinates x',y',z' of a 3D model of the object, which has been provided by calculations based on u,v,t and known operating conditions and relationships used in light triangulation.

[0081] The method and actions thus involve providing 2D image data associated with 3D image data from light triangulation. The 2D image data are associated with the same location on the measurement object from which the 3D image data originate, because, as explained above and as further explained and exemplified below, the 2D image data are obtained in association with and preferably in conjunction with obtaining an image frame (the first image (IM1) having a first intensity peak (SP1) for light triangulation) and by using the same camera and image sensor, but not limited to obtaining the 2D image data from illumination performed by the same light source used to obtain the 3D image data. As a result, various 2D image data associated with the 3D image data can be obtained, for example based on using different kinds of light and / or illumination, including illumination from different directions and locations.

[0082] The second light 251 or a plurality of second lights (e.g., in the case of several second light sources) are beneficially diffused, and / or may be provided by one or more light emitting diodes (LEDs), so the second light source 250 may be based on the light emitting diodes (LEDs). At least one second light source may provide diffuse second light. Diffuse light contrasts with the usually very directional first light 111 (such as, laser). Diffuse light enables reflectivity without laser speckle. Another advantage of diffuse second light is a simpler system setup, and it is easier to achieve the right intensity at SP2 and also at SP1. Diffuse light is also closer to the illumination that occurs normally, and the expected 2D image data about the measured object can therefore be better captured. Some embodiments may also become easier to implement using diffuse second light.

[0083] Figure 3A -B schematically illustrates the method disclosed in the earlier application and combined with the above Figure 2B Two main groups of scenarios of the described embodiments. In short: Figure 3A With respect to the first main group, the second image (IM2) is identical to the first image (IM1), and therefore, the second exposure period (EXP2) is also identical to the first exposure period (EXP1). Figure 3B Alternatively, with respect to the second main group, the second image (IM2) is separated from the first image (IM1), whereby it also has its own separate exposure time period (EXP2), which is located between the first exposure time periods, here between EXP1-1 and EXP1-2 of the first images IM1-1, IM1-2. The second image IM2-1 and its second exposure time period EXP2-1 are used for this group and are therefore generated between IM1-1 and IM1-2, that is, between the images used for light triangulation.

[0084] Regarding the first major subgroup, Figure 3AMore specifically and schematically, a second image 342a-1 corresponding to IM2-1 is shown, and in this example, the second image 342a-1 is the same image as the first image 341a-1 corresponding to IM1-1. The second image 342a-1 is produced using a second exposure time period 362a-1 corresponding to EXP2-1, and for this group, the second exposure time period 362a-1 is the same as EXP1-1 and the first exposure time period 361a-1. Another first image 341a-2 corresponding to IM1-2 is also shown, the first exposure time period 361a-2 corresponding to EXP2. The images IM1-1 and IM1-2 shown therefore correspond to images for light triangulation. The first image IM1-1 341a-1 is associated with a capture time t1, and the second image 341a-2 is associated with a capture time t2. The difference T between t2 and t1 3D (That is, T 3D = t2-t1) is a time period 363a which corresponds to the time period between subsequent image portions of the optical triangulation used to provide the 3D image data, typically corresponding to the scan rate. It should be noted that the time length of the exposure time period illustrated is only to show that there is also an extended portion of time involved, as in conventional 3D imaging performed by optical triangulation, but what is shown is schematic and does not illustrate any actual time period T between subsequent image frames used in the optical triangulation. 3D The exposure time period and the time period between subsequent images can be the same as conventional light triangulation, but in some cases it may be interesting to adjust the exposure time period to accomplish appropriate sensing of both the reflected first light 211 and the reflected second light 251 in the same image.

[0085] In the case of the first main group, like Figure 2B As in action 202 of the method in , a respective second sensor position (SP2) is selected in the first image 341a-1 (ie, in IM1-1) having a respective sensor coordinate difference (d) relative to the respective first sensor position (SP1).

[0086] Regarding the second main group, Figure 3BMore specifically and schematically, a second image 342b-1 corresponding to IM2-1 is shown, and in this example, the second image 342b-1 is an image separate from the first image 341b-1 corresponding to IM1-1. The second image 342b-1 is generated using a second exposure time period 362b-1 corresponding to EXP2-1, which is therefore a new and / or separate exposure time period and relates to an image that is not used for light triangulation as such but is located in a time period between such images, here in a time period between a first exposure time period 361b-1 corresponding to EXP1-1 and another first exposure time period 361b-2 corresponding to EXP1-2. The second exposure time period 362b-1 is an exposure time period of the second image 342b-1 corresponding to IM2-1. It should be appreciated that the figure shows an example in which the one or more second exposure time periods (EXP2) and the one or more second images (IM2) are only one. However, in reality, there are usually additional subsequent IM2s having the second exposure period, such as IM2 - 2 with EXP2 - 2 following IM2 - 1 with EXP2 - 1 , and so on.

[0087] The first images IM1-1, IM1-2 and the first exposure time periods EXP1-1, EXP2-1 and the time period T 3D Can be with them Figure 3A The first exposure time period for the first image may be the same as the corresponding part in , and / or may be the same as conventional light triangulation, and the first exposure time period for the first image may typically have the same length.

[0088] exist Figure 3A -B in both and for both main groups, the first images IM1-1 and IM1-2 correspond to capturing the first light and are therefore used in the above combination Figure 2B The image IM2 - 1 in and used for the light triangulation discussed, and the second image IM2 - 1 are used to capture the reflected second light 251 , and thereby capture 2D image data.

[0089] The difference compared to the first main group is that for the second main group, during the one or more second exposure time periods (EXP2), such as during 362b-1, the first light 211 may be prevented and beneficially prevented from illuminating the object 220, for example by being switched off, attenuated or directed elsewhere. This is possible due to the fact that the second image IM2 is separate from the first image IM1, that is, a separate second image is captured reflecting the second light 251. This can be used to remove the risk of light interference from the first light (e.g. laser) in IM2 and thereby also achieve greater freedom in the selection of SP2.

[0090] Figure 4A- C schematically illustrates an example associated with a first main group of the main groups, and Figure 4B -C is an example associated with the second primary group among the primary groups.

[0091] Related to the first major group Figure 4A 21 and 25. The light intensity along the image sensor coordinate v is shown, corresponding to reflected light from an object (e.g., object 220), such as sensed by the image sensor 231 during any of the first exposure time periods (EXP1) and, for example, included in one of the first images (IM1). As described above, for embodiments according to the first main group, IM2=IM1 and EXP2=EXP1. For reasons of explanation, the contributions from reflected first light 211 and reflected second light 251 are shown separately in the figures. The contribution from reflected first light (such as first light 211 reflected from object 220 and sensed by the image sensor 231) corresponds to the first light intensity distribution 413a in the figures. The contribution from reflected second light (that is, second light 251 reflected from object 220 and sensed by the image sensor 231) corresponds to the second light intensity distribution 453a in the figures. In other words, first light intensity distribution 413a is an example of first light 211 that has been reflected (i.e., reflecting the first light), and second light intensity distribution 453a is an example of second light 251 that has been reflected (i.e., reflecting the second light). In practice, light intensity distributions 413a and 453a are superimposed, corresponding to adding the distributions shown, thereby forming a single light intensity distribution corresponding to the light intensity distribution sensed by the image sensor.

[0092] The first light intensity distribution 413a is relatively narrow and has an intensity peak position 471a corresponding to SP1, which is usually the case when the illumination is structured light (e.g., a light plane) that causes light (such as laser light) on the object. The second light intensity distribution 453a is at a substantially constant level, which is usually the case when the second light is provided as diffuse light to illuminate the object.

[0093] Combination of the above Figure 3A Referring to action 202, a corresponding second sensor position SP2 is selected in the first image IM1 with a corresponding sensor coordinate difference (d) relative to the corresponding first sensor position (SP1). The second sensor position 473a corresponding to SP2 is shown in Figure 4A The difference d is shown in the figure as difference 475a.

[0094] Additionally, above under action 202, it is noted that the illumination with the one or more second lights 251 should be provided such that during the one or more second exposure time periods (EXP2), any reflected second light from an object, such as according to the second light intensity distribution 453a, results in a higher intensity than the reflected first light 211 at the selected second sensor position (SP2). An example of how this might look like when, like the first main group, IM2=IM1 and EXP2=EXP is shown in Figure 4A In the example, at the second sensor position 473a, namely, at SP2, the second light intensity distribution 453a is higher than the first light intensity distribution 413a. Therefore, at SP2, there is a difference in light intensity contribution between the first light and the second light, namely, I DIFF (SP2), which is shown in the accompanying drawing as light intensity difference 479a.

[0095] In contrast, at the first sensor position 471a, i.e., at SP1, the second light intensity distribution 453a is lower than the first light intensity distribution 413a. This is also consistent with the above combination Figure 3A As indicated by action 201, illumination may be provided so that during the first exposure period (EXP1), any reflected second light should result in a lower intensity than the intensity of the reflected first light from the first sensor position (SP1). Therefore, there is also a light intensity difference between the first light and the second light at SP1, i.e., I DIFF (SP1), which is shown in the accompanying drawing as light intensity difference 479a.

[0096] From this figure, it should be appreciated why SP2 should be selected with a difference d relative to SP1 for the first main group, i.e. where the first light and the second light are sensed in the same image, i.e. where IM1 and IM2 are identical. The reason is that the second light should dominate at SP2 and preferably be affected as little as possible by the first light, but this may be difficult to avoid completely when IM2 = M1 and the reflected first and second light are imaged simultaneously. Of course, SP2 should also be a position close enough to SP1 so that the 2D image data at SP2 is considered to be related to the 3D image data based on SP1.

[0097] Similarly, the reflected first light should advantageously dominate at SP1 so that the intensity peak and thus the light triangulation is not adversely affected. Despite the presence of any second light, the intensity peak position should be identifiable, e.g. as conventional. The first light intensity peak should preferably be affected as little as possible by the second light.

[0098] In practice, a skilled person can easily provide a suitable intensity level and / or light distribution for the second light, such as illustrated in the accompanying drawings. In the case of, for example, diffuse light and uniform distribution of the second light, for a given SP2, an increased IDIFF (SP1) means reduced I DIFF (SP2). It is then about finding a suitable balance between these, and / or selecting SP2 appropriately. For most practical scenarios, e.g. with a uniform second light distribution, this will not be a problem because the second light intensity level will generally have no problem staying well below the first light peak intensity level, which also decreases rapidly when moving away from SP1, i.e. the peak is very narrow. If required for a particular application, the skilled person will be able to select a suitable SP2 and second light intensity level for SP2, e.g. by routine testing and experimentation.

[0099] Figure 4B -C is associated with the second primary group and shows the light intensity along the image sensor coordinate v, corresponding to reflected light from an object (eg, object 220 ) and sensed by the image sensor 231 . Figure 4B shows reflected first light 211 sensed during EXP1 and, for example, included in IM1, while Figure 4C Reflected second light 251 is shown sensed during EXP2 and, for example, included in IM2, which is separated from IM1 as explained above for the second main group. Figure 4B A first light intensity distribution 413b is shown which corresponds to and may be identical to the first light intensity distribution 413a. Figure 4C A second light intensity distribution 453b is shown, which corresponds to and may be identical to the second light intensity distribution 453a. In this scenario, although not shown in the drawings, in the case of an additional second image IM2 (such as IM 2-2), the image Figure 4C Another figure of the same can be provided, with another second light intensity distribution obtained by sensing of reflected second light during another EXP2, such as EXP2-2 resulting in the another IM2 (that is, IM 2-2-).

[0100] The first sensor position 471b corresponding to SP1 is shown at Figure 4B The second sensor position 473b corresponding to SP2 is shown in Figure 4C middle.

[0101] Combination of the above Figure 3B It is noted that the first light 211 can be prevented and beneficially prevented from illuminating the object 220 during the one or more exposure time periods (EXP2) (e.g., the second exposure time period 362b-1). Any first light can thus not be captured in IM2, which is clearly Figure 4CSimilarly, as also described above, the second light 251 may be prevented from illuminating the object 220 during the first exposure period 361b-1 (ie, during EXP1-1), and therefore, is not captured in IM1. Figure 4B By Figure 4B -C with Figure 4A By comparison, it can be appreciated that this means that the reflected first light and the second light do not interfere with each other, regardless of the selection of SP2 relative to SP1. Therefore, since only the reflected first light or the second light is present in the image at the same time, there is no need for the image Figure 4A SP2 is positioned relative to SP1 at a distance d as in , alternatively, Figure 4B -C, SP2 can be, for example, located at the same sensor position as SP1, which makes the selection of SP2 very simple. In other words, the corresponding second sensor position SP2 in the one or more second images (IM2) (e.g., 342b-1) can be selected at the same corresponding first sensor position SP1 (e.g., 471b) as the intensity peak in the first image (IM1) (e.g., 340-1). Of course, for these embodiments, the distance of the sensor position v corresponding to d is generally acceptable.

[0102] It should be noted that since optical triangulation is usually performed with continuous movement of the measuring object and / or the camera / light source relative to each other, an IM2 at another time that is later than IM1 at time t2, other than, for example, IM1 at time t2, means that there has still been some change in the object position mapped to the same sensor position. In other words, even SP2 at the same sensor position as SP1 may still be mapped with some offset to the actual object position on the measuring object that reflected the light that resulted in the intensity peak position SP1. However, typically, and in principle for all conventional and practical optical triangulation systems, the object position difference between two consecutive images used in and for optical triangulation (such as, between IM1-1 and IM1-2) will typically be smaller than as shown in the combined Figure 4A Any d discussed. However, in order to make the 2D image data at SP2 as relevant as possible to SP1 when located at the same sensor position (e.g., for the same u,v), IM2 may still be provided as quickly as possible after IM1, minimizing the distance caused by any movement between t1 and t2. For example, the one or more second exposure time periods (EXP2) may be exposure time periods adjacent to the first exposure time period (EXP1), such as before EXP1 or as close as possible after EXP1. In the case where EXP2 is several exposure time periods, these may be consecutive and follow each other as quickly as possible and as close as possible to EXP1.

[0103] Of course, it is also possible to select SP2 according to a certain offset of the sensor position to compensate for the movement and change of the measured object position between t1 and t2, but this will generally not be necessary. For embodiments according to the first main group, it may be necessary to accept a larger offset between the object position mapped to SP1 and the object position mapped to SP2 than for the second main group.

[0104] A solution to completely remove any offset is to ensure that there is no relative movement between the object 220 and the image sensor 231 between IM1 and IM2 by temporarily stopping the relative movement used in the optical triangulation (e.g. stopping at t1 and starting again at t2). However, for practical reasons this is usually not a desirable situation and if achieved by modification of an existing optical triangulation system this may also result in reduced optical triangulation system throughput.

[0105] Figure 5A A simplified example of an imaging system 500 that can be configured to perform optical triangulation and embodiments herein is schematically illustrated. The imaging system 500 is thus used for 3D imaging based on optical triangulation and for capturing information about both 2D and 3D properties of one or more objects. Such an object is illustrated in the figure by object 520 and is shown within a field of view 532 of a camera 530 of the imaging system 500. The system shown corresponds to a basic configuration and comprises:

[0106] The first light source 510 is used to illuminate an object 520 corresponding to a measurement object with a first light 511 (thus structured light, such as a light or laser plane) used in light triangulation.

[0107] One or more second light sources for illuminating the object 220 with two or more second lights 551 , which are exemplified in the figure by second lights 551 a - 1 and 551 a - 2 . These lights are used for 2D imaging of the object 520 .

[0108] The camera 530 with image sensor 531 is arranged for sensing reflected first light from object 520 as part of the 3D imaging performed by light triangulation and for sensing reflected second light from object 520 for the 2D imaging.

[0109] Camera 530, image sensor 531 and first light source 510 are configured and arranged relative to each other for optical triangulation and may thus be like in conventional optical triangulation. For optical triangulation purposes, system 500 may correspond to system 100 or system 100 in FIG. 1. Figure 2AThe system 200 in FIG. 1 is similar to the system 200 in FIG. 1 , i.e. with respect to the first light source 510, the camera 530, the image sensor 531, how they are positioned relative to each other and the object 520, etc., but the system 500 is additionally configured to perform according to the embodiments herein. The image sensor 531 should of course also be sensitive to the second light being used. This will generally be the case for all conventional image sensors used for optical triangulation, at least with respect to human visible light, but there is often also sensitivity to light also outside the visible spectrum, at least with respect to adjacent wavelengths, such as with respect to infrared light. If necessary, a conventional 3D imaging system based on optical triangulation can be modified by switching to a camera and / or image sensor that is also sensitive to the second light used in the embodiments herein in addition to the first light used for optical triangulation.

[0110] The imaging system 500 may correspond to the imaging system 200 discussed above for performing the actions and methods of the earlier application, but with the difference that there is one or more light sources configured to provide illumination by at least two second lights having different light wavelength contents (that is, having multi-spectral contents). The imaging system 500 should also be operated and / or configured in a different manner to perform actions according to the embodiments herein that are different from the actions described above with reference to the imaging system 200. Therefore, in order to avoid duplication of information, in the following, the focus is mainly on the differences compared to what has been described above with respect to the imaging system 200 and its corresponding components and in conjunction with the imaging system 200 and its corresponding components. Therefore, generally, what is not described in a different manner below can be assumed to be as described above with respect to the imaging system 200 and with respect to the corresponding components therein.

[0111] As in imaging system 200, image frames provided by camera 530 and image sensor 531 and / or information derived therefrom may be desirable to be communicated (e.g., transmitted) to, for example, computing device 533, such as a computer or the like, for further processing outside of camera 530, which may correspond to computing device 233 and may thus be part of imaging system 500 or located external to imaging system 500. Such further processing may additionally (e.g., in a distributed manner) or alternatively be performed by some remote and / or separate computing unit or device (not shown) (e.g., a remote server and / or a computer portion of a computer cloud). The further processing may involve performing one or more actions of the embodiments herein, such as described below.

[0112] In some embodiments not shown in the figures but similar to the imaging system 200, the computing device 533 is separate from the image processor 531, but is still included in the camera 530 (e.g., integrated with the camera 530) or included in a unit that includes the camera 530. The computing device 533 or other similar device as described above (not shown) can be configured to control devices involved in light triangulation and / or involved in the embodiments herein so that actions related to the embodiments herein are performed, including, for example, both 2D imaging and 3D imaging. This can include: the computing device 533 or a similar device as indicated in the figures is configured to control the illumination performed by the second light, for example, by controlling the one or more light sources (such as, the second light source 550a) that provide the illumination performed by the second light. This can involve controlling when to turn on / off the second light, how long, the illumination on duration of the corresponding second light, etc.

[0113] The second light source 550a may be located at a fixed position relative to the camera 530 and the first light source 510, but the exact position and relationship relative to the camera 530 and the image sensor 531 is not used as in the light triangulation, and the second light source can therefore be arranged more freely for the purpose of providing the second light and illumination according to various embodiments herein. However, for the embodiments herein, the following situation is generally advantageous: the light sources providing the second light of different wavelengths are kept together, so that the illumination of the second light of different wavelengths comes from the same or substantially the same location, with the same or substantially the same illumination direction. This is because it is generally of interest to have the multi-spectrum second light all illuminate the same object location. This is facilitated by the illumination of the second light from the same location and in the same direction, because this reduces the risk that a certain second light does not successfully illuminate the same object location as another second light. For this reason, the following situation may also be beneficial: the second light source is placed at one or more locations near the first light source, and / or provides a major portion of the illumination in a direction close to the illumination direction of the first light. This scenario is illustrated in the accompanying drawings.

[0114] If the second light is provided as diffuse illumination (which is typically the desired situation as described above), the same or approximately the same location should suffice, as the exact direction then becomes less relevant. Diffuse light illumination performed in a similar manner by multi-spectral second light from approximately the same location can be accomplished, for example by using a single illumination unit as light source 550a, as discussed next.

[0115] It should be noted that the same location and / or direction here refers to multi-spectral light of different wavelengths, such as second light in the same illumination unit. Then, in a manner similar to that disclosed in the aforementioned earlier application regarding several second light sources, it can be the second light or other second light having different locations and / or different illumination directions in the imaging system. One or more additional such illumination units.

[0116] Figure 5B 5 is a schematic example of a second light source 550 implemented as a single illumination unit 552, which includes a plurality of second light sources, here two or more (sub) light sources 550-1, 550-2 for providing second light 551-1, 551-2, respectively. It is also shown how such a unit can also include one or more second light sources for providing further second light in a similar manner, which are exemplified in the figure by a third second light source 550-3 for providing a third second light 551-3.

[0117] The respective second lights from the second light sources may pass through a common lens and / or diffuser of the illumination unit so that the illuminations of the second lights provided from the illumination units 552 will be more similar to each other except for the difference in wavelength.

[0118] As Figure 5B In a first example of an alternative to what is shown in , there are multiple second light sources, but are provided separately (such as, in separate illumination units), for example, arranged side by side in the imaging system 500 and / or gathered together for providing the second light.

[0119] As Figure 5B In a second example of an alternative to what is shown in , there is a single second light source that provides illumination of the multi-spectral second light via an apparatus having one or more filters (e.g., electrically and / or mechanically controlled filters) that pass the second light of different wavelengths. The second light source can thus provide all wavelengths (e.g., a range), and the corresponding filters then select one or more wavelengths (usually a corresponding sub-range) from these wavelengths as the corresponding second light.

[0120] Figure 5C Schematically illustrated: the second light may generally be non-overlapping in wavelength, or two or more second lights (such as, second light 551-1, 551-2) may be non-overlapping in wavelength, for example, second lights having non-overlapping wavelength ranges. Non-overlapping wavelengths are preferred in many applications and can also be simply accomplished by using different light sources, for example based on different types of LEDs.

[0121] Figure 5DSchematically illustrated: Instead of or in addition to having non-overlapping wavelengths, the second light may generally partially overlap, or two or more second lights may partially overlap, illustrated in the figure by second lights 551-1' and 551-2' as an alternative to exemplifying second lights 551-1, 551-2. Partial overlap means that some wavelengths of the second light but not all wavelengths are also present in the other second light. When such a situation exists, it may be preferred but not necessarily, which is mutual, if one of the second lights is formed by a sub-range of the wavelength of the other second light (not shown in the figure), then this is not the case. This can be interesting in some applications, but it is usually more useful to have no overlap or only partial overlap, and thereby have a greater separation in wavelength content between the second lights.

[0122] Figure 6 is a flow chart for schematically illustrating a method according to an embodiment herein (thus having a multi-spectral second light). The content disclosed herein can be regarded as a combination of the above Figure 2B A special case of the method discussed and as disclosed in said earlier application, or at least based on the same principles as disclosed therein.

[0123] The following actions forming the method are used to associate 2D image data with 3D image data generated from optical triangulation performed by an imaging system (here illustrated by imaging system 500) for 3D imaging of an object (e.g., object 520). The imaging system thus includes: a first light source, illustrated by first light source 510, for illuminating the object with a first light (illustrated by first light 511); and a camera, illustrated by camera 530, having an image sensor illustrated by image sensor 531. The imaging system also includes: one or more second light sources, illustrated by second light source 550, for illuminating the object 520 with two or more second lights, illustrated by second light 551, the second lights being multispectral, differing from each other by including different light wavelengths. Optical triangulation can thus be like in conventional 3D imaging based on optical triangulation, and thus includes illuminating different successive parts of the object 520 with the first light 511 and sensing, by the image sensor 531, the reflected first light from the corresponding parts in a corresponding first image (IM1). IM1 may correspond to the above in combination with Figure 2B The methods discussed in IM1 are further discussed and illustrated below in conjunction with FIGS. 7-8 regarding the embodiments herein.

[0124] The following Figure 6 The method and / or actions indicated in the may be performed by a device (i.e., one or more devices, such as camera 530 and / or computing device 533), or by imaging system 500 and / or a suitable device thereof or a suitable device connected to it. The device for performing the method and its actions is further described below.

[0125] It should be noted that the following actions may be performed in any suitable order, and / or performed with full or partial overlap in time when possible and appropriate.

[0126] Action 601

[0127] The 3D data is obtained as a first sensor position (SP1) of IM1 generated by camera 530 and image sensor 531. SP1 corresponds to the location of the peak intensity of the reflected first light from object 520 as part of the light triangulation. Here, SP1 may correspond to the peak intensity of the reflected first light from object 520 as described above in combination with Figure 2B SP1 is discussed in conjunction with the method of FIG. 7-8 , and is further discussed and illustrated below in conjunction with the embodiments herein.

[0128] Action 602

[0129] Two or more second images (IM2) generated by the camera 530 and the image sensor 531 are obtained. During the illumination respectively performed by the two or more second lights 551, the IM2 images the object. The corresponding IM2 is associated with the corresponding IM1 among the IM1. The corresponding IM2 may be the corresponding IM1 among the IM1 and associated with the corresponding IM1 among the IM1, or may be generated between two IM1 among the IM1 and associated with any IM1 among them.

[0130] IM2 may thus be IM1, in other words, IM2 and IM1 may be the same image, and / or IM2 may be separate from IM1, more specifically, generated between IM1, preferably between consecutive IM1. In the former case with IM2=IM1, the IM1 associated with IM2 should be the same image as IM2, and in the latter case, the IM1 associated with IM2 should be the IM1 generated before or after the corresponding IM2, typically the IM1 generated in the closest time before or after IM2.

[0131] Here, a single IM2 may correspond to the above combination Figure 2B IM2 as a group (eg, in a sequence) according to embodiments herein is further discussed and illustrated below in conjunction with FIGS. 7-8 .

[0132] Action 603

[0133] In the corresponding IM2, a corresponding second sensor position (SP2) is selected for and in conjunction with the corresponding SP1 of the corresponding IM1 to which the corresponding IM2 is associated. The corresponding SP2 is a position in the corresponding IM2 at which the reflected second light from the object 520 has a higher intensity than the reflected first light from the object 520.

[0134] SP2 may correspond to and / or be selected accordingly as above in combination with Figure 2B The SP2 according to the embodiments herein is further discussed and illustrated below in conjunction with FIGS. 7-8 .

[0135] Action 604

[0136] The intensity values ​​of the selected SP2 are respectively associated with the SP1 for which they were selected. As a result, the multispectral 2D data corresponding to the intensity values ​​of SP2 from the reflected multispectral second light becomes associated with the 3D data corresponding to the SP1 for which SP2 was selected.

[0137] There will thus be SP1 corresponding to 3D data of two or more IM1 mapping two consecutive portions on the object according to light triangulation, SP1 being associated with SP2 having 2D image data corresponding to intensity values ​​of the sensed second light reflected from the object. Since the intensity values ​​are obtained by said illumination of the second light of different wavelengths, they correspond to multispectral 2D image data.

[0138] From the above, it can be further appreciated that if IM2 is generated between IM1 and associated with the same IM1, the corresponding SP1 in this image will be associated with as many intensity values ​​as second lights of different wavelengths. There will therefore be the same resolution of the multispectral 2D image data as the 3D image data. However, if the corresponding IM2 is the corresponding IM1, e.g. for all images, IM2=IM1, e.g. the corresponding IM1 in the consecutive IM1 is also IM2, then the resolution of the 2D image data will be lower, because the "time dimension" in laser triangulation is mapped to the third dimension of the 3D image data, in the example shown here, to the "y dimension". For example, if there are two different second lights and the corresponding IM2 is the corresponding IM1, two IM1s are required, one IM1 for each second light, to form the multispectral image data. Therefore, the resolution of the 2D image data is half of that of the 3D image data. In practice, the 2D image data resolution will usually be made lower than or equal to the 3D data resolution, but the opposite is unlikely to be of interest.

[0139] In order to obtain combined image data with the same resolution of 3D image data and 2D image data, images with the image data may be formed "per light", e.g. one image with the 3D image data and one image for each second light with the 2D image data. In the example of 2 second lights and half the resolution of the 2D image data compared to the 3D image data, the 2D image data may then be upsampled by a factor of 2. The obtained image data with the same resolution can then easily be combined into a single combined image, wherein each pixel or voxel corresponding to an object point according to the 3D image data and therefore according to light triangulation will be associated with 2 intensity values ​​(one for each second light) and therefore with the multispectral 2D image data. Combination Fig. 9A -D discusses a practical example of this principle. In general, it should be understood that the underlying principles of the embodiments herein can be used independently of which data has a lower or higher resolution than other data, by using upsampling and / or downsampling to achieve the same resolution of the corresponding image data and then combining. Of course, it is also possible to extract relevant information from 2D image data associated with 3D image data, rather than first ensuring that all image data has the same resolution.

[0140] Action 605

[0141] Referring to the above discussion, in some embodiments, the 3D image data and the 2D image data are provided at the same resolution.

[0142] If the resolutions of the 3D image data and the 2D image data are different after the execution of the actions described above (which is the case in some embodiments herein), the provision of image data of the same resolution may be accomplished by up-sampling and / or down-sampling of the 3D image data and / or the 2D image data. That is, as the skilled person will appreciate, this can be accomplished in various ways by appropriate up-sampling and / or down-sampling.

[0143] Action 606

[0144] The 3D image data and the 2D image of the same resolution can be combined into a combined image. As a result, the corresponding 3D image data points of the image become associated with the multispectral 2D image data corresponding to the intensity values ​​obtained by reflecting the two or more second lights including different wavelengths.

[0145] As explained above and elsewhere herein, the difference in resolution of 3D image data and 2D image data is typically a result when IM1 and IM2 for different second lights are generated at different rates. For example, if there is a sequence of three consecutive IM2s with different second lights (e.g., second lights corresponding to the colors "red", "green", "blue" (R, G, B)) generated during the same time as 3 IM1s with first light, or in other words, the corresponding IM2s are the corresponding IM1s, then the rate of "IM2 for each color" will be 1 / 3 of the rate of IM1, and therefore, the second image data for each color R, G, B will have 1 / 3 of the resolution of the 3D image data. According to laser triangulation, the reduced resolution is in the dimension corresponding to time, which in the example here is the dimension corresponding to y. (In other dimensions, such as z, x in the example here, the resolution is the same, which can also be achieved by selecting a corresponding SP2 for each SP1 and thus having a 1:1 relationship between them.) By upsampling the corresponding second image data corresponding to R, G, B in that dimension by a factor of 3, the 3D image data and the 2D image data will have the same resolution. In this example, an alternative could be to downsample the 3D image data by 1 / 3, but this is generally not preferred because 3D information is lost when the resolution is reduced like this.

[0146] As indicated above, in some embodiments, the second light includes one or more of the following: red light (R), green light (G) and blue light (B). Using all three colors, full-color 2D image data can be completed. As used herein, the typical definition of R is light with one or more wavelengths in the range of 620-750nm, with 625nm as a typical wavelength, and can be provided by a red LED. As used herein, the typical definition of G is light with one or more wavelengths in the range of 495-570nm, with 525nm as a typical wavelength, and can be provided by a green LED. As used herein, the typical definition of B is light with one or more wavelengths in the range of 450-495nm, with 460nm as a typical wavelength, and can be provided by a blue LED. In some embodiments, the second light includes R, G, B, and the difference in the duration of illumination of R, G, B can be used for white balance during imaging. That is, the on and / or off time of the corresponding second light can be controlled separately, for example, set to different suitable on durations, so as to be able to achieve the desired white balance directly in the obtained 2D image data. Fig. 8E , further discusses and illustrates this point.

[0147] In some embodiments, the second light comprises a second light corresponding to infrared light (IR) or near infrared light (NIR). As used herein, NIR is defined as light in the wavelength range of 850-1000nm, and IR is defined as light with a longer wavelength (that is, any wavelength or range of wavelengths above 1000nm).

[0148] It should be noted that although some light (including, for example, ultraviolet light) is not explicitly mentioned above, the principles herein are of course also applicable to such light or generally different light according to conventional definitions.

[0149] In some embodiments, according to the sequence of the two or more second lights, the illumination of the two or more second lights is provided sequentially, and the second images are generated in a corresponding order. The sequence may be a certain sequence, which may be application-specific. The sequence may be repeated one or more times, generating additional IM2 and its associated one or more IM1 in each repetition. In other words, during the repetition, additional IM1 and IM2 as described above, that is, IM2 corresponding to IM1 or generated between IM1, etc. are generated according to light triangulation. This can continue, so that the entire object becomes covered, for example scanned, and 3D image data with associated 2D image data about the entire object has been generated. Therefore, according to the sequence repetition, in each sequence repetition, there will be one or more involved IM1 and different IM1. Therefore, different parts of the object will be mapped in each repetition because IM1 is related to 3D imaging of different object parts. The sequence may thus be repeated, for example during scanning of the object by the first light as part of the light triangulation, and 3D image data captured during scanning according to the light triangulation and SP1 , while a multispectral 2D image is also captured at SP2 in conjunction with SP1 .

[0150] In some embodiments, in the sequence, one or more of the second lights occur more frequently than one or more other second lights in the second lights. That is, one or more second lights may occur two or more times before all irradiations of the second lights and the generation of the corresponding IM2 have occurred and / or before the sequence is repeated. For example, when the sequence of second lights includes R, G and B, G may occur more frequently than R and B in the sequence, for example, twice as frequently. Therefore, the sequence may be RGBG, etc., or include RGBG, etc. It is known that more occurrences of G enhance the perceived sharpness of color images. In this way, the obtained 2D image data can be perceived as clearer in an image with combined 2D image data. Of course, there may also be other reasons why some second lights occur more frequently than other second lights.

[0151] Examples of different sequences are further discussed and illustrated below in conjunction with Figures 7-8.

[0152] In some embodiments, the object is organic. That is, the object being imaged may be organic, and / or the method may be applied in applications where the object is an organic object. For organic objects, which often differ in 3D appearance but also differ in 2D (e.g., also have 2D surface variations), details about both and combinations are often of interest. Therefore, the embodiments herein may be of particular interest for application in applications with organic objects. For example, in applications where the object is one of the following types and / or categories: plants, trees, logs, wood, meat, vegetables, bread, food, waste.

[0153] For example, the embodiments herein may be beneficially applied in the following application areas:

[0154] Agriculture, where the first light can be a green laser to get good data from green leaves and be able to measure e.g. plant size and height through 3D image data, and at the same time get information about plant condition and health from 2D image data by using the second light with RG and NIR. This can be used to optimize nutrition and water usage for the plant.

[0155] Food sorting, e.g. being able to manage and sort (such as by grasping) food products using 3D image data, and also by using 2D image data to detect foreign objects and / or for quality assurance reasons, where it may be desirable to apply different classifications based on 2D data.

[0156] Waste sorting, for example waste / junk food can be managed and sorted (such as by grasping) using 3D image data, and this can be done in different ways based on material, color and shape, where multispectral 2D image data can provide very useful information.

[0157] Wood inspection, where the information gained via 3D image data is important, but also multispectral 2D image data, in order to be able to better differentiate between heartwood and sapwood (on the surface), for example.

[0158] Thus, in some embodiments, the object is one of the following types and / or categories: plant, tree, log, wood, meat, vegetable, food, waste.

[0159] Fig. 7A -B schematically illustrate the corresponding two exemplary main groups of the embodiments herein, similar to the above combination Figure 3A -B and the main group discussed in said earlier application, but here in the context of a multi-spectral second light and combined with the above Figure 6 The method according to the embodiments described herein is related. Figure 3A-B are shown in a similar manner to facilitate comparison and understanding of the differences. Like in the embodiments here, the case with a multi-spectral second light can be considered as an extension or special case. Since more images are involved in these examples, Figure 3A - The exposure period indicated in B has been Fig. 7A -B is removed to save space, but the exposure period is of course actually present.

[0160] Fig. 7A Regarding the first main example group of embodiments, IM2a is identical to IM1 , that is, the same image used for light triangulation and, therefore, also has the same exposure period of the image sensor.

[0161] Figure 7B Regarding the second main example group of embodiments instead, where IM2 is separated from IM1 and located between IM1 , that is, between the images used for light triangulation, and therefore, IM2 also has a separate exposure period.

[0162] In both figures, the time period T between successive IM1 3D 763 are all indicated and correspond to the time period between successive IM1 parts of the light triangulation and are therefore relevant to the provision of 3D image data. 3D 763 typically corresponds to the scan rate used in optical triangulation.

[0163] Furthermore, to facilitate understanding, the types of light involved in the respective images and the kinds of image data that the respective lights are used for generating are indicated in the drawings. Since the first light is used for the generation of 3D image data in light triangulation, the first light is therefore indicated as “Light1 for 3D”, and since the second light is used for the generation of 3D image data, the two or more second lights are indicated as “Light2 for 2D”. The first second light is named “Light2-1 for 2D” and the second second light is named “Light2-2 for 2D” because these are different, since the two or more second lights are different in wavelength content as discussed above.

[0164] In addition, in both figures, the time period T SEQ 765 are all shown and indicate by their time period such a sequence discussed above, that is, a sequence of IM2s, respective IM2s being exposed to one of the second lights until all of the two or more second lights have been used according to the sequence, which sequence can thereafter be and typically is repeated.

[0165] two Fig. 7A -B shows each T SEQ765 and thus has an example of two IM2s in the sequence, but the principle can be easily extended to more than two IM2s, as discussed above and also illustrated below.

[0166] Regarding the first main example group of embodiments herein Fig. 7A More specifically and schematically showing T SEQ 765a, T SEQ 765a includes two IM2s that are also IM1s, namely IM2-1 741a-1 that is identical to IM1-1 742a-1 and IM2-2 741a-2 that is identical to IM1-2 742a-2. It can be seen that in this case, T SEQ 765a is T 3D In the case of more than two second lights and / or more than two IM2s in the sequence, T SEQ will certainly be relative to T 3D 763a further increased.

[0167] Furthermore, in the drawings, the corresponding SP1 and SP2 in each image are schematically illustrated: SP1-1 771a-1 and SP2-1 772a-1 in IM1-1 741a-1 which is the same as IM2-1 742a-1. SP1-2 771a-2 and SP2-2 772a-2 in IM1-2 741a-2 which is the same as IM2-2 742a-2. As schematically indicated in the drawings, and in accordance with the above discussion, here, since IM2=IM1, SP2 has an offset in sensor coordinates relative to SP1, so that the captured reflected second light at SP2 is dominant relative to the captured reflected first light, while the first light should be dominant relative to the second light at SP1, that is, similar to the combination of Figure 2B Methods and examples discussed. In other words, and generally with respect to the embodiments herein, at SP2, the captured reflected second light from the object should have a higher intensity than the reflected first light from the object. Since SP1 is a result from light triangulation, it means that the first light is dominant there, and therefore, at SP1, the captured reflected second light from the object has a lower intensity than the reflected first light from the object, or at least the light intensity of the second light is low enough at SP1 not to adversely affect the detectability of the first light at SP1.

[0168] The figure also shows two further IM1 and IM2 belonging to a repetition of the sequence, merely to indicate that what is shown is generally repeated, e.g. whereby the entire object becomes 3D scanned by light triangulation and in conjunction with this and in relation to the 3D image data multispectral 2D image data is captured.

[0169] Regarding the second main example group of embodiments herein Figure 7B More specifically and schematically showing T SEQ 765b, T SEQ 765b includes two IM2 separated from IM1, namely IM2-1 741b-1 and IM2-2 741a-2 located between the first image IM1-1 742b-1 and the other first image IM1-2 741b-2. It can be seen that in this case, T SEQ 765a and T 3D 763a, but are shown in the figure as having an offset. When IM2 is separated from IM1 and generated between IM1, IM2 is typically but not necessarily evenly spaced between IM1, that is, so that the image frame rate at which the image sensor operates can be the same for all images, that is, the same for IM1 and IM2.

[0170] In the accompanying drawings, it is schematically illustrated that since in this case, as already described above, if the first light is not present in IM2 / when the first light is not present in IM2, that is, when the first light does not illuminate the object when IM2 is captured, for example, by keeping the first light “on” only during the exposure of IM1 and keeping the first light “off” during the exposure of IM2 / if the first light does not illuminate the object when IM2 is captured, for example, by keeping the first light “on” only during the exposure of IM1 and keeping the first light “off” during the exposure of IM2, SP2 is the same as SP1.

[0171] SP1-1 771b-1 is shown in IM1-1 741b-1. SP2-1 772b-1 which is the same sensor position as SP1-1 is shown in IM2-1 742b-1, and SP2-2 772b-2 1 which is the same sensor position as SP1-1 is shown in IM2-2 742b-2.

[0172] The figure also shows two additional images (repeated IM1-1 and IM2-1 parts of the sequence) just to indicate the SEQ The indicated sequence is typically repeated, for example, whereby the entire object becomes 3D scanned by light triangulation, and in conjunction with this and in correlation with the 3D image data, multispectral 2D image data is captured.

[0173] It should be noted that, since there are two or more second lights and therefore two or more IM2 in each sequence, the embodiments herein can be implemented as a combination between the first main group and the second main group of embodiments, but this is not shown in the drawings. That is, in a sequence of, for example, two or more IM2 and associated one or more IM1, one or more IM2 may be respectively identical to one or more IM1 of the sequence, and one or more other IM2 of the sequence may be separated from the IM1 and generated between the IM1. Therefore, the following situation will be possible, for example: in a sequence like Figure 7B In the corresponding example as in , there is only one separated IM2, such as for example IM2-1=IM1-1, IM2-2 is separated from IM1-1 and located between IM1-1, and then the next additional IM1-2 part of the sequence is repeated.

[0174] Fig. 8A -E schematically illustrates examples of different sequences with second light. For embodiments corresponding to the first main example group, examples are shown, that is, IM2=IM1, but based on the above description, it is realized that the corresponding sequence also applies to another second main example group and to combinations between the two main example groups.

[0175] Fig. 8A is with Fig. 7A The example in FIG. 1 corresponds to the illustrative example in FIG. 2 , but using the time period T SEQ 865a has three second lights R, G, B (red, green, blue) such as mentioned above and three IM2=IM1 in the sequence. The sequence can therefore be described as R, G, B. There is IM2-1 842a-1 which is the same as IM1-1 841a-1, IM2-2 842a-2 which is the same as IM1-2 841a-2, and IM2-3 842a-3 which is the same as IM1-3 841a-3. Therefore: during the exposure period of the image sensor 531 which causes the generation of IM2-1, the second light irradiating the object 520 is red light. During the exposure period of the image sensor 531 which causes the generation of IM2-2, the second light irradiating the object 520 is green light. During the exposure period of the image sensor 531 which causes the generation of IM2-3, the second light irradiating the object 520 is blue light.

[0176] Figure 8B is another schematic example of a sequence as discussed above but in a more comprehensive perspective, simply indicating the type of second light used for each image frame of the sequence. Thus, in this example, the sequence is R, G, B, IR, and thus four IM2s (and four IM1s, since IM2=IM1 as described above). The sequence is represented in the figure by a sequence time period T SEQ 865b instructions.

[0177] Figure 8C is as discussed above and like Figure 8B Another schematic example of a sequence is shown in the same type of full angle as in FIG. Here, in addition, the type of second light used for each image frame of the sequence is indicated. Thus, in this example, the sequence is R, G, NIR, and thus there are three IM2s in the sequence (and four IM1s, since IM2=IM1 as described above). The sequence is represented in the figure by a sequence time period T SEQ 865c instructions.

[0178] Fig.8D is as discussed above and like Figure 8C Another schematic example of a sequence shown in full perspective as in FIG. Here, in addition, the type of second light used for each image frame of the sequence is indicated. Thus, in this example, the sequence is R, G, B, G, so that, as described above, green light occurs more frequently than red and blue. Although there are only three different second lights in this example, the sequence contains four IM2s (and four IM1s, since here, as described above, IM2=IM1). The sequence is represented in the figure by a sequence time period T SEQ 865c instructions.

[0179] Fig. 8E Corresponds to Fig. 8A , with the difference that the “on time” (that is, the illumination duration) of the corresponding second light is different here according to IM2. For example, as described above, for white balance purposes, the on duration (and therefore, the corresponding off duration) of the corresponding second light during the exposure time period resulting in IM2 (and IM1, because IM2=IM1 in this example) is therefore different in some embodiments. In these embodiments, the on / off duration of the corresponding second light can generally be controlled by the imaging system 500, for example by its computing device 533, so that the duration of the appropriate second light illumination is achieved during the exposure time period. Therefore, in Fig. 8A In it is shown that:

[0180] IM2-1 842e-1, same as IM1-1 841e-1, having an exposure period EXP;

[0181] IM2-2 842e-2, identical to IM1-2 841e-2, having an exposure period EXP; and

[0182] IM2-3 842e-3, like IM1-3 841e-3, has an exposure period EXP.

[0183] It is noted that the exposure period EXP as shown in this example is typically, but not necessarily, the same for all images, at least for the first group with IM2=IM1.

[0184] During the exposure time period EXP of the image sensor 541 resulting in the generation of IM2-1 842e-1, during the red light on time period 866e-1 which may be less than EXP, the second light irradiation of the object 520 is red light (R). During the exposure time period EXP of the image sensor 541 resulting in the generation of IM2-2 842e-2, during the green light on time period 866e-2 which may be less than EXP, the second light irradiation of the object 520 is green light (G). During the exposure time period EXP of the image sensor 541 resulting in the generation of IM2-3 842e-3, during the blue light on time period 866e-3 which may be less than EXP, the second light irradiation of the object 520 is blue light (B).

[0185] It should be noted that in the case of the second main example group where IM2 is separated from IM1, effective illumination of the second light can be accomplished alternatively by controlling the exposure period EXP of IM2, and then the corresponding second light can have the same on-duration covering the exposure period. Fig. 8E This may not be possible or at least not suitable, as it would mean that different exposure time periods are also used for the first light and light triangulation, and therefore would mean different exposures of the first light in different IM1s, which is usually not a desirable situation. However, by controlling the on / off duration of the first light, so that it remains "on" during the same time period during each exposure time period, exposure time periods of the same first light can also be achieved in this case.

[0186] The above are examples only, and the principles of course apply to any type of second light of different wavelengths, not just RGB.

[0187] From what has been discussed above, it can be understood that, in order to provide the second light, it can be a matter of controlling which second light is switched on / off and for how long and in what order during imaging, for example by controlling the illumination unit and / or the light source providing the second light. In practice, this can correspond to a "flickering" of the second light, with a different second light in each "flicker" according to the sequence used, which sequence can then be repeated, for example, as long as the scanning of the object continues. During such time, and / or according to the light triangulation performed by the system, the first light (e.g., a laser) can be permanently switched on. It should be noted that the light triangulation does not need to be affected by the second light in this way, at least in principle. However, in some embodiments, the first light (e.g., a laser) may also "flicker", that is, be switched off, if / when the first light should not be present in one or more second images. Furthermore, in some embodiments, there are several single second light sources and / or illumination units for providing illumination of the second light from several different directions, for example as disclosed in the said earlier application. In this way, advantages similar to and / or corresponding to those discussed in said earlier application can be obtained by irradiating from different directions and / or in different ways, and also multi-spectral 2D image data associated with 3D image data is obtained, as in the embodiments herein. Furthermore, it should be noted that, for example, the second light in said single irradiation unit may provide illumination in various ways, for example as disclosed in said earlier application regarding the provision of the second light.

[0188] Fig. 9A -D shows exemplary images and results from a real and practical application of the embodiments herein. The embodiments resulting in the images are based on the first main example group with IM2=IM1, and the sequence of the multispectral second light is NIR, R, G, B, that is, four different second lights, and therefore, according to the above explanation, the corresponding 2D image data resolution is 1 / 4 of the resolution of the 3D image data. The test object being imaged is formed by a piece of wood, on which a plurality of candies of different colors are placed to form a composite test object for imaging with many different colors, structures and materials. By optical triangulation, here the object is imaged in 3D by scanning by the first light and the second light. The scanning involves the generation of more than 5000 IM1s (and therefore first images, with SP1 in each first image), resulting in a 3D profile according to optical triangulation. Approximately 5120 IM1s are used in the example shown in the accompanying drawings, thus corresponding to 5120 profile images of the object, and therefore corresponding to the corresponding 3D image data resolution in the dimension corresponding to the scanning direction (in the example, in the y′ direction).

[0189] Fig. 9AA combined image of 3D image data in the form of 3D image points corresponding to pixels (and therefore voxels) of 3D according to an embodiment herein and associated multispectral 2D image data is shown. Since true 3D cannot be shown here, a 2D view of the 3D image points taken "from above" is shown, so that the object is shown in the x'-y' plane. The associated 2D image data is clearly visible as a texture on the object formed by the 3D image points.

[0190] Fig. 9B yes Fig. 9A , where it can be seen that in the y' direction, for every fourth line of 3D image data, that is to say for every fourth line of 3D image points, there is different 2D image data. Thus, as expected from the sequence used with 4 IM2s and 4 different second lights, respectively. Thus, in the y' direction, the 2D image data resolution is 1 / 4 of the 3D image data resolution, thus 5120 / 4=1280.

[0191] Fig. 9C 2D image data separated into 4 images is shown, each image having a resolution of 1280 in the y' direction. It can be clearly seen that the respective images shown have captured different 2D information about the object through the illumination of the respective second light. As explained above, the 2D image data of each image includes intensity values ​​as a result of the sensed reflected second light from the object being illuminated with the second light during the exposure of IM2 resulting in the acquisition of the 2D image data (from the SP2 position). The respective 2D image data can therefore be viewed as a monochrome (e.g., grayscale) image, but if each image is represented by a separate monochrome color (e.g., RGB and some suitable color scale for NIR), it is appreciated that a full-color image with a mixture of all colors can be accomplished. Of course, it is also possible, and sometimes sufficient and / or desirable, to use and / or view the 2D image data separately rather than mixing or combining with each other. For example, viewing 2D image data associated with 3D image data in separate images, e.g., side by side for comparison, like Fig. 9C In other cases, measurements may be performed on 2D image data associated with 3D image data without actually needing to view the 2D image data.

[0192] In any case, it will be appreciated that it is often desirable to have all image data have the same resolution. Fig. 9B The line effect and the associated distortions seen in FIG. 1 can be avoided. Thus, as already described above, the corresponding 2D image data can be provided, for example, converted to the same resolution. In the case of 2D image data at a lower resolution, it may be appropriate to convert, for example, the image data to Fig. 9CThe 2D image data in the form of the same image as in the example is converted into the same resolution as the 3D image data. This conversion can be performed by resampling (hence, upsampling in this example).

[0193] Fig.9D A combined image with 3D data and 2D image data is shown after the 2D image data has first been upsampled to the same resolution as the 3D image data. As a result, there will be multispectral 2D image data represented by 4 intensity values ​​at each 3D image data point, and one intensity value at each second light used. Therefore, full color images and textures on 3D objects can be accomplished, such as Fig.9D Note that the images are in color, but because patent drawings and illustrations are often reproduced in black and white and / or grayscale, Fig.9D The image may still appear in grayscale in the attached figure.

[0194] Fig.10 is a schematic block diagram for illustrating an embodiment of one or more devices 1000 (ie, (one or more) devices 1000), which may correspond to the above-mentioned embodiments for performing the embodiments herein (such as, for performing the above-mentioned embodiments in combination with Figure 6 The device 1000 may correspond, for example, to a computing device 533 and / or a camera 530 or a suitable device included in the imaging system 500 and / or forming the imaging system 500 or one or more computing devices located at a distance from and separate from the imaging system involved. For example, a computing device that operates on the output from the imaging system 500. Therefore, compared to the time when the imaging system 500 is operated and provides IM1, IM2, etc., the device 1000 that performs the method can achieve this at a later time. Therefore, the method can be performed by a device that is located at a distance in time and space relative to the imaging system and imaging. The output from the imaging system required by the method can be uploaded to a server or a computer cloud, for example, which includes a computing device configured to perform the method. However, it is generally preferred that the device is configured to perform the method as part of and / or in conjunction with the imaging system and imaging involved, because this can achieve faster execution and less image data must be temporarily stored and / or transmitted over a longer distance. Therefore, it may be preferred that the device 1000 is the camera 533 and / or the computing device 533, or in other words, the camera 530 and / or the computing device 533 is configured to perform the method.

[0195] The schematic block diagram is used to illustrate how the apparatus 1000 can be configured to perform the above combination Figure 6Embodiments of methods and actions discussed. Thus, apparatus 1000 is used to associate multispectral 2D image data with 3D image data generated from light triangulation performed by an imaging system (such as imaging system 500) for 3D imaging of an object.

[0196] The apparatus 1000 may include: a processing module 1001, such as a processing tool; one or more hardware modules, including, for example, one or more processing circuits, circuit systems, such as a processor; and / or one or more software modules for performing the methods and / or actions.

[0197] The apparatus 1000 may also include a memory 1002, which may include (such as, contain or store) a computer program 1003. The computer program 1003 includes "instructions" or "codes" for performing the methods and / or actions, respectively, which may be directly or indirectly executed by the apparatus 900. The memory 1002 may include one or more memory units, and may further be arranged to store data, such as configurations, data and / or values, involved in or used to perform the functions and actions of the embodiments herein.

[0198] In addition, the corresponding device 1000 may include a processing circuit 1004 that participates in processing and, for example, encodes data as an exemplary hardware module, and may include or correspond to one or more processors or processing circuits. The processing module 1001 may include, for example, "embodied in the form of such a processing circuit 1004" or "implemented by such a processing circuit 1004". In these embodiments, the memory 1002 may include a computer program 1003 that can be executed by the processing circuit 904, respectively, so that the corresponding device 1000 can be operated or configured to perform the method and / or its actions.

[0199] Typically, the device 1000 (e.g., processing module 1001) includes an input / output (I / O) module 1005, which is configured to participate in (e.g., by executing) any communication to and / or from other units and / or devices, such as sending information to other devices and / or receiving information from other devices. When applicable, the I / O module 1005 can be exemplified by an obtaining (e.g., receiving) module and / or a providing (e.g., sending) module.

[0200] In addition, in some embodiments, the device 1000 (e.g., the processing module 901) includes one or more of a selection module, an association module, a selection module, and a combination module as an exemplary hardware and / or software module for performing the actions of the embodiments herein. These modules may be implemented in whole or in part by the processing circuit 1004.

[0201] therefore:

[0202] The apparatus 1000 and / or the processing module 1001 and / or the processing circuit 1004 and / or the I / O module 1005 and / or the obtaining module may be operable or configured to obtain the 3D data as the SP1 of the IM1 generated by the camera and the image sensor.

[0203] The device 1000 and / or the processing module 1001 and / or the processing circuit 1004 and / or the I / O module 1005 and / or the acquisition module are operable or configured to obtain the two or more IM2s generated by the camera and the image sensor, and to image an object during illumination by the two or more second lights, respectively, wherein a corresponding IM2 is associated with a corresponding IM1 among the IM1s or is generated between two IM1s among the IM1s and associated with any IM1 among them.

[0204] The device 1000 and / or the processing module 1001 and / or the processing circuit 1004 and / or the I / O module 1005 and / or the selection module can be operated or configured to select the corresponding SP2 in the corresponding IM2 for the corresponding SP1 of the corresponding IM1 associated with the corresponding IM2 and in combination with the corresponding SP1, in which the reflected second light from the object has a higher intensity than the reflected first light from the object.

[0205] The device 1000 and / or the processing module 1001 and / or the processing circuit 1004 and / or the I / O module 1005 and / or the association module are operable or configured to respectively associate the intensity value of the selected SP2 with the SP1 for which the SP2 is selected, so that the multispectral 2D data corresponding to the intensity value at SP2 from the reflected multispectral second light becomes associated with the 3D data corresponding to the SP1 for which the SP2 is selected.

[0206] The apparatus 1000 and / or the processing module 1001 and / or the processing circuit 1004 and / or the I / O module 1005 and / or the providing module may be operable or configured to provide 3D image data and 2D image data at the same resolution.

[0207] The device 1000 and / or the processing module 1001 and / or the processing circuit 1004 and / or the I / O module 1005 and / or the combining module are operable or configured to combine the 3D image data and the 2D image of the same resolution into the combined image, so that the corresponding 3D image data points of the image become associated with the multispectral 2D image data, and the multispectral 2D image data corresponds to the intensity values ​​obtained by reflecting the two or more second lights including different wavelengths.

[0208] Fig.11is a schematic diagram illustrating some embodiments related to a computer program 1003 and its carrier for causing the apparatus 1000 discussed above to perform the methods and actions.

[0209] The computer program 1003 comprises instructions which, when executed by the processing circuit 1004 and / or the processing module 1001, cause the apparatus 1000 to perform as described above. In some embodiments, one or more carriers, that is, (one or more) carriers, or more specifically, data carriers, such as a computer program product comprising a computer program, are provided. The corresponding carrier may be one of an electronic signal, an optical signal, a radio signal, and a computer-readable storage medium (e.g., a computer-readable storage medium 1101 as schematically shown in the accompanying drawings). The computer program 1003 may therefore be stored on a computer-readable storage medium 1101. The carrier may be excluded as a transient propagation signal, and the data carrier may be correspondingly named as a non-transient data carrier. Non-limiting examples of data carriers as computer-readable storage media are memory cards or memory sticks, disk storage media, or mass storage devices typically based on hard disk drives or solid-state drives (SSDs). The computer-readable storage medium 1101 may be used to store data accessible via a computer network 1102 (e.g., the Internet or a local area network (LAN)). The computer program 1003 may additionally be provided as a pure computer program or included in one or more files. The one or more files may be stored on a computer-readable storage medium 1001 and may be obtained, for example, by downloading via a computer network 1102 (such as, via a server) as indicated in the accompanying drawings. The server may be a web or file transfer protocol (FTP) server or the like. The one or more files may be executable files for downloading directly or indirectly to the device and executing on the device to, for example, enable it to be executed as described above by executing it by the processing circuit 1004. The one or more files may also be used or alternatively for intermediate downloading and compiling involving the same or another processor to make it executable before further downloading and execution, enabling the device 1000 to be executed as described above.

[0210] It should be noted that any of the aforementioned processing modules and circuits may be implemented, for example, as software and / or hardware modules in existing hardware, and / or as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. In addition, it should be noted that any of the aforementioned hardware modules and / or circuits may be included, for example, in a single ASIC or FPGA, or distributed among several separate hardware components, whether they are individually packaged or assembled into a system on a chip (SoC).

[0211] Those skilled in the art will also understand that the modules and circuits discussed herein may refer to a combination of hardware modules, software modules, analog and digital circuits and / or one or more processors, wherein the one or more processors are configured with software and / or firmware stored in a memory, for example, and when the software and / or firmware are executed by the one or more processors, the software and / or firmware may cause devices, sensors, etc. to be configured as and / or perform the above-mentioned methods and actions.

[0212] The identification by any identifier herein may be implicit or explicit. In a certain context, such as for a certain computer program or program provider, the identification may be unique.

[0213] As used herein, the term "memory" may refer to a data storage device for storing digital information, typically a hard disk, magnetic storage device, medium, portable computer floppy disk or disk, flash memory, random access memory (RAM), etc. In addition, the memory may be an internal register memory of a processor.

[0214] In addition, it should be noted that any enumerated terms (such as, first device, second device, first surface, second surface, etc.) should be considered as non-restrictive, and the terms do not imply a certain hierarchical relationship. In the absence of any explicit information to the contrary, naming by enumeration should only be considered as a way to achieve different names.

[0215] As used herein, the expression "configured to" may mean that, through software or hardware configuration, the processing circuit is configured to or adapted to perform one or more actions described herein.

[0216] As used herein, the term "number" or "value" may refer to any kind of number, such as a binary number, a real number, an imaginary number, or a rational number, etc. In addition, a "number" or "value" may be one or more characters, such as a letter or a string of letters. In addition, a "number" or "value" may be represented by a bit string.

[0217] As used herein, the expressions "may" and "in some embodiments" have been generally used to indicate that the features described can be combined with any other embodiments disclosed herein.

[0218] In the drawings, features that may be present only in some embodiments are often drawn using dotted or dashed lines.

[0219] When the word "comprising" or "including" is used, it shall be interpreted as being non-limiting, ie meaning "consisting at least of."

[0220] The embodiments herein are not limited to the above embodiments. Various alternatives, modifications and equivalents may be used. Therefore, the above embodiments should not be considered to limit the scope of the present disclosure defined by the appended claims.

Claims

1. A method for associating multispectral 2D image data with 3D image data resulting from light triangulation performed by an imaging system (500) for 3D imaging of an object (520), wherein the imaging system (500) comprises a first light source (510) for illuminating the object (520) with a first light (511), a camera (530) having an image sensor (531) and one or more second light sources (550) for illuminating the object (520) with two or more second lights (551) that differ from each other by including different light wavelengths, the two or more second lights (551) being multispectral, wherein the light triangulation comprises illuminating different successive portions of the object (520) with the first light (511) and sensing, by the image sensor (531), reflected first lights from the respective portions in respective first images "IM1" (541), wherein the method comprises: - obtaining (601) the 3D data as a first sensor position "SP1" (571) of IM1 (541) generated by the camera (530) and the image sensor (531), SP1 corresponding to the location of the peak of the intensity of the reflected first light from the object (520) as part of the light triangulation, - obtaining (602) two or more second images "IM2" (742) generated by the camera (530) and the image sensor (531), imaging the object during illumination by the two or more second lights (551), respectively, wherein a corresponding IM2 is a corresponding IM1 among the IM1s (741) and is associated with a corresponding IM1 among the IM1s (741), or is generated between two IM1s among the IM1s (741) and is associated with any IM1 of them, - in the respective IM2 (542), for the respective SP1 (571) of the respective IM1 (541) to which the respective IM2 (542) is associated and in combination with the respective SP1 (571), selecting (603) a respective second sensor position "SP2" (572) at which the reflected second light (551) from the object (520) has a higher intensity than the reflected first light from the object (520), and - Associating (604) the intensity value of the selected SP2 (572) with the SP1 (571) for which SP2 (572) was selected, respectively, whereby the multispectral 2D data corresponding to the intensity value at SP2 (572) from the reflected multispectral second light (551) becomes associated with the 3D data corresponding to the SP1 (571) for which SP2 (571) was selected.

2. The method of claim 1, wherein the method further comprises: - providing (605) 3D image data and 2D image data at the same resolution, and - combining (606) the 3D image data and the 2D image of the same resolution into a combined image, whereby corresponding 3D image data points of the image become associated with multispectral 2D image data corresponding to intensity values ​​obtained by reflecting the two or more second lights comprising different wavelengths.

3. The method of claim 3, wherein if the resolutions of the 3D image data and the 2D image data are different, the providing at the same resolution is accomplished by up-sampling and / or down-sampling the 3D image data and / or the 2D image data.

4. The method according to any one of claims 1 to 3, wherein the second light comprises one or more of the following: red light "R", green light "G" and blue light "B". 5 . The method of claim 4 , wherein the second light comprises R, G, B, and wherein a difference in irradiation duration of R, G, B is used for white balance during imaging.

6. The method of any one of claims 1 to 5, wherein the second light comprises second light corresponding to infrared "IR" light or near infrared "NIR" light.

7. The method according to any one of claims 1 to 6, wherein according to a sequence of the two or more second lights, illumination of the two or more second lights is provided sequentially, and the second image is generated in accordance with the corresponding sequence. 8 . The method of claim 7 , wherein in the sequence one or more of the second lights appear more frequently than one or more other of the second lights.

9. The method of claim 8, wherein the sequence of the second light comprises red light "R", green light "G" and blue light "B", and wherein in the sequence, G appears more frequently than R and B.

10. The method of any one of claims 1 to 9, wherein the object is organic.

11. The method of claim 10, wherein the object is or corresponds to one of the following types and / or kinds of objects: plant, tree, log, wood, meat, vegetable, food, waste.

12. One or more devices (500; 530; 533; 1000) for associating multispectral 2D image data with 3D image data, the 3D image data being generated from light triangulation performed by an imaging system (500) for 3D imaging of an object (520), wherein the imaging system (500) comprises a first light source (510) for illuminating a measurement object (520) with a first light (511), a camera (530) having an image sensor (531) and a second light source (551) for illuminating the object (520) with two or more second lights (551). one or more second light sources (550) differ from each other by including different light wavelengths (231) for sensing reflected first light (413; 513) from a measurement object (220; 520), the two or more second lights (551) being multispectral, wherein light triangulation comprises illuminating first different successive portions of the measurement object (520) with the first light (511) and sensing, by the image sensor (531), the reflected first light from the respective first portions in respective first images "IM1" (541), wherein the one or more means are configured to: obtaining the 3D data as a first sensor position "SP1" (571) of IM1 (541) generated by the camera (530) and image sensor (531), SP1 corresponding to the location of a peak intensity of reflected first light from the object (520) as part of the light triangulation, obtaining two or more second images "IM2" (742) generated by the camera (530) and the image sensor (531) to image the object during illumination by the two or more second lights (551), respectively, wherein a corresponding IM2 is a corresponding IM1 among the IM1s (741) and is associated with a corresponding IM1 among the IM1s (741), or is generated between two IM1s among the IM1s (741) and is associated with any IM1 among them, In the corresponding IM2 (542), for the corresponding SP1 (571) of the corresponding IM1 (541) to which the corresponding IM2 (542) is associated and in combination with the corresponding SP1 (571), selecting (603) a corresponding second sensor position "SP2" (572) at which the reflected second light (551) from the object (520) has a higher intensity than the reflected first light from the object (520), and The intensity values ​​of the selected SP2 (572) are associated (604) with the SP1 (571) for which SP2 (572) is selected, respectively, whereby the multispectral 2D data corresponding to the intensity value at SP2 (572) from the reflected multispectral second light (551) becomes associated with the 3D data corresponding to the SP1 (571) for which SP2 (571) is selected.

13. One or more computer programs (1003) comprising instructions which, when executed by one or more processors (1004), cause the one or more apparatuses of claim 12 to perform the method of any one of claims 1-11.

14. One or more carriers, comprising one or more computer programs (1003) as claimed in claim 13, wherein the one or more carriers are one or more of the following: an electronic signal, an optical signal, a radio signal or a computer-readable storage medium (1101).

Images