DEVICE AND METHOD FOR PRODUCEING PHOTOMETRICAL STEREO IMAGES AND A COLOR IMAGE

Description

[0001] The present invention relates to an apparatus and a method for generating photometric stereo images and a color image of an object moving relative to an image sensor arrangement.

[0002] In numerous industrial and scientific applications, there is a need to examine objects using imaging techniques. In the manufacturing industry, for example, the objects can be individual or multiple products. To better identify and sort out rejects, images of the products are often taken and evaluated to determine whether the respective product complies with specified product specifications.

[0003] Product specifications, for example, may specify requirements regarding the surface finish of the products. It is often necessary, for example, to eliminate any unevenness or scratches. In other applications, however, it may be required that an embossed pattern applied to the products be completely and properly embossed. In photometric stereo images, such as inclination images, texture images, curvature images, gradient images, or height images, such three-dimensional or spatially distinct features of objects can be clearly visualized and therefore easier to examine.

[0004] Product specifications often also include requirements for two-dimensional or two-dimensional characteristics, such as color. For example, unwanted discoloration must be detected or the color match of a print on the objects with a print template must be ensured. For this purpose, color images of the objects must be captured. Color sensors or color cameras are usually used for this purpose, but these are primarily disadvantageous due to their comparatively high acquisition costs. A device for generating photometric stereo images and a color image is known, for example, from US 2018 / 0328855 A1. Furthermore, Rajarshi Ray, et al. "Error Analysis of Surface Normals Determined by Radiometry"; IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Computer Society, USA, Vol. 30, No. 6, November 1, 1983, p.631-645, XP011242705, ISSN: 0162-8828 a method for determining geometric errors in surface detection is known.

[0005] The present invention is therefore based on the object of creating possibilities that allow images of the type mentioned at the outset to be generated at low cost, in particular without the use of color sensors or color cameras.

[0006] The object is achieved according to the invention by a device for generating photometric stereo images and the color image as defined in the appended claim 1.

[0007] Advantageously, the device according to the invention can be used to generate photometric stereo illumination by means of the different illumination directions of the illumination pulses, which simultaneously enables color differentiation due to the different wavelength ranges of the illumination pulses.

[0008] The device according to the invention preferably comprises an image data processor for calculating photometric stereo images and a color image of an object moving at a predetermined relative speed relative to an image sensor arrangement with a predetermined sensor geometry, wherein the image data processor is designed to calculate the photometric stereo images and the color image from at least four image data sets, wherein at least three of the image data sets represent brightness values ​​of different locations of a region of the object illuminated in respectively different wavelength ranges and wherein at least two of the image data sets represent brightness values ​​of these locations of the object illuminated from respectively different illumination directions, wherein the image data processor is designedbased on sensor data representing the sensor geometry and speed values ​​representing the relative speed, brightness values ​​of individual pixels at the same location in the object area from the image data sets with different wavelength ranges are combined as a multi-colored pixel tuple, and brightness values ​​of individual pixels at the same location in the object area from the image data sets with different illumination directions are combined as a multi-directional pixel tuple, and wherein the image data processor is configured to generate a pixel of the color image based on each multi-colored pixel tuple and the wavelength ranges, and to generate a pixel of a photometric stereo image based on each multi-directional pixel tuple and the illumination directions.

[0009] The brightness values ​​of the image data sets are, in particular, brightness values ​​or intensity values ​​measured by pixels of the image sensor array. In other words, each image data set contains brightness values ​​or intensity values ​​measured by pixels of the image sensor array under illumination in a predetermined wavelength range from a predetermined illumination direction, wherein at least three of the image data sets differ in the wavelength ranges represented therein, and wherein at least two of the image data sets differ in the illumination directions represented therein. However, the specific information about the different wavelength ranges and illumination directions does not have to originate from the image sensor array, since it is sufficient if this information can merely be assigned to the image data sets and, in particular, to the brightness values ​​contained therein.

[0010] If all brightness values ​​can be clearly assigned to a wavelength range, they can advantageously be present as monochromatic gray values, and the pixels of the color image can still be reconstructed as follows. In combination with the associated wavelength range, the brightness value of an individual pixel of each multicolored pixel tuple necessarily corresponds to the brightness value of a corresponding pixel of a color channel of the color image, if the color channel reproduces the said wavelength range. The superposition of the pixels of the color channels of all different wavelength ranges represented in the image data sets from one and the same multicolored pixel tuple, reconstructed in this way, consequently results in the complete pixel of the color image. The entire color image results from the totality of the generated pixels.

[0011] A similar procedure is possible when generating the pixels of a photometric stereo image. This procedure is based on the so-called photometric stereo method, which is known to those skilled in the art from the prior art and is therefore only briefly explained below. According to the photometric stereo method, the brightness value of an individual pixel of each multidirectional pixel tuple and the associated illumination direction are inserted into an equation of a system of equations for calculating a surface normal at the location in the object area that the respective pixel tuple represents.

[0012] The system of equations reflects the qualitative relationship that an object surface facing a light source reflects more light and therefore appears brighter to an observer (here: image sensor array) than an object surface facing away from that light source. Object surfaces that are laterally positioned relative to the light source accordingly exhibit an average brightness. With enough individual pixels in the multidirectional pixel tuple, a sufficient number of linearly independent equations of the system of equations can be set up, the solution set of which determines the surface normal. In the simplest case, a difference image can be generated from just two illumination directions, which makes many topological defects in a specific orientation visible. By calculating further difference images with different combinations of directions, ultimately all defects in all orientations can be detected.

[0013] Using an arbitrarily defined color coding, a color value or gray value can be assigned to the orientation of the surface normal calculated for the respective multidirectional pixel tuple. This color value or gray value corresponds to the color value or gray value of the pixel in the photometric stereo image. An entire photometric stereo image results from all pixels generated in this way.

[0014] The present invention therefore has the advantage that both photometric stereo images and the color image can be calculated based on recorded monochromatic brightness values. This eliminates the need for a color sensor or color camera. The cost is correspondingly low when the image data processor is used to generate photometric stereo images and the color image.

[0015] A method for generating the photometric stereo image and the color image also solves the problem set out at the outset, the method being defined in the appended claim 6.

[0016] This method offers the same advantages as the previously described device according to the invention.

[0017] The wavelength ranges can be continuous, interrupted, and / or composed of individual spectral lines. The illumination directions describe the spatial directions in which the respective illumination pulses propagate toward the illumination area.

[0018] The inventive solution can be further improved by various embodiments, each advantageous in itself and combinable with one another as desired. These embodiments and the associated advantages are discussed below. The advantages described with regard to the image data processor and the device also apply to the inventive methods, and vice versa.

[0019] According to a first possible embodiment, the image data processor can be configured to read the image data sets directly from the image sensor array after they have been recorded there. Additionally or alternatively, the image data processor can be configured to read the image data sets from the storage medium. Accordingly, the image data sets can be stored or temporarily stored on the storage medium. If necessary, a temporal decoupling between the recording of the image data sets and the calculation of the images can thus advantageously be achieved.

[0020] According to one possible embodiment of the device and / or method according to the invention, the illumination sequence can also comprise more than four illumination pulses. In particular, at least two illumination pulses from different illumination directions can have the same wavelength range. Thus, the applied photometric stereo method is less susceptible to distortion due to absorption of the illumination pulses. In other words, by using the same wavelength range, it can be ensured that the measured brightness values ​​are based purely on the orientation of the object surface and are not influenced by the color of the object surface.

[0021] Additionally or alternatively, at least two illumination pulses of different wavelength ranges can have the same illumination direction.

[0022] According to another possible embodiment, the wavelength ranges of the illumination pulses can cover an RGB color space. In other words, each of the wavelength ranges can represent or form a portion of the RGB color space. The wavelength ranges can thus be the wavelength ranges of red light, green light, and blue light. This embodiment is advantageously suitable for generating RGB color images.

[0023] Additionally or alternatively, the wavelength ranges can also correspond to UV radiation and / or IR radiation. The color image to be calculated can then be a multispectral image in false color representation, in particular a multispectral image or a hyperspectral image.

[0024] Optionally, at least one illumination pulse, preferably each illumination pulse, can form a light sheet or light disk. In other words, the illumination pulses can be configured as light sheets or light disks directed onto the illumination area. This allows the illumination pulses to be distributed as evenly as possible across the illumination area.

[0025] According to the invention, the device comprises a monochromatic image sensor array with a recording area, wherein the recording area of ​​the image sensor array and the illumination area overlap, and wherein the image sensor array is configured to measure brightness values ​​in the recording area during each illumination pulse. By using a monochromatic image sensor array, this embodiment is advantageously characterized by particularly low acquisition costs.

[0026] According to the invention, the method step of illuminating comprises the steps of emitting an illumination pulse in a first wavelength range from a first illumination direction, from a second illumination direction, from a third illumination direction, and from a fourth illumination direction; emitting an illumination pulse in a second wavelength range from each of the first, second, third, and fourth illumination directions; and emitting an illumination pulse in a third wavelength range from each of the first, second, third, and fourth illumination directions, wherein the first, second, and third wavelength ranges are not congruent with one another, and wherein the first, second, third, and fourth illumination directions are not parallel to one another. These steps are preferably repeated periodically.

[0027] The use of four different illumination directions sufficiently improves the accuracy of the photometric stereo method for most applications without disproportionately increasing the computational effort.

[0028] Of course, for applications that require even more accurate photometric stereo images, the illumination pulses can be emitted from more than four different illumination directions.

[0029] According to another possible embodiment, the illumination pulses can be emitted individually in succession, each generating a unidirectional illumination configuration. Optionally, simultaneous emission of multiple illumination pulses with the same wavelength range and different illumination directions can result in a bidirectional illumination configuration or a multidirectional diffuse illumination configuration. This is explained below.

[0030] For example, at least two illumination pulses in the same wavelength range can be emitted simultaneously, creating the bidirectional illumination constellation. Vectors representing the illumination directions of these two illumination pulses preferably have two codirectional spatial components and one oppositely directed spatial component. At least two additional illumination pulses in this same wavelength range can also be emitted simultaneously, creating yet another bidirectional illumination constellation.

[0031] The aforementioned multi-sided diffuse illumination configuration arises, for example, when four illumination pulses in the same wavelength range are emitted simultaneously. Vectors representing the illumination directions of these four illumination pulses exhibit only one co-directional spatial component and two oppositely directed spatial components for two of the four illumination pulses. Optionally, four additional illumination pulses in the same wavelength range can also be emitted simultaneously, and the remaining illumination pulses can be emitted sequentially. The multi-sided diffuse illumination configuration for the latter can be subsequently reconstructed computationally by averaging the brightness values ​​measured during the consecutively emitted illumination pulses.

[0032] The multi-sided diffuse illumination configuration provides uniform lighting conditions for recording the brightness values ​​for the color image. The unidirectional or bidirectional illumination configurations primarily serve to record the brightness values ​​for a photometric stereo image. Depending on the number of different unidirectional or bidirectional illumination configurations, a tilt image, a texture image, a curvature image, a gradient image, and / or a height image can be calculated as a photometric stereo image. The tilt images, texture images, curvature images, gradient images, and height images can also be converted into one another using numerical integration or differentiation.

[0033] The device according to the invention can comprise the aforementioned image data processor and thus advantageously fulfill its function without any additional devices. Additionally or alternatively, the device can also be designed to be connectable to the aforementioned computer.

[0034] The image data processor can be designed to read out the brightness values ​​from the image sensor arrangement as image data sets after each illumination pulse and to assign the wavelength range and the illumination direction of this illumination pulse to each brightness value.

[0035] According to the invention, the image sensor arrangement has at least four pixel fields spaced apart from one another at a predetermined spatial distance, wherein the time interval between two successive illumination pulses of the illumination sequence is equal to the time during which an image of the object projected onto the image sensor arrangement advances by the relative speed by a distance which corresponds to the spatial distance between two adjacent pixel fields.

[0036] In this way, pixel synchronicity can be achieved between the brightness values ​​recorded by the pixel fields. For this purpose, the object is preferably moving in a straight line, and the pixel fields are arranged exactly one behind the other with respect to the direction of movement of the object. This means that every point on the object captured by the first pixel field is also captured by all pixel fields arranged behind it.

[0037] The time interval corresponds to a time span and / or a time duration calculated from the pixel pitch on the object and the relative speed of the moving object to the image sensor arrangement.

[0038] In other words, the illumination arrangement is designed such that the time interval between two consecutive illumination pulses of the illumination sequence is selected such that a specific object area is imaged sequentially onto the at least four pixel fields. This advantageously ensures pixel synchronicity, while the generated measured values ​​are comparable.

[0039] The pixel fields of the image sensor array can, for example, be pixel rows. Accordingly, the spatial distance represents a row spacing between the pixel rows. Optionally, the image sensor array can have at least one pixel row for each different illumination constellation resulting from the illumination sequence. This ensures that for each multi-colored pixel tuple, a multi-directional pixel tuple can also be recorded that images the same location on the object. For this purpose, the image sensor array can be configured, for example, as a multi-row line-scan camera or a matrix camera that can be read line by line. Optionally, a combination of several such cameras can also form the image sensor array.

[0040] According to an alternative embodiment, the number of pixel rows of the image sensor array can also be smaller than the number of different illumination constellations resulting from the illumination sequence. Thus, although pixel rows are missing to capture every area of ​​the moving object in each of the different illumination constellations, the pixel synchronicity defined above can still be achieved with sufficient accuracy. For each "missing" pixel row, the brightness values ​​for one illumination constellation each must be recorded "between" the existing pixel rows. In other words, the time interval to the illumination pulse of the previous illumination constellation must be smaller than the time during which the image of the object projected onto the image sensor array advances due to the relative speed by the distance corresponding to the line spacing between two adjacent pixel rows.

[0041] For example, in one application, the image sensor array may have only two pixel rows, while the illumination array emits an illumination sequence with four different illumination constellations. In another application, the image sensor array may have four pixel rows, while the illumination array emits an illumination sequence with six different illumination constellations. The number of pixel rows and illumination sequences is, in principle, variable, although in any case, the number of pixel rows is smaller than the number of illumination sequences.

[0042] In these cases, however, there are generally not enough pixel rows available to capture every area of ​​the moving object in each of the different lighting configurations. To at least partially achieve the pixel synchronicity mentioned above, the time interval between two consecutive illumination pulses must be shorter, at least in two lighting configurations, than the time during which the image of the object projected onto the image sensor array advances due to the relative speed by the distance corresponding to the spatial distance between two adjacent pixel rows. The brightness values ​​for these lighting configurations must therefore be recorded "between" the pixel rows. The offset between the two abbreviated pixel rows is, for example, 50%, optionally less than 25%, and preferably less than 10% of the pixel width of a single pixel of the image sensor array.Such accuracy is sufficient for most applications and is still considered pixel synchronicity within the scope of the present invention. If this somewhat imprecise pixel synchronicity is not sufficient, the brightness values ​​can be further approximated by interpolation or averaging of two pixel rows.

[0043] Even with pixel rows that are not directly adjacent, the aforementioned pixel synchronicity can be sufficiently achieved if the "interposed" illumination constellations are generated as close as possible to the previous illumination constellation and the brightness values ​​are measured immediately. In this case, this means that many "interposed" illumination constellations are generated while the image of the object projected onto the image sensor array advances by the distance corresponding to the line spacing of two adjacent pixel rows due to the relative speed.

[0044] The above-mentioned linear movement of the object can be generated, for example, by a conveyor device. Accordingly, the device according to the invention can comprise the conveyor device. The method according to the invention comprises the steps (optionally controlling the conveyor device) of moving the object through the illumination area and through the recording area of ​​the monochromatic image sensor array, and recording the brightness values ​​in the recording area during each illumination pulse. This embodiment enables continuous recording, making it particularly suitable for applications with elongated or continuous objects.

[0045] The relative speed and sensor geometry are preferably constant. Depending on the application, the speed values ​​and / or the sensor data may change over time. In both cases, the image data processor can be configured to read or receive the speed values ​​and the sensor data, for example, via one or more interfaces.

[0046] Furthermore, the method according to the invention provides for a monochromatic image sensor arrangement having at least four pixel fields spaced apart from one another at a predetermined spatial distance, wherein the time interval between two successive illumination pulses of the illumination sequence is selected to be less than or equal to the time during which an image of the object projected onto the image sensor arrangement advances due to the relative speed by a distance which corresponds to the spatial distance between two adjacent pixel fields. The invention is explained in more detail below by way of example with reference to the drawings. The combination of features represented by way of example in the embodiments shown can be supplemented by further features in accordance with the above statements, depending on the properties of the image data processor and / or the device according to the invention required for a specific application.Also, subject to the above explanations, individual features may be omitted from the described embodiments if the effect of this feature is not important in a specific application. In the drawings, the same reference numerals are always used for elements with the same function and / or structure.

[0047] They show: Fig. 1: a schematic representation of an image data processor according to an exemplary embodiment; Fig. 2: a schematic representation of an inventive device according to an exemplary embodiment; and Fig. 3: a schematic representation of an inventive device according to another exemplary embodiment.

[0048] The following is an image data processor 1 with reference to Fig. 1 Furthermore, exemplary embodiments of a device 2 according to the invention are described with reference to Fig. 2 and 3described. Although some aspects of the invention are described merely in the context of a device, it is of course possible that these aspects also represent a description of the corresponding method, wherein, for example, a block, a module, a unit, or a device corresponds to a method step or a function of a method step. Analogously, aspects described in the context of a method step also correspondingly represent a description of a block, a module, a unit, or a property of the device.

[0049] In Fig. 1 A highly simplified, schematic representation of an exemplary embodiment of the image data processor 1 is shown. The image data processor 1 can have a stand-alone processor board 4 and / or be mounted on a board (not shown) of the device 2 (see Fig. 2 and 3). The blocks, modules, and units of the image data processor 1 described below can each be implemented in hardware, software, or a combination of both.

[0050] The image data processor 1 is designed to calculate photometric stereo images 6 and a color image 8 of a monochromatic image sensor array 12 (see Fig. 2 and 3 ) with a predetermined sensor geometry. For this purpose, the image data processor 1 is configured to calculate the photometric stereo images 6 and the color image 8 from at least four image data sets 18 in an image calculation unit 16. This will be explained in more detail below.

[0051] In Fig. 1 On the left, it is indicated for the image data sets 18 that these can, for example, be read directly from monochromatic pixels 20 of the image sensor array 12 after they have been recorded there consecutively, as will be explained further below. Alternatively, the image data sets 18 can also be stored on a storage medium 22 and read from it. For the readout (see thought bubble 201 in Fig. 1 ), the image data processor 1 can have a corresponding readout unit 24 and a readout interface 26.

[0052] The image data sets 18 contain, according to the invention, monochromatic gray values ​​that represent brightness values ​​28 of different locations in a region of the object 14. At least three of the image data sets 18 represent brightness values ​​28 of the object 14 illuminated in different wavelength ranges 30. Such image data sets 18 are in Fig. 1 exemplified as image data set group 32a. At least two of the image data sets 18 represent brightness values ​​28 of the object 14 illuminated from different illumination directions 34. Such image data sets 18 are in Fig. 1 exemplified as image data set group 32b.

[0053] In other words, each image data set 18 contains brightness values ​​28 measured by the image sensor array 12 at different locations on the object 14 under illumination in a predetermined wavelength range 30 from a predetermined illumination direction 34, wherein at least three of the image data sets 18 differ in the wavelength ranges 30 represented therein (i.e., image data set group 32a) and wherein at least two of the image data sets 18 differ in the illumination directions 34 represented therein (i.e., image data set group 32b). As can be seen, the same image data sets 18 can simultaneously belong to image data set groups 32a and 32b.

[0054] As explained further below, the different wavelength ranges 30 can be, for example, the wavelength ranges of red light 36, green light 38, and blue light 40. Accordingly, the wavelength ranges 30 used for illumination can cover an RGB color space, and the color image 8 to be calculated can be an RGB color image 42. Additionally or alternatively, UV radiation and / or IR radiation can also be used for illumination. The color image 8 to be calculated can then be a multispectral image in false color representation, in particular a multispectral image or a hyperspectral image.

[0055] As will also be explained further below, the different illumination directions 34 can result from a unidirectional illumination constellation 44, a bidirectional illumination constellation 46, or a multidirectional diffuse illumination constellation 48. The unidirectional or bidirectional illumination constellations 44, 46 primarily serve to record the brightness values ​​28 for the photometric stereo image 6. Depending on how many different unidirectional or bidirectional illumination constellations 44, 46 are present, an inclination image, a texture image, a curvature image, a gradient image, and / or a height image can be calculated as the photometric stereo image 6. The multidirectional diffuse illumination constellation 48 provides uniform illumination conditions for recording the brightness values ​​28 for the color image 8.

[0056] The precise formation of the illumination constellations 44, 46, 48 in the different wavelength ranges 30 and from the different illumination directions 34 is explained in more detail below in connection with the device 2 according to the invention. The image data processor 1, in particular the readout unit 24, can be configured to assign the corresponding wavelength range 30 and the corresponding illumination direction 34 to each brightness value 28.

[0057] To improve understanding, the image data sets 18 are shown in Fig. 1 the brightness values ​​28 are assigned to the individual pixels 20 of the image sensor array 12 by which they were recorded and are imaged together with the entire object 14, the different wavelength ranges 30 and the different illumination directions 34. These are each top views 50 of the object 14 from the perspective of the image sensor array 12. The respective illumination constellation 44, 46, 48 is indicated in the top views 50 by arrows, with different arrow directions representing different illumination directions 34, while different arrow lines (e.g. dot-dash, solid line, thick line) represent the different wavelength ranges 30.

[0058] The brightness values ​​28 of the image data sets 18 can be measured by the image sensor array 12 using an electronic shutter. Accordingly, it can be seen that the image sensor array with which the image data sets 18 are recorded has four pixel rows 52a, 52b, 52c, 52d. The sensor geometry already mentioned above is defined in this case primarily by the mutual spatial spacing 54 of the pixel rows 52a, 52b, 52c, 52d (i.e., row spacing 124).

[0059] The image data processor 1 is further configured to identify brightness values ​​28 of individual pixels 20 of the same location of the object area based on the spatial distance 54 and on the speed values ​​56 representing the relative speed 10 from the image data sets 18 with different wavelength ranges 30, in particular from the respective image data set groups 32a, and to combine them as a multi-colored pixel tuple 58 (see thought bubble 202 in Fig. 1 ). In addition, the image data processor 1 is designed to identify brightness values ​​of individual pixels 20 of the same location in the object area based on the spatial distance 54 and the speed values ​​56 from the image data sets 18 with different illumination directions 34, in particular from the respective image data set groups 32b, and to combine them as a multidirectional pixel tuple 60 (see thought bubble 203 in Fig. 1 ). For this purpose, the image calculation unit 16 may have a pixel tuple identification module 62 with a block 64 for identifying and combining multi-colored pixel tuples 58 and a block 66 for identifying and combining multi-directional pixel tuples 60.

[0060] Each multicolored pixel tuple 58 is a tuple of length n and contains as elements the brightness values ​​28 from the image data sets 18 with different wavelength ranges 30, which were measured by pixels of the image sensor arrangement 12 at one and the same location in the object area, wherein the length n the number of different wavelength ranges 30. Similarly, each multidirectional pixel tuple 60 is a tuple of length mand contains as elements the brightness values ​​28 from the image data sets 18 with different illumination directions 34, which were measured by pixels of the image sensor arrangement 12 at one and the same location in the object area, wherein the length m the number of different illumination directions corresponds to 34.

[0061] Preferably, the image data sets 18 are recorded such that for each multi-colored pixel tuple 58, there is also a multi-directional pixel tuple 60 that images the same location in the object area, and vice versa. This will be explained in more detail below in connection with the device 2 according to the invention and enables the generation of pixel synchronicity between the color components of the color image 8 and the photometric stereo image 6.

[0062] The brightness values ​​28 of each multi-colored pixel tuple 58 can, for example, be uniquely assignable to a wavelength range 30 by their arrangement within the pixel tuple 58. Accordingly, the brightness values ​​28 of each multi-directional pixel tuple 60 can be uniquely assigned to an illumination direction 34 by their arrangement within the pixel tuple 60. This is shown in Fig. 1 purely exemplary for pixel tuples 58, 60 with three / two elements 70 (see thought bubbles 202, 203). The multicolored pixel tuples 58 and the multidirectional pixel tuples 60 can, of course, differ in their number of elements 70.

[0063] The image data processor 1 is further configured to generate a pixel 72 of the color image 8 based on each multi-colored pixel tuple 58 and the wavelength ranges 30, in particular the wavelength ranges 30 of the multi-sided diffuse illumination constellations 48 (see thought bubble 204 in Fig. 1 ). Since all brightness values ​​28, as already described above, are only present as monochromatic gray values, but can be assigned to a wavelength range 30, the pixel 72 of the color image 8 can be reconstructed as follows. The brightness value 28 of an element 70 of the multicolored pixel tuple 58 in combination with the associated wavelength range 30 corresponds to the brightness value 28 of a pixel 74 of a color channel of the color image 8, wherein the color channel reproduces this wavelength range 30. The superposition of the pixels 74 of different wavelength ranges 30 from one and the same multicolored pixel tuple 58 reconstructed in this way consequently results in the complete pixel 72 of the color image 8. The entire color image 8 results from the totality of the generated pixels 72 (see thought bubble 206 in Fig. 1 ).

[0064] In addition, the image data processor 1 is designed to generate a pixel 76 of the photometric stereo image 6 based on each multidirectional pixel tuple 60 and the illumination directions 34, in particular the illumination directions 34 of the directed illumination constellations 44, 46 (see thought bubble 205 in Fig. 1 ). The generation of the pixel 76 is carried out according to the so-called photometric stereo method, which is known to those skilled in the art from the prior art and is therefore only briefly explained below. The brightness value 28 of each element 70 of the multidirectional pixel tuple 60 and the associated illumination direction 34 are inserted, according to the photometric stereo method, into an equation of a system of equations for calculating a surface normal at the location of the object area that the pixel tuple 60 depicts.

[0065] The system of equations reflects the qualitative relationship that an object surface facing a directed light source reflects more light and therefore appears brighter to an observer (here: image sensor array 12) than an object surface facing away from this light source. Object surfaces positioned laterally to the light source exhibit an average brightness. With enough elements 70 in the pixel tuple 60, a sufficient number of linearly independent equations of the system of equations can be established, the solution set of which determines the surface normal.

[0066] Using an arbitrarily definable color coding, a color value or a gray value 78 can be assigned to the orientation of the surface normal calculated for the pixel tuple 60. This color value or gray value 78 then corresponds to the color value or gray value 80 of the pixel 76. The entire photometric stereo image 6 results from the totality of the pixels 76 generated in this way (see thought bubble 207 in Fig. 1 ).

[0067] For the purpose of generating the pixels 72, 76, the image calculation unit 16 can have a pixel generation module 82 with a block 84 for generating the pixels 72 of the color image 8 and a block 86 for generating the pixels 76 of the photometric stereo image 6. Due to the aforementioned pixel synchronicity, the color image 8 and the photometric stereo image 6 can optionally be combined in the image calculation unit 16 to form a photometric stereo color image.

[0068] Optionally, the image data processor 1 can have an internal data memory 88, which is designed to store or temporarily store intermediate results of the process steps running in the image data processor 1. The thought bubbles 201, 202, 203, 204, 205, 206 and 207 of Fig. 1 are intended merely to represent a symbolic representation of the process steps taking place in the image data processor 1. Their placement in the area of ​​the data memory 88 is due to space constraints and is not intended to imply that such graphical reproduction must take place in the image data processor 1 or in the data memory 88.

[0069] Fig. 2 and 3 show schematic representations of exemplary embodiments of the device 2 according to the invention, which has the image sensor arrangement 12 already mentioned above and an illumination arrangement 90. The structure and function of the illumination arrangement 90 are explained below with reference to Fig. 2 in which the image sensor arrangement 12 is shown only in a very simplified manner. The image sensor arrangement 12 is again described with reference to Fig. 3 described, in which the lighting arrangement 90 is shown only in a highly simplified manner for reasons of better clarity.

[0070] The device 2 serves to generate the photometric stereo image 6 and the color image 8. In particular, the illumination arrangement 90 of the device 2 serves to generate the illumination constellations 44, 46, 48, and the image sensor arrangement 12 of the device 2 serves to record the brightness values ​​28 for the image data sets 18. In order to generate the relative speed 10, the device 2 can further comprise a conveyor device 92, for example a conveyor belt 94, on which the object 14 is movable relative to the device 2, in particular relative to the image sensor arrangement 12. Additionally or alternatively, the device 2, in particular the image sensor arrangement 12, can also be designed to be movable relative to the object 14.

[0071] The illumination arrangement 90 is directed toward an illumination region 96 and configured to cyclically emit an illumination sequence 98 into the illumination region 96. The illumination sequence 98 comprises at least four illumination pulses 100, which preferably repeat periodically, with each illumination pulse 100 having a wavelength range 30 and an illumination direction 34. The wavelength range 30 can be continuous, interrupted, or formed from individual spectral lines. The illumination direction 34 is the spatial direction in which the respective illumination pulse 100 propagates toward the illumination region 96.

[0072] At least three of the illumination pulses 100 differ in their respective wavelength ranges 30 and at least two of the illumination pulses 100 differ in their respective illumination directions 34, which is reflected in the image data set groups 32a, 32b already mentioned above.

[0073] The illumination pulses 100 can be emitted individually by the illumination arrangement 90 and each generate a unidirectional illumination constellation 44. Optionally, by simultaneously emitting several illumination pulses 100 with the same wavelength range 30 and different illumination directions 34, a bidirectional illumination constellation 46 (see Fig. 3 ) or a multi-sided diffuse lighting constellation 48. This is explained below.

[0074] The illumination arrangement 90 can, for example, be configured to emit a first illumination pulse 100a in a first wavelength range 30a from a first illumination direction 34a, a second illumination pulse 100b in a second wavelength range 30b from a second illumination direction 34b, and a third illumination pulse 100c in a third wavelength range 30c from a third illumination direction 34c. The wavelength ranges 30a, 30b, 30c are not congruent with one another, and the illumination directions 34a, 34b, 34c are not parallel to one another. However, all illumination directions can intersect in the illumination region 96.

[0075] As shown in the top views 50 of the Fig. 1 As shown top left, each illumination pulse 100a, 100b, 100c is emitted individually and generates exactly one unidirectional illumination constellation 44. The illumination pulses 100a, 100b, 100c can repeat periodically.

[0076] The illumination arrangement 90 may further be configured to emit a fourth illumination pulse 100d in the third wavelength range 30c from the illumination direction 34d, wherein the illumination directions 34c, 34d are not parallel to one another.

[0077] In other words, the illumination arrangement 90 can be configured to emit exactly one illumination pulse in the first wavelength range from each of the first, second, third, and fourth illumination directions, exactly one illumination pulse in the second wavelength range from each of the first, second, third, and fourth illumination directions, and exactly one illumination pulse in the third wavelength range from each of the first, second, third, and fourth illumination directions. This is not shown separately. In this case, correspondingly more pixel rows would also have to be provided. Namely, exactly the number of wavelengths times the number of directions.

[0078] Two illumination pulses 100 in the same wavelength range 30 can also be emitted simultaneously to create a two-way illumination constellation 46. The illumination directions of the two simultaneously emitted illumination pulses preferably have a horizontal component 102 directed in the same direction and an opposite horizontal component 104. Preferably, the remaining illumination pulses in the same wavelength range 30 are then also emitted simultaneously. This is shown in the top views 50 of the Fig. 3 shown.

[0079] As in Fig. 3 As shown, the illumination arrangement 90 can have at least three electromagnetic radiators 106 directed towards the illumination region 96, each of which is designed to emit exactly one of the illumination pulses 100 in exactly one wavelength range 30 and exactly one illumination direction 34, so that the illumination pulses 100 shown in the plan views 50 of the Fig. 1 The illumination sequence 98 shown in the top left is created. In particular, the Fig. 3 In the lighting arrangement 90 shown, the electromagnetic radiators 106 are designed with red light 36, with green light 38 and with blue light 40, wherein the red light 36, the green light 38 and the blue light 40 are present alternately and radiate onto the lighting area 96 from different spatial directions per electronic radiator 106.

[0080] As in Fig. 2 As shown, the electromagnetic radiators 106 are designed, if possible, to distribute the illumination pulses 100 homogeneously across the illumination region 96. For this purpose, each electromagnetic radiator 106 can have an optic (not shown) directed toward the illumination region.

[0081] Preferably, the illumination arrangement 90 has at least one combined electromagnetic radiator 106 for each different wavelength range 30 of the illumination pulses 100 and / or for each different illumination direction 34 of the illumination pulses 100. Thus, the illumination arrangement 90, as shown by way of example in Fig. 3 shown, four electromagnetic radiators 106 configured as light strips 108. In each light strip 108, LEDs 110 of each of the different wavelength ranges 30 can be arranged alternately (e.g., in the sequence: RGB-...-RGB). LEDs of the same wavelength range 30 can be controlled independently of the LEDs of the other wavelength ranges 30. Furthermore, the light strips 108 can be configured to radiate onto the illumination area 96 in four different spatial orientations. This makes it possible to generate the illumination sequences 98 shown in the top views 50.

[0082] Optionally, at least one illumination pulse 100, preferably each illumination pulse 100, can form a light sheet 112 or a light disk 114. For this purpose, each of the electromagnetic radiators 106 can have an optic (not shown) directed toward the illumination area 96.

[0083] For recording the brightness values ​​28 as already mentioned above, the image sensor arrangement 12 has a recording area 116 that overlaps with the illumination area 96. The image sensor arrangement 12 is configured to measure the brightness values ​​28 in the recording area 116 during each illumination pulse 100.

[0084] As already mentioned above, the image sensor arrangement 12 has at least four pixel rows 52a, 52b, 52c, 52d arranged at a spatial distance 54 from one another. In particular, the image sensor arrangement 12 can have at least one pixel row for each different illumination constellation 44, 46, 48 resulting from the illumination sequence 98. This ensures that for each multi-colored pixel tuple 58, a multi-directional pixel tuple 60 can also be recorded, which images the same location of the object area. For this purpose, the image sensor arrangement 12 can be designed, for example, as a multi-line line scan camera 118 (see Fig. 3 ) or a line-by-line readable matrix camera 120 (see Fig. 2 ). Alternatively, a combination of several such cameras can also form the image sensor arrangement 12.

[0085] To achieve the aforementioned pixel synchronicity in a simple manner, the temporal interval 122 between two consecutive illumination pulses 98 of the illumination sequence 98 is preferably equal to the time during which the image of the object 14 projected onto the image sensor array 12 advances due to the relative velocity 10 by a distance corresponding to the spatial distance 54 between two adjacent pixel rows. To achieve this, the spatial distance 54 and the relative velocity 10 can be predetermined, and the temporal interval 122 between two consecutive illumination pulses 100 can be adapted accordingly. For example, this adaptation can be achieved by controlling the illumination array 90.

[0086] Alternatively, the spatial distance 54 and the temporal distance 122 can be specified, and the relative speed 10 can be adjusted accordingly. For this purpose, the conveyor device 92 can be controlled accordingly.

[0087] Furthermore, alternatively, the temporal distance 122 and the relative speed 10 can be predetermined, and the spatial distance 54 can be adjusted accordingly. The spatial distance 54 can be adjusted in advance by design on the image sensor array 12. For the matrix camera 120, it is also conceivable that the line spacing 124 is influenced by which lines of the matrix camera 120 are read out. To increase the spatial distance 54, for example, only every second pixel line can be read out instead of every pixel line.

[0088] The number of pixel lines of the image sensor arrangement 12 can optionally also be smaller than the number of different illumination constellations 44, 46, 48 resulting from the illumination sequence 98. This is in Fig. 3 There, the image sensor arrangement 12 has four pixel rows 52a, 52b, 52c, 52d, while the illumination arrangement 90 is designed to emit an illumination sequence with six different illumination constellations 46, 48 (see top views 50 in Fig. 3 ).

[0089] In principle, there are therefore not enough pixel rows available to capture every area of ​​the moving object 14 in each of the six different illumination constellations 46, 48. To at least partially achieve the pixel synchronicity already mentioned above, the temporal interval 122 between two consecutive illumination pulses 98 must be smaller, at least in two illumination constellations 46, than the time during which the image of the object 14 projected onto the image sensor array 12 advances by the distance corresponding to the spatial distance 54 between two adjacent pixel rows due to the relative speed 10. The brightness values ​​28 for these illumination constellations 46 must therefore be recorded "between" the pixel rows. This is Fig. 3 indicated by the offset top views 50. If the pixel rows are immediately adjacent, the brightness values ​​28 can be approximated by interpolation or averaging. Even with pixel rows that are not immediately adjacent, the above-mentioned pixel synchronicity can be sufficiently achieved if the "interposed" illumination constellations 46 are generated at the shortest possible time interval from the respective previous illumination constellation and the brightness values ​​are measured immediately.

[0090] In Fig. 3 An imaginary raster projection 126 is shown, which indicates the object areas captured by the individual pixel rows 52a, 52b, 52c, 52d of the image sensor array 12. The spatial distance 54 and the pixel width 68 are indicated on the raster projection 126.

[0091] As an alternative to pixel rows, the image sensor array 12 can also comprise other types of pixel arrays. In this case, the image data processor 1 is configured to identify the multicolored pixel tuples 58 and the multidirectional pixel tuples 60 based on other sensor data 57 representing the sensor geometry.

[0092] The speed values ​​56 and sensor data 57 are preferably constant. Depending on the application, the speed values ​​56 and / or sensor data 57 can change over time. For inputting or reading the speed values ​​56, the image data processor 1 can have an interface 128. Via the interface 128, the image data processor 1 can be connected to a rotary encoder (not shown) of the conveyor device 92 (see Fig. 2) or receive the speed values ​​56 from it in real time. A further interface 130 for inputting or reading the sensor data 57 can optionally also be provided on the image data processor 1.

[0093] The device 2 can comprise the image data processor 1. Alternatively, the device 2 can also be connected to a computer 132 on which a computer program can be executed, which includes instructions that cause the computer 132 to execute the method steps already described for the image data processor 1.

Claims

1. Device (2) for generating photometric stereo images (6) and one color image (8) of an object (14) moved by a predefined relative speed (10) relative to the device (2), whereby the device (2) has a lighting area (96) and a lighting arrangement (90) oriented to the lighting area (96), whereby the lighting arrangement (90) is designed to cyclically emit a lighting sequence (98) in the lighting area (96), whereby the lighting sequence (98) has at least four lighting pulses (100), whereby every lighting pulse (100) has a wavelength range (30) and a lighting direction (34), whereby at least three of the lighting pulses (100) are distinguished by their respective wavelength range (30), whereby at least two of the lighting pulses (100) are distinguished by their respective lighting directions (34) and whereby at least two lighting pulses (100) of different lighting directions (34) having the same wavelength range (30), whereby the device (2) comprises a monochromatic image sensor arrangement (12) with a capture area (116), whereby the capture area (116) of the image sensor arrangement (12) overlaps with the lighting area (96), and whereby the image sensor arrangement (12) is configured to measure brightness values (28) in the capture area (116) during each lighting pulse (100), characterized in that the image sensor arrangement (12) has at least four pixel fields (52) spaced apart from one another by a predefined spatial distance (54), whereby the temporal interval (122) between two consecutive lighting pulses (100) of the lighting sequence (98) is less than or equal to the time during which an image of the object (14) projected onto the image sensor arrangement (12) shifts, due to the relative speed (10), by a distance corresponding to the spatial distance (54) between two adjacent neighboring pixel fields (52).

2. Device (2) according to Claim 1, whereby at least two lighting pulses (100) of different wavelength ranges (30) have the same lighting direction (34).

3. Device (2) according to Claim 1 or 2, whereby the wavelength ranges (30) of the lighting pulses (100) correspond to an RGB color space.

4. Device (2) according to one of the Claims 1 to 3, whereby at least one lighting pulse (100) forms a light sheet (112).

5. Device (2) according to Claim 1, whereby the image sensor arrangement can be designed as a multiline linescan camera or a matrix camera which can be read out line by line.

6. Method for generating photometric stereo images (6) and one color image (8) of an object (14), whereby the method comprises the following steps: - cyclic illumination of a lighting area (96) in which the object (14) is located, with a lighting sequence (98), whereby the lighting sequence (98) has at least four lighting pulses (100), whereby each lighting pulse (100) has a wavelength range (30) and a lighting direction (34), whereby at least three of the lighting pulses (100) are differentiated by their respective wavelength ranges (30), whereby at least two lighting pulses (100) are distinguished by their respective lighting directions (34) and have the same wavelength range (30), whereby the illumination method step comprises: ▪ emitting a lighting pulse (100) in a first wavelength range (30a) from a first lighting direction (34a), from a second lighting direction (34b), from a third lighting direction (34c) and from a fourth lighting direction (34d), ▪ emitting a lighting pulse (100) in a second wavelength range (30b) from each of the first, second, third and fourth lighting direction (34a, 34b, 34c, 34d), and ▪ emitting a lighting pulse (100) in a third wavelength range (30c) from each of the first, second, third and fourth lighting direction (34a, 34b, 34c, 34d), whereby the first, second and third wavelength range (30a, 30b, 30c) are not congruent with on another, and whereby the first, second, third and fourth lighting direction (34a, 34b, 34c, 34d) are not parallel with one another, - moving the object (14) through the lighting area (96) and through a recording area (116) superimposed by the lighting area (96) of a monochromatic image sensor arrangement (12) and - recording brightness values (28) in the recording area (116) of the monochromatic image sensor arrangement (12) during each lighting pulse (100), characterized by: - providing the monochromatic image sensor arrangement (12) with at least four pixel fields (52) spaced apart from each other at a predefined spatial distance (54), whereby the time interval (122) between two consecutive lighting pulses (100) of the lighting sequence (98) is selected smaller / equal the time during which one picture of the object (14) projected onto the image sensor arrangement (12) pushes forward by means of the relative speed (10) by a distance which is equivalent to the spatial distance (54) of two neighboring pixel fields (52).

7. Method according to Claim 6, whereby at least two lighting pulses (100) are emitted simultaneously in the same wavelength range (30) and whereby at least two additional lighting pulses (100) are emitted simultaneously in the same wavelength range (30).

8. Method according to Claim 6 or 7, whereby four lighting pulses (100) are simultaneously emitted in the same wavelength range (30), whereby four additional lighting pulses (100) are simultaneously emitted in the same wavelength range, and whereby the remaining lighting pulses (100) are emitted simultaneously or consecutively.