System and method for determining the three-dimensional contour of a surface using a plenoptic camera and structured lighting

Abstract

The invention relates to the field of determining the three-dimensional profile of a surface. The invention relates to a system and a method for determining such a three-dimensional profile using a plenoptic camera and a structured lighting system. According to the invention, the method (30) comprises the following steps: # acquiring (31) a sequence of plenoptic images, each plenoptic image being formed by a set of pixels and comprising a plurality of sub-images of the surface observed from different viewing angles, the set of pixels being each associated with a light intensity of an imaged surface element, # projecting (32) the sequence of structured images onto the surface, the sequence of structured images being synchronized with the sequence of plenoptic images such that each imaged surface element is illuminated by a sequence of light intensities that is different from the series of light intensities illuminating the other imaged surface elements, # constructing (33) for each pixel of the plenoptic image an intensity vector representing the series of light intensities of the imaged surface element, and # matching (34) each pixel of a sub-image with a pixel of another sub-image based on the similarity of the intensity vectors of the pixels.

Description

Technical Field

[0001] The present invention relates to the field of determining the three-dimensional profile of a surface. The invention relates to a system for determining such a profile, the system comprising: an illumination system configured to project structured illumination onto the surface; a plenoptic camera configured to acquire a plenoptic image of the surface; and a processing unit configured to reconstruct depth information from the image acquired by the plenoptic camera. The invention also relates to the field of methods for determining the three-dimensional profile of a surface.

[0002] In particular, the invention finds application in controlling and inspecting the surfaces of components in an industrial environment, but may also be applied in other fields involving determining the three-dimensional contour of one or more surfaces of an object, such as video surveillance or driver assistance systems. Background Art

[0003] Quality control of mechanical components is a common problem in the manufacturing industry. This type of control can be performed, in particular, using ultrasonic sensors and probes or through image processing. The goal of this control can be to check whether the surface of a mechanical component reproduces the desired contour within a predetermined tolerance margin. To this end, image processing-based control typically involves acquiring a pair of stereo images, matching the pixels of the stereo images by searching for shape similarities in the images, and determining depth information through triangulation based on the corresponding positions of the pixels in the images.

[0004] Instead of a stereo image acquisition system, a plenoptic camera can also be used, which generates multiple images from different perspectives. Plenoptic cameras offer an interesting compromise between measurement performance and system compactness. However, the determination of depth information still relies on pixel matching between two or more images by searching for shape similarities. However, the process of searching for shape similarities requires a lot of computing resources and can be particularly complex or even impossible for surfaces with few features (such as changes in shape, color or texture) or no features. One solution to facilitate image matching is to add physical markers to the object to be inspected. However, this solution is cumbersome because it involves the arrangement and possible removal of these physical markers. Moreover, this solution is not always applicable and depends on the object to be inspected.

[0005] Another solution to facilitate image matching is to associate image acquisition with specific illumination. This measurement method is therefore referred to as an "active measurement method." This measurement method involves projecting a known structured illumination onto the object and searching for spatial or phase similarities associated with this structured illumination. In particular, laser interferometry or a video projector can be used to project stripes, or a set of patterns with a regular or pseudo-random spatial distribution can be generated. However, these techniques still involve a spatial similarity search process, which is computationally expensive. Furthermore, plenoptic cameras have a large number of subapertures, so the illuminated pattern must have low redundancy to avoid pixel matching ambiguity. In practice, active measurement methods are difficult to use with plenoptic cameras.

[0006] In view of the above, the present invention is directed to a method for determining the three-dimensional profile of a surface using a simple and robust plenoptic camera. The present invention is also directed to a system for determining the three-dimensional profile of a surface, the design cost, manufacturing cost, and maintenance cost of which are compatible with use on an industrial scale. Summary of the Invention

[0007] To this end, the invention is based on projecting a varying structured image onto a surface whose contour is to be determined, acquiring a sequence of plenoptic images of the surface, and searching for temporal similarities between pixels of different sub-images constituting the plenoptic image.

[0008] More particularly, the present invention relates to a system for determining a three-dimensional profile of a surface, the system comprising:

[0009] a plenoptic camera configured to acquire a sequence of plenoptic images of the surface, each plenoptic image being formed by a set of pixels each associated with a light intensity of an imaged surface element and comprising a plurality of sub-images of the surface observed from different viewing angles,

[0010] an illumination system configured to project a sequence of structured images onto the surface, the sequence of structured images being synchronized with the sequence of plenoptic images such that each imaged surface element is illuminated by a series of light intensities that is different from the series of light intensities illuminating the other imaged surface elements, and

[0011] a processing unit configured to construct, for each pixel of the plenoptic image, an intensity vector representing a series of light intensities of the imaged surface elements corresponding to the pixel in question, and to match each pixel of a sub-image with a pixel of another sub-image according to similarities between the intensity vectors of the pixels.

[0012] By varying the structured image during the sequence, the light intensity projected onto the imaged surface elements, and therefore the light intensity reflected from these imaged surface elements, also varies. Consequently, each imaged surface element can reflect a light intensity that varies uniquely compared to other imaged surface elements. The sequence of plenoptic images is synchronized with the sequence of structured images. In other words, each plenoptic image is acquired during the projection of the structured image. Therefore, during the sequence, the variation in light intensity of one pixel must differ from the variation in light intensity of another pixel, as these two pixels are associated with different imaged surface elements. Conversely, the presence of pixels whose light intensity exhibits the same or similar variation indicates that these pixels are associated with the same imaged surface element.

[0013] In particular, the number of structured and plenoptic images in the sequence (denoted as N) depends on the number of pixels in the sub-image. The higher the number of pixels, the larger N must be. Preferably, N is greater than or equal to 10. For example, for sub-images each formed by an array of 100 pixels by 100 pixels, N can be greater than or equal to 50, or greater than or equal to 100.

[0014] According to a particular embodiment, the illumination system is configured such that each structured image is formed by a set of projection patterns having a random light intensity distribution. Random light intensity distribution means that the projection patterns have no periodicity or pseudo-periodicity in any direction or plane.

[0015] Lighting systems can include:

[0016] a transmissive mask formed by a set of patterns, each of which has a transmittance that can take at least one of two different values, and

[0017] - a light source arranged to illuminate the surface by projection through the transmission mask.

[0018] For example, the size of the patterns of the transmission mask is about 10 μm (micrometers). In particular, each pattern can be rectangular or square. The set of patterns is, for example, in the form of an array arranged in rows and columns.

[0019] Preferably, the illumination system is configured such that the patterns each have a size smaller than the size of the imaged surface elements.

[0020] The transmittance of each pattern of the transmission mask can take a value of zero or one. That is, each pattern of the transmission mask can completely block the energy of the light beam or allow the light beam to pass through completely. The transmittance can also take a first value and a second value, where the first value is between 0% and 20% and the second value is between 80% and 100%. For example, the transmission mask has a total transmittance of 50%.

[0021] Alternatively, the transmittance of each pattern of the transmission mask may take multiple values, such as 8, 64, 128, or 256.

[0022] According to a particular embodiment, the illumination system further comprises displacement means arranged to displace the transmission mask and / or the light source relative to the surface in order to project a sequence of different structured images.

[0023] The light source can be a point light source. Thus, the light source can be individually displaced to obtain different structured images. The light source can also take the form of a backlight. In particular, the light source can comprise an array of light-emitting diodes. When the range of the light source is greater than the range covered by the transmissive mask at the positions associated with the different structured images in the sequence, the light source need not be displaced.

[0024] In a particular embodiment, the displacement device comprises:

[0025] a support arranged to receive the transmissive mask, and

[0026] - a positioning plate arranged to displace the support along at least one axis of rotation or at least one axis of translation.

[0027] In particular, the support may comprise a frame, within which the transmissive mask may be positioned and fixed.

[0028] The positioning plate is, for example, arranged to displace the support portion along a rotation axis parallel to the projection axis of the light source. Preferably, the rotation axis is eccentric with respect to the center of the transmission mask. Alternatively, the positioning plate can be arranged to displace the support portion along a first translation axis and a second translation axis, wherein the first translation axis and the second translation axis define a plane perpendicular to the projection axis of the light source.

[0029] Furthermore, in a particular embodiment, the pattern of the transmission mask has a first dimension along the first axis and a second dimension along the second axis, and the shifting device is arranged to shift the transmission mask and / or the light source between two consecutive structured images by a shifting step length that is greater than or equal to the larger of the first dimension and the second dimension. Preferably, the shifting step length is greater than or equal to twice the larger of the first dimension and the second dimension.

[0030] According to a specific embodiment, each pattern of the transmissive mask includes liquid crystal, and the illumination system further includes: a set of electrodes arranged on both sides of the pattern; and a control unit configured to supply power to the electrodes to individually control the transmittance of each pattern. Thus, the transmissive mask forms a liquid crystal display. Optionally, this specific embodiment can be combined with an embodiment in which the illumination system includes a shifting device for shifting the transmissive mask. Thus, a plenoptic image is formed based on the transmittance of each pattern and the position of the transmissive mask relative to the surface.

[0031] Preferably, the light source is monochromatic. For example, the light source emits a light beam with a wavelength of 405 nm (nanometer), 465 nm, 525 nm or 625 nm. A monochromatic light source avoids dispersion, thereby generating a projection pattern with sharp edges.

[0032] Furthermore, the light source is preferably incoherent or has low temporal coherence. Coherent light sources can produce light spots on the illuminated surface. These light spots thus add to the projected pattern and can disrupt pattern recognition.

[0033] According to a specific embodiment, a plenoptic camera comprises:

[0034] - a light sensor comprising a set of sensing elements, and

[0035] - a set of microlenses, each microlens being associated with a subgroup of sensing elements of the light sensor, such that each subgroup of sensing elements is capable of generating a sub-image.

[0036] The sensing elements of the light sensor are, for example, arranged in rows and columns. Similarly, the microlenses can be arranged in rows and columns. For example, the light sensor can include 15,000 rows by 15,000 columns of light sensors and 150 rows by 150 columns of microlenses. Thus, each microlens is associated with a group of 100 rows by 100 columns of sensing elements.

[0037] In one particular embodiment, the processing unit is configured to: for each sequence of sub-images associated with a given microlens, and for each pixel of the sub-image under consideration, search for one or more pixels having the most similar intensity vector from the pixels of the sequence of sub-images associated with the given microlens. The neighboring microlenses of a given microlens include at least two neighboring microlenses located in the same column and in the previous and next rows, and two neighboring microlenses located in the same row and in the previous and next columns. The neighboring microlenses may also include four microlenses located in the previous or next row and in the previous or next column. More generally, the neighboring microlenses may include a neighborhood of rank P, where P represents the maximum offset between the row rank of a given lens and the row rank of a neighboring lens, and the maximum offset between the column rank of a given lens and the column rank of a neighboring lens. Thus, a neighborhood of rank 2 includes twenty-four neighboring microlenses. When a plurality of pixels of different sequences of sub-images have similar intensity vectors as the pixel under consideration, all of these pixels may be considered.

[0038] The processing unit may also be configured to determine depth information of each imaged surface element based on the matched pixels associated with the imaged surface element. In particular, the corresponding positions of the sensing elements generating the matched pixels enable distance information of the imaged surface elements to be determined by triangulation.

[0039] The invention also relates to a method for determining the three-dimensional profile of a surface, the method comprising the following steps:

[0040] - acquiring a sequence of plenoptic images, each plenoptic image being formed by a set of pixels each associated with a light intensity of an imaged surface element and comprising a plurality of sub-images of the surface observed from different viewing angles,

[0041] - projecting a sequence of structured images onto the surface, the sequence of structured images being synchronized with the sequence of plenoptic images such that each imaged surface element is illuminated by a series of light intensities that is different from the series of light intensities illuminating the other imaged surface elements,

[0042] - constructing for each pixel of the plenoptic image an intensity vector representing the series of light intensities of the imaged surface element corresponding to the pixel in question, and

[0043] - Matching each pixel of a sub-image with a pixel of another sub-image based on the similarity between the intensity vectors of the pixels. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Other characteristics, details and advantages of the invention will appear on reading the following description, which is provided for illustrative purposes only and is given with reference to the accompanying drawings, in which:

[0045] - Figure 1 A first example of a system for determining a three-dimensional profile of a surface according to the present invention is schematically shown;

[0046] - Figure 2 A second example of a system for determining a three-dimensional profile of a surface according to the present invention is schematically shown;

[0047] - Figure 3 An example of a method according to the invention for determining a three-dimensional profile of a surface is shown. DETAILED DESCRIPTION

[0048] Figure 1 A first example of a system for determining the three-dimensional profile of a surface according to the invention is schematically shown. Reference numeral 1 denotes a surface whose three-dimensional profile is to be reconstructed. The system 2 comprises a plenoptic camera 3, an illumination system 4 and a processing unit 5.

[0049] Plenoptic camera 3 includes an optical acquisition lens 6, a microlens array 7, and a light sensor 8. The optical acquisition lens 6 is arranged to direct a portion of the light beam reflected from surface 1 toward the microlens array 7. The light sensor 8 is a planar sensor comprising a set of sensing elements 9 organized into an array. For example, the light sensor comprises a set of 15,000 rows by 15,000 columns of sensing elements. Light sensor 8 is sensitive to visible light wavelengths, for example. It is sensitive to at least the wavelength of the light beam emitted by illumination system 4. Thus, light sensor 8 can generate a plenoptic image formed from 15,000 by 15,000 pixels, each pixel representing the intensity of light emitted from the surface element imaged by that pixel. The microlens array 7 includes a set of microlenses 10, also organized into rows and columns. For example, the microlens array comprises 150 rows by 150 columns of microlenses. Each microlens 10 is associated with a separate subset of sensing elements 9. Each subset is formed from adjacent sensing elements, enabling the generation of a sub-image corresponding to a portion of the plenoptic image. In this case, each microlens 10 and each associated subgroup of sensor elements 9 can generate a subimage having 100 rows by 100 columns of pixels. The microlenses 10 are arranged so that different subimages represent the surface 1 in sections from different viewing angles.

[0050] The illumination system 4 includes a light source 11, a transmission mask 12, a motor 13, and an optical projection lens 14. The light source 11 is arranged to uniformly illuminate the transmission mask 12. The light source includes, for example, an array of light-emitting diodes. Preferably, the light source 11 emits an incoherent light beam having a single wavelength (e.g., a wavelength of 405 nm). The transmission mask 12 is formed by a set of patterns 15, the transmittance of each pattern in the set of patterns can be selected as a value of zero or a value close to zero, or a value of unity or a value close to unity. In other words, each pattern 15 is arranged to block the light beam emitted by the light source 11 or to allow the light beam emitted by the light source to pass through. Each pattern 15 is, for example, in the form of a square having a side length of 10 μm (micrometers). The transmission mask 12 includes, for example, an opaque plate in which openings are manufactured to form an alternating open and closed pattern. The patterns 15 are, for example, organized in the form of an array. Preferably, the patterns are arranged to have randomly distributed transmittance values. The motor 13 is arranged so that the motor rotates the transmission mask 12 around an axis that is parallel to the axis of the light beam emitted by the light source 11. For example, the motor is a stepper motor. The motor 13 is arranged to arrange the transmission mask 12 in at least N different positions, where N is a natural number greater than or equal to 10. Preferably, N is greater than or equal to 100. The optical projection lens 14 is arranged to focus the light beam on the surface 1, the light beam being emitted by the light source 11 and filtered by the transmission mask 12. In order to obtain a large depth of field, the optical projection lens 14 can be a telecentric lens. Thus, the illumination system 4 enables the generation of so-called structured images on the surface 1, each structured image being formed by projecting a pattern 15 of the transmission mask 12 onto the surface 1. When the pattern 15 has randomly distributed transmittance values, each structured image has a corresponding randomly distributed light intensity.

[0051] The processing unit 5 is configured to control the plenoptic camera 3 and the illumination system 4 in a synchronized manner. The processing unit is configured to instruct the illumination system 4 to project N structured images in succession, and to instruct the plenoptic camera 3 to acquire N plenoptic images synchronized with the N structured images. Thus, each plenoptic image is acquired during the projection of the structured image. Figure 1In the exemplary embodiment shown in FIG, N plenoptic images are obtained by shifting a transmission mask. The shift is arranged such that each surface element imaged by the plenoptic camera 3 is illuminated by a series of light intensities that is different from the series of light intensities illuminating the other imaged surface elements. Thus, each imaged surface element can be individually identified in a different sub-image of the plenoptic image. To this end, the processing unit 5 is configured to construct, for each pixel of the plenoptic image, an intensity vector representing the series of light intensities of the imaged surface element corresponding to the pixel in question. The processing unit 5 can also be configured to match each pixel of a sub-image with a pixel of another sub-image based on the similarity between the intensity vectors of the pixels. Different similarity methods can be used, such as methods based on correlation coefficients, methods based on squared intensity differences (SD), methods based on absolute intensity differences (AD), or methods based on mean absolute differences (MAD). Finally, the processing unit 5 can be configured to determine depth information for each imaged surface element based on the matched pixels associated with the imaged surface element. Typically, this depth information can be obtained through triangulation.

[0052] Figure 2 A second example of a system for determining the three-dimensional profile of a surface according to the present invention is schematically shown. The system 20 comprises a plenoptic camera, an illumination system and a coaxial configuration. Figure 1 Similar to system 2 in FIG, system 20 includes an optical acquisition lens 6, a microlens array 7, a light sensor 8, a light source 11, a transmission mask 12, and a motor 13. The system also includes a common optical lens 21 and an intensity separation plate 22. The intensity separation plate 22 is arranged to transmit a portion of the light beam from the transmission mask 12 toward the common optical lens 21, and to transmit a portion of the light beam from the common optical lens 21 toward the light sensor 8. Thus, the structured image is projected onto the surface 1 along the same optical axis as the optical axis along which the plenoptic image is acquired.

[0053] Figure 3 An example of a method for determining a three-dimensional profile of a surface according to the present invention is shown. By way of illustration, the method 30 is considered to use Figure 1The method 30 is implemented by the system 2 in FIG. The method 30 includes a step 31 of acquiring a sequence of N plenoptic images, a step 32 of projecting a sequence of N structured images, a step 33 of constructing intensity vectors, a step 34 of pixel matching, and a step 35 of reconstructing a three-dimensional profile of the surface. The steps 31 of acquiring the plenoptic images and 32 of projecting the structured images are synchronized. The step of acquiring the plenoptic images is performed by the plenoptic camera 3, and the step of projecting the structured images is performed by the illumination system 4. Each plenoptic image is acquired during the projection of the corresponding structured image. The step 33 of constructing the intensity vector is performed by the processing unit 5. As described above, the step of constructing the intensity vector includes forming a vector having a dimension of N for each pixel of the plenoptic image, the vector comprising different light intensities quantified by corresponding sensing elements of the light sensor 8. The pixel matching step 34 includes, for each pixel of a sub-image, searching for pixels of one or more other sub-images having the same or similar intensity vectors. Preferably, the search is performed in adjacent sub-images, i.e., in sub-images associated with microlenses 10 adjacent to the microlens associated with the sub-image under consideration. The three-dimensional profile reconstruction step 35 comprises determining distance information for each imaged surface element from the matched pixels corresponding to that imaged surface element. This distance information is determined along the optical axis of the plenoptic camera 3.

Claims

1. A system for determining a three-dimensional profile of a surface (1), said system comprising: a plenoptic camera (3) configured to acquire a sequence of plenoptic images of the surface (1), each plenoptic image being formed by a set of pixels each associated with a light intensity of an imaged surface element of the surface (1) and comprising a plurality of sub-images of the surface (1) observed from different viewing angles, an illumination system (4) configured to project a sequence of structured images onto the surface (1), the sequence of structured images being synchronized with the sequence of plenoptic images such that each imaged surface element of the surface (1) is illuminated by a series of light intensities that is different from the series of light intensities illuminating the other imaged surface elements of the surface (1), and - a processing unit (5) configured to construct, for each pixel of the plenoptic image, an intensity vector representing a series of light intensities of imaged surface elements of the surface (1) corresponding to the pixel in question, and to match each pixel of a sub-image with a pixel of another sub-image based on similarities between the intensity vectors of the pixels.

2. The system according to claim 1, wherein: The illumination system (4) is configured such that each structured image is formed by a set of projection patterns having a random light intensity distribution.

3. The system according to claim 1, wherein: The lighting system (4) comprises: a transmission mask (12) formed by a set of patterns (15), each of which has a transmittance capable of taking at least one of two different values, and - a light source (11) arranged to illuminate the surface (1) by projection through the transmission mask (12).

4. The system according to claim 3, wherein: The illumination system (4) further comprises a displacement device (13) configured to displace the transmission mask (12) and / or the light source (11) relative to the surface (1) in order to project the sequence of different structured images.

5. The system according to claim 4, wherein: The shifting device comprises: a support arranged to receive the transmissive mask, and - a positioning plate arranged to displace the support along at least one axis of rotation or at least one axis of translation.

6. The system according to claim 4, wherein: The pattern (15) of the transmission mask (12) has a first size along a first axis and a second size along a second axis, and the shifting device (13) is configured to shift the transmission mask (12) and / or the light source (11) between two consecutive structured images with a shift step size that is greater than or equal to the larger of the first size and the second size.

7. The system according to claim 3, wherein: Each pattern (15) of the transmission mask (12) includes liquid crystal, and the illumination system (4) further includes: a group of electrodes arranged on both sides of the pattern; and a control unit configured to supply power to the electrodes to individually control the transmittance of each pattern.

8. The system according to claim 3, wherein: The light source (11) is monochromatic.

9. The system according to claim 3, wherein: The light source (11) is incoherent.

10. The system according to any one of claims 1 to 9, wherein: The plenoptic camera (3) comprises: - a light sensor (8) comprising a set of sensing elements (9), and - a set of microlenses (7), each microlens (10) being associated with a subgroup of sensing elements (9) of the light sensor, such that each subgroup of sensing elements is capable of generating a sub-image.

11. The system according to claim 10, wherein: The processing unit (5) is configured to search, for each sequence of sub-images associated with a given microlens (10), and for each pixel of the sub-image under consideration, for the pixel having the most similar intensity vector from among the pixels of the sequence of sub-images associated with the adjacent microlenses of the given microlens.

12. The system according to any one of claims 1 to 9, wherein: The processing unit (5) is further configured to determine depth information of each imaged surface element based on the matched pixels associated with the imaged surface element.

13. A method for determining the three-dimensional profile of a surface (1), said method comprising the steps of: - acquiring (31) a sequence of plenoptic images, each plenoptic image being formed by a set of pixels each associated with a light intensity of an imaged surface element of the surface (1) and comprising a plurality of sub-images of the surface (1) observed from different viewing angles, - projecting (32) a sequence of structured images onto the surface (1), the sequence of structured images being synchronized with the sequence of plenoptic images such that each imaged surface element of the surface (1) is illuminated by a series of light intensities that is different from the series of light intensities illuminating other imaged surface elements of the surface (1), - constructing (33) for each pixel of the plenoptic image an intensity vector representing the series of light intensities of the imaged surface elements of the surface (1) corresponding to the pixel in question, and - matching each pixel of a sub-image with a pixel of another sub-image based on similarities between the intensity vectors of the pixels (34).

Images