Method for generating depth images and corresponding depth image calculation device

Description

[0001] The invention relates to a method for generating depth images, wherein images of a scene are recorded with at least two cameras, wherein depth information is derived from content correspondences between the images, wherein the scene is illuminated with structured light during the recording of the images.

[0002] Such methods are known, using structured light to enhance the detail of an illuminated object. This makes it easier to identify local image regions with corresponding content, allowing local depth information to be calculated with improved accuracy, especially for structureless areas of the scene.

[0003] Often, content correspondences between details of related images from the two cameras are used to determine a difference in viewing angles or a binocular disparity, from which local depth information about these details results.

[0004] The invention further relates to a depth image calculation device. Such a depth image calculation device can be used, for example, to implement the described methods.

[0005] EP 2 166 304 A1 discloses a lighting unit with a divergent coherent light source for generating an at least locally self-dissimilar pattern in a monitoring area (12). The lighting unit has an optical phase element in the beam path of the light source, which has unevennesses designed to generate local phase differences between adjacent light components incident on the phase element and thus generate the self-dissimilar pattern through interference.

[0006] Chen et al., Recovering Dense Stereo Depth Maps Using A Single Gaussian Blurred Structured Light Pattern, 2013 International Conference on Computer and Robot Vision, DOI 10.1109 / CRV.2013.22, describes a method for depth estimation using a stereo image pair with structured light.

[0007] Kim et al., Visual Measurement of a 3-D Plane Pose by a Cylindrical Structured Light, Proceedings of the 1993 IEEE / R SJ International Conference on Intelligent Robots and Systems Yokohama, Japan, Vol. 3, 26 July 1993, describes a method for measuring the depth and orientation of a local area of ​​an object surface by projecting a cylindrical structured light onto the object surface using a laser.

[0008] The invention is based on the object of increasing the accuracy of the calculation of local depth information.

[0009] To achieve the stated object, the invention provides the features of claim 1. According to the invention, to achieve the stated object, a method of the type described above proposes smoothing contrast gradients of the structured light. This smoothing can easily reduce or even eliminate homogeneous intensity ranges in the structured light. This makes it easy to prevent noise from introducing significant artifacts that lead to miscalculations of the depth information.

[0010] The invention has thus recognized that a departure from bivalent or oligovalent illumination patterns is beneficial for the accuracy of the calculation of depth information.

[0011] According to the invention, it is further provided that a local smoothing width in at least one partial area is greater than a characteristic length of diffraction effects and / or aberrations of a projection of the structured light. Thus, it is also detectable on the object that the structured light exhibits clearly visible transitions across a smoothing width at intensity changes or contrasts. Thus, a quasi-continuous intensity modulation can be achieved in areas of strong intensity changes or contrasts, which can be used, for example, to prevent the occurrence of artifacts. This can have a positive effect on the result of a filtering process, for example, using a census transformation.

[0012] A smoothing width can, for example, be described as a characteristic length over which a change in intensity of an image contrast takes place.

[0013] A depth image can, for example, be characterized as a data field with location-dependent distance information, in particular for each entry in the data field. The data field preferably also includes further image information, in particular brightness and / or color values. For example, the content-related correspondences can be characterized and / or calculated as stereoscopic disparities, in particular for pixels of the images, or by artificial neural networks trained according to a stereoscopic principle, or generally as correspondences resulting from the assumption that both images show the same scene or object from different viewing angles or poses. These correspondences can, for example, be calculated explicitly or determined or processed implicitly to calculate the depth information.

[0014] In one embodiment, the structured light can have a pattern in at least one partial area, using different, connected pattern elements. The invention thus provides at least one additional degree of freedom, for example, in the sense of variability of the properties of the pattern elements, in the pattern design, whereby local self-similarity of the pattern can be reduced or avoided.

[0015] For example, a connected pattern element can be characterized as all points in the pattern that lie above or below an intensity threshold and that can be connected to each other by a path that only leads via points in the pattern element.

[0016] For example, the pattern elements can be configured to differ in size. This describes an easily variable parameter for an optimization method that minimizes the self-similarity of certain pattern environments, for example, in the context of a stereoscopic correspondence search.

[0017] For example, alternatively or additionally, the pattern elements can be provided with respect to a shape. Thus, for example, a discrete set of different shape classes can be used to reduce or avoid the similarity of certain pattern environments. However, it can also be provided that the shape can be varied by a continuous parameter, for example, by an angle between two details. This can open up additional possibilities for reducing remaining local self-similarities.

[0018] For example, alternatively or additionally, the pattern elements can be provided with different smoothing widths. This makes it easy to eliminate very local self-similarities in homogeneous image areas.

[0019] In one embodiment, the structured light can have an irregular pattern. This describes a simple means of generating a pattern that is as unself-similar as possible. Low self-similarity, for example, in the context of a stereoscopic correspondence search for image environments to be compared, preferably after rectification, is advantageous if mismatches of image content are to be avoided when calculating disparities.

[0020] For example, the pattern can be designed irregularly with respect to the position of the pattern elements. The invention has thus recognized that a deliberate deviation from a regular position, for example, a deliberate deviation from grid points of a regular grid, introduces at least one additional degree of freedom, which makes it possible to reduce local self-similarity of the pattern.

[0021] For example, it can also be provided that the pattern is irregular with respect to the size of the pattern elements. A size, for example, a largest dimension or a dimension in a specific direction, can denote a degree of freedom whose variation from pattern element to pattern element can be used to reduce self-similarity.

[0022] For example, it can also be provided that the pattern is irregular with respect to the shape of the pattern elements. Thus, irregular arrangements of pattern elements are to be formed—for example, at regularly arranged positions. This provides a further means of reducing self-similarity, particularly in a search direction for correspondence searches.

[0023] For example, the pattern can be irregularly formed with respect to the smoothing width of the pattern elements. The invention has recognized that an increased smoothing width has a positive effect on reducing mismatches that can arise in nearly homogeneous image components due to the algorithms used, for example, due to the information reduction associated with a census transformation.

[0024] This effect can be exploited by varying the smoothing width to avoid unwanted content correspondences.

[0025] In this case, it can be provided that a preferably local dissimilarity measure on the pattern essentially assumes values ​​above a threshold value other than zero. Dissimilarity measures can be an easily manageable means for generating a structure that is as unself-similar as possible and / or for verifying a low degree of self-similarity in a given structure.

[0026] The measure for determining the pairwise dissimilarity of image environments to generate an optimal illumination pattern can, for example, correspond exactly to the dissimilarity measure used for the stereoscopic correspondence search. A Hamming distance of a preferably two-valued census transformation of image environment pairs is a suitable measure here. This measure is characterized by low memory requirements and computational effort, while resulting in relatively low information loss in view of stereoscopic requirements. Especially with regard to the census transformation, it has proven advantageous to use smoothed intensity curves, as this allows noise artifacts to be suppressed.

[0027] Alternatively or additionally, it can be provided that a local irregularity of the pattern can be described by at least one characteristic parameter that can assume a multitude of values. Thus, a parameter space can be defined with which—for example, starting from a regular arrangement or an arrangement randomly modified from a regular arrangement—an optimization process can be carried out in order to achieve the lowest possible self-similarity of the pattern. A particularly favorable starting point for the optimization process can be selected, for example, by a starting configuration with the highest possible degree of dissimilarity due to a uniform distribution of the pattern element properties as states in the parameter space.

[0028] A list of examples of such a characteristic parameter may include, for example, a position and / or a size and / or a shape and / or a smoothing width of a pattern element or a sub-element of a pattern element.

[0029] In an advantageous embodiment, the illumination pattern can be composed of congruent basic cells with periodic boundary conditions. This allows for the simple assembly of larger patterns.

[0030] A stereoscopic measurement system typically has a defined depth range and, associated with it, a disparity range—for example, measured in pixels—which it must be capable of resolving. This disparity range can essentially define the width of the basic cell as the smallest possible unit within the illumination pattern. Within this width (or longitudinal edge length), the self-similarity of pattern neighborhoods should be minimal along the stereoscopic correspondence search direction. Perpendicular to the stereoscopic correspondence search direction, however, the height of the basic cell (or transverse edge length) should preferably correspond at least to the size of the pattern neighborhoods used for the stereoscopic correspondence search (matching block size). This provides an example of the minimum size of congruent rectangular basic cells with periodic boundary conditions from which the illumination pattern can be composed in a tiled manner.

[0031] By choosing the dimensions of the base cell in this way, it is ensured that corresponding pixels are always searched for within the base cell dimensions, and that repetitions of pattern environments outside the base cell, especially perpendicular to a search direction, do not negatively influence the discovery of truly corresponding details. Furthermore, by limiting the pattern calculation to the minimum size of the base cell detailed here, the computational effort is kept to a minimum.

[0032] The stereoscopic correspondence search direction of the data acquisition system, i.e., the epipolar direction of the rectified stereo images, is not necessarily identical to the direction along which the self-similarity of the pattern surroundings of the illumination pattern is minimized. Manufacturing tolerances in the manufacture of the light source and the image acquisition system, as well as optical aberrations, can lead to deviations between these two directions, which can cause an angular mismatch between the illumination pattern and the data acquisition system. The invention recognizes that such an angular mismatch can be taken into account when calculating an illumination pattern optimized for minimal self-similarity.This can be achieved, for example, by minimizing the pairwise self-similarity of pattern environments not only exactly along the supposed stereoscopic correspondence search direction, but also within a cone around this search direction with an opening angle corresponding to the maximum expected angular mismatch (cf. ). Fig. 5 ).

[0033] In an advantageous embodiment, the pattern, in particular each pattern element, can be composed of sub-elements that have a uniform parameter set for the entire pattern to fully describe them. The optimization of an illumination pattern suitable for stereoscopic image processing associated with the invention is thus achieved by successively maximizing the pairwise diversity of pattern environments of all sub-element parameter spaces of the pattern. In this case, a local optimum is sought in the union of the parameter spaces of all sub-elements.

[0034] With already comparatively low-dimensional (e.g. three-dimensional) sub-element parameter spaces, it is possible to generate very high dissimilarity measures with the method according to the invention compared to the prior art.

[0035] In general, it can thus be said for many embodiments that the light intensity distribution in the base cell of the structured illumination pattern results from the maximization of a metric for measuring the pairwise dissimilarity of all possible pattern environments of defined size along the stereoscopic correspondence search direction.

[0036] Here, the mentioned metric for measuring the pairwise dissimilarity of all possible pattern environments of defined size along the stereoscopic correspondence search direction in the base cell can be calculated depending on the position of all pattern or sub-elements superimposing these pattern environments as well as their further properties (e.g. size, smoothing width or shape) that span the said sub-element parameter space.

[0037] This can enable the aforementioned metric to be maximized in the sequential traversal of so-called unit cells corresponding to the sub-elements, which structure the basic cell into uniform sub-areas, while varying all properties of the sub-elements within a defined parameter space. For example, all pattern environments can be considered that lie within the range of substantial intensity changes due to changes in the position, size, or shape parameters of the respective pattern or sub-elements.

[0038] It can be provided that the mentioned metric for describing the pairwise dissimilarity of pattern environments of defined size is structured in suitable distributions for each pattern environment, wherein one distribution captures the pairwise dissimilarity of all pattern environments within a very small local area (a few, e.g. three, pixels of the data acquisition system) along the stereoscopic correspondence search direction around a respective pattern environment of interest (local distribution), while another distribution captures the pairwise dissimilarity of all further pattern environments lying along the stereoscopic correspondence search direction around a pattern environment of interest (epiglobal distribution).

[0039] This may result in the local and / or epiglobal distributions of the mentioned metric over all pattern environments of a base cell being summarized into a suitably aggregated scalar metric, the maximum of which characterizes an optimal illumination pattern.

[0040] The aforementioned metric can be represented, for example, by a Hamming distance of census transforms of sample neighborhoods. Sample neighborhoods can be understood and / or described, for example, as the preferably square pixel blocks of the data acquisition system or captured image that frame a pixel of interest.

[0041] The mentioned aggregated scalar metric can be represented, for example, by a sum of the epiglobal distribution minima of all sample environments of the base cell that are different from 0 (these - as mentioned - are represented, for example, by preferably square pixel blocks of the data acquisition system) multiplied by the sum of the local distribution minima of all sample environments of the base cell that are different from 0 or 1 or 2.

[0042] It can further be provided that, when calculating said local and epiglobal distributions of said metric, not only sample environments represented by discrete pixel blocks of the data acquisition system are used, but also their interpolations, wherein sample environments are interpolated in the form of preferably square pixel blocks, preferably on a subgrid of 2x2 or 4x4 subpixels. The already known consideration of subpixel shifts between the illumination pattern and the data acquisition system can be advantageously used for optimizing the illumination pattern.

[0043] When calculating the aforementioned epiglobal distribution of the metric described above, a possible angular mismatch between the supposed stereoscopic correspondence search direction, based on the design of the illumination pattern, and the true correspondence search direction of the data acquisition system can be taken into account. For example, all pattern neighborhoods within a bounding cone around the correspondence search direction, such as the aforementioned one, can be considered, each starting from a pattern neighborhood of interest.

[0044] It may further be provided that when calculating the above-mentioned distributions for the described metric, the blur of the optical image of both the illumination pattern on its projection surface and its images in the data acquisition system (cameras) is taken into account - for example by convolving the illumination pattern with a two-dimensional Gaussian function of characteristic width.

[0045] For further improvement, when calculating the metrics for pattern optimization described above, it can be provided that when calculating the Hamming distance of the census transform of two pattern elements or sub-elements, only those pixels contribute to the Hamming distance whose intensity differences stand out significantly from the noise of the data acquisition system (cameras).

[0046] In an advantageous embodiment, it can be provided that the pattern is composed of sub-elements, for example the sub-elements already mentioned, wherein a basic cell, in particular the basic cell already mentioned, can be subdivided into equally sized, preferably square, unit cells, so that each unit cell is assigned at least one sub-element, in particular exactly one sub-element or a constant number of sub-elements.Thus, an irregular arrangement of pattern elements and / or sub-elements can be easily traced back to a regular arrangement in which the irregularity appears as a deviation from the regular arrangement, for example with regard to a position of the pattern elements and / or sub-elements in the associated unit cell, a size of the pattern element and / or the sub-element, an orientation of the pattern element and / or the sub-element relative to the unit cell, a shape of the pattern element and / or the sub-element and / or a smoothing width of the pattern element and / or the sub-element.

[0047] The use of sub-elements from which the pattern elements are composed can help to reduce the complexity of the pattern and thus reduce the computational effort required to minimize self-similarity.

[0048] For example, it can be provided that an assignment of sub-elements to unit cells is determined by the position of an intensity extreme value of the sub-element within the unit cell assigned to it. Thus, a unique assignment can be easily defined and / or verified, especially if each pattern element or each sub-element has exactly one intensity extreme value (maximum for light pattern elements, minimum for dark pattern elements).

[0049] In an advantageous embodiment, the pattern can be provided with rotationally symmetrical or elliptical, connected sub-elements. Such sub-elements can be produced particularly easily in almost any desired configuration, particularly with diffractive optical elements.

[0050] For example, it can be provided that the pattern elements or the sub-elements each have the shape of two-dimensional Gaussian light intensity distributions, with the optimization parameters of the horizontal and vertical center of gravity position as well as the distribution width.

[0051] In this case, the subelements can be designed with different sizes and / or distribution or smoothing widths. It has been shown that this – combined with a freely selectable position, for example, in the unit cell – creates sufficient freedom to achieve a highly locally diverse pattern.

[0052] In general, any projector capable of projecting a desired pattern onto an object as an illumination pattern can be used. For example, the pattern could be created using mirror arrays and / or transilluminated masks and / or arrays of miniaturized light elements.

[0053] In an advantageous embodiment, the structured light can be generated using a diffractive optical element. It has been found that suitable patterns can be easily generated in this way.

[0054] In this case, or in general, it can be provided that overlapping pattern elements or sub-elements form interference. This interference pattern can create an additional structure that can further reduce self-similarity.

[0055] In an advantageous embodiment, the light intensity maxima of all sub-elements can be arranged within a predetermined intensity band. This allows the dynamic range of the cameras to be optimally utilized.

[0056] It can thus be provided that the light intensity amplitude of each smoothed pattern element or sub-element utilizes as large a part as possible of the dynamics of the data acquisition system, in particular so that local light intensity minima are as small as possible and local light intensity maxima are characterized by an amplitude as homogeneous as possible over the pattern area.

[0057] It can be provided that the intensity band, for example as a band of permissible maximum intensities, has a width which is less than 25%, preferably less than 10%, of an average maximum intensity of all sub-elements.

[0058] It has been shown that the narrower the range of permissible maximum intensities, the better the utilization of the cameras' dynamic range. Ideally, the light intensity maxima are equally strong.

[0059] In order to arrive at an optimization result as quickly as possible, it can be provided that a parameter space of properties of all pattern elements arranged along one, in particular the mentioned, correspondence search direction is populated in a state density that is as homogeneous as possible in a step preceding the optimization.

[0060] It can be provided here or generally that the sizes of the smoothed pattern elements or sub-elements permissible during the optimization lie within limits suitable for the said metric, in particular wherein the smallest possible pattern element sizes or sub-element sizes are selected to be sufficiently large and the largest possible pattern element sizes or sub-element sizes are selected to be sufficiently small in order to prevent light intensity profiles within the illumination pattern that are too homogeneous and therefore disadvantageous for the said metric.

[0061] It can also be provided that a typical shape of optical interference effects when smoothed pattern elements or sub-elements are superimposed is used to maximize the said metric, whereby the characteristics of optical interference of different pattern elements can be empirically anticipated when simulating the illumination pattern.

[0062] Alternatively or additionally, the features of the independent claim directed to a depth image calculation device are provided according to the invention to achieve the stated object. In particular, to achieve the stated object, the invention proposes that a depth image calculation device be configured to implement a method according to the invention, in particular as described above and / or according to one of the claims directed to a method. Thus, a device is provided that allows application of the method according to the invention, in particular as described above and / or according to one of the claims directed to a method.

[0063] In general, the pattern elements and / or the sub-elements can, for example, define bright or dark areas of the structured light.

[0064] The invention will now be described in more detail using an exemplary embodiment, but is not limited to the exemplary embodiment. Further exemplary embodiments arise from combining the features of one or more claims with one another and / or with one or more features of the exemplary embodiment.

[0065] It shows: Fig. 1 a highly simplified schematic drawing of a depth image calculation device according to the invention, Fig. 2 a basic cell of a pattern of structured light for use in a method according to the invention, Fig. 3 a pattern element with two overlapping sub-elements of a pattern according to Fig. 2 in a highly simplified representation, Fig. 4 a subdivision of a basic cell into unit cells, Fig. 5 a basic cell with a stereoscopic correspondence search direction and Fig. 6 a flow chart of a generation of a pattern for use in a method according to the invention.

[0066] Fig. 1 shows a depth image calculation device designated as a whole by 1.

[0067] The depth image calculation device 1 has a projector 2 with which structured light 3 is projected onto an object or a scene 4.

[0068] The depth image calculation device 1 further has two cameras 5, 6, which are aligned parallel to each other at a predetermined distance.

[0069] Each of the cameras 5, 6 takes a stereoscopic image of the scene.

[0070] In a processing unit 7, depth information is extracted from apparently different representations of corresponding image components in a conventional manner. This results in a depth image with location-dependent depth information.

[0071] Figure 2 shows an example of a periodic basic cell of pattern 8, from which the illumination pattern is composed in a tiled manner.

[0072] Shown is a basis cell 9 which has periodic boundary conditions such that basis cells can be smoothly arranged in a way that ensures minimal self-similarity of pattern environments along the correspondence search direction even across basis cell boundaries.

[0073] In the base cell 9, pattern elements 10 are visible in the form of complex bright spots composed of rotationally symmetric sub-elements 11. The pattern elements 10 are each formed in a connected manner.

[0074] In Figure 2 a pattern element 10 is designated, which is composed of a plurality of sub-elements 11, whereby only a few sub-elements 11 are designated here.

[0075] The rotationally symmetric sub-elements 11 do not have a hard boundary, but each have a transition to the environment with a smoothing width that is significantly above a value that typically results from diffraction or alternative sources of blurred imaging.

[0076] Figure 3 shows an example of a pattern element 10 which is composed of two rotationally symmetrical sub-elements 11.

[0077] The sub-elements 11 of pattern 8 in Figure 2 differ from each other in their size and distribution width as well as their position within the unit cells assigned to them.

[0078] Figure 4 shows a subdivision of the base cell 9 from Figure 2 into square, equally sized unit cells 12 ("unit cells"). In the example, each unit cell 12 is composed of nine pixels 13.

[0079] Each unit cell 12 has exactly one subelement 11 whose intensity maximum lies in the unit cell 12, and in this sense each subelement 11 belongs to a unit cell 12.

[0080] The unit cells 12 can define a regular pattern (not shown) in which, for example, a uniform sub-element 11 is located in the center of a unit cell 12.

[0081] The pattern 8 from Figure 2 In contrast, it has 12 unit cells whose subelements 11 are different from the described regular arrangement. In the example, there are differences in size, smoothing width, and position within the unit cell 12. This creates an irregular pattern. These parameters span a parameter space in which the subelements 11 can be described.

[0082] The arrangement of the cameras 5, 6 relative to each other provides an epipolar direction 14, which corresponds to a correspondence search direction 15, along which corresponding image contents are searched (cf. Fig. 5 ).

[0083] The base cell 9 thus has a longitudinal edge length 16 along the epipolar direction, which is determined by a disparity range of an image recording and data acquisition system comprising the cameras 5, 6, here given by the depth image calculation device 1.

[0084] Transverse to this direction, the base cell 9 has a significantly smaller transverse edge length 17, which is given by a size of a pattern environment used for the stereoscopic correspondence search (matching block size).

[0085] The projector 2 has, for example, a diffractive optical element (not shown in detail) with which the rotationally symmetrical sub-elements 11 can be displayed as a diffraction image.

[0086] The light maxima of the sub-elements 11 are scaled so that all sub-elements 11 have almost the same maximum value.

[0087] Figure 5 shows the base cell 9 with a pixel 13, for which a corresponding image content is to be searched in the other image of the stereoscopic recording with the cameras 5, 6.

[0088] The correspondence search direction 15 is shown here, which approximately corresponds to the epipolar direction 14, whereby deviations are permitted to take manufacturing tolerances into account.

[0089] Figure 2 shows, by way of example, an overlap area 19 between sub-elements 11 in which interferences 20 develop. These interferences 20 contribute to the individualization of the sub-elements 11 and further reduce the self-similarity.

[0090] Figure 6 shows a schematic flow chart for generating a pattern 8. 1. Division of the above-mentioned (preferably rectangular) area to be illuminated in a structured manner into identical rectangular basic cells 9, a. each of which has at least one pixel width corresponding to the maximum permissible disparity range in the respective stereoscopic image recording system, here the depth image calculation device 1, plus the maximum width of the pixel blocks to be compared, b. and each of which has at least one pixel height corresponding to the maximum height of the pixel blocks to be compared. 2. Division of each basic cell 9 into an integer number of preferably square so-called unit cells 12 with an edge length corresponding to an integer number α of pixels 13.This discretization affects the size of the basic cell 9, as it enforces a pixel width m·α corresponding to an integer multiple m of the unit cell length α and a pixel height n·α corresponding to an integer multiple n of the unit cell length α. The basic cell 9 thus consists of a matrix-like arrangement of . n·mUnit cells 12. The unit cell length α is preferably 2, 3, or 4 pixels. 3. Assignment of one (or more) elementary excitations (preferably in the form of a Gaussian illumination intensity distribution) to each unit cell 12, characterized by a center of gravity within the unit cell area and by a width (smoothing width) – preferably in the form of a standard width σ of the Gaussian illumination intensity distribution. Thus, each excitation has three degrees of freedom, namely the vertical and horizontal center of gravity position of the light excitation within the unit cell 12, as well as the width of the excitation, which should range within a value range suitable for maximum dissimilarity measures of pixel blocks to be compared.Light intensities of neighboring unit cells 12 can overlap and (particularly when using an illumination device consisting of a coherent light source and a diffractive optical element) interfere with each other, thus forming a pattern element 10. The elementary excitations thus correspond to the sub-elements of a pattern element. a. Implementation of periodic boundary conditions at the edges of the basic cell 9. This is done by considering the influence of the light intensity distribution of neighboring base cells 9 that are identical to the base cell 9 of interest. 4. Determination of an initial configuration of the center of gravity positions and the widths of all light excitations in their associated unit cells 12 within the base cell 9. This initial configuration should already increase the degree of dissimilarity of pixel blocks (pattern environments) to be compared in a suitable manner.This is done by exploiting the fact that the greatest possible variability of both the vertical center of gravity position and the width of the light excitations of all unit cells 12 arranged in an epipolar direction 14 (horizontally) favors the degree of dissimilarity. All three degrees of freedom of each light excitation are initially determined randomly. However, when determining the vertical center of gravity position and the width, there is a reservation as to a maximum density that must not be exceeded: This means that if a vertical center of gravity or a width of a light excitation is critically close (i.e. below a defined threshold value) to an already existing vertical center of gravity or an already existing width of any already positioned light excitation of a unit cell 12 of the same height within the base cell 9, this value is discarded and determined again randomly.If the space of permissible parameters for a light excitation is already so densely populated that no further suitable value can be found with the threshold value of the respective maximum permissible density, the maximum permissible density is increased until optimal initial parameters are assigned to all light excitations of the unit cells 12 of a height within the base cell 9. 5. Normalization of the maximum intensities of all light excitations to an approximately equal target value. This is done iteratively, since the intensities of directly or even indirectly neighboring unit cells 12 can overlap. The amplitude of each excitation is multiplied by the ratio of the desired maximum intensity to the actual maximum intensity. 6. Application of a smearing filter (e.g.Gaussian) on the illumination structure to represent the characteristic image sharpness of the illumination structure in the object space as well as the structured illuminated object space in the image recording system. 7. Determination of suitable limits of permissible light spot widths based on the following criteria: a. If there are areas of negligible light intensity in cross-sectional profiles through the base cell 9 that extend over several pixels 13, the smallest permissible width chosen is too small. This makes image areas with insufficient diversity from the image environments to be compared likely. Expressed in distribution widths of a Gaussian function, a value of . σ ≥ 0.15 αproved to be a suitable minimum value. b. If the ratio of the maximum light intensity to a fictitious homogeneous light intensity under homogeneous illumination with the same integral light output falls below a factor of 1.8, the largest permissible width is too large. This would result in the available light output not being used optimally for contrast, and very wide light spots would be created with relatively little variation in the image surroundings within them. Expressed in distribution widths of a Gaussian function, a value of σ ≤ 0,6· α proved to be a suitable maximum value. 8. Optimization of the normalized initial light intensity distribution thus obtained by stepwise maximization of a scalar dissimilarity measure in the space of the three ogDegrees of freedom of each light excitation within the basic cell 9. All unit cells 12 are successively traversed, and within each unit cell 12 number i the following procedure is followed: a. Calculation of the light intensity distribution of the entire basic cell 9 for a new, suitably varied parameter set, unless the initial illumination structure is included. b. Normalization of the maximum intensities and application of a smearing filter (e.g. Gaussian) to impose a smearing measure typical for the device according to steps 0 and 0. c. Calculation of distributions of the dissimilarity measures for all pixels 13 in the sphere of influence of the light excitation belonging to the respective unit cell 12 number i. The sphere of influence includes all pixels 13 of the basic cell 9 in which the light excitation makes a non-negligible intensity contribution.These do not necessarily have to be pixels 13 of the respective unit cell 12, but can also be pixels of the directly or indirectly neighboring unit cells 12. i. Two distributions of disparity measures are calculated in each case, which are derived from two optimization objectives: The first optimization objective is to achieve the greatest possible disparity between closely adjacent pixel blocks to be compared (local). The second optimization objective is to achieve the highest possible minimum disparity between all pixel blocks to be compared along an epipolar line within the permissible disparity range (epiglobal). ii. The distribution of disparity measures per pixel 13 results from the comparison of a pixel 13 of interest and its surroundings (its surrounding pixel block) with all closely adjacent (local) or neighboring (epiglobal) pixels 13 and their surroundings along the epipolar direction 14.Thus, each pixel 13 has a local and an epiglobal distribution of dissimilarity measures. iii. The distribution of dissimilarity measures per pixel 13 includes all so-called subpixel configurations. This is intended to account for the fact that the relative position between the illumination structure and the pixel grid of the image acquisition system does not have a precision much smaller than the pixel dimension. The pixel grid discretizes an initially continuous lateral intensity distribution both in its lateral resolution and in the form of integer intensity values. Thus, when maximizing dissimilarity, all possible relative positions between the illumination structure and the pixel grid must be taken into account, which greatly increases the number of dissimilarity measures to be compared. Here, the area of ​​each pixel 13 and its surroundings up to half of all its neighbors is divided into . k 2< possible subpixel layers (preferably with k=4)underdiscretized. The distribution of local dissimilarity measures then compares the k 2< subpixel blocks 18 belonging to the pixel 13 of interest with the associated nearest, next-next, and next-next-next neighbors in both directions, from which 3·2· k 2< Dissimilarity measures are contained in the local distribution. The distribution of epiglobal dissimilarity measures, on the other hand, compares all k 2< subpixel blocks 18 belonging to the pixel 13 of interest with all subpixel blocks lying in the permissible disparity range along the (horizontal) epipolar direction 14. Taking into account an angular mismatch between the epipolar direction 14 of the illumination structure and the epipolar direction 14 of the rectified stereo images, which also has to be anchored here, the number of disparity measures in the epiglobal distribution at a mismatch angle β and a permissible disparity range of D pixels with( D 2< k 4< tan β ) / 2be estimated. iv. The contribution of so-called weak bits to the dissimilarity measure, i.e. those pixels 13 within a pixel block with only a very small deviation (in the range of the image noise) from the (central) pixel 13 of interest, is eliminated by a constraint. d. Calculation of an aggregated scalar dissimilarity measure which characterizes the dissimilarity of all pixel blocks to be compared with one another within the entire basic cell 9. i. An advantageous embodiment of the aggregated scalar dissimilarity measure is, for example, the product of the sum of all non-zero minima of the epiglobal dissimilarity distributions (of all pixels) within the basic cell 9 with the sum of all minima of the local dissimilarity distributions (of all pixels) within the entire basic cell 9 that exceed zero (or one, or two). e.Maximizing the aggregated scalar dissimilarity measure for each unit cell 12 in the space of the three free parameters of its associated light excitation (center of gravity positions and width). i. This always determines a local optimum in a three-dimensional subspace of the 3 n m dimensional parameter space of the basic cell optimization problem. ii. An advantageous embodiment consists in calculating the aggregated scalar dissimilarity measure initially on a sufficiently finely discretized three-dimensional grid in the space of the free optimization parameters, followed by multiple gradient-based approximations to the point of maximum dissimilarity in the parameter space, each starting from the (for example, three) discrete grid points of highest dissimilarity. 9.Verification of the optimization dynamics by before-and-after comparison of the aggregated scalar dissimilarity measure of the entire basic cell 9, and depending on this, decision on a further run 0. [optional] 10. After identifying a local optimal state achieved by performing all previous process steps, all process steps can be run through while varying the unit cell length α in order to maximize the aggregated scalar dissimilarity measure of the basic cell also as a function of the unit cell length α. [optional] .

[0091] In the method for calculating a depth image, it is thus proposed to use structured light 3 with a smoothed and / or irregular pattern 8 and / or a pattern composed of different pattern elements 10. List of reference symbols

[0092] 1Depth image calculation device 2Projector 3Structured light 4Scene 5, 6Camera 7Calculation unit 8Pattern 9Basic cell 10Pattern element 11Subelement 12Unit cell 13Pixel 14Epipolar direction 15Correspondence search direction 16Longitudinal edge length 17Transverse edge length 18Subpixel block 19Overlap area 20Interference

Claims

1. Method for generating depth images, wherein images of a scene (4) are recorded using at least two cameras (5, 6), wherein depth information is derived from content correspondences between the images, wherein the scene (4) is illuminated with structured light (3) during the recording of the images, wherein contrast gradients of the structured light (3) are smoothed, characterized in that a local smoothing width in at least one subregion is greater than a characteristic length of diffraction effects and / or imaging errors of a projection of the structured light (3).

2. Method for generating depth images according to claim 1, characterized in that the structured light (3) has, at least in a subregion, a pattern (8) in which different contiguous, preferably light or dark, pattern elements (10), in particular contiguous pattern elements (10) of different size and / or shape and / or smoothing width, are used.

3. Method for generating depth images according to claim 1 or 2, characterized in that the structured light (3) has an irregular pattern (8), in particular with regard to a position and / or a size and / or a shape and / or a smoothing width of contiguous, preferably light or dark, pattern elements (10), in particular wherein a preferably local measure of dissimilarity on the pattern assumes substantially values above a threshold value different from zero and / or wherein a local irregularity of the pattern (8) can be described by at least one characteristic parameter which can assume a plurality of values.

4. Method according to one of the preceding claims, characterized in that the pattern (8) is composed of congruent, preferably rectangular, base cells (9) which are aligned along an epipolar direction (14) distinguished by the cameras (5, 6) and have periodic boundary conditions, in particular wherein a longitudinal edge length (16) of the base cell (9) along a stereoscopic correspondence search direction (15) corresponds to at least one disparity range of an image recording and data capture system comprising the cameras (5, 6) and / or wherein a transverse edge length (17) of the base cell (9) corresponds to at least one size of pattern environments used for a stereoscopic correspondence search.

5. Method according to one of the preceding claims, characterized in that the pattern (8), in particular each pattern element (10), is composed of sub-elements (11) which can be parameterized by means of a parameter space whose dimension is smaller than a number of sub-elements, in particular smaller than a number of pattern elements (10) of the base cell, or smaller than ten.

6. Method according to one of the preceding claims, characterized in that the pattern (8) is composed of sub-elements (11), wherein the or a base cell (9) can be subdivided into equally sized elementary cells (12) so that at least one sub-element (11), in particular exactly one sub-element (11) or a constant number of sub-elements (11), is associated with each elementary cell (12), in particular by a position of an intensity extreme value of the sub-element (11).

7. Method according to one of the preceding claims, characterized in that the pattern (8) has rotationally symmetrical or elliptical contiguous, preferably light or dark, sub-elements (11), in particular sub-elements (11) with different sizes and / or smoothing widths.

8. Method according to one of the preceding claims, characterized in that the structured light (3) is generated using a diffractive optical element, in particular wherein superimposed pattern elements (10) or sub-elements (11) can form interferences (20).

9. Method according to one of the preceding claims, characterized in that light intensity maxima of all sub-elements (11) lie in a predetermined intensity band, in particular with a width which is less than 25%, preferably less than 10%, of an average maximum intensity of the sub-elements.

10. Depth image calculation device (1) having means for carrying out a method according to one of the preceding claims.