Stereo Camera System

Abstract

The present invention relates to a stereo camera system for measuring three-dimensional surface shapes, which may be used to obtain depth information from an object moving along a conveyor belt in front of the stereo camera system, which may form part of a production monitoring system.

Description

[Technical field]

[0001] The present invention relates to a stereo camera system for the measurement of three-dimensional surface shapes. This may be used to obtain depth information from objects moving along a conveyor belt in front of the stereo camera system. The stereo camera system may form part of a production monitoring system.

[0002] Production monitoring systems comprising conveyor systems and cameras for optically scanning products transported by the conveyor system are known and are used, for example, in package sorting facilities and production lines.

[0003] Typically, the height of the product or object to be scanned is determined by a measuring device, e.g. a light curtain. The camera has an adjustable lens, which is set to a pre-determined height of the object to be scanned in order to correctly scan the surface of the object. The camera may have a zoom lens with a variable focal length so that it can be set appropriately to scan objects of different heights. It is also possible to use a lens with a fixed focal length, the focus of which can be tracked to ensure that objects of different heights are sharply depicted on the sensor.

[0004] In the case of a zoom lens, multiple lenses are moved relative to each other. A zoom lens has a movable mechanism inside the lens that presses multiple lenses against each other. In the case of a zoom lens, the scale can be kept constant along the adjustment path.

[0005] When using a lens with a fixed focal length, the entire lens is moved along the optical axis so that the area of ​​sharpness is located at the desired height. As the optical path length decreases, the imaging scale increases.

[0006] Both zoom lenses and fixed focal length lenses require moving mechanical parts. This movement leads to wear. Industrial applications require millions of cycles of use, which is usually far beyond the lifespan of these moving parts.

[0007] Where a product surveillance system requires the evaluation of both optical and height information, various methods have been established.

[0008] One possibility is the so-called light section method, in which a thin, straight strip of light is shone on the object to be measured and recorded by an area scan camera. The distortion caused by the object in the strip of light may be used to estimate the surface of the object.

[0009] Stereoscopic vision offers a further option: a camera system can record an object from two different viewing directions and determine the topological surface data of the object from the corresponding pixel offsets. However, the disadvantage of this method is that it is limited to one focal plane.

[0010] Such a stereo camera system may comprise two camera modules with which the same object area may be scanned from two different viewing directions. The camera modules are equipped with lenses for imaging the object area onto the sensors of the respective camera modules. The optical axis of the lens is the axis of symmetry of the lens, which axis runs approximately transverse to the surface of the lens or lenses contained in the lens. The object-facing principal surface of the lens is called the lens surface. The sensor surface is the plane on which the face of the light-sensitive sensor lies. The object surface is the surface that is sharply imaged by the lens onto the face of the sensor.

[0011] In a conventional stereo camera system, the lens plane, the object plane, and the sensor plane are arranged parallel to each other. In the following, the parallel alignment of the object plane, the lens plane, and the sensor plane is referred to as a "parallel arrangement."

[0012] What is important for a stereo camera system to resolve a three-dimensional surface shape is the base angle (vertex angle, Basis-Winkel) α, which is the angle that the viewing directions of the two camera modules of the stereo camera system occupy on the target area. The base angle α is basically set by the size of the base (base) B, which is the distance between the faces of the two sensors of the two camera modules S1 and S2. In other words, the larger the base B, the larger the base angle α, and the higher the resolution of the stereo camera system in the height direction (Figure 8).

[0013] In the following, unless otherwise specified, it is assumed that the mutual distances between the sensor surface SP, the lens surface LP, and the object surface OP are constant, and only the base B determines the base angle α.

[0014] A lens has an aperture angle γ that gives the maximum angle of incidence of the incoming light with respect to the optical axis, within which an object may be imaged substantially without distortion (Figure 8). Depending on the lens design, the maximum aperture angle of the lens may vary. In the following, the beam passing through the lens is described as the "beam path". The beam path includes all beams within the aperture angle and extends between the face of the sensor and the object.

[0015] However, in the parallel stereo camera system described above, the base may not be as large as desired because if the base is too large, the portions enclosed by the incidence angles will not overlap or will otherwise only slightly overlap in the object region, resulting in a sharply imaged portion of the object region being small or no longer present.

[0016] Only the overlap region, i.e. the region of the object that can be captured simultaneously by both sensors, can be used for the evaluation of the three-dimensional surface shape. The edge region of this overlap region is imaged with a large lens aperture angle γ. Therefore, the distortion is greatest at the edge region of the overlap region. In addition, the imaging performance, which may be characterized inter alia by color defects and the modulation transfer function (MTF), is worse at the edge region of the overlap region than in the center of the overlap region. Color defects describe the loss of color imaging. The MTF describes the reduction in contrast of details.

[0017] "Scheimpflug stereocamera for particle image velocimetry in liquid flow", Ajay K. Prasad and Kirk Jensen, APPLIED OPTICS 34, 30 (1995), describes a stereo video camera with Scheimpflug conditions and an area sensor. This system is used to determine the velocity distribution of small particles in liquids.

[0018] In such a Scheimpflug camera, the parallel arrangement is eliminated by tilting the lens plane LP with respect to the sensor plane, so that the object area is not limited by the angle of incidence and a large base is possible. In a Scheimpflug camera, the so-called Scheimpflug condition is fulfilled, in which the lens plane, the sensor plane and the object plane intersect at a common intersection axis. There is a Scheimpflug angle β between the object plane OP and the sensor plane SP (FIG. 9). If the Scheimpflug condition is fulfilled, the object area is sharply imaged by the lens onto the sensor. In the above mentioned documents, intersection axes at an infinite distance from the stereo camera system are sometimes described. In this special case, the Scheimpflug condition corresponds to a parallel arrangement. However, in the following considerations, it applies that the Scheimpflug condition is not fulfilled if these planes or axes intersect at infinity. Therefore, the above mentioned planes or axes must intersect at a finite point so that the Scheimpflug condition is fulfilled. One or more mirrors may be provided in the beam path such that the beam path is folded so that the actual lens, sensor and object planes do not intersect at a common intersection axis. In such an arrangement, the Scheimpflug condition is met if the respective lenses, sensors and objects are equally effectively positioned in an unfolded beam path without an associated mirror, and the lens, sensor and object planes intersect at a common intersection axis.

[0019] On the other hand, for a stereo camera system with the Scheimpflug condition, a wider base angle may be selected for a constant lens and image quality than for a stereo camera system without the Scheimpflug condition. A larger base may result in a smaller angle of incidence and therefore be optimized due to the tilted position of the lens. As a result, lenses with a smaller aperture angle γ are also suitable. Due to the smaller aperture angle, the image quality, especially the MTF, is more constant in the overlap region in a stereo camera system with the Scheimpflug condition compared to a stereo camera system without the Scheimpflug condition. Linear distortions along the overlap region are taken into account, since they can be corrected by a parallelization process. The larger the base chosen, the more the sensor and lens planes need to be tilted with respect to the object plane in order to reduce the angle of incidence.

[0020] A further problem of stereo camera systems without the Scheimpflug condition is back reflections. Due to the narrow base angle in the parallel arrangement, the lens not only images the object on the sensor, but also the sensor on the object. This image of the lens can be imaged back to the sensor again by reflection on the reflective surface of the object. In this way, the sensor can capture a mirror image of itself. A wider base angle can be selected, for example by tilting the lens with a Scheimpflug, thereby avoiding such back reflections. Even if the reflection remains, the image of the reflection may also show, for example, a neutral background. The effect of the reflection is therefore reduced.

[0021] Other measurement systems with stereo cameras in which the Scheimpflug condition is met are disclosed in WO 2014 / 017977 A1, US Pat. No. 6,671,397 B1 and WO 02 / 040970 A1.

[0022] WO 2014 / 017977 A1 discloses a method and apparatus for determining coplanarity in an integrated circuit housing. To determine the planarity, two sensors, each having two beam paths, are arranged to follow the Scheimpflug principle.

[0023] US Patent No. 6,671,397 B1 discloses a measurement system with a camera with a lens and a separate sensor. The measurement system has a first camera with a lens and a separate sensor, which are arranged so that their planes intersect at the object plane according to the Scheimpflug principle. The second camera is arranged perpendicular to the object plane. The camera system is an area scan camera.

[0024] WO 02 / 040970 A1 discloses an optical method and apparatus for inspecting large planar objects which in one embodiment uses two wafer-holding area-scan cameras arranged in a Scheimpflug condition.

[0025] The advantages of a stereo camera system with the Scheimpflug condition have been explained above. However, in the case of a stereo camera system with the Scheimpflug condition, a negative effect on the quality of the stereo vision occurs when using area sensors. This is due to the so-called keystone effect, which describes the distortion of the projected sensor face.

[0026] A two-dimensional image of an object area is acquired using an area sensor. In a conventional stereo camera system, one side of the area sensor is arranged parallel to the base. Therefore, the main axis of the acquired two-dimensional image extends in either a direction parallel to the base or a direction lateral to the base. Hereinafter, the direction parallel to the base is referred to as the base direction, and the direction lateral to the base is referred to as the lateral direction.

[0027] When imaging an object area with a stereo camera system, compression of the image occurs not only in the lateral direction, but also in the base direction. This compression occurs because the individual points of a planar object are located at different distances from the face of the sensor. The further two points are from the face of the sensor, the closer they are imaged. The larger the base, the stronger the compression. In the lateral direction, lateral compression occurs, and in the base direction, base compression occurs.

[0028] For example, for a square object area, if the base and center placement of the camera system is made as small as possible, the square object area will be imaged with little distortion. However, if the base is made larger, the image of the square will become trapezoidal or keystone shaped. Corners that are farther away from each other will appear smaller than the distance between the corners that are closer to the lens.

[0029] This effect occurs in stereo camera systems with and without the Scheimpflug condition. In the non-Scheimpflug condition, the effect is similarly limited by the narrowness of the base. In the Scheimpflug condition, the effect is particularly pronounced because the base may be particularly large.

[0030] The keystone effect results in a loss of resolution. Resolution is the minimum distance two point-like objects need to be apart before they are recognized as separate objects. Compression causes two points far from the face of the sensor to be imaged closer together. At a certain distance from the lens, these points may no longer be recognized as separate. The resolution parallel to the base is hereafter referred to as the "base resolution" and the resolution lateral to the base as the "lateral resolution".

[0031] In addition, the keystone effect increases the calculation times. In order to use the images for the evaluation of the three-dimensional surface shape, both the base and lateral compressions have to be calculated from the image data by a rectification process. Rectification is a method to remove geometric distortions of the image data that can occur, for example, due to central projection photography. Rectification shifts the individual pixels so that the same position of the object is represented when the pixels of the two sensors are superimposed. In addition, the distortion is corrected so that the calculated disparity depends linearly on the height profile of the object.

[0032] In stereoscopic vision, each point in the image of one camera module is assigned to each point in the image of the second camera module. The rectangular sensor records a trapezoid T1 of the object surface if the sensor position is tilted relative to the object due to the keystone effect (Figure 10). The other area sensor records a second trapezoid T2 of the object surface. In the point assignment, these trapezoids overlap, each with its long edge coinciding with the short edge of the other trapezoid. This results in a hexagonal overlap area (hatched in Figure 10).

[0033] The disadvantage here is that some of the information recorded in the non-overlapping areas is lost, and only the overlapping areas can be used to evaluate the 3D surface shape.

[0034] Furthermore, the aperture angle of the lens must be large enough to completely cover the overlap area, which applies when the diagonal D of the hexagonal overlap area is detected by the lens.

[0035] DE 10 2015 11 11 20 A1 discloses a method for scanning a surface with a stereo camera comprising at least two camera modules, which are arranged to acquire one image each from a common area of ​​the surface to be scanned. The camera modules may comprise a line camera with several different sensor lines. The individual line images are superimposed by geometric imaging.

[0036] Line sensors and parallel-positioned stereo camera systems have the same problems as area sensors and parallel-positioned stereo camera systems: as mentioned above, if the base is too large, the portions of the object area enclosed by the incidence angles will overlap slightly or not at all, resulting in a small or non-existent portion of the object area that can be sharply imaged, and the base cannot be made any desired size.

[0037] In principle, stereo camera systems that do not satisfy either the Scheimpflug condition or the parallel arrangement are conceivable. For example, only the object plane OP and the sensor plane SP could be parallel to each other, but the lens plane LP could be tilted with respect to the other two planes, so that the object field is not limited by a limited angle of incidence. The imaging plane, which corresponds to the plane in which the lens sharply images the object field, is arranged at an angle (obliquely) with respect to the object plane or the object axis. Only in the area of ​​intersection of the imaging plane and the sensor plane is the object field sharply imaged on the camera sensor (Fig. 11). The greater the inclination of the lens axis with respect to the object axis, the narrower the focus area in which the object is sharply imaged. Also, the greater the depth of field TS, the wider the focus area. With a narrow focus area, only a small part of the object provides the information necessary for the evaluation of the three-dimensional surface shape.

[0038] However, for imaging three-dimensional surface features, sharp imaging is very important and it is desirable to use the entire surface of the sensor.

[0039] In the following, a "stereo camera system not fulfilling the Scheimpflug condition" means a stereo camera system having a parallel arrangement.

[0040] Known from "Inspection of a groove in an O-ring nut", Photonik 5 / 2012, is a system in which three line cameras are operated in parallel, each imaging one side of the groove. The side walls of the groove are imaged by the line cameras with the Scheimpflug condition.

[0041] US Patent No. 7,724,362 B1 discloses a system for inspecting macro wafers at angled incidence, where the wafer is detected by a line camera whose beam path is directed at the wafer at an angle that satisfies the Scheimpflug condition.

[0042] DE 10 2013 103 897 A1 discloses a camera module for scanning an object line by line. The camera module comprises a linear sensor and a lens for imaging the object onto the sensor. The camera module has a number of line sensors which are arranged at varying distances from the lens, so that image lines at different distances from the lens are imaged on the respective line sensors. [Prior art documents] [Patent documents]

[0043] [Patent Document 1] International Publication No. 2014 / 017977A1 ​​Brochure [Patent Document 2] U.S. Patent No. 6,671,397B1 [Patent Document 3] International Publication No. 02 / 040970A1 Brochure [Patent Document 4] DE Published Patent Application No. 10 2015 11 11 20A1 [Patent Document 5] U.S. Patent No. 7,724,362B1 [Patent Document 6] DE 10 2013 103 897 A1 [Non-patent literature]

[0044] [Non-Patent Document 1] "Scheimpflug stereocamera for particle image velocimetry in liquid flow", Ajay K. Prasad and Kirk Jensen, APPLIED OPTICS 34, 30 (1995) [Non-Patent Document 2] "O-Ring-Nut", Photonik 5 / 2012 Summary of the Invention [Problem to be solved by the invention]

[0045] The invention is based on the problem of creating a stereo camera system which in a simple manner provides good spatial resolution over a large scan area.

[0046] A further object of the invention is to create a stereo camera system which allows reliable scanning of spatial objects.

[0047] A further object of the invention is to create a stereo camera system which is able to cover a large imaging area in a simple manner.

[0048] A further object of the present invention is to create a stereo camera system with a compact design.

[0049] A further object of the invention is to create a stereo camera system that is suitable for imaging different object planes. [Means for solving the problem]

[0050] One or more of the problems are solved by the subject matter of the independent claims. Advantageous developments and preferred variants form the subject matter of the dependent claims.

[0051] A stereo camera system according to the invention for measuring a three-dimensional surface shape of an object has at least two linear sensor areas and at least one lens for imaging the object onto the sensor areas, which lens is arranged in such a way that the object area is imaged onto one sensor area by two independent beam paths, respectively. The stereo camera system is characterized in that the Scheimpflug condition is fulfilled along at least one of the beam paths.

[0052] As explained above, a stereo camera system that takes the Scheimpflug condition into account has a larger base than a parallel stereo camera system without the image being distorted by the lenses, which results in a higher resolution of the 3D surface shape.

[0053] Such a stereo camera system may be used for line-by-line scanning of an object area intersecting in a line direction, with the stereo camera system being moved relative to the object. The direction parallel to the line sensor is called the line direction, and the direction perpendicular to the line sensor is called the scan direction. The distance of two points located towards each other in the scan direction are recorded with the same resolution, regardless of the keystone effect and their position in the line direction. This means that the image is not compressed in the scan direction. This improves the resolution in the scan direction.

[0054] Similarly, trapezoid formation is prevented. Both line sensors can cover the same area with their entire sensor line, so no information is lost. The overlapping area of ​​the line sensors is hereafter referred to as the overlapping line.

[0055] In the case of a line sensor, the aperture angle of the lens needs to cover only the overlap line, which is smaller than the diagonal of the overlap area of ​​two area sensors that form a hexagon as described above if the sensors have the same length. Therefore, the required aperture angle of a stereo camera system with Scheimpflug condition and area sensor is larger than that of a stereo camera system with Scheimpflug condition and line sensor. Therefore, a stereo camera system with Scheimpflug condition and line sensor does not meet the Scheimpflug condition and Area Sensor This results in less distortion than a stereo camera system equipped with

[0056] Reduced compression also reduces the need for parallelization, which reduces computation time and therefore makes image data available sooner.

[0057] In summary, it can be said that by using a stereo camera system with the Scheimpflug condition, a large base may be chosen without the occurrence of distortions as in a stereo camera system without the Scheimpflug condition. A large base results in a precise (high) resolution in the height direction. In the Scheimpflug condition, the aperture angle of the lens can be small, resulting in a good image quality. However, due to the keystone effect, in the case of area sensors, the Scheimpflug condition leads to undesirable distortions in both directions. If a line sensor is used, the distortions only occur in one direction, resulting in a high image quality. Furthermore, since the projection width of the line sensor is smaller than the projection diagonal of the area sensor, the required aperture angle of the lens is smaller for the line sensor than for the area sensor. In addition, in contrast to the area sensor, all pixels of the sensor are used.

[0058] Therefore, a stereo camera system that satisfies the Scheimpflug condition and has a line sensor may be able to obtain an image with high resolution in the height direction using relatively simple means.

[0059] It is preferred that the Scheimpflug condition is satisfied for all beam paths. In principle, the beam paths may have different sensor areas and lens arrangements. Since all beam paths satisfy the Scheimpflug condition, a larger base may be selected compared to a stereo camera system in which only one beam path satisfies the Scheimpflug condition.

[0060] Preferably, the beam paths are arranged with mirror symmetry. This arrangement makes it easier to adjust the camera system. Asymmetries, which sometimes occur during adjustment, may be corrected. The evaluation of the image data is also easier if the sensors are equidistant from the object region. Therefore, the acquired pixels are also offset by the same distance.

[0061] Preferably, one or more of the beam paths are deflected by at least one mirror. The beam path may thereby be folded by the mirror. The image of the measured object is reflected one or more times by the mirror, thereby maintaining the length of the beam path. However, the volume captured by the arrangement of optical lenses is partly considerably reduced. In particular, a large base can be selected for the stereo camera system, but the two sensor halves are still close to each other. This means that a more compact design is possible compared to a structure without a folded beam path.

[0062] In connection with the folding of the beam path, a distinction may be made between object-side folding, in which the mirror folding the beam path is arranged between the lens and the object, sensor-side folding, in which the mirror folding the beam path is arranged between the lens and the sensor, and double-sided folding, in which the mirror folding the beam path is arranged not only between the lens and the object but also between the lens and the sensor. In the case of object-side folding, the working distance, which is the distance between the lens and the object, is reduced.

[0063] The beam paths are preferably each projected onto a separate portion of one sensor. These portions each form a sensor area of ​​one of the two or more camera modules. The sensor may be divided into a number of sensor areas, each of which is assigned a beam path. For example, in the case of a stereo camera system with two beam paths, the sensor is divided into two areas. In the case of a stereo camera system with three beam paths, the sensor is divided into three. This also increases the stability of the system. In addition, it allows for an easy-to-adjust and compact design.

[0064] Alternatively, the camera can be designed such that each beam path is projected onto a separate sensor, with one mirror for each beam path, and the beam paths are arranged in a Scheimpflug configuration, allowing for a compact geometry with a larger sensor surface than a single sensor, leading to improved resolution.

[0065] Yet another method is characterized in that the beam paths are projected onto separate sensors, each of which has two mirrors and is arranged in a Scheimpflug configuration, which allows for a compact geometry.

[0066] Preferably, the linear sensor areas are arranged adjacent to each other on a line, each represented by a line sensor. This linear arrangement may facilitate adjustment since the linear sensor areas may be fixed to a common support. For example, there is no need to realign (readjust) the sensor areas if the support is tilted.

[0067] For stereoscopic methods it is advantageous if the offset is only in one direction, meaning that the sensors are arranged parallel or perpendicular to the sensor axis, which is supported by the line arrangement.

[0068] According to a further modification, the linear sensor areas may be arranged parallel and adjacent to one another, again with the offset of the two beam paths only in one direction.

[0069] The linear sensor area can be a pixel line of an area sensor, and existing stereo line camera systems can be easily adapted to do so.

[0070] A common lens may be provided for all beam paths. In this way, a particularly compact stereo line camera system may be created. The reduced number of lenses may also potentially reduce costs.

[0071] The two beam paths may have different distances between the object plane and the sensor plane, in which case it may be advantageous to smooth the difference in distance by means of one or more glass elements in each of the shorter beam paths.

[0072] This glass element slows down the speed of light within the glass element, so it takes longer for the light to reach the sensor, which has the same effect as lengthening the light path, in this way distance variations may be evened out.

[0073] In a further aspect of the invention, a stereo line camera system for measuring three-dimensional surface shapes includes at least one color sensor having a plurality of pixels sensitive to different colors and at least one hyperchromatic lens for imaging the object onto the sensor, the color sensor being arranged towards the object such that different focal planes are each imaged onto pixels of the same color.

[0074] In a hyperchromatic lens, the focus is wavelength dependent, therefore each color line of the sensor is assigned to a different focal plane.

[0075] This may be used for the measurement of three-dimensional surface shapes. Different heights of the object area each have a different focal plane. This ensures that each height in a particular focus area is sharply imaged on the sensor. The height resolution here corresponds to the color resolution. This method is described below as hyperchromatic depth determination. It is particularly suitable for detecting small differences in height.

[0076] Preferably, the hyperchromatic depth determination is applied together with a stereoscopic method that comprises the Scheimpflug condition: in the hyperchromatic depth determination, fine height differences are detected, whereas in the stereoscopic method, height differences are detected over a larger range.

[0077] Preferably, the color lines are tilted relative to the lens. The tilting is performed about an axis parallel to the individual color lines, and the Scheimpflug condition is maintained along the lines. This tilting changes the distance of the color lines from the lens and also moves the focal plane, which may effectively change the maximum height difference measurable in hyperchromatic depth determination.

[0078] Preferably, the color channels of the color sensor are evaluated separately, so that different focal planes can be calculated separately.

[0079] In a further embodiment of the invention, a stereo line camera for measuring three-dimensional surface shapes comprises at least two area sensors, which are designed with a plurality of pixel lines arranged parallel and adjacent to one another, and at least two lenses for imaging an object onto one of the area sensors, characterized in that an object field with two independent beam paths is imaged simultaneously onto one of the sensors, respectively, such that a linear object field is imaged onto the pixel lines of the area sensors, but in a line plane of the beam paths intersecting the linear object field, the Scheimpflug condition is fulfilled, and the area sensors are tilted about an axis such that the surface is not parallel to the object field to be scanned, such that the individual lines of the area sensors image different planes of the linear object field.

[0080] Due to the inclination, each pixel line has a different distance from the lens. Along the line, the Scheimpflug condition is maintained. Each line therefore has its own focal plane. In this way, different heights of the object area are sharply imaged on the respective different pixel lines. The height resolution here corresponds to the line resolution. This method is referred to below as "inclination depth determination" (Latin: inclinatio = inclination).

[0081] The maximum difference in height measurable in a slope depth determination may be varied by varying the slope.

[0082] Height resolution may be improved by combining the dip depth determination with a stereoscopic method with the Scheimpflug condition, which reduces measurement errors since the height is detected by two different methods.

[0083] Preferably, the sensor is a color sensor and the lens is a hyperchromatic lens, which may further split the individual focal planes caused by the tilt of the area sensor, improving the focal plane resolution.

[0084] Preferably, the image data are rectified online, which harmonizes the image data of the stereo image pair line by line and corrects for offsets and distortion differences due to the two lenses. This rectification, which can also be done offline, adjusts the position data of each distortion, so that the image data can be displayed undistorted and further calculated. This position data does not require an additional transformation step, which increases the calculation speed.

[0085] Line sensors using charge-coupled device (CCD) sensors have shift registers that store the detected luminance values. These shift registers are read out in succession. Typically, when reading out the shift registers for each line, each pixel is assigned a uniform positional value.

[0086] Preferably, the pixel lines of the sensors are synchronized to record line-synchronously and start imaging at the same time. This significantly reduces the time required for the parallelization process. Especially in combination with the parallelization process, this allows the acquired pixels of one camera module to be assigned to the same point on the object at the same time as the second acquired pixels of the second camera module. This has the advantage that in the stereo calculation, each line of the sensor area is automatically assigned to the appropriate line of the second sensor area. This means that the stereo calculation is only required in one dimension.

[0087] A further aspect of the invention relates to a surface profile detection apparatus having at least one stereo camera system as described above for acquiring a three-dimensional surface profile of an object to be measured, the surface profile detection apparatus including a transport device for transporting the object to be measured or the camera system, a synchronization device for synchronizing the transport speed with a linear scanning of the three-dimensional surface profile, and an evaluation unit for evaluating the measurement result of the three-dimensional surface profile.

[0088] The conveying device may for example be a conveyor belt.

[0089] The synchronizing device provides a signal to the stereo camera system and / or the evaluation unit to synchronize the transport speed with the measurement of the 3D surface shape. If this signal is sent to the stereo camera system, it can trigger the measurement. If the signal is sent to the evaluation unit, the speed of the transport device is determined. Also, if the image generation frequency is known, it can be determined how far one image line is from the next image line.

[0090] The synchronizing device may for example be an incremental encoder which is triggered by the transport device. If the transport device is a conveyor belt, for example by means of a wheel coupled to the conveyor belt, the position of the wheel can be determined by the synchronizing device. At a certain wheel position, a signal is emitted. Alternatively, the synchronizing device could be provided with a camera unit which detects markings on the conveyor belt and with its help emits a signal.

[0091] The synchronisation device may be in the form of a module of the evaluation unit.If the image generation frequency and the speed of the transport device are known, the distance between two recorded image lines may be calculated.

[0092] The evaluation unit is a module on the computing unit which calculates the three-dimensional surface shape from the recorded image data. [Brief description of the drawings]

[0093] The invention will now be described in more detail, by way of example, with the aid of embodiments shown in the drawings, which are in each case shown diagrammatically in side view.

[0094] [Figure 1] 1 is a first embodiment of a camera with two lenses and two sensors. [Figure 2a] 2 is a second embodiment of a camera with two lenses, four mirrors, and one sensor. [Figure 2b] 1 is a variation of the second embodiment of a camera with two lenses, four mirrors and two sensors. [Figure 2c] 13 is a further modification of the second embodiment of the camera with two color filters, two lenses, four mirrors and two sensors. [Diagram 3] 3 is a third embodiment of a camera with two lenses, two mirrors, and two sensors. [Figure 4] It is a hyperchromatic lens in which the various focal planes lie on different color lines. [Diagram 5] 4 is a fourth embodiment of a camera with three lenses, four mirrors, and a color line sensor. [Figure 6] 11 is a variation of the fourth embodiment of the camera with one lens, four mirrors, a glass element, and a color line sensor. [Figure 7] 1 is a block diagram showing a surface shape detection device using a stereo camera according to the present invention; [Figure 8] 1 is a schematic layout diagram of sensors and lenses of two camera modules of a stereo camera not according to the present invention. [Figure 9] FIG. 2 is a schematic layout diagram of sensors and lenses of two camera modules of a stereo camera. [Figure 10] This is the sensor plane projected onto the object plane. [Figure 11]1 is a block diagram showing a surface shape detection device using a stereo camera according to the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0095] In a first embodiment, a stereo camera system 25 according to the present invention includes two camera modules 1a, 1b (FIG. 1).

[0096] Each camera module 1 is formed by a lens 2 and a line sensor 3. The line sensors 3 acquire (capture) by means of their respective sensor areas 13 an image formed thereon by the lens 2. The two camera modules 1 are arranged to scan a common object area 6 from two different viewing directions.

[0097] For each camera module 1, a camera axis 5 passes respectively through the centre of the sensor area 13 and the centre of the lens 2. The two camera axes 5 therefore run in the direction of the field of view of the sensor area 13 to the object area 6.

[0098] The two camera axes 5 are positioned on a single optical plane. Also, on this optical plane, there are two line sensors 3 and their sensor areas 13. This optical plane coincides with the drawing plane in FIG.

[0099] The base B is the distance between the centres of the two sensor areas 13. The camera axis 5 intersects with the object axis 10, forming (encompassing) a base angle α.

[0100] Since this camera system uses a line sensor 3, only the sensor axis 8, lens axis 9 and object axis 10 are relevant for further consideration, which are straight lines that lie in the optical plane and at the same time lie in the sensor, lens and object planes defined above.

[0101] The line sensors 3 acquire an image of the object area 6 reproduced on the line sensor 3 by the lens 2. The line sensors 3 are CCD sensors and contain a shift register and a one-dimensional array of photodetectors, each of which is responsible for recording one pixel. For each line sensor 3, a sensor axis 8 runs coincident with the sensor line.

[0102] The lenses 2 of the camera module 1 are arranged relative to the respective line sensor 3 such that the object area 6 is sharply depicted in the plane of the line sensor 3. The lens 2 may be formed by one lens or by several lenses. The optical axis of the lens 2 is the axis of symmetry of the lens 2 and runs perpendicular to the lens axis 9 in the optical plane.

[0103] The object area 6 lies on the object axis 10 and is sharply imaged onto the sensor area 13 .

[0104] The sensor axis 8, lens axis 9, and object axis 10 intersect at a common point, but this common point does not exist at infinity, thereby satisfying the Scheimpflug condition.

[0105] In the following, a second embodiment (FIG. 2a) will be described, in which the same elements as in the first embodiment are given the same reference numerals, and the above description is applicable to the same elements unless otherwise specified below.

[0106] In the second embodiment, the stereo camera system 25 again includes two camera modules 1a, 1b. Each camera module 1 includes a lens 2, two mirrors 11, 12 and a line sensor area 13. The camera axis 5 is deflected by the two mirrors 11, 12. The line sensor areas 13 of each camera module 1a, 1b are aligned, one behind the other, and together form a line sensor 3.

[0107] The mirrors 11 and 12 are disposed between the lens 2 and the line sensor area 13 so that the object area 6 falls on (enters) the line sensor area 13.

[0108] Sensor axis 8 is reflected by mirror 11. Sensor axis 8' (not shown) is the reflected sensor axis 8. Sensor axis 8' is now reflected by mirror 12. Sensor axis 8" (not shown) is the reflected sensor axis 8', i.e. the doubly reflected sensor axis 8.

[0109] The base B of the stereo camera system 25 is given by the distance between the centers of the projections of the two line sensors 3, the projections of the line sensors being the points where the line sensors 3 would be if the image of the object area 6 was not reflected.

[0110] The sensor axis 8'', lens axis 9 and object axis 10 intersect at a common point, but this common point does not exist at infinity. This satisfies the Scheimpflug condition.

[0111] The second embodiment differs from the first embodiment in that two mirrors 11, 12 per camera module 1a, 1b deflect the focal points of the object axes 10 such that they lie on a common axis, the associated sensor axis 8. It is therefore not necessary to have two separate sensors, but instead both beam paths may be projected onto a line sensor 3 that is split into two line sensor areas 13.

[0112] In this embodiment, the sensor axis 8 is parallel to the object axis 10. However, it is also conceivable that these two axes are not parallel, in which case both camera modules are not of mirror symmetric design.

[0113] Alternatively, the stereo camera system 25 has a gap between the two line sensor areas 13 (FIG. 2b). Each line sensor 3 includes only one line sensor area 13. This embodiment has some similarity to the HyperSkop, however, it does not fall under the Scheimpflug condition.

[0114] A further option of this variant is that the two optical axes intersect and a first image of the object area 6 is projected onto the opposite line sensor 3b and line sensor area 13b, respectively. Without the Scheimpflug condition, this arrangement is known as a Pseudoskop.

[0115] In a further embodiment, the stereo camera system 25 has a design according to the second embodiment (FIG. 2c), but the sensor areas 13 are not arranged adjacent to each other in a line along the sensor axis 8, but instead are arranged parallel to each other perpendicular to the sensor axis 8. These sensor areas 13 may each be formed by a different color line of the line sensor 3, as will be explained in detail below. Both beam paths are arranged perpendicular to the line color sensor 3 offset from each other, such that the two different beam paths meet slightly offset on the line color sensor 3. The two mirrors 11 are mounted adjacent to each other. And a color filter 26 is arranged along each beam path. As a result, the image of one camera module 1a appears on the line color sensor 3 as, for example, a red image (image), and the image of the other camera module 1b appears on the line color sensor 3 as, for example, a green image (image). Thereby, the line color sensor 3 may simultaneously acquire two or even more images of the object area 6. Here, the surface of the object must not be one of the colors of the color filters 26. Therefore, the beam path with the color filter of the first color hits the sensor line that detects the first color, and the beam path with the color filter of the second color hits the sensor line that detects the second color.

[0116] An object surface that is white or grayscale is preferred. White or grayscale means that each color in the color spectrum has the same intensity value. If light passes through the respective color filter 26, the integrated intensity of the light is reduced equally for all colors, regardless of the color of the color filter. If the color channels of the line sensor 3 are evaluated as grayscale, the resulting images of both beam paths will only differ in grayscale intensity due to different viewing angles to the object, and not due to different colors in the beam paths.

[0117] In this embodiment, the color filter 26 is mounted in front of the lenses 2a, 2b (FIG. 2c), but the color filter 26 may also be located between the mirrors 11 and 12 or between the line color sensor 3 and the mirror 11.

[0118] Alternatively, instead of color filters, different polarizing filters may be used along the beam paths, as long as the line sensor 3 can individually acquire suitably different polarized images of the object region 6. Each beam path of the stereo camera system 25 therefore has a different polarizing light when it hits the line sensor 3. If the phase and intensity of the images are acquired, the method may also be used for holography. In this case, it may be ensured that the object surface does not change the polarizing light, since the polarizing filters would otherwise filter out image data required for calculating the height data.

[0119] The third embodiment (FIG. 3) will be described below, with the same reference numerals used for the same elements as in the first and second embodiments. The above description applies to the same elements unless otherwise specified below.

[0120] In the third embodiment, a stereo camera system 25 again includes two camera modules 1a, 1b. Each camera module 1 includes a lens 2, a mirror 11 and a line sensor 3. A camera axis 5 is deflected by the mirror 11.

[0121] A mirror 11 is disposed between the lens 2 and the line sensor 3 so that the object area 6 falls on the line sensor 3 .

[0122] The sensor axis 8 is reflected by a mirror 11. A sensor axis 8' (not shown) is the reflected sensor axis 8. The two sensor axes 8 of the two camera modules 1 are not necessarily parallel to each other in this embodiment.

[0123] The base B of the stereo camera system 25 is given by the distance between the centers of the projections of the two line sensors 3, the projections of the line sensors being the points where the line sensors 3 would be if the image of the object area 6 was not reflected.

[0124] The sensor axis 8', lens axis 9, and object axis 10 intersect at a common point, which is not at infinity, thereby satisfying the Scheimpflug condition.

[0125] In contrast to the second embodiment, the two line sensor areas 13 are no longer aligned, one behind the other, and in addition, due to the mirror 11, the sensor 3 may be arranged perpendicular or nearly perpendicular to the object area 6, which makes it possible to choose a very compact design.

[0126] The fourth embodiment (FIG. 5) will be described below, with the same reference numerals used to refer to the same elements as in the first embodiment. The above description applies to the same elements unless otherwise specified below.

[0127] In the fourth embodiment, a stereo camera system 25 includes three camera modules 1a, 1b, 1c. Instead of a simple line sensor, the stereo camera system 25 includes a three-line sensor 15. The three-line sensor 15 includes three parallel line sensor areas 13 arranged next to each other. The sensor axis 8 runs perpendicular to the optical plane.

[0128] Each camera module 1a, 1c includes a lens 2, two mirrors 11, 12, and one of three line sensor areas 13. Camera module 1b includes a lens 2 and one of three line sensor areas 13. The camera modules 1 scan a common object area 6 from three different viewing directions.

[0129] For the outer camera modules 1a, 1c, the sensor axis 8 is reflected by mirror 11. Sensor axis 8' (not shown) is the reflected sensor axis 8. Sensor axis 8' is now reflected by mirror 12. Sensor axis 8" (not shown) is the reflected sensor axis 8', i.e. the doubly reflected sensor axis 8.

[0130] If three camera modules 1 are used in the stereo camera system 25, three base lengths B are possible for each combination of two camera modules 1. The base B is given by the distance between the centers of the projections of the two line sensors 3, the projections of the line sensors being the points where the line sensors 3 would be if the image of the object region 6 was not reflected.

[0131] The sensor axis 8 or 8'', the lens axis 9, and the object axis 10 intersect at a common point, but this common point does not exist at infinity. This satisfies the Scheimpflug condition.

[0132] Alternatively, the stereo camera system 25 includes only a single lens 2 that surrounds all three camera modules 1a, 1b, 1c, instead of the three lenses 2a-2c. To compensate for this, the middle camera module 1b has a glass element 19 (FIG. 6).

[0133] The fourth embodiment differs from the preceding embodiments in that the line sensor areas 13 are arranged parallel to one another, rather than in line, one behind the other.

[0134] In an alternative variant of the above-described embodiment, the lens of each camera module 1 is a hyperchromatic lens 14, the focus of which depends, inter alia, on the respective wavelength (see FIG. 4). In addition, the three-line sensor 15 is a three-line color sensor 15 having red 16a, green 16b and blue 16c lines. Due to the hyperchromatic lens 14, each red, green and blue line (16a, 16b, 16c) of the three-line color sensor 15 has a different focal plane 17a, 17b, 17c, respectively. Thus, each color line 16a, 16b, 16c measures a different height measurement area 18a, 18b, 18c. In this case, the height resolution corresponds to the color resolution.

[0135] Preferably, the color lines 16 are tilted relative to the lens 14. The tilt is about an axis parallel to the individual color lines 16. The Scheimpflug condition is maintained along the color lines 16. The tilt changes the distance of the color lines 16 from the lens 14, which also moves the focal plane 17.

[0136] A further option of the present invention is that the camera module 1 comprises an area sensor instead of a line sensor 3. An area sensor is a two-dimensional array of photodetectors, each photodetector corresponding to one pixel. In this embodiment, the area sensor can be considered as a series of parallel line sensors 3.

[0137] The area sensors are tilted about a tilt axis parallel to the individual line sensors 3. Each line, together with its assigned sensor axis 8 or 8' or 8", intersects the lens axis 9 and the object axis 10 at one point and is therefore positioned to satisfy the Scheimpflug condition.

[0138] Here, the amount of angle between the normal and the optical axis of the area sensor is less than 90°. Each line of the area sensor has a different height measurement area 18. Adjacent height measurement areas lead to an enlargement of the height area. As a result, the same camera setup may be used for objects of different heights. Each height of the object is assigned at least one corresponding line of the area sensor with an associated height measurement area.

[0139] In an alternative variant of the embodiment described above, instead of a straight mirror 11, a curved mirror is used so that the image of the object 4 to be measured reaches the line sensor 3 without distortion.

[0140] The above-mentioned embodiments each include two or three camera modules 1. In principle, any desired number of camera modules may be provided for the stereoscopic determination of the surface shape, but at least two camera modules are necessary. The more camera modules there are, the smaller the statistical error. However, the costs also increase. The number of base lengths follows a triangular number. For example, with two camera modules 1 only one base is possible, with three camera modules 1 three, with four six options and with five ten different base lengths are possible.

[0141] The surface shape detection device 20 (FIG. 7) will now be described.

[0142] The surface shape detection device includes a stereo camera system 25 described in one of the above embodiments relating to any one of Figures 1 to 6 for measuring the three-dimensional surface shape of the object 4 to be measured, a conveying device 21 for conveying the object 4 to be measured, a synchronization device 22 for synchronizing the conveying speed with the measurement of the three-dimensional surface shape, and an evaluation device 23 for evaluating the measurement results of the three-dimensional surface shape.

[0143] In this embodiment, the conveying device 21 is a conveyor belt.

[0144] The synchronization device 22 in this embodiment is an incremental encoder 22. The incremental encoder is connected to the conveyor belt 21 so as to rotate during the movement of the conveyor belt. In the course of the rotation around a given rotation angle, a clock signal is generated. Thereby, the conveyor belt covers a given distance between two successive clock signals. These clock signals are passed to the evaluation device 23. The evaluation device 23 is also connected to the camera module 1 so as to obtain the image signals acquired by the sensors 3 arranged in the camera modules. The evaluation device 23 is designed to control the time at which each camera module scans the object 4 to be measured. In this way, the evaluation device 23 is able to synchronize the scan of the object 4 to be measured to the respective movement of the conveyor belt and of the object 4 to be measured according to the clock signals received from the incremental encoder 22. This synchronization is preferably arranged in such a way that the object to be measured is scanned in the direction of movement 7 at the same local distance. Several successive one-dimensional scans may be combined to give a two-dimensional image in the evaluation device 23. Three-dimensional data on the surface shape are thus obtained.

[0145] The acquired image may be corrected: on the one hand, by successively and sensitively switching the individual pixels of the line camera, it is possible to correct shift errors resulting from the movement of the measured object 4 relative to the camera, so that at high speeds the image has a constant tilt position with respect to the direction of movement 7.

[0146] In addition, the image may be rectified, which removes geometric distortions in the image data, which may occur due, inter alia, to the Scheimpflug geometry.

[0147] Subsequent corrections, called parallelization, may be performed online directly during image registration. After taking a picture of the image line, the data is directly corrected. This involves digitizing the sensor signal followed by pre-processing of the image signal in the camera's FPGA (Field Programmable Gate Array). If the distortion parameters of the two camera modules are known, e.g. by system calibration, the same correction models may be applied as before on the PC. Therefore, no further corrections are required on the computing device, which increases the processing speed.

[0148] The images thus corrected are used for depth reconstruction of the imaging plane of the object 4 to be measured. A typical method of depth reconstruction is stereotriangulation. It involves assigning to each point of a first image a second point of a second image. The distance between the two points depends on the actual spatial depth of the point, more precisely on its distance from the object region 6 of the sensor 3 and on the base B of the stereo camera system 25. In this way, each point of the first image is assigned depth information.

[0149] Alternatively, blocks, i.e. groups of pixels, such as 3x3 matrices, may be assigned to each other, a method known as block matching.

[0150] In this way, for each image line acquired, an image line containing depth information is calculated. If multiple image lines of the object 4 to be measured are acquired, the lines with depth information may be combined to generate a three-dimensional surface profile of the object to be measured. The distances of the individual lines may be calculated by information from the synchroniser 22.

[0151] If instead of several line sensors 3 a line sensor area 13 is used, the image data of all camera modules 1 are generated on the same line sensor 3. The resulting digital double image is split in the evaluation device 23. The two resulting images are corrected as described above and used for depth construction.

[0152] If a three-line sensor 15 is used to acquire the object to be measured, a separate image is generated for each color. For each color, a separate image is corrected as described above and used for depth construction. If a hyperchromatic lens is used, each color has a different height measurement area 18a, 18b, 18c, so that an additional estimate of the depth profile of the surface of the object to be measured may be calculated via image sharpness detection. Each point of the image is a point of the image of the other two color areas, but assigned to the same camera module 1. The position of the point does not differ in the three images. Thus, the sharpness of the image of this point differs depending on the depth and color of the actual point on the object to be measured 4. If the point is deep, it is displayed sharply in the blue image, at a medium depth in the green image and at a low depth in the red image. The sharpness detection may be performed, for example, by a contrast comparison of the images in a pixel area. The sharper a point is, the greater the contrast.

[0153] In this way, by combining with stereoscopic calculations, the surface shape can be determined in detail.

[0154] Rather than calculating each color image separately, the RGB images may be combined to calculate a grayscale image that is then analyzed.

[0155] If an area sensor is used to capture the object to be measured, a separate image is generated for each line. For each line, the separate image is corrected and used for depth construction as described above. Then, in each separate image, the sharpness of an actual point on the object to be measured 4 is determined. Thus, the line that captures the point the sharpest may be determined. Since the position and focal length of the line are known, the distance of the point is determined from these factors. If all the points are connected together, the height profile of the object to be measured 4 is obtained.

[0156] In addition to the three-line sensor 15 described above, other combinations are also possible, such as a sensor with three RGB sensor points lined up, a sensor with three sensor points lined up that also recognize grayscale, a sensor with three sensor points lined up that only recognize grayscale, etc.

[0157] The sensor may be a CCD sensor, but other sensor types are also contemplated, such as a Complementary Metal-oxide Semiconductor (CMOS) sensor.

[0158] Reference number 1 Camera module 2 Lenses 3 Sensors 4. Objects to be measured 5 Camera Axis 6 Object Area 7. Direction of movement 8 Sensor Axis 9 Lens axis 10 Object Axis 11. Mirror 1 12. Mirror 2 13 Line sensor area 14 Hyperchromatic Lens 15 3 line sensor 16 Color Line 17 Focal plane 18 Height measurement area 19 Glass Elements 20 Surface shape detection device 21 Transport device 22 Synchronous Device 23 Evaluation Equipment 25 Stereo Camera System 26 Color Filters α Base angle β Scheimpflug angle γ Lens aperture angle B-Base

Claims

1. A stereo camera system for measuring a three-dimensional surface shape of an object, comprising: At least two linear sensor areas; at least one lens for imaging an object onto the sensor area; Equipped with the sensor areas and the at least one lens are arranged such that an object area is imaged simultaneously by two independent beam paths, each onto one of the sensor areas; A stereo camera system, characterized in that the Scheimpflug condition is satisfied along at least one of the beam paths.

2. 2. The stereo camera system of claim 1, wherein the Scheimpflug condition is satisfied along all beam paths.

3. 3. A stereo camera system according to claim 1 or 2, characterized in that one or more of the beam paths are deflected by at least one mirror.

4. 4. The stereo camera system according to claim 1, wherein each of the beam paths is projected onto a different part of a sensor, and the sensor is divided into the number of beam paths.

5. 4. The stereo camera system according to claim 1, wherein the beam paths are projected onto separate sensors, each beam path is provided with at least one mirror, and the beam paths are configured in a Scheimpflug arrangement for each beam path.

6. 4. The stereo camera system according to claim 1, wherein the beam paths are projected onto separate sensors, each beam path is provided with at least two mirrors, and the beam paths are configured in a Scheimpflug arrangement for each beam path.

7. 7. The stereo camera system according to claim 1, wherein the linear sensor areas are arranged adjacent to each other on a line, and each linear sensor area is represented by a line sensor.

8. 6. The stereo camera system according to claim 1, wherein the linear sensor areas are arranged parallel to and adjacent to each other.

9. 9. The stereo camera system according to claim 8, wherein the linear sensor region is a pixel line of an area sensor.

10. 10. The stereo camera system according to claim 1, wherein a common lens is provided for all beam paths.

11. 11. The stereo camera system according to any one of the preceding claims, characterized in that a glass element is provided in at least one of the beam paths.

12. A stereo camera system for measuring a three-dimensional surface shape of an object according to any one of claims 1 to 11, at least one color sensor having a plurality of pixels sensitive to different colors; at least one hyperchromatic lens for imaging the object onto the color sensor; Equipped with 11. A stereo camera system, comprising: a color sensor arranged with respect to the lens such that different focal planes are each imaged into pixels of the same color.

13. 13. The stereo camera system of claim 12, wherein the color channels of the color sensor are evaluated separately.

14. A stereo camera system for measuring a three-dimensional surface shape of an object, comprising: At least two area sensors, the area sensors being designed to have a plurality of pixel lines arranged parallel and adjacent to each other; at least two lenses for imaging an object onto one of the area sensors; Equipped with an object region having two independent beam paths is imaged simultaneously onto one of the area sensors, A linear object area is imaged onto the pixel line of one area sensor; - in a line plane of the beam path intersecting the linear object region, the Scheimpflug condition according to any one of claims 1 to 11 is satisfied; and the area sensor is tilted about an axis such that the surface is not parallel to the object area being scanned, such that individual lines of the area sensor image different planes of the linear object area; A stereo camera system characterized by:

15. The stereo camera system as described in claim 14, characterized in that the lens is a hyperchromatic lens as described in claim 12.

16. A stereo camera system as described in any one of claims 1 to 15, characterized in that the acquired data is recorded online for correction of subsequent parallelization processing.

17. 17. A stereo camera system according to any one of claims 1 to 16, characterized in that the pixel lines of the sensors are synchronized so as to record line synchronously and start imaging at the same time.

18. A surface shape detection device, comprising at least A stereo camera system according to any one of claims 1 to 17 for measuring a three-dimensional surface shape of an object to be measured; A conveying device for conveying the object to be measured; a synchronization device for synchronizing a conveying speed with the measurement of the three-dimensional surface shape; an evaluation device for evaluating the measurement results of the three-dimensional surface shape; A surface shape detection device comprising:

Images