Photoelectric sensor and method for detecting an object

Abstract

The present application relates to a photoelectric sensor and a method for detecting an object. A photoelectric sensor (10) for detecting an object in a monitoring area (16, 120, 212) is provided, wherein the sensor (10) has: a laser scanner (110) having a deflection unit (112) rotatable about a rotation axis (118), the deflection unit being used to scan the monitoring area (120) with at least one scanning beam (126, 128); and a first distance measuring unit (140) being used to determine a 3D measurement point of a corresponding object scanned by the scanning beam (126, 128) using a light time-of-flight method; a panoramic camera (210) having a panoramic optical device (216) and an image sensor (218), the image sensor having a plurality of light receiving elements for detecting image points; and a control and evaluation unit (14, 140, 222) configured to fuse 3D measurement points and image points. In this case, the optical axis (220) of the panoramic camera and the rotation axis (118) coincide.

Description

Technical Field

[0001] The present invention relates to a photoelectric sensor and a method for detecting objects in a surveillance area using a laser scanner and a panoramic camera. Background Art

[0002] In a laser scanner, a scanning beam scans an area and the diffusely reflected or reflected light is evaluated. In order to also obtain information about the distance, outline or shape of an object, it is usually necessary not only to determine the presence of an object but also its distance at the same time. Such distance-measuring laser scanners operate according to the principle of light time-of-flight, in which the flight time from the scanner to the scene and back is measured and the distance data is calculated based on the speed of light. Two types of light time-of-flight methods are widely used. In phase-based methods, a light transmitter modulates the scanning beam and the phase between the reference and received scanning beam is determined. Pulse-based methods modulate a distinct pattern (e.g., narrow pulses of a few nanoseconds duration) into a pulse. The pattern is then transmitted onto the scanning beam and the reception time of the pattern is determined. In a general design known as the pulse averaging method, multiple pulses or a pulse train are transmitted and the received pulses are statistically evaluated.

[0003] The laser scanner has a rotating mirror or a multifaceted mirror wheel in order to periodically scan the monitoring plane or a section of the monitoring plane. Alternatively, an active measuring head with a light transmitter and a light receiver rotates. A 3D laser scanner detects a three-dimensional spatial area, not just a surface. This can be achieved by a movement around another axis (pivoting movement or full rotational movement). For example, in DE 10 2008 032 216 A1, the entire scanning unit incorporating a transmitter, a receiver and a rotating mirror is arranged on a deflection disk. EP 1 965 225 A2 discloses various solutions for combining the rotational movement of a laser scanner with an additional tilting movement.

[0004] Furthermore, multi-layer laser scanners are known, which have a plurality of scanning beams arranged one above the other at an elevation angle. Thus, it is known from US Pat. No. 8,767,190 B2 to provide a light transmitter and a light receiver for each of the plurality of scanning planes, so that each scanning plane can be adjusted in a desired height grid or elevation grid. US 2017 / 0289524 A1 discloses an optical system for detecting distance information, wherein a light source arranged in series is provided on the transmitting side, and an image sensor is provided on the receiving side, which is behind a large receiving lens with an aperture array.

[0005] Cameras are an alternative to acquiring information optically. Compared to laser scanners, which detect angles up to 360°, cameras usually have a significantly narrower field of view. By using panoramic optics, a wider field of view can be achieved, up to a full 360° field of view. Aki, Mika Aikio, and Matti Kumpulainen, “Fisheye optics for omnidirectional perception”, 2014 IEEE 10th International Conference on Intelligent Computer Communications and Processing (ICCP), IEEE, 2014, use a fisheye lens. Alternatively, a catadioptrisch lens is also used, see for example Aikio, Mika, Jukka-Tapani and Bo Yang, “Omnidirectional camera,” 2013 IEEE 9th International Conference on Intelligent Computer Communications and Processing (ICCP), IEEE, 2013, or see Gimkiewicz, Christiane et al., “Ultra-miniaturecatadioptrical system for an omnidirectional camera,” Micro-Optics 2008, vol. 6992, International Society for Optics and Photonics, 2008.

[0006] In principle, this type of panoramic camera can also generate depth values ​​itself, i.e. can be configured as a 3D camera. For example, WO 2012 / 038601 A1 introduces a stereo camera with an omnidirectional lens. DE 20 2006 014 939 U1, DE 20 2011 052 106 U1 or EP 2 354 806 A1 respectively describe cameras with omnidirectional lenses based on the light transmission principle.

[0007] None of these systems can produce completely satisfactory 3D image data. The point cloud of a laser scanner usually has a lower resolution in the vertical direction than in the horizontal direction. Although stereo cameras can detect smaller objects horizontally and vertically, they have relatively low depth resolution due to their principle. Cameras based on time-of-flight with panoramic optics have been discussed in papers but still have to prove their worth in practice.

[0008] Thus, there is the possibility of combining systems. While the laser scanner is used to scan the geometry of the scene, the camera can detect the visual impression of the scene and thus detect properties such as lighting, material or texture. Alternatively, it is conceivable to also generate 3D data with the camera, so that the two systems complement each other in terms of geometry detection.

[0009] However, this requires a great deal of effort for calibration and subsequent data fusion. To this end, the so-called extrinsics must be known, i.e. the transformation between the optical center of the laser scanner and the camera. This transformation generally includes a translation in three spatial dimensions and three rotation angles in space, thus having six degrees of freedom. Determination of the transformation is called extrinsic calibration. This can also be described as determining the conversion rules between the corresponding local coordinate systems or the rules for conversion into the global coordinate system, for which the term registration is also commonly used.

[0010] In the article "Extrinsic calibration of a 3d laser scanner and an omnidirectional camera" by Pandey, Gaurav et al., IFAC Bulletin, Vol. 43.16 (2010): 336-341, a well-defined chessboard is used, which must be viewed from different angles. In addition, the camera system is equipped with multiple optical channels.

[0011] According to the article "Calibration between color camera and 3D LIDAR instruments with a polygonal planer board" by Park, Yoonsu et al., Sensors 14.3 (2014): 5333-5353, a precisely defined diamond-shaped object is used instead of a chessboard as a calibration target. A precisely defined calibration target may be difficult or expensive to obtain, or may not be available when it is needed, for example, during a measurement campaign. Furthermore, in this case the camera detects only a subsection of the scan area.

[0012] However, in Scaramuzza, Davide, Ahad Harati, and Roland Siegwart, “Extrinsic self calibration of a camera and a 3d laser range finder from natural scenes”, Intelligent Robots and Systems, 2007, IROS2007, IEEE / RSJ International Conference, IEEE, 2007, no calibration objects are used. Instead, the correspondences used for transformation estimation must be defined manually, which is time-consuming and can be empirically inaccurate.

[0013] DE 10 2017 109 039 A1 relates to the fusion of a camera and a laser scanner. In this regard, in addition to the 3D measurement points of the laser scanner, diffuse reflections are also detected with the laser scanner, and a mutual calibration is achieved by comparing the diffuse reflection image with the camera image. From GB 2507560 A, an external calibration of a camera and a laser scanner is known, wherein in this regard, the task is additionally made difficult by the fact that the system is in motion and the calibration also reconstructs the motion path from the detected measurement data. In US 2016 / 0070981 A1, in order to fuse the camera and the laser scanner on the vehicle, the reference data of the laser scanner are first localized according to the motion path and then localized in the camera image. In all these examples, the camera is not a panoramic camera.

[0014] In US2018 / 0139431 A1, panoramic images and depth data are detected and combined. In this regard, the panoramic image is obtained by moving a camera or multiple cameras arranged around. CN 105678783B describes a complex calibration model for any arrangement of panoramic cameras and laser scanners. Due to the large number of degrees of freedom, calibration can only be performed with great effort.

[0015] Recently, a new camera technology has been created, the so-called event-based camera, which is also called a neuromorphic camera based on the visual cortex. Conventional cameras expose all their pixels at a regular image refresh rate and then read out these pixels simultaneously to obtain a number of still two-dimensional images corresponding to the image refresh rate for each observed time period. In event-based cameras, there is also a pixel matrix, but there is neither a fixed image refresh rate nor a common pixel readout. Instead, each pixel is individually checked to see whether it is determined that the intensity has changed. Only in this case is the image information output or read out, and only from this pixel. Therefore, each pixel is an independent motion detector. The detected motion is reported separately as an event. As a result, event-based cameras react extremely quickly to dynamics in the scene. Image data cannot be so intuitively detected by the human eye due to the lack of a static image component, but can provide advantages in machine evaluation.

[0016] In this regard, an overview is given in the article “Neuromorphic vision chips” by Wu, Nanjian, Science China: Information Sciences 61.6 (2018): 060421. For example, event-based cameras are described in a white paper by Prophesee, which can be downloaded from its webpage. Pixel circuits for event-based cameras are known from WO 2015 / 036592 A1, WO 2017 / 174579 A1 and WO 2018 / 073379 A1, respectively. Gallego, Guillermo, Henri Rebecq and Davide Scaramuzza, in the article “Aunifying contrast maximization framework for event cameras, with applications to motion, depth, and optical flow estimation”, published in IEEE International Conference on Computer Virtual Pattern Recognition (CVPR), Vol. 1, 2018, proposes a method for determining motion, distance and optical flow from data of event-based cameras. US 2013 / 0335595 A1 discloses a device and a method for event-based image processing, in particular a device and a method for determining optical flow, which is not particularly important in the reading of codes.

[0017] Event-based image sensors have their advantages, but in this respect only replace conventional 2D image sensors and therefore do not yet make any contribution to improved 3D detection or even in particular to the fusion of laser scanners with panoramic cameras. Summary of the invention

[0018] It is therefore an object of the present invention to simplify the combination of a laser scanner and a camera.

[0019] This task is achieved by a method for detecting objects in the monitoring area using a photoelectric sensor and a laser scanner and a panoramic camera. The two combined systems - laser scanner and panoramic camera - have any known structure. The laser scanner emits one or more scanning beams, which are periodically guided on the scanning plane by a deflection unit rotating around the rotation axis or possibly pivoting back and forth around the rotation axis. These scanning beams are received again and the light flight time is measured. The 3D measurement point is obtained using the distance, the azimuth from the angular position of the deflection unit and the elevation from the known emission direction of the corresponding scanning beam. On the other hand, the panoramic camera includes a panoramic optical device and an image sensor with a large number of light receiving elements or pixels. Due to the panoramic optical device, image points are realized over a large angle range of at least 90° or even 180° up to 360°. Preferably, the angular ranges of the laser scanner and the panoramic optical device are at least almost the same.

[0020] The control and evaluation unit fuses the 3D measurement points of the laser scanner and the image points of the panoramic camera. The corresponding measured values ​​are thus associated with one another or converted into a common coordinate system, if necessary using additional steps, such as for filling gaps in the output data and / or the fused data or for smoothing interpolation. Thus, for example, depth values ​​or geometric structures are superimposed with diffuse reflection measurement values ​​or the texture of the scene. This presupposes (external) calibration, so the control and evaluation unit must know the corresponding transformation or the parameters for determining the transformation, either by a fixed mechanical structure, the input of corresponding parameters or a calibration phase. According to an embodiment, the control and evaluation unit is implemented as one or more functional blocks in the laser scanner, the panoramic camera or a system connected thereto.

[0021] The invention is based on the basic idea of ​​arranging the laser scanner and the panoramic camera on the same axis. The sensor is configured so that the rotation axis of the laser scanner and the optical axis of the panoramic camera are identical, i.e., lie on the same straight line. In particular, the optical axis of the panoramic camera is the axis of symmetry of the panoramic optics.

[0022] The invention has the advantage that particularly high-quality detection data are obtained by fusing a laser scanner with a scanning angle or azimuth of up to 360° with a camera. In particular, both the vertical resolution and also the lateral resolution of the laser scanner can be refined by the camera. As a result, a more reliable, faster and more accurate object recognition and object tracking can be achieved compared to known technical solutions. The camera has only one optical channel, but nevertheless detects the scanning field of the laser scanner as comprehensively as possible. Due to the special structure on a common axis, a relatively simple system is achieved and calibration is therefore greatly simplified. Complex calibration models are unnecessary.

[0023] Preferably, the panoramic camera is configured as a light time-of-flight camera. Thus, in addition to the color image data or grayscale image data of the panoramic camera, the panoramic camera also detects 3D points. In this way, it is possible to check whether the 3D measurement points of the laser scanner are reasonable, or to obtain 3D points with a higher resolution overall.

[0024] Preferably, the image sensor is arranged on the axis of rotation, in particular oriented perpendicularly to the axis of rotation and / or centered on the axis of rotation. This results in a rotationally symmetrical arrangement of the actual detection of the image points, which further simplifies the design and calibration. Alternatively, however, slightly offset or tilted positions of the image sensor can also be compensated by means of calibration.

[0025] Preferably, the control and evaluation unit is configured to associate at least one angular position of the deflection unit with a light receiving element on a radial line of the image sensor. This enables an association of the scanning angle of the laser scanner with the azimuth angle of the panoramic camera, so that an agreement is reached on a common angular coordinate relative to a rotation about the axis of rotation. The angular position is in particular a zero position, but it is also conceivable to determine and ensure other or further angular positions of the mentioned association. Preferably, the origin or reference point is a penetration point through the axis of rotation of the image sensor, wherein a translation of the common world coordinate along the axis of rotation or in any other direction is of course still possible.

[0026] In order to understand the connection, in particular with radial lines on the image sensor, it should be explained that the scanning planes of the laser scanner are imaged by the image sensor as concentric rings around the penetration point of the axis of rotation. Each radius starting from the penetration point then corresponds to a fixed azimuth angle or scanning angle. This identity of azimuth angle and scanning angle is a result of the design in which the axis of rotation and the optical axis coincide. Therefore, there is only one zero position or angle 0° to be calibrated, which of course is only a name for a general reference angle. The zero position is in turn determined by the geometry, so that with a fixed, calibrated arrangement of the image sensor and an angle of 0° for the laser scanner, an actual calibration can be omitted, since this is achieved by the combination. However, the zero position can also be calibrated at the factory or taught in that the laser scanner is held in its zero position and the control and evaluation unit positions the illuminated pixel on the image sensor and marks the corresponding radius as the azimuthal zero position.

[0027] Preferably, the control and evaluation unit is configured to associate the radial distance of the light receiving element from the optical axis with the scanning beam. Due to the special design of the sensor, this is the only degree of freedom that remains to be calibrated, in addition to the azimuth and the scanning angle, in which the corresponding scanning beam is associated with a specific one of the concentric circles (i.e. the radial distance from the penetration point) according to its azimuth at a certain object distance. Again, all of this is determined by the geometry, in this case mainly by the parallax, which in turn depends on the mutual distance of the laser scanner and the panoramic camera on the axis of rotation. As an alternative to the calculation and adjusted assembly of the calibration, a calibration procedure outside the factory or on site can be envisaged. For this purpose, a specific scanning beam of the laser scanner is activated and the concentric rings imaged on the image sensor are positioned with a constant object spacing. The discrete sequence of these concentric rings in the radial direction corresponds clearly to the order of the superimposed arrangement of the scanning beams in elevation.

[0028] Preferably, the laser scanner and the panoramic camera are arranged directly one above the other on the axis of rotation. The distance between the two systems is kept as small as possible in order to obtain low parallax and 3D measurement points and image points that are matched to one another as well as possible. The distance correction of the image points can be achieved in an unambiguous manner based on the corresponding distance information of the scanning points, the distance between the laser scanner and the panoramic camera, and the known imaging properties of the panoramic optics in the radial direction. In this regard, it is advantageous to configure the panoramic optics in such a way that the radial position on the image sensor is as proportional as possible to the elevation angle, also in order to achieve a resolution that is as constant as possible over the entire scanning range.

[0029] Preferably, the control and evaluation unit is configured to detect the light spot generated by the scanning beam on the image sensor and to associate the 3D measuring point and the image point with one another based on the light spot. This even allows dynamic calibration or tracking of the calibration during operation, in particular in order to make the time relationship of the corresponding measurement data particularly accurate.

[0030] Preferably, the image sensor is an event-based image sensor. The opening section briefly introduces this type of event-based or neuromorphic image sensors. These image sensors are particularly suitable for monitoring movable or rapidly changing scenes, such as in the case of mobile applications of sensors.

[0031] Preferably, the light receiving elements are configured to determine when the intensity detected by the respective light receiving element changes and then accurately provide image information in an event-based manner. The light receiving elements check whether the detected intensity changes. Only in the event, and only in the case of an event, is the image information output or read out. A hysteresis type can be envisaged, in which the light receiving elements ignore defined, too small intensity changes and do not regard them as events.

[0032] Preferably, the corresponding light receiving element provides the difference information whether the intensity decreases or increases as image information. Therefore, the information read from the light receiving element is, for example, a sign of +1 or -1, depending on the direction of change in intensity. Here, thresholds for intensity changes can be set, up to which the light receiving element does not trigger an event. In a subsequent internal representation for further evaluation, the value 0 can be supplemented for the time when no event was detected.

[0033] Preferably, the corresponding light receiving element provides the integrated intensity in the time window determined by the intensity change as image information. Here, the information is not limited to the direction of the intensity change, but the incident light is integrated in the time window determined by the event and the gray value is determined from this. The measured values ​​thus correspond to those of a conventional camera, but the time points of the detection are still event-based and associated with the intensity change.

[0034] Preferably, the corresponding light receiving element outputs image information with an update frequency of at least 1KHz or even at least 10KHz. The update frequency of a conventional camera is an image refresh frequency or a frame rate. An event-based camera does not have such a common image refresh frequency, because the light receiving element outputs or refreshes its image information individually and based on an event. However, there is an extremely short response time in this regard, which can only be achieved at a huge cost of thousands of images per second or more images using a conventional camera. In the case of an event-based camera, the update frequency may even be 10KHz or even tens of KHz, which is technically impossible to display using a conventional camera. This high temporal resolution also has the advantage of having virtually no motion artifacts (motion blurr). In an extremely short update cycle (corresponding to a conventional exposure time), the object no longer moves further multiple pixels, and therefore no blurred image is recorded. Preferably, extremely fast recording is used to detect the laser point with a high temporal resolution, and thereby achieve or track high-precision calibration.

[0035] Preferably, the laser scanner and / or the panoramic camera are configured as modules which can be removed from or added to the sensor, wherein in particular the control and evaluation unit is configured for mutual calibration of the laser scanner and the panoramic camera when the modules are subsequently added. Due to the modular construction, it is also possible to supplement the panoramic camera or the laser scanner at the operating site later if necessary. The calibration is then carried out, for example, by triggering a teach-in process. Due to the construction on a common axis, this is very easy to implement. In contrast, conventional universal calibration models would require calibration objects in the scene and very complex calculations with a correspondingly large amount of calibration data.

[0036] The method according to the invention can be developed in a similar manner and at the same time exhibits similar advantages. Such advantageous features are exemplary and non-exhaustive in the dependent claims following the independent claim.

[0037] Preferably, the fusion links the scanning angle of the scanning unit to the azimuth angle on the image sensor and links the elevation angle of the scanning beam to the radial distance from the rotation axis on the image sensor. It was explained above that the scanning plane of the laser scanner is imaged as concentric rings around the rotation axis on the image sensor based on this simple calibration and conversion: the radius corresponds to a fixed azimuth angle and scanning angle at the same time, and the radial distance of the individual rings is clearly associated with a specific scanning beam at a specific object distance. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Other features and advantages of the present invention are described in more detail below based on exemplary embodiments and with reference to the accompanying drawings. In the accompanying drawings:

[0039] Figure 1 shows a schematic cross-sectional view of a laser scanner;

[0040] Figure 2 shows a schematic cross-sectional view of a panoramic camera;

[0041] Figure 3 A schematic cross-sectional view showing a photoelectric sensor which is a combination of a laser scanner and a panoramic camera; and

[0042] Figure 4 Shown according to Figure 3 Illustration of the field of view of a sensor and its imaging on the image sensor of a panoramic camera. DETAILED DESCRIPTION

[0043] Figure 1A schematic cross-sectional view of an exemplary laser scanner 110 is shown. In a rough division, the laser scanner 110 comprises a movable scanning unit 112 and a base unit 114. The scanning unit 112 is an optical measuring head, while other elements such as a power supply, evaluation electronics, ports, etc. are also installed in the base unit 114. In operation, the scanning unit 112 is set in a rotational movement about an axis of rotation 118 by means of a drive 116 of the base unit 114 in order to periodically scan a monitoring area 120.

[0044] In the scanning unit 112, a light emitter 122 having a plurality of light sources 122a (e.g., LEDs or lasers in the form of edge emitters (Kantenemitter) or VCSELs) generates a plurality of emission light beams 126 with mutual angular offsets by means of a common emission optical device 124, which are emitted into the monitoring area 120. Alternatively, the plurality of emission light beams 126 can be generated by fewer or only one light source and at least one beam splitter element or pattern generating element (Mustererzeugungselement). The number of four emission light beams 126 is only an example and can be more or less, or even only a single emission light beam.

[0045] If the emission light beam 126 falls on an object in the monitoring area 120, the corresponding diffusely reflected light beam 128 returns to the laser scanner 110. The diffusely reflected light beam 128 is guided by the receiving optical device 130 to a light receiver 132 having a plurality of light receiving elements 132a, which respectively generate electrical receiving signals. The light receiving elements 132a can be individual components or pixels arranged in an integrated matrix, such as photodiodes, APDs (avalanche diodes) or SPADs (single-photon avalanche diodes).

[0046] exist Figure 1 In the embodiment shown, the light emitter 122 and the light receiver 132 are arranged together on a printed circuit board 134, which is located on the rotation axis 118 and is connected to the shaft 136 of the drive 116. This is to be understood only by way of example, and in fact any number and arrangement of printed circuit boards can be envisaged. The optical basic structure with a light emitter 122 and a light receiver 132 placed adjacent to each other in a double axis is also not mandatory and can be replaced by any structural form known per se for a single-beam photoelectric sensor or laser scanner. For this purpose, an example is a coaxial arrangement with or without a beam splitter.

[0047] A contactless power supply and data interface 138 connects the movable scanning unit 112 to a stationary base unit 114, in which a distance measuring unit 140 is located, which can also be arranged at least partially on the printed circuit board 134 or at another location in the scanning unit 112. The distance measuring unit 140 controls the light transmitter 122 and receives the received signal of the light receiver 132 for further evaluation. In addition, the distance measuring unit controls the drive 116 and receives the signal of an angle measuring unit (not shown) which is generally known from laser scanners and determines the corresponding angular position of the scanning unit 112.

[0048] Preferably, for evaluation, the distance to the scanned object is measured using the known light flight time method. Together with the information about the angular position from the angle measuring unit, the two-dimensional polar coordinates of all object points in the scan plane can be obtained after each scanning cycle using the angle and the distance. The individual scan planes are also identified by the corresponding diffusely reflected light beam 128. and its detection in one of the light receiving elements 132a is known, thereby scanning a three-dimensional spatial area as a whole. Therefore, the object position or object contour is known and can be output via the sensor interface 142, for example as a 3D point cloud output. In turn, the sensor interface 142 or another port (not shown) is used as a parameterization interface.

[0049] The laser scanner 110 shown has a rotating measuring head, i.e. a scanning unit 112. In this case, not only can the transmitting-receiving modules rotate together as shown here, but also other modules of this type with a height offset or an angle offset relative to the axis of rotation 118 are conceivable. Alternatively, a periodic deflection by means of a rotating mirror or a facet mirror wheel is also conceivable. This requires additional considerations, because how a plurality of transmission light beams 126 fall into the monitoring area 120 depends on the respective rotational position. These transmission light beams rotate around each other, so to speak, with the movement of the rotating mirror, but this is done in a predictable and therefore compensable manner in the 3D measuring point.

[0050] During the rotation of the laser scanner 110, each emission beam 126 scans a surface. Figure 1 In the case where the emission beams not present in the monitoring area 120 are horizontal, the plane of the monitoring area 120 is scanned. The remaining emission beams scan the outer surface of the cone, which is configured into different tips according to the deflection angle. In the multiple emission beams 126 deflected upward and downward at various angles, multiple hourglass nesting types are generated as scanning structures. Here, these cone outer surfaces are sometimes also referred to as scanning planes.

[0051] Figure 2 A cross-sectional view of an exemplary panoramic camera 210 is shown. Received light 214 is directed from a monitored area 212 via a panoramic optical device 216 to an image sensor 218. The panoramic optical device is capable of receiving light in a large azimuth range of up to 360°. For example, the panoramic optical device is designed as a fisheye lens or a catadioptric optical device or an omnidirectional optical device. Some specific embodiments are listed in the above introduction.

[0052] Preferably, the image sensor 218 is perpendicular to the symmetry axis 220 of the panoramic camera 210 or the panoramic optical device 216, and again preferably such that the point of penetration of the symmetry axis 220 is located at the center of the image sensor 218. In addition to the embodiment as a CCD or CMOS matrix with pixels for color or monochrome recording, the embodiment as event-based or neuromorphic image sensor mentioned at the beginning is also possible. Due to the extremely fast detection of up to 50 kHz or more of the event-based image sensor, such an event-based image sensor is particularly suitable for fast and continuous object tracking. This particularly advantageously supplements the relatively slow refresh rate of the laser scanner 110 (typically a rotation frequency of 50 Hz in terms of time diversity). In the combined system composed of the laser scanner 110 and the panoramic camera 210, which is still to be described, the spot of the moving emission light beam 126 can be recorded with high time resolution, so that the measurement points of the two systems 110, 210 can be temporally correlated with the high time resolution of the event-based image sensor.

[0053] The position of the corresponding pixel of the image sensor 218 carries the position information of the corresponding detected object point, namely the azimuth angle in the angular direction of the penetration point around the symmetry axis 220 and the elevation angle in the radial direction. This will be referred to below Figure 3 and Figure 4 Elaborate in more detail.

[0054] The image readout unit 222 is connected to the image sensor 218 and outputs the detected image points via an output terminal 224. In an alternative embodiment, the panoramic camera has an additional illuminator, which is coaxially coupled into the optical path of the panoramic optical device 216, for example by means of a frequency splitter mirror (Teilerspiegel), or the illuminator has its own panoramic optical device. Thus, the image readout unit 222 can be expanded and the light flight time can also be measured by appropriately modulated illumination. In practice, such 3D panoramic cameras usually already perform light flight time measurement or at least preparatory demodulation in the pixels of the image sensor 218, thereby at least partially eliminating the problem of Figure 2The subdivision shown in FIG. 21 is into two separate components 218, 222. With a 3D panoramic camera, 3D pixels are generated in addition to or instead of the color pixels or grayscale pixels for fusion to be described below.

[0055] Figure 3 A schematic cross-sectional view of an optoelectronic sensor 10 is shown as a combination of a laser scanner 110 and a panoramic camera 210. The two individual systems - the laser scanner 110 and the panoramic camera 210 - can be configured in one of the described ways or in another way known per se. From now on, these systems are only shown purely symbolically as functional blocks. The fusion and further processing of the corresponding measurement data can actually be distributed arbitrarily on the distance measuring unit 140 of the laser scanner 110, the image readout unit 222 of the panoramic optics, the common data processing unit 14 and / or other connected systems (e.g., a higher-level controller or even the cloud).

[0056] The laser scanner 110 and the panoramic camera 210 are integrated into the jointly constructed sensor 10, so that the rotation axis 118 and the symmetry axis 220 of the laser scanner 110 coincide on the common axis 12. The mechanical coupling is therefore such that the rotation axis 118 and the symmetry axis 220 are identical, or the image sensor 218 is perpendicular to the rotation axis 118 of the laser scanner 110. As a result, the external calibration and the association between the 3D measurement points of the laser scanner 110 and the image points of the panoramic camera 210 are greatly simplified.

[0057] Firstly, this is because only two systems have to be calibrated in total by using the panoramic camera 210. In addition, due to the special structure of the sensor 10, the scanning angle of the rotational movement of the laser scanner 110 and the azimuth angle of the panoramic camera are determined by a common angle. Here, only the convention about the zero position is missing. The corresponding altitude angle or elevation angle θ can also be associated with the radial distance just as easily. This is shown below in Figure 4 2 is depicted in a top view of image sensor 218 .

[0058] The sketch is in Figure 4 The right side of the figure is shown again in an enlarged manner. On the left side of the figure, the sensor 10 with a common monitoring area 16 is shown again, wherein in the common monitoring area 16, Figure 3 The respective monitoring areas 120, 212, which are displayed separately on a large display scale very close to the system, preferably overlap as completely as possible. This can be supported if the mutual distance a on the common axis 12 is selected to be as small as possible. In particular, a large angular range of at least 90°, at least 180° or even 360° or a suitable intermediate value (eg 270°) is detected.

[0059] exist Figure 4 On the right side of the diagram, it can be seen how the common monitoring area 16 is imaged onto a ring on the image sensor 218. The inner boundary circle with a radius of K1 corresponds to the top elevation angle, and the outer boundary circle with a radius of K2 corresponds to the bottom elevation angle. The fixed angle is The object point is imaged onto a line segment r along the radius. Where this is located depends largely on the parallax, so it is advisable to keep the distance a as small as possible. Therefore, the scanning beams 126, 128 describe a circle with a radius between K1 and K2 during their scanning movement according to their respective elevation angles θ, and the scanning beams 126, 128 superimposed on each other describe a line segment along the radius r at a fixed scanning angle.

[0060] Thus, the calibration can be limited to the adjustment of the scanning angle and the azimuth, for example using a common zero position and associating the points along the line segment r with the respective scanning beams 126, 128. These relationships are given by the geometry in the sensor 10 and can therefore be calculated. Alternatively, during the periodic scanning movement, the spots of the scanning beams 126, 128 are positioned on the image sensor 218 with the deflection unit 112 and / or the ring stationary. In particular when applying an event-based image sensor 218, this can even be dynamic in operation. Complex calibration models are no longer necessary.

Claims

1. A photoelectric sensor (10) for detecting an object in a monitoring area (16, 120, 212), wherein: The sensor (10) comprises: a laser scanner (110) having a deflection unit (112) rotatable about a rotation axis (118), the deflection unit being used to scan the monitoring area (120) using at least one scanning beam (126, 128); and a first distance measuring unit (140) being used to determine a 3D measurement point of a corresponding object scanned by the scanning beam (126, 128) using a light time-of-flight method; a panoramic camera (210) having a panoramic optical device (216) and an image sensor (218), the image sensor having a plurality of light receiving elements on radial lines for detecting image points; and a control and evaluation unit (14, 140, 222) configured to fuse the 3D measurement points and the image points, wherein the optical axis (220) of the panoramic camera coincides with the rotation axis (118) (12); and The control and evaluation unit (14, 140, 222) is configured to associate a plurality of angular positions of the deflection unit (112) with the plurality of light receiving elements on radial lines of the image sensor (218), so that the scanning planes of the laser scanner are imaged by the image sensor as concentric rings around a penetration point of the rotation axis, wherein each radius starting from the penetration point corresponds to one of the plurality of angular positions.

2. The sensor (10) according to claim 1, wherein: The panoramic camera (210) is configured as a light time-of-flight camera.

3. The sensor (10) according to claim 1 or 2, wherein: The image sensor (218) is arranged on the rotation axis (12, 118).

4. The sensor (10) according to any one of claims 1 to 2, wherein: The control and evaluation unit (14, 140, 222) is configured to associate a light receiving element at a radial distance from the optical axis with a scanning beam (126, 128).

5. The sensor (10) according to any one of claims 1 to 2, wherein: The laser scanner (110) and the panoramic camera (210) are arranged directly above one another on the rotation axis (12, 118).

6. The sensor (10) according to any one of claims 1 to 2, wherein: The control and evaluation unit (14, 140, 222) is configured to detect a light spot generated by a scanning beam (126, 128) on the image sensor (128) and to associate 3D measuring points and image points with one another based on the light spot.

7. The sensor (10) according to any one of claims 1 to 2, wherein: The image sensor (218) is an event-based image sensor.

8. The sensor (10) according to claim 7, wherein: The light receiving elements are configured to determine when the intensity detected by the corresponding light receiving element changes, and then accurately provide image information based on the event.

9. The sensor (10) according to any one of claims 1 to 2, wherein: The laser scanner (110) and / or the panoramic camera (210) are configured as modules that can be removed from or added to the sensor (10).

10. The sensor (10) according to claim 3, wherein: The image sensor (218) is oriented perpendicularly to the rotation axis (12, 118) and / or is centered on the rotation axis (12, 118).

11. The sensor (10) according to claim 8, wherein: The corresponding light receiving element provides difference information of whether the intensity decreases or increases as image information, and / or provides an integrated intensity within a time window determined by the change in the intensity.

12. The sensor (10) according to claim 9, wherein: The control and evaluation unit (14, 140, 222) is configured to calibrate the laser scanner (110) and the panoramic camera (210) relative to one another when modules are subsequently added.

13. A method for detecting objects in a monitoring area (16, 120, 212), wherein on the one hand a laser scanner (110) having a deflection unit (112) rotatable about a rotation axis (118) scans the monitoring area (120) with at least one scanning beam (126, 128) in order to determine 3D measurement points of the respective object scanned by the scanning beam (126, 128) using the light time-of-flight method, and on the other hand a panoramic camera (210) having panoramic optics (216) and an image sensor (218) with a plurality of light receiving elements on radial lines detects the monitoring area (120) in order to obtain image points, and wherein the 3D measurement points and the image points are fused by a control and evaluation unit (14, 140, 222), in, The optical axis (220) of the panoramic camera (210) coincides with the rotation axis (118) (12); and The control and evaluation unit (14, 140, 222) associates the multiple angular positions of the deflection unit (112) with the multiple light receiving elements on the radial lines of the image sensor (218), so that the scanning planes of the laser scanner are imaged by the image sensor as concentric rings around the penetration point of the rotation axis, wherein each radius starting from the penetration point corresponds to one of the multiple angular positions.

14. The method according to claim 13, wherein: The fusion relates the scanning angle of the laser scanner (110) to the azimuth angle on the image sensor (218), and relates the elevation angle of the scanning beam (126, 128) to the radial distance from the rotation axis (12, 118) on the image sensor (218).

Images