CONFIGURATION OF A 3D SENSOR FOR SECURE OBJECT TRACKING

DE502023000236D1Active Publication Date: 2024-11-21SICK AG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE502023000236
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Filing Date
2023-01-10
Publication Date
2024-11-21
Estimated Expiration
2043-01-10

AI Technical Summary

Technical Problem

Current 3D sensor configuration processes for secure object tracking in safety technology are complex, time-consuming, and prone to errors, particularly in industrial environments where high safety standards must be met, such as EN 13849 and IEC 61496, making them difficult to automate and requiring manual intervention.

Method used

An automated method for configuring a 3D sensor system that identifies a reference surface, such as the ground level, to define a detection area using 3D image processing and filtering techniques, allowing for the identification of moving objects and expanding the detection area to include sloping parts, with user confirmation and iterative refinement.

Benefits of technology

Simplifies the configuration process, reduces user complexity, and enhances safety by automating the detection area definition, reducing commissioning time and error rates, ensuring compliance with safety standards by accurately tracking moving objects and preventing accidents.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The invention relates to a method for automatically configuring a 3D sensor, in particular a 3D camera, for secure object tracking and to a 3D sensor system according to the preamble of claim 1 and 9 respectively.

[0002] A 3D camera measures a distance and thereby obtains depth information. The captured three-dimensional image data with distance or range values ​​for the individual pixels is also referred to as a 3D image, range image, or depth map. Various methods are known for determining depth information. A time-of-flight (TOF) camera, for example, illuminates a scene with amplitude-modulated light. The light returning from the scene is received and demodulated at the same frequency that is also used to modulate the transmitted light (lock-in method). After repeated measurements with different relative phase positions between the signals for the transmit-side modulation and the receive-side demodulation, the time-of-flight-related absolute phase shift between the transmitted and received signals can be determined. This phase shift is proportional to the object distance in the scene.Other 3D cameras use the stereoscopic principle, a light-section or projection method, or are designed as plenaptic cameras.

[0003] A special application of a 3D camera is safety technology for accident prevention, especially in industrial environments. This involves monitoring people in the vicinity of a hazardous location, such as a machine or, in particular, a robot. If a hazardous situation is detected, a safety-related response is initiated to prevent an accident. Sensors used in safety technology must operate with exceptional reliability and therefore meet stringent safety requirements, such as the EN1 3849 standard for machinery safety and the IEC 61496 or EN 61496 device standard for electro-sensitive protective equipment (ESPE). To comply with these safety standards, a number of measures must be implemented, such as secure electronic evaluation using redundant, diverse electronics, function monitoring, or, specifically, monitoring for contamination of optical components.The safety standards further specify so-called safety levels or safety requirement levels, which classify the achievable safety. Examples include SIL according to IEC 61508 / IEC 61511 (Safety Integrity Level), with 1 as the lowest and 4 as the highest, or PL (Performance Level) according to EN ISO 13849-1, with a "low" to e "high."

[0004] The most common protection approach is based on manually configured protective fields that are not accessible. A much more flexible approach is the detection of a safety distance violation based on the movement speed (speed-and-separation). However, this requires new safety functions that go beyond simple functions such as protective field evaluations. This includes object tracking, which determines the current position and movement history of the detected objects.

[0005] Such extended safety functions pose particular challenges even during commissioning and configuration. While it would be conceivable, in principle, to construct a detection zone for object tracking from 3D objects in a type of CAD program, similar to a conventional protective field configuration, this would be extremely complex and often overwhelming for users. In any case, it is time-consuming and error-prone. In the field of safety technology, however, errors are particularly critical because a source of danger may not be adequately protected, leading to accidents.

[0006] DE 10 2006 048 166 A1 discloses a method for observing a person in an industrial environment. A 3D model of the person is created based on 3D point data, for example, captured with a stereo camera. Knowing the position and movement behavior of a machine and the virtual body model, a hazard potential is determined and, if necessary, safety measures are taken on the machine. However, no configuration of the stereo camera is explained.

[0007] EP2 825 812 B1 describes a method and device for safeguarding a hazardous work area of ​​an automated machine. This method uses a hybrid of protective field monitoring and object tracking. This still requires the conventional manual configuration of protective fields.

[0008] DE 10 2005 063 217 B4 discloses a method for configuring a monitoring system for monitoring a room area. The floor is detected using so-called setup markers, which must be specifically placed in the scene. Protection is achieved using protective fields, which are then manually defined with respect to the floor in a type of CAD program, as described above.

[0009] In EP 3 203 263 B1, a laser scanner projects its currently configured protective fields onto the floor. For this purpose, the scanner's orientation can be determined using an inclination sensor, or its height can be determined using a detector, ensuring that the projection is accurate and without distortion from the laser scanner's perspective. The conventional protective field approach remains; object tracking is not provided. Even with regard to protective fields alone, the visualization merely facilitates their configuration, but does not change the fact that the appropriate protective field must be set manually.

[0010] EP 3 521 860 B1 describes a multi-layer laser scanner mounted on a vehicle that compares the detected contour with a ground model. The type of sensor and the mounting position are designed to detect obstacles in the direction of travel and are fundamentally unsuitable for object tracking.

[0011] From the work of Rougier, Caroline, et al., "Fall detection from depth map video sequences." Toward Useful Services for Elderly and People with Disabilities: 9th International Conference on Smart Homes and Health Telematics, ICOST 2011, Montreal, Canada, June 20-22, 2011. Proceedings 9. Springer Berlin Heidelberg, 2011, a 3D monitoring system is known that precedes ground plane detection. To avoid a computationally intensive RANSAC plane fit, a so-called V-disparity image is generated, based on row-by-row computed histograms of disparities.

[0012] EP 3 611 422 A1 discloses a sensor arrangement for securing a hazardous area around a machine. If a new object, such as an automated vehicle, enters the monitoring area, it is possible to dynamically generate and monitor an additional hazardous area.

[0013] US 2019 / 0138818 A1 discloses an automatic ground plane calibration. A machine learning algorithm is used to generate a depth map from 2D images. The ground plane is then derived from the depth map. The depth map and the ground plane are updated when a minimum period of time has elapsed since their generation and a background model has changed. The background model averages 2D images over a certain period of time to obtain an image that changes only very slowly or does not change at all.

[0014] US 2015 / 0269427 A1 discloses a multi-camera system that captures a workspace from multiple perspectives and detects people in synchronous shots. The search can be limited to an area of ​​interest, which can be defined as the visible floor plus a tolerance for a person standing at the very edge.

[0015] US 2015 / 0036887 A1 discloses a method for object recognition, wherein a ground surface is recognized based on pixels selected in a depth image that are at a distance corresponding to the ground plane. Temporal and spatial filtering of these pixels allows for more accurate detection of the ground plane.

[0016] Some image processing programs, and now even smartphones, offer the ability to eliminate moving objects from a shot by applying a median filter pixel by pixel to a series of images. This makes it possible, for example, to photograph a seemingly deserted landmark. However, intentionally ignoring people is the exact opposite of what security surveillance should achieve.

[0017] The known 3D configuration processes are therefore complex and not amenable to automation. Therefore, the object of the invention is to simplify the configuration process.

[0018] This problem is solved by a method for automatically configuring a 3D sensor for safe object tracking and a 3D sensor system according to claim 1 and 9, respectively. The 3D sensor can in principle use any desired sensor principle. It is a 3D camera, as briefly explained in the introduction. After configuration, it can be used for object tracking, with which the respective position of the moving objects within a detection range of the 3D sensor is determined, which in turn can be used as a prerequisite for further analysis for safeguarding a machine or accident prevention. The object tracking is safe; this preferably also applies to the 3D sensor itself. In this description, terms such as safe and safety mean fulfilling the requirements of a safety standard for non-contact protective devices or machine safety to prevent accidents.Examples of such standards are listed in the introduction, including other equivalent standards as well as regional variants or successor standards. A safety standard, in particular, specifies safety levels with clearly defined error control.

[0019] The 3D sensor captures at least one 3D image (range image, distance image, depth map) that encodes respective distances or separations in an XY grid of pixels. The specific representation is not important. By evaluating the 3D image, a detection zone is defined within the detection area. Object tracking should only take place within the detection area, because no movement of safety-relevant objects is expected in other parts of the detection area. The goal of the configuration is to find this detection zone, in which potentially safety-critical objects, such as people, vehicles, machine parts, and the like, are moving.

[0020] The invention is based on the basic idea of ​​finding the detection area based on a reference surface on which objects will move. In most cases, this is the ground plane, or simply the floor. For this purpose, those pixels in the 3D image that correspond to the reference surface are identified. The detection area is then determined from these pixels. This is not necessarily just the union of the pixels; filters can also be provided that expand the detection area, round it off, or, for example, exclude inaccessible areas of the reference surface.

[0021] The process runs automatically; it is a computer-implemented process. For this purpose, at least one computing unit is provided in the 3D sensor and / or connected to it at least temporarily for configuration or permanently for further safety monitoring. Examples of an internal computing unit are digital computing components such as a microprocessor or a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), an ASIC (Application-Specific Integrated Circuit), an AI processor, an NPU (Neural Processing Unit), a GPU (Graphics Processing Unit), or the like. A connected computing unit can be a computer of any type, including notebooks, smartphones, tablets, a (safety) controller, a local network, an edge device, or a cloud.

[0022] The invention has the advantage of significantly simplifying configuration and avoiding unnecessary complexity for the user. This allows configuration to be performed even by a user who lacks in-depth knowledge of the specific sensor technology used. Automation not only significantly shortens commissioning time, but the reduced complexity also prevents errors from the outset, thus ensuring greater safety.

[0023] Preferably, the 3D position of the reference surface is determined by fitting a plane that best matches the pixels recorded in the at least one 3D image. The 3D position in space refers to the orientation and position of the reference surface, for example, relative to a camera coordinate system or a world coordinate system in which the camera position is known. Each pixel of the 3D image contains the distance or Z value measured at its XY position. It is therefore algorithmically possible to find a reference surface that best matches the pixels (plane fit). Tolerances are permitted in this process; the pixels will not lie exactly on a mathematical plane in the real 3D image of a real scene. Preferably, the largest possible reference surface with the most matching pixels is sought.Illustratively, the floor will comprise most of the pixels, which will have a certain amount of noise between them, but this is a much more meaningful result than a small, almost perfectly flat measured area of, for example, a shelf.

[0024] The 3D sensor preferably measures its own orientation using a position sensor. From this, an expected orientation of the reference surface is derived, and the expected orientation is used to identify pixels corresponding to the reference surface. This is an alternative or complement to a purely algorithmic search for the reference surface. From the 3D sensor's own orientation, an expectation of the orientation of the reference surface or the ground can be derived. This, in turn, is a strong starting point for an algorithm that identifies the pixels corresponding to the reference surface, for example, by forming distance ranges with respect to the expected orientation in the correct perspective and counting the corresponding pixels. If the mounting height of the 3D sensor is also calibrated or measured in advance, the pixels can be directly classified as matching or not matching.

[0025] An image sequence with numerous 3D images is recorded while objects move through the detection area. The configuration is then no longer based solely on a single 3D image, but rather on a chronological sequence of 3D images. This provides more information for an even more precise configuration. In particular, moving sections can be distinguished from stationary ones, and the configuration is prevented from being based on a 3D image captured in an unfavorable constellation. To support the configuration, people and other safety-relevant objects can perform typical movements during the recording of the image sequence, which are as representative as possible of the conceivable movements.

[0026] Using the 3D images from the image sequence of the reference surface, corresponding pixels are repeatedly identified to detect even temporarily obscured parts of the reference surface. By observing the detection area over a certain period of time and evaluating the data accordingly, occlusions are resolved because the objects have moved further.

[0027] Pixels are identified in which the measured distance across the 3D images of the image sequence changes by more than a noise tolerance, and these pixels are assigned to the detection area. A change in the distance value that is more than noise-related means that an object has moved at the position of the affected pixel. Such change areas can form the detection area or a core detection area as a starting point for defining the detection area, or the previous detection area can be expanded accordingly. When fitting a reference area, the furthest measured distance values ​​in a change area are preferably used as a basis, because shorter distances are due to moving objects. Change areas do not necessarily belong to the reference area, as the example of a person near a wall shows.Part of this person is captured above the floor, while another part is captured in front of the wall. Both are identified as areas of change.

[0028] For neighboring pixels corresponding to the reference surface, it is preferably checked whether the measured distance values ​​are compatible with the distance to the reference surface within a distance tolerance, and a neighboring pixel for which this is the case is added to the detection region. This is a type of region growing function. The initially or previously determined detection region with pixels corresponding to the reference surface is expanded to include neighboring pixels, provided that these have measured a similar distance within the distance tolerance. This condition can at least partially overlap with the fitting of a plane or the original identification of pixels corresponding to the reference surface, depending on the algorithm used. Typically, however, filling will set a new condition that leads to a different result.Furthermore, the filling process can be iterated one or more times, so that neighbors of neighbors in increasingly higher orders are also included, as long as the distance tolerance is met. At least the accumulated distance tolerance will differ from the original membership condition of an image point to the reference surface.

[0029] The distance tolerance is preferably chosen large enough so that a stair or inclined ramp is still compatible with the reference surface. Thus, a relatively generous distance tolerance is intentionally chosen. The distance tolerance can, for example, be selected from the interval 5–50 cm. This is also conceivable for identifying the original reference surface, but carries the risk of resulting in a supposed reference surface that bears little resemblance to a floor plane. Gradual filling, on the other hand, can be very effective, especially for subsequently expanding the reference surface once it has been identified to include inclined sections, for which a staircase or ramp is mentioned here as examples.

[0030] Preferably, the configured detection area is displayed, and confirmation is requested before the detection area is used for secure object tracking. The display is provided, for example, as a 2D or 3D image, preferably as an image of the scenery with an overlaid marking of the configured detection area. Confirmation gives the user or security expert the final say as to whether the detection area provides the required protection. Such final approval may be a requirement of a safety standard, regardless of the performance of an automated configuration procedure. Beyond mere confirmation, functions for subsequently correcting or editing the detection area can be offered to the user.This can even go so far that the automatic configuration is only a first step in the configuration, which, however, already provides very targeted raw data with which the user can finally define the detection area.

[0031] Following the configuration of the detection area, additional 3D images are preferably recorded cyclically and objects are tracked in them in order to output the current position of moving objects in the detection area as a safety function. In other words, the configuration is followed by the actual operation, in which the configured detection area is used to safely track objects. This ensures that the current position of all safety-relevant objects is always reliably known. It is conceivable to use 3D images recorded during operation to further refine the configuration of the detection area. However, as explained in the previous paragraph, approval may be required before such an adjustment of the detection area is actually used for further operation.

[0032] The positions of the objects are preferably evaluated, particularly taking into account their previous movement, in order to decide whether there is a risk of accident. The safe object positions are not an end in themselves, but the basis for hazard monitoring. In particular, they can be used to assess whether an object is coming too close to a machine or other source of danger. The movement history is also known from object tracking and can be included in the risk assessment, particularly the direction of movement and / or speed. If a risk of accident is detected, a safety measure is preferably taken, in particular to put a monitored machine into a safe state. Depending on the situation, the machine can be slowed down or stopped completely, or an evasive movement can be initiated.

[0033] The 3D sensor system according to the invention comprises at least one 3D sensor and at least one control and evaluation unit in which one of the explained embodiments of the method for automatically configuring the 3D sensor for secure object tracking is implemented. The 3D sensor is, in particular, a 3D camera as introduced above. A network system comprising several 3D sensors is also conceivable. The possible configurations of the control and evaluation unit correspond to the computing units explained above on which the automatic method can run. The 3D sensor system is secure in the sense of a security level of a relevant security standard.

[0034] The control and evaluation unit is preferably further configured to perform secure object tracking following configuration by evaluating cyclically recorded 3D images. This reliably provides current positions and, if necessary, other movement behavior of the safety-relevant objects for safety-related monitoring.

[0035] The invention will be explained in more detail below with regard to further features and advantages, using exemplary embodiments and with reference to the accompanying drawings. The figures of the drawing show: Fig. 1 shows a schematic block diagram of a time-of-flight camera as an example of a 3D sensor; Fig. 2 shows an example 3D image recorded by a 3D sensor with objects detected therein; Fig. 3 shows an example flowchart for the configuration of a detection area for object tracking with a 3D sensor; Fig. 4 shows an example alternative flowchart using a position sensor; Fig. 5 shows an example image to explain image areas in which a movement of a safety-relevant object takes place outside of a reference surface; Fig. 6 shows an example flowchart in a further alternative with determination of changes in measured distances; and Fig. 7 shows an example 3D image recorded by a 3D sensor with safety-relevant moving objects and irrelevant static objects.

[0036] Figure 1shows a schematic block diagram of a time-of-flight camera as an example of a 3D sensor 10. An illumination unit 12 emits modulated transmitted light 16 into a detection area 18 through a transmission optics 14. LEDs or lasers in the form of edge emitters or VCSELs can be considered as the light source. The illumination unit 12 can be controlled such that the amplitude of the transmitted light 16 is modulated at a frequency in the range of 1 MHz to 1000 MHz. The modulation is, for example, sinusoidal or rectangular, or in any case, a periodic modulation.

[0037] If the transmitted light 16 strikes an object in the detection area 18, a portion is reflected back to the 3D sensor 10 as received light 20 and there guided through a receiving optics 22, for example a single lens or a receiving objective, to an image sensor 24. The image sensor 24 has a plurality of receiving elements or receiving pixels 26, arranged, for example, in a matrix or row. The resolution of the image sensor 24 can range from a few to thousands or millions of received pixels 26. The pixel arrangement is preferably a matrix, resulting in a lateral spatial resolution in an X-direction and a Y-direction, which is complemented by the Z-direction of the distance measurement to the three-dimensional image data. The received pixels 26 are each designed for demodulation according to a lock-in method.By performing several measurements with different phases between the modulation frequency used for the transmitted light 16 and the modulation frequency used for demodulating, the absolute phase offset due to the light propagation time through the detection area 18 is reconstructed, which is converted into a distance value per received pixel 26.

[0038] A control and evaluation unit 30 is connected to the image sensor 24 and the illumination unit 12. The functionality for modulating the transmitted light 16 and demodulating the received light 20 with a suitable timing for a time-of-flight measurement can be distributed in any manner between the control and evaluation unit 30 and the illumination unit 12, or between the image sensor 24 and its received pixel 26. As a result of the image acquisition and time-of-flight measurement, a 3D image, distance image, or depth image is generated. The 3D image is evaluated in the control and evaluation unit 30 itself or output unprocessed or preprocessed at an interface 32. The interface 32, or alternatively, one or more other connections (not shown), conversely serve to input control signals or parameterizations of the 3D sensor 10.

[0039] The to Figure 1The time-of-flight camera explained is only one example of a 3D sensor 10. Other 3D sensors are conceivable, in particular the variants of a 3D camera briefly mentioned above, and in principle also non-optical 3D sensors. Instead of just one 3D sensor 10, a sensor network can be used. The invention does not essentially concern the design of the 3D sensor 10, but rather its configuration for reliable object tracking, which will be described below with reference to Figures 2 to 7 The corresponding configuration method is implemented in any desired distribution in the control and evaluation unit 30 of the 3D sensor and / or a computing unit connected thereto.

[0040] Figure 2shows an example 3D image captured by a 3D sensor with objects detected in it. Object detection and thus object tracking are limited to a detection area 34 within the entire detection area 18 and thus the 3D image, in which safety-relevant objects 36 may be located. This means that, in particular, static objects 38 at the edge are ignored. Finding the appropriate detection area 34 for reliable protection is the goal of the configuration to be described below.

[0041] Based on the configured detection area 34, secure object tracking, i.e., object tracking as a safety function, can then be implemented during subsequent operation. This involves detecting moving objects 36 with sufficient reliability for safety applications and thus accident prevention for personal protection, and tracking their movement. With each detection cycle, a current position of the detected objects 36 in the image is obtained. In addition, the movement history is known, from which additional information such as the direction of movement and speed can be derived. In the case of secure object tracking integrated into the 3D sensor 10, positions and, if necessary, additional information can be provided cyclically at the interface 32.The results of object tracking can be used for autonomous path planning of vehicles and for the localization of objects and people in a large operational environment. The data is also diversely redundant with the location data of a radio positioning system and can be used for its verification.

[0042] The moving, at least potentially (safety-)relevant objects 36 are detected in relation to a reference surface 40, preferably the ground. For example, in each detection cycle, the scene captured in the 3D image is compared with a known reference scene in order to identify the relevant objects 36 based on the geometric differences. This requires that the configuration precisely defines the location of the reference surface 40, including its position and orientation in space. In addition, the detection area 34 defines areas outside the detection area where detection is irrelevant. This is Figure 2represented by the static objects 38 at the edge. Such irrelevant objects can be, for example, shelves, pallets, or boxes, where it is clear from the outset that they are not suitable as movement areas for relevant objects 36. Therefore, corresponding areas can be masked out or classified as stationary obstacles to movement. This reduces the effort and potential misdetections.

[0043] Figure 3 shows an example flow chart for the configuration of the detection area 34. After the mechanical attachment and electrical connection of the 3D sensor 10, the secure object tracking function is configured in an automatic process.

[0044] In a step S1, the 3D sensor 10 captures one or more 3D images of the scene in the detection area 18. Multiple 3D images can be used to repeat a single image evaluation. However, an evaluation of multiple 3D images is preferably performed comprehensively, for example, to compensate for obscured areas in individual 3D images with other 3D images.

[0045] In a step S2, a reference surface 40 is fitted into the at least one 3D image. Various 3D algorithms are known for this purpose, which, for example, in an optimization process involving all or only a portion of the pixels, minimize an error measure of the remaining offsets from the reference surface 40 by rotating, shifting, and changing the shape and size of the reference surface 40. Preferably, the fitting is performed under the assumption that the reference surface 40 is the largest surface or plane that is compatible with the pixels of the 3D image in terms of small residual offsets or another error measure.

[0046] In step S3, the pixels that captured the reference surface 40 are identified. The criterion for this is that the distance measured from the pixel is, at least up to a noise tolerance, the same as that of the reference surface 40. The two steps S2 and S3 are clearly separated, primarily for the sake of explanation. Depending on the algorithm, the fitting of the reference surface 40 and the finding of the pixels corresponding to the reference surface 40 occur at least partially simultaneously.

[0047] In a step S4, the detection area 34 is defined based on the pixels of the reference surface 40 identified in step S3. In the simplest case, this can simply be the union of these pixels. Preferably, additional processing is performed to expand, round, or reduce the detection area 34, for example, to adjust measurement outliers to their surroundings, to give the detection area 34 a regular shape, or to exclude irrelevant objects 38.

[0048] Finally, in step S5, the configured detection area 34 is displayed for the user to confirm its correctness. This adds another level of security, and final responsibility for safety remains with the user or safety expert, as required by many safety standards. The display can be shown, for example, as a 2D or 3D image of the scene with the detection area 34 superimposed. At this point, the user can be given the opportunity to modify the detection area 34 before release. Tools similar to those used in a drawing or CAD program are provided for this purpose.

[0049] The procedure according to Figure 3is a simple way of automatically configuring the detection area 34 with pixel-precise differentiation of the reference area 40 to be monitored from irrelevant areas that can be masked out. Various alternatives, modifications, and extensions are explained below.

[0050] Figure 4 shows an exemplary alternative flow diagram using a position sensor. The position sensor (e.g., IMU, Inertial Measurement Unit) is preferably integrated into the 3D sensor 10 and determines its orientation in space, in particular relative to the Earth's gravitational field. Figure 4 The process shown corresponds in principle to that of the Figure 3 , so only the differences are explained.

[0051] Following the acquisition of a 3D image in step S1 or prior to or in parallel thereto, the orientation of the 3D sensor 10 is determined using the position sensor in a step S6. If the mounting position of the 3D sensor 10 does not change, a one-time or infrequent determination of the orientation is sufficient, whereas, for example, for mobile applications of the 3D sensor 10, a current determination of the orientation should be made. Provided that the reference surface 40 is oriented perpendicular to the Earth's gravitational field, as in the case of the ground, or alternatively that a deviation is known, the orientation of the reference surface 40 relative to the 3D sensor 10, and thus in the respective 3D image, can be predicted from the orientation of the 3D sensor 10.

[0052] Accordingly, steps S2 and S3 are carried out according to Figure 3modified. In a modified step S2a, the fitting of the reference surface 40 now takes place with knowledge of the orientation of the reference surface 40 determined with the position sensor. This significantly limits the search space of an optimization method with which the reference surface 40 is fitted. Accordingly, in a modified step S3a, the pixels corresponding to the reference surface 40 are identified with less effort and, as a rule, more precisely.

[0053] It is conceivable to additionally determine the orientation of the reference surface 40 algorithmically, as in step S2 of the Figure 3 explained. This means that the Figures 3 and 4 on the one hand, they represent alternatives, but also a possible combination. In combination, the reliability of the configuration is further increased through redundant determination or plausibility check options, which is particularly valuable in safety-related applications.

[0054] The possibility of capturing and evaluating multiple 3D images or an image sequence has already been briefly discussed. On the one hand, this allows a certain averaging of measurement errors in a static scene. However, it is preferable to extend the configuration sequence to include an additional movement phase. During this phase, the user is required to move the people and moving objects in the accessible areas as an example. For example, people should walk through the area or vehicles should cross the detection area 18 along their programmed path. During the movement phase, a continuous series of images is then captured, and the entire image series is searched for pixels that are compatible with the reference surface 40. The movement ensures that all pixels belonging to the movement area are free at some point and are identified as corresponding to the reference surface 40.This avoids dead zones in which people or moving objects temporarily obscure part of the reference surface 40 and would therefore be incorrectly excluded from the detection area 34, or during a movement phase. Furthermore, the distances measured with significant differences due to movement can serve as an important indicator for belonging to the detection area 34. In addition to a dedicated movement phase, it is also possible to continue the configuration parallel to the actual operation and to collect data for further optimization of the configuration.

[0055] Figure 5illustrates a special situation with a relevant object 36 at the edge of the reference surface 40, for example, near a wall. The person's head area is detected outside the reference surface 40. In this case, it may be useful to extend the detection area 34 beyond the reference surface 40 and to add pixels that detect the head area or the wall behind it. Such pixels are recognized, for example, by the fact that the distance values ​​measured by one and the same pixel change significantly during a movement phase of the configuration, for example, by more than a noise tolerance. This idea will now be explored in more detail, as it allows for further advantages to be achieved.

[0056] Figure 6shows an exemplary flow diagram of the configuration with determination of changes in measured distances. In a step S11, an image sequence with a plurality of 3D images is recorded, corresponding to the previous step S1, preferably during a movement phase in which relevant objects 36 move in the detection area 18. In a step S12, optionally and analogous to step S6, the Figure 4 The orientation of the 3D sensor 10 and thus the orientation of the reference surface 40 can be determined by means of a position sensor.

[0057] In step S13, areas of change in the detection area 18 are determined. These are pixels or groups of pixels in which the measured distance value changes significantly over the image sequence. The criterion for this can be the exceeding of a noise tolerance.

[0058] In a step S14, a reference surface 40 is fitted analogously to step S2 or S2a. If the optional step S12 was previously performed, its orientation is known. Based on the change ranges determined in step S13, the distance of the reference surface 40 can be determined. Figure 5In the special situation explained above, the change regions can be evaluated individually beforehand in order to distinguish between pixels in change regions that correspond to the reference surface 40 and that indicate the distance of the reference surface 40, and those that do not, for example, due to an edge location on a wall. Thus, prior knowledge about the orientation and / or distance of the reference surface is available for fitting the reference surface 40, whereby alternatively or additionally the reference surface 40 can be fitted without prior knowledge, as in step S2. A further alternative is to no longer specifically fit a reference surface 40, but to derive the reference surface 40 directly from the change regions.

[0059] In a step S15, the Figures 3 and 4A preliminary detection area 34 is defined based on the change areas and / or the reference area 40. In this case, the pixels corresponding to the reference area 40 can be found beforehand, analogous to steps S3, S3a.

[0060] In an optional step S16, the preliminary detection area 34 can be filled to determine the entire accessible area, which is why the result of step S15 was referred to as a preliminary detection area 34. Starting from the previously determined detection area 34 and thus already known areas of the reference surface 40 in the adjacent neighborhood of pixels, a check is made to determine whether a neighboring pixel also belongs to the reference area 40 and, if so, is added to the detection area 34. The criterion here is preferably that the distance value measured in a neighboring pixel differs by at most one distance tolerance from its neighboring pixel already belonging to the detection area 34. This neighborhood check can be repeated iteratively until all accessible areas have been detected in the expanded detection area 34.

[0061] By allowing a larger distance tolerance, detection zones that do not lie in a single plane can also be automatically configured. For example, if a staircase lies within the detection zone 18, the individual steps are defined as a change zone during the movement phase in step S13 and later added to the detection zone 34 in step S16. An inclined ramp can be added in a similar way. In principle, a similar result can be achieved by fitting a reference surface 40, which may exhibit a certain inclination or curvature. However, in contrast to iterative filling, this creates difficulties in appropriately parameterizing the optimization algorithm to ensure that only suitable reference surfaces 40 are found.

[0062] In a step S17 concluding the configuration, the configured detection area is displayed, edited if necessary, and released analogously to step S5.

[0063] Figure 7 shows another example of a 3D image captured by the 3D sensor 10 with safety-relevant moving objects 36 and irrelevant static objects 38. The static objects 38 can result in certain parts 42 of the floor, and thus of the reference surface 40, actually being inaccessible. They should therefore preferably be excluded from the detection area 34. This can only be achieved subsequently with greater algorithmic effort, which tests possible paths within the detection area 34. Through the iterative filling in step S16 starting from change areas, such inaccessible parts 42 are not added to the detection area 34 in the first place.

[0064] Although the Figures 3, 4 and 7While each of the steps separately shows an exemplary process, numerous mixed forms are conceivable that transfer individual or multiple steps from one process to the other, either replacing or cumulatively. This applies in particular to the use of 3D image sequences instead of individual 3D images, the use of a position sensor to determine the orientation of the reference surface 40, the fitting of the reference surface 40 with and without prior knowledge of its orientation and / or distance, the locating of change areas as starting points for pixels of the detection area 34, or the filling of this detection area 34 based on neighborhood comparisons.

Claims

1. A method for automatically configuring a 3D sensor (10) for safe object tracking with the aim of the configuration to find a detection area (34) within a detection range (18) of the 3D sensor (10) in which possible safety-critical objects move and to which the object tracking is restricted, wherein at least one 3D image is recorded with the 3D sensor (10) and the detection area (34) is determined on the basis of the at least one 3D image, wherein the term safe means the fulfillment of requirements of a safety standard for non-contact protective equipment or machine safety for accident prevention, characterized in that in the at least one 3D image, image points are identified that correspond to a reference surface (40), in particular floor plane, for which the distance measured by the image point is that of the reference surface (40) except for a noise tolerance, and the detection area (34) is defined as the union of these pixels or as the union of these pixels processed with a filter, the detection area (34) being expanded, rounded or reduced, being given a regular shape, or inaccessible areas of the reference surface (40) or irrelevant objects (38) being excluded by the filter, and in that an image sequence with a plurality of 3D images is recorded while objects (36) move through the detection area (18), in that image points corresponding to the reference surface (40) are repeatedly identified on the basis of the 3D images from the image sequence, in order to also detect temporarily obscured parts of the reference surface (40) because objects have moved on, and in that image points are determined in which the respectively measured distance over the 3D images of the image sequence changes by more than a noise tolerance, and these image points are added to the detection area (34).

2. The method according to claim 1, wherein the 3D position of the reference surface (40) is determined by fitting a plane which corresponds as closely as possible to the pixels recorded in the at least one 3D image.

3. The method according to claim 1 or 2, wherein the 3D sensor (10) measures its own orientation by means of a position sensor, an expected orientation of the reference surface (40) is derived therefrom, and the expected orientation is used to identify pixels corresponding to the reference surface (40).

4. The method according to any of the preceding claims, wherein the detection area (34) is filled up by checking, starting from the previously determined detection area (34), in the adjacent neighborhood of pixels whether the distance value measured in an adjacent pixel differs by at most a distance tolerance from its adjacent pixel already belonging to the detection area (34), and wherein this neighborhood check is repeated, in particular iteratively.

5. The method according to claim 4, wherein the distance tolerance is selected large enough that a step or inclined ramp is still compatible with the reference surface (40).

6. The method according to any of the preceding claims, wherein the configured detection area (34) is displayed and a confirmation is requested before the detection area (34) is used for safe object tracking.

7. The method according to any of the preceding claims, wherein, following the configuration of the detection area (34), further 3D images are cyclically recorded and objects (36) are tracked therein in order to output the respective current position of moving objects (36) in the detection area (34) as a safety function.

8. The method according to claim 7, wherein the positions of the objects (36) are evaluated, in particular taking into account the previous movement, in order to decide whether there is a risk of an accident, in particular in order to place a monitored machine in a safe state in this case.

9. A 3D sensor system having at least one 3D sensor (10) and having at least one control and evaluation unit (30), with a method for automatic configuration of the 3D sensor (10) for reliable object tracking according to any of the preceding claims being implemented in the control and evaluation unit (30).

10. The 3D sensor system according to claim 9, wherein the control and evaluation unit (30) is furthermore configured to carry out reliable object tracking following the configuration by evaluating cyclically recorded 3D images.