METHOD FOR AUTOMATIC CALIBRATION OF A NETWORK OF IMAGERING DEEP SENSORS

Abstract

Method for the automatic calibration of a network (5) of imaging depth sensors (1, 4), wherein, for the creation of a topology, detection ranges (7, 8) of several depth sensors (1, 4) of the network (5) are directed towards different areas of a space to be jointly monitored, at least one marker is placed in the space to be monitored, and the depth sensors (1, 4) each determine whether the marker is located in their detection range (7, 8), wherein at least one depth sensor (1, 4) projects a reference image (3) as a marker into the space to be monitored by means of a projector (2), characterized in that the overlap of a detection range (7) of one depth sensor (1) into the detection ranges (8) of the other depth sensors (4) is detected, and the depth sensors (1, 4) determine a position and orientation of the marker in a separate,The inner coordinate system is captured and, using a central evaluation unit (6), the position and location of at least one marking in the inner coordinate systems of the depth sensors (1, 4) are related to each other and / or to a common outer coordinate system by determining or estimating a transformation.

Description

[0001] The present invention relates to a method for automatically calibrating a network of imaging depth sensors, wherein, in order to create a topology, detection areas of several depth sensors of the network are aligned to different areas of a space to be monitored together, at least one marker is set in the space to be monitored and the depth sensors each determine whether the marker is located in their detection area, wherein at least one depth sensor projects a reference image as a marker into the space to be monitored by means of a projector.

[0002] Such a method is already known from US 2012 / 0194650 A1. Reference is also made to EP 3 171 129 A1.

[0003] Other known methods use a combination of natural two-dimensional and three-dimensional features, moving objects, manually placed markers within the monitored space as markers, or they dispense with such markers entirely in favor of entirely manually selected point correspondences. Such markers are suitable for detection by depth sensors. They can be used to determine overlaps between depth sensors, and it is possible to create a topology of the depth sensors by determining transformations between the individual sensors.

[0004] However, these known methods are highly dependent on the specific scene. This can depend on the existing two-dimensional or three-dimensional features, but also on the walking paths of people, or in the case of manually selected marker points, which can represent prominent spatial positions such as furniture or wall corners, floor patterns, and the like, on the manual intervention of a user. The only decisive factor for manual selection is that such a marker point can be detected and identified in all participating sensor images.

[0005] In general, such a sensor network serves to seamlessly record situations in a larger, possibly even branched, spatial area. Cross-sensor tracking of people is the basis of many applications, such as people counting in public buildings, digital customer flow analysis in supermarkets, at trade fairs, and the like, or public safety.

[0006] However, before deploying such a sensor network, the sensors must be calibrated, which has so far been achieved with extensive manual work and numerous iterations of movement sequences. The most promising approaches in this regard are those that use people moving in the monitored space to determine point correspondences. If sufficient walking paths are recorded over a longer period of time, which can take several hours depending on the frequency of the scene, and if it is possible to associate the individual walking paths of disjoint sensors with each other, automatic calibration can also be achieved this way.

[0007] However, the methods described above have some limitations. First, the result depends on the number of people and the walking paths in the space to be monitored. Second, in heavily frequented areas, it can be very challenging to associate the individual walking paths of different sensors with each other. This problem is of combinatorial exponential complexity and can only be approximated in practice using robust model estimations. Calibration is based on correspondences between person detections. However, these are subject to measurement inaccuracies and are therefore only of limited use as point correspondences alone.

[0008] Based on this, the present invention is based on the object of automatically carrying out a calibration of a network of imaging depth sensors, in particular without both the distribution of marks and two- or three-dimensional markings in space, as well as without manual intervention in the evaluation of the sensor images to find suitable natural markings in space.

[0009] This object is achieved by a method according to the features of independent claim 1. Useful embodiments of such a method can be found in the subsequent dependent claims.

[0010] For this purpose, a method is provided for the automatic calibration of a network of imaging depth sensors. To create a topology, the detection ranges of several depth sensors in the network are aligned with different areas of a space to be monitored jointly. At least one marker is placed in the space to be monitored, and the depth sensors each determine whether the marker is located within their detection range. According to the invention, such a method is characterized in that at least one depth sensor projects a reference image as a marker into the space to be monitored using a projector, and the overlap of a detection range of one depth sensor with the detection ranges of the other depth sensors is detected.

[0011] What is essential here is that the invention uses a projector associated with and optionally integrated into the depth sensors, generally stereo sensors, RGBd, or time-of-flight sensors with additional imaging. Instead of manually applying markers in the space to be monitored, the stereo sensors are each capable of projecting a reference image into the space and its boundaries. Neighboring depth sensors whose detection ranges overlap with the depth sensor emitting the reference image will capture the reference image, at least in sections. By observing patterns emitted by other sensors in the network, the relative positions of the sensors can be deduced. The sensors are treated as adjacent if a reference image from another depth sensor extends into their detection range.

[0012] Due to such use of artificially projected reference images, the proposed method, in contrast to the known methods from the state of the art, is independent of the specific nature of the room to be monitored, and manual intervention by specially assigned personnel can essentially be eliminated.

[0013] In a specific embodiment of the method described above, it can be provided that the projector has a defined position and orientation with respect to a depth sensor assigned to the projector. Because a known relationship already exists between the projector and the associated depth sensor, which may be due to a previous measurement or a design-related dependency due to a shared arrangement in a housing, locating the depth sensor's own reference image in the internal coordinate system is not necessary. However, the additional acquisition of the reference image by the emitting sensor itself can also make it possible to establish such a correspondence.

[0014] Furthermore, it can be provided that the reference image at least approximately completely fills the detection range of the assigned depth sensor and / or the reference image lies entirely within the detection range of the assigned depth sensor. In particular, if both conditions are met, the reference image corresponds to the detection range of the depth sensor, so that the largest possible overlap area is shared with the other depth sensors. The larger the overlap area, the more area is available in the reference image for determining the transformation and the better the mutual alignment can be determined.

[0015] First of all, it is of course possible for a network to consist of only a few depth sensors. In this case, it may be sufficient to have just one depth sensor, possibly located centrally, which has a projector. In principle, the projector could be dispensed with for edge-mounted depth sensors. Preferably, however, a projector can be assigned to several or all depth sensors in the network, with the topology creation being repeated for each depth sensor. On the one hand, this makes it possible to further distance ourselves from individualizing the problem and to prepare the depth sensors for general operation. On the other hand, measurement accuracy is improved when measurements are taken in both directions, i.e. when the depth sensors can detect each other in pairs.In general, during calibration, exactly one depth sensor can be active in a network at a time, i.e., emit a reference image. All others can then acquire the reference image within an overlap range with the active depth sensor. The role of the active depth sensor is passed on after acquisition, so that each depth sensor emits a reference image once, and the others have acquired it, as long as the active depth sensor has an overlap with their detection range.

[0016] In particular, the reference image can be a pseudorandom pattern of individual points. Such a distribution, which can also be repeated repeatedly, allows for a particularly accurate determination of the orientation of the reference image relative to the internal coordinate system of the recording, i.e., passive, depth sensor.

[0017] Furthermore, the projector can be an infrared projector for projecting a reference image in the infrared range. Firstly, such projections are not disruptive to people in the area, and secondly, there is less stray light in this area that could influence the measurements, thus improving the detection of the reference image by the depth sensors.

[0018] In an advantageous development, it can further be provided that the depth sensors detect a position and orientation of the marking in their own internal coordinate system. With the help of a central evaluation unit, based on the position and orientation of at least one marking in the internal coordinate systems of the depth sensors, these are related to each other and / or to a common external coordinate system by determining or estimating a transformation. In this further step, the sensor network is further calibrated by determining and taking into account not only the topological arrangement but also the alignment of the individual depth sensors relative to each other.

[0019] Finally, to determine or estimate the transformation between the internal coordinate systems of two depth sensors, at least one model can be moved through the space to be monitored, and each depth sensor determines its trajectories. This complements the method according to the invention with the method already known from the prior art in order to obtain the best measurement results.

[0020] The invention described above is explained in more detail below using an exemplary embodiment.

[0021] It shows Fig. 1 a schematic representation of an emission of a reference image by a depth sensor and detection by an adjacent depth sensor.

[0022] Fig.1 shows two depth sensors 1 and 4 arranged above a room to be monitored, which are connected together in a network 5 with an evaluation unit 6. The type of network 5 can be largely freely selected and can be adapted to the respective local conditions. The network 5 can in particular be both wired and wireless. Measurements carried out using the depth sensors 1 and 4 and the images they record are received and processed by the evaluation unit 6. The evaluation unit 6 determines in particular from the image data acquired by the depth sensors 1 and 4 a transformation between the two depth sensors 1 and 4, which sets the detection areas 7 and 8 covered by them in relation to one another.

[0023] In a first configuration step, the mutual alignment of the depth sensors 1 and 4 is determined. To do this, it is necessary to determine the overlap 9 between a first detection area 7 of the first depth sensor 1 and a second detection area 8 of the second depth sensor 4. In this context, the first depth sensor 1, which projects a reference image 3 onto the floor of the room to be monitored with the aid of an integrated projector 2, is referred to as the active sensor, while the second depth sensor 4, which only records the reference image 3 of the first depth sensor 1, is referred to as the passive sensor. Although the depth sensor 4 is only shown as a passive sensor, it can be assumed that in a possible larger network 5, there will be more passive sensors but always only one active sensor.

[0024] A first sub-step for the passive second depth sensor 4 involves checking the illumination of all existing passive sensors. Only if a subsequent projection of the reference image 3 also produces sufficient contrast can the next sub-step proceed. This sub-step involves creating a background image for each passive sensor, such as the second depth sensor 4 here, as a comparison pattern for the subsequent illumination with the reference image 3. This image can be compared with the reference image 3.

[0025] In a further sub-step, the active sensor, here depth sensor 1, projects a reference image 3 into the space to be monitored using a projector 2. If the reference image 3 of an active depth sensor, here the first depth sensor 1, is recognizable in the second detection range 8 of a passive, second depth sensor 4, a topological proximity can be inferred. If this process is repeated until each depth sensor 1 or 4, as an active sensor, has sent out a reference image 3 and each passive sensor has checked which other depth sensor 4 or 1 the reference image 3 lies in its own detection range 7 or 8, a topology can be created, thus completing the first step of calibration.

[0026] In a second calibration step, the alignment of depth sensors 1 and 4 is then checked. The reference image 3 is a pseudorandom pattern in which unevenly distributed points repeat in groups at larger intervals. This allows the alignment of the reference image 3 to an internal coordinate system of the depth sensor 4 to be accurately determined based on a comparatively small area of ​​the overlap 9. The reference image 3 essentially corresponds to the first detection area 7 of the depth sensor 1, so that the alignment of the overlap 9 with the depth sensor 4 can be carried out with the greatest possible information and is still possible even if the two depth sensors 1 and 4 are located further apart from each other.

[0027] As soon as the passive second depth sensor 4, and any further passive depth sensors, have detected an area of ​​overlap 9, the corresponding image is transmitted to the evaluation unit 6, which then uses the pattern of the reference image 3 to determine the location and position, as well as the orientation of the depth sensor 4 with respect to the depth sensor 1, estimates or determines a transformation between the two depth sensors 1 and 4 and thus creates a topology.

[0028] The quality of this estimation can be improved in a further step by additionally working with moving models that move through the space to be monitored.

[0029] Thus, a method is described above which is capable of automatically calibrating a network of imaging depth sensors, in particular without the need for distributing marks and two- or three-dimensional markings in space, as well as without manual intervention in the evaluation of the sensor images to locate suitable natural markings in space. LIST OF REFERENCE SYMBOLS 1 first depth sensor 2 projectors 3 Reference image 4 second depth sensor 5 Network 6 Evaluation unit 7 first detection area 8 second detection area 9 Overlap

Claims

[1] Method for the automatic calibration of a network (5) of imaging depth sensors (1, 4), wherein, in order to create a topology, detection areas (7, 8) of several depth sensors (1, 4) of the network (5) are aligned with different areas of a space to be monitored jointly, at least one marker is set in the space to be monitored and the depth sensors (1, 4) each determine whether the marker is located in their detection area (7, 8), wherein at least one depth sensor (1, 4) projects a reference image (3) as a marker into the space to be monitored by means of a projector (2), characterized bythat the overlap of a detection range (7) of a depth sensor (1) into the detection ranges (8) of the other depth sensors (4) is detected, the depth sensors (1, 4) detect a position and location of the marking in their own internal coordinate system and with the aid of a central evaluation unit (6) based on the position and location of the at least one marking in the internal coordinate systems of the depth sensors (1, 4), these are related to one another and / or to a common external coordinate system by determining or estimating a transformation. [2] Method according to claim 1, characterized by that the projector (2) has a defined position and location with respect to a depth sensor (1) assigned to the projector (2). [3] Method according to claim 2, characterized bythat the reference image (3) at least approximately completely fills the detection range (7) of the associated depth sensor (1) and / or the reference image (3) lies completely within the detection range (7) of the associated depth sensor (1). [4] Method according to one of the preceding claims, characterized by that a projector (2) is assigned to several or all depth sensors (1, 4) of the network (5), wherein the creation of a topology is repeated for each depth sensor (1, 4). [5] Method according to one of the preceding claims, characterized by that the reference image (3) is a pseudorandom pattern of individual points. [6] Method according to one of the preceding claims, characterized by that the projector (2) is an infrared projector for projecting a reference image (3) in the infrared range. [7] Method according to one of the preceding claims, characterized byin that, in order to determine or estimate the transformation between the inner coordinate systems of two depth sensors (1, 4), at least one model is moved through the space to be monitored and each depth sensor (1, 4) determines its trajectories.

Images