A system that automatically provides UGVs and UAVs with their lidar data for reference and 3D detection

Abstract

The present disclosure relates to a system for UGVs and UAVs to automatically provide their lidar data for reference for 3D detection. The system includes two lidar devices, one of which is mounted on an AGV and the other is mounted on a UAV. The system also includes a reference unit having a first marker and a second marker in a spatially fixed arrangement relative to each other. Automatic detection of the first marker is performed, and collaborative measurement of the first marker by the first lidar device is performed to determine relative position data that provides relative position information of the first marker relative to the first lidar device. The relative position data and spatial 3D information about the spatially fixed arrangement of the first marker and the second marker relative to each other are taken into account to perform automatic detection of the second marker and collaborative measurement of the second marker by the second lidar device. The collaborative measurement of the first marker and the second marker is taken into account for the reference of the lidar data of the UGV and UAV lidar devices relative to a common coordinate system.

Description

Technical Field

[0001] The present invention relates to a system for providing 3D detection of an environment by an unmanned ground vehicle (UGV) and an unmanned aerial vehicle (UAV). Background Art

[0002] For example, 3D detection is used to assess the actual conditions in areas of interest (e.g., confined or hazardous areas such as construction sites, industrial plants, commercial complexes, or caves). The results of 3D detection can be used to efficiently plan the next work step or appropriate action to respond to the determined actual conditions.

[0003] Decision-making and planning of work steps are further assisted by means of dedicated digital visualizations of the actual status (eg in the form of a point cloud or vector file model) or by means of augmented reality functions utilizing 3D detection data.

[0004] 3D detection typically involves optically scanning and measuring the environment using a laser scanner, which, for example, uses pulsed electromagnetic radiation to emit a measuring laser beam. By receiving echoes from backscattered surface points in the environment, the distance to the surface point is determined and associated with the angular emission direction of the associated measuring laser beam. This generates a three-dimensional point cloud. For example, distance measurements can be based on the time of flight, shape, and / or phase of the pulses.

[0005] For additional information, the laser scanner data can be combined with the camera data, for example by means of an RGB camera or an infrared camera, in particular to provide high-resolution spectral information.

[0006] However, collecting 3D data can be cumbersome and in some cases even dangerous for human workers. Often, certain areas are off-limits or have strictly restricted access to human workers.

[0007] Nowadays, robotic vehicles (especially autonomous robotic vehicles) are increasingly used to facilitate data acquisition and reduce risks to human workers. 3D detection devices used in combination with such robotic vehicles are typically configured to provide detection data during movement of the robotic vehicle, wherein referencing data provides information about the trajectory of the data acquisition unit, such as position and / or pose data, making it possible to combine detection data acquired from different positions of the data acquisition unit into a common coordinate system.

[0008] This 3D probe data can then be analyzed using feature recognition algorithms (e.g., using shape information provided by virtual object data from a CAD model) to automatically identify semantic and / or geometric features captured by the probe data. Such feature recognition (particularly for identifying geometric primitives) is now widely used to analyze 3D data.

[0009] Many different types of autonomous robotic vehicles are known. For example, ground-based robotic vehicles may have multiple wheels for propelling the robot, typically with complex suspensions to handle different types of terrain. Another widely used type is legged robots, such as four-legged robots, which are generally capable of handling harsh terrain and steep slopes. Aerial robotic vehicles (e.g., quadrotor drones) are further versatile for exploring inaccessible areas, but often at the expense of less exploration time and / or sensor complexity due to generally limited payload capacity and battery power.

[0010] Unmanned aerial vehicles and unmanned ground vehicles are the most advanced platforms used by the multilateral sector. These platforms are equipped with imaging sensors and lidar sensors to collect 3D detection and reality capture data for autonomous path planning and autonomous movement.

[0011] For movement control and path planning, autonomous robotic vehicles are typically configured to autonomously create a 3D map of the new environment using data from the robotic vehicle's sensors, for example, with the aid of Simultaneous Localization and Mapping (SLAM) functionality.

[0012] In the prior art, mobile control and path planning for exploration activities are primarily managed by leveraging the autonomous robot's built-in visual perception sensors. The acquisition and use of 3D exploration data is typically separated from the acquisition and use of control data for the mobile robot.

[0013] In prior art robotic vehicles, a compromise must typically be made between field of view and viewing distance on the one hand, and reactivity (e.g., for obstacle detection and initiating evasive maneuvers) on the other, which limits the robot's mobility. Typically, the robot only "sees" its immediate surroundings, which provides efficient reactivity to obstacles and terrain changes, while larger-scale path control is provided by predefined environmental models and guidance instructions. This limits the applicability of autonomous robotic vehicles for mobile 3D exploration in unknown terrain, for example. In known terrain, following a predefined path is cumbersome and typically involves a technician taking into account various measurement requirements, such as desired point density, measurement speed, or measurement accuracy.

[0014] The combination of multiple autonomous robotic vehicles provides flexibility when surveying large and varying areas (e.g. different soil conditions, measurements from the ground and from the air, etc.). Each mobile surveying device can provide 3D survey data. For example, the surveying device generates a so-called local 3D point cloud from each measurement position, which provides a plurality of measurement points referenced to a common coordinate system related to the surveying device. When the surveying device is moved, the local point clouds determined in the different positions of the surveying device have to be related to each other by a process known as referencing, point cloud registration, point set registration or scan matching in order to form the so-called 3D survey point cloud of the corresponding surveying device. In addition, the 3D survey point clouds from different surveying devices located on different autonomous robotic vehicles have to be referenced to each other in order to form a so-called "combined" 3D survey point cloud.

[0015] In order to refer to local point clouds of the same detection device, additional information provided by the detection device is usually used, such as data from an inertial measurement unit and a simultaneous localization and mapping (SLAM) unit.

[0016] Typically, referencing different measured point clouds from different measuring devices is cumbersome and only possible in post-processing. This is made more difficult, for example, by the fact that different types of measuring sensors are often used (e.g., where the sensors of different measuring devices offer different point densities, fields of view, and distance resolutions). When using image data, different image distortions, etc., must be taken into account.

[0017] To save computation time, point matching typically involves an operator manually identifying and linking matching features in different measured point clouds from different measurement devices. Consequently, substantially real-time fusion of different 3D data from different devices is often impossible or prone to errors and interruptions. Summary of the Invention

[0018] It is therefore an object of the present invention to provide an improved system for mobile 3D detection having increased applicability, in particular in view of detecting varying and extensive terrains.

[0019] A further object is to provide a system for mobile 3D detection which is easier to handle and more robust against interruptions.

[0020] These objects are achieved by realizing at least some of the features of the independent claims. Features which further develop the invention in an alternative or advantageous manner are described in the dependent patent claims.

[0021] The present invention relates to a system for providing 3D detection of an environment, wherein the system comprises a first lidar device and a second lidar device. One of the first lidar device and the second lidar device (hereinafter referred to as the UGV lidar device) is specifically foreseen for montage on an unmanned ground vehicle and is configured to generate UGV lidar data to provide a collaborative scan of the environment in relation to the UGV lidar device. The other of the first lidar device and the second lidar device (hereinafter referred to as the UAV lidar device) is specifically foreseen for montage on an unmanned aerial vehicle and is configured to generate UAV lidar data to provide a collaborative scan of the environment in relation to the UAV lidar device. The system is configured to provide a reference of the UGV lidar data and the UAV lidar data relative to a common coordinate system for determining a (combined) 3D detection point cloud of the environment.

[0022] In particular, the system is configured to provide positioning of the UGV lidar device in the digital 3D model based on the UAV lidar data, and vice versa, provide positioning of the UAV lidar device in the digital 3D model based on the UGV lidar data.

[0023] The system includes a reference unit, which includes a first marker and a second marker, wherein the first marker and the second marker are in a spatially fixed arrangement relative to each other, and each of the first marker and the second marker is configured as a target for collaborative measurement of the corresponding marker by a lidar device. The system is configured to perform automatic detection of the first marker and perform collaborative measurement of the first marker by the first lidar device to determine relative position data, which provides relative position information of the first marker relative to the first lidar device. Moreover, the system is configured to take into account the relative position data and the spatial 3D information about the spatially fixed arrangement of the first marker and the second marker relative to each other to perform automatic detection of the second marker and perform collaborative measurement of the second marker by the second lidar device. The collaborative measurement of the first marker and the collaborative measurement of the second marker are then taken into account to provide a reference of the UGV lidar data and the UAV lidar data relative to a common coordinate system.

[0024] For example, both the UGV lidar device and the UAV lidar device are provided with a combined start / end reference. The first marker and the second marker are referenced to each other by design (e.g., mechanically), wherein one marker allows good detection by a starting / landing UAV. For example, in a nominal installation of the reference unit, the first marker is set in the vertical plane and the second marker is set in the horizontal plane. The first marker then allows good detection by a passing UGV, while the second marker allows good detection by a starting / stopping UAV. Thus, the two markers provide a basic reference between the UAV and the UGV, allowing the UGV lidar data and the UAV lidar data to be spatially fused.

[0025] Thus, in one embodiment, one of the first and second markers (hereinafter referred to as the UGV marker) is configured to be spatially positioned in a nominal installation of the reference unit in such a way that the UGV LiDAR device can perform collaborative measurements of the UGV marker, wherein the collaborative measurements of the UGV marker are performed from a side-looking field of view associated with a picture composition of the UGV LiDAR device on the UGV. The other of the first and second markers (hereinafter referred to as the UAV marker) is configured to be spatially positioned in a nominal installation of the reference unit in such a way that the UAV LiDAR device can perform collaborative measurements of the UAV marker, wherein the collaborative measurements of the UAV marker are performed from a top-looking field of view associated with a picture composition of the UAV LiDAR device on the UAV.

[0026] In another embodiment, the system is configured such that collaborative measurement of the first marker is performed by the UGV lidar device, and collaborative measurement of the second marker is performed by the UAV lidar device, wherein automatic detection of the second marker and collaborative measurement of the second marker by the UAV lidar device are performed at each takeoff and landing of the unmanned aerial vehicle.

[0027] For example, the system is configured to continuously update the relative position data so that the relative position information provides continuously updated spatial information about the arrangement between the first marker and the UGV lidar device.

[0028] The additional markers on the UGV can be used to provide a link between the UAV and the UGV along the mobile map building process. For example, this provides a larger baseline to overcome inaccuracies in the referenced starting marker. Therefore, in another embodiment, the system also includes the additional markers (in addition to the first marker and the second marker), which are specifically foreseen for use in picture synthesis on the unmanned ground vehicle. The UAV lidar device is configured to automatically perform collaborative measurements of the additional markers, and the system is configured to take into account the collaborative measurements of the additional markers to provide a reference of the UGV lidar data and the UAV lidar data relative to a common coordinate system.

[0029] The spatial positioning between the UAV lidar device and the UGV lidar device can be based on a sparse map from imagery and / or lidar data. For example, the sparse map is generated by a camera and / or lidar device of the UAV or UGV. The corresponding UGV or UAV is then positioned (in real time) within the sparse map. Therefore, in another embodiment, the system includes a visual pickup device that is configured to be set on an unmanned ground vehicle or an unmanned aerial vehicle, for example, wherein the visual pickup device is a camera or one of the UGV lidar device or the UAV lidar device. The system is configured to use the visual pickup device to generate a sparse map and to perform positioning of the UGV lidar data or the UAV lidar data in the sparse map.

[0030] For example, the sparse map is generated by photogrammetric triangulation (e.g., so-called structure from motion), and the positioning includes a first referencing between the UGV lidar data and the UAV lidar data. Then, after the first referencing, a second referencing between the UGV lidar data and the UAV lidar data is performed based on point cloud matching between the UGV lidar data and the UAV lidar data, wherein the sparse map is referenced relative to a known digital model of the environment.

[0031] For example, known methods for point cloud matching include iterative closest point to point, iterative closest point to plane, robust point matching, and kernel correlation point set registration. The known digital model can be at least one of a digital building information model (BIM), a computer-aided design model (CAD), and a digital model (e.g., a vector file model), the known digital model being generated based on collaborative scanning data provided by a terrestrial laser scanner (TLS), a mobile mapping system, or a photogrammetric capture device.

[0032] In another embodiment, the system is configured to access assignment data providing spatial 3D information about a spatially fixed arrangement of the first marker and the second marker relative to each other.

[0033] Alternatively or additionally, at least one of the first marker and the second marker includes a visual code (e.g., a barcode or a matrix barcode) that provides spatial 3D information about the spatially fixed arrangement of the first marker and the second marker relative to each other. Here, the system is configured to determine the spatial 3D information about the spatially fixed arrangement of the first marker and the second marker relative to each other by using a visual pickup device (e.g., a camera or a barcode laser scanning device).

[0034] For example, the first and second markers can be implemented to be mechanically compatible with standard markers used in the prior art for surveying control points. These markers can be implemented so that they can be measured using standard surveying equipment such as a total station. The markers can also be implemented so that they can be automatically identified and detected in point cloud software. For example, the markers can contain coded information that can be read by UGV and / or UAV lidar equipment and point cloud software during post-processing, while also having visual features that allow measurement using a total station.

[0035] In another embodiment, the first marking and the second marking are provided on a common component such that the relative spatial arrangement of the first marking and the second marking is mechanically fixed.

[0036] For example, the common component also includes an alignment indicator (eg, a bubble level) to provide visual confirmation of the alignment of the common component relative to an external coordinate system or relative to cardinal directions to establish a nominal installation.

[0037] In another embodiment, at least one of the first and second markers (or the further marker, see above) comprises a visually detectable pattern, for example, a pattern provided by areas of different reflectivity, different gray levels and / or different colors. The system is configured to determine the 3D orientation of the pattern by determining geometric features in an intensity image of the pattern, wherein the intensity image of the pattern is acquired by scanning the pattern with a lidar measurement beam of a UGV lidar device or a UAV lidar device and detecting the intensity of the returned lidar measurement beam. A plane fitting algorithm is performed by analyzing the appearance of geometric features in the intensity image of the pattern to determine the orientation of the pattern plane, and the system is configured to take into account the 3D orientation of the pattern for providing a reference of the UGV lidar data and the UAV lidar data relative to a common coordinate system.

[0038] For example, the pattern includes circular features, and the system is configured to identify an image of the circular features within an intensity image of the pattern. The plane fitting algorithm is configured to fit an ellipse to the image of the circular features and determine the orientation of the pattern plane based on the fit. For example, the system is further configured to determine the center of the ellipse and derive aiming information for aiming the center of the ellipse using the lidar measurement beam. The center of the ellipse can then be used as an aiming point to further determine the 3D position of the marker, for example, allowing the marker's 6DoF pose (six degrees of freedom, position and orientation) to be determined and taken into account.

[0039] In particular, the pattern includes internal geometric features, for example, a rectangular feature surrounded by a circular feature, and the internal geometric features provide information about the calibration of a common coordinate system or an external coordinate system, and / or spatial 3D information about the spatially fixed arrangement of the UGV marker and the UAV marker relative to each other.

[0040] In another embodiment, the first marker and the second marker each include a visual indication of the direction of at least two of the three principal axes, in particular the three axes, spanning the common coordinate system, wherein the system is configured to determine the directions of the three principal axes by using the UGV lidar device and the UAV lidar device, and to consider the directions of the three principal axes for providing a reference of the UGV lidar data and the UAV lidar data relative to the common coordinate system.

[0041] In another embodiment, collaborative scanning of the environment by the UGV lidar device is provided based on a UGV scanning pattern provided locally by the UGV lidar device, wherein the UGV scanning pattern has multiple scanning directions relative to the UGV lidar device. Similarly, collaborative scanning of the environment by the UAV lidar device is provided based on a UAV scanning pattern provided locally by the UAV lidar device, wherein the UAV scanning pattern has multiple scanning directions relative to the UAV lidar device. Moreover, the UGV scanning pattern and the UAV scanning pattern provide the same local angular distribution of the multiple scanning directions, the same angular point resolution for each scanning direction, and the same range resolution. Therefore, the UGV lidar data and the UAV lidar data are inherently provided with the same measurement parameters, which provides a simplified referencing of the UGV lidar data and the UAV lidar data relative to a common coordinate system.

[0042] For example, UGV and UAV lidar devices are each implemented as a laser scanner configured to generate lidar data by rotating a laser beam about two rotational axes. The laser scanner includes a rotating body configured to rotate about one of the two rotational axes and provide variable deflection of the exit and return portions of the laser beam, thereby providing rotation of the laser beam about one of the two rotational axes (typically referred to as the fast axis). The rotating body rotates about the fast axis at a rate of at least 50 Hz, while the laser beam rotates about the other of the two rotational axes (typically referred to as the slow axis) at a rate of at least 0.5 Hz. The laser beam is emitted as a pulsed laser beam, for example, including 1.5 million pulses per second, thereby providing a point acquisition rate of lidar data of at least 300,000 points per second. For the rotation of the laser beam about the two axes, the field of view about the fast axis is at least 130 degrees, while the field of view about the slow axis is 360 degrees.

[0043] For example, a UGV lidar device / laser scanner is implemented so that when mounted on a UGV, the slow axis is substantially vertical, and a UAV lidar device / laser scanner is implemented so that when mounted on a UAV, the slow axis is substantially horizontal. Thus, the UAV lidar device has real-time coverage of the area above the UAV, in front of the UAV, and the surface below the UAV. The UGV lidar device has real-time coverage of the floor and the area in front of, above, and behind the UGV.

[0044] In another embodiment, the system includes: a UGV simultaneous localization and mapping unit (UGV SLAM unit), and a UAV simultaneous localization and mapping unit (UAV SLAM unit). The UGV SLAM unit is configured to: receive UGV lidar data as UGV perception data, the UGV perception data providing a representation of the surroundings of the UGV lidar device at a current location; use the UGV perception data to generate a UGV map of the environment; and determine a trajectory of a path that the UGV lidar device has traversed within the UGV map of the environment. The UAV SLAM unit is configured to: receive UAV lidar data as UAV perception data, the UAV perception data providing a representation of the surroundings of the UAV lidar device at a current location; use the UAV perception data to generate a UAV map of the environment; and determine a trajectory of a path that the UAV lidar device has traversed within the UAV map of the environment.

[0045] To provide sufficient data processing capacity, the system can include connectivity for data exchange between the UGV and UAV lidar devices and a data cloud that provides cloud computing, for example, to determine a 3D detection point cloud or perform at least a portion of the processing described above for estimating alternative trajectories for the UGV or UAV, respectively. On the UGV side in particular, the system can benefit from onboard computing, for example, using a dedicated computing unit provided with the UGV lidar device or a computing unit of an unmanned ground vehicle. This significantly expands computing power in the event of a loss of cloud connectivity or when data transfer rates are limited. The same is true for UAVs, but payload capacity and battery power are typically limited for UAVs. Another possibility is to include connectivity with a companion device (e.g., a tablet computer) that can be configured to determine a 3D detection point cloud or perform at least a portion of the processing described above for estimating alternative trajectories for the UGV or UAV, similar to cloud processing. The local companion device can then take over processing for areas with limited or no cloud connectivity, or it can serve as a cloud interface, acting as a relay between onboard computing and cloud computing. For example, switching between onboard computing, cloud processing, and processing by companion devices is performed dynamically based on the connectivity between the three processing locations.

[0046] In one embodiment, the system includes an onboard computing unit, specifically foreseen to be located on an unmanned ground vehicle, and configured to perform at least a portion of the system processing, wherein the system processing includes performing a SLAM process for a UGV or UAV, thereby providing reference to UGV lidar data and UAV lidar data, and performing additional trajectory estimation for the UGV, the UAV, or both. The system also includes an external computing unit configured to perform at least a portion of the system processing. The system's communication module is configured to provide communication between the onboard computing unit and the external computing unit, wherein the system includes a workload selection module configured to: monitor the available bandwidth of the communication module for communication between the onboard computing unit and the external computing unit; monitor the available power of the onboard computing unit, the UGV lidar device and / or the UAV lidar device, the UGV's SLAM unit and the UAV's SLAM unit, and the path planning unit; and dynamically change the assignment of at least a portion of the system processing to the onboard computing unit and the external computing unit based on the available bandwidth and available power assigned to the external processing unit.

[0047] For example, localization is processed locally on a computing device (either part of the UGV or UAV, or a separate computing base station), or in the "cloud." Similarly, computations for landmark detection and reference transformation, sparse map generation, scan area definition, and gap filling can be distributed across different onboard, local, and cloud-based computing units.

[0048] For example, to minimize the computational weight on the UAV, fast connectivity between the UAV, UGV, supporting equipment, and the cloud is implemented. For example, communication to the cloud is based on a 4G / 5G uplink, where a local connection (e.g., WLAN) is used between the UAV and the UGV and / or supporting equipment to download data from the UAV to the UGV and / or supporting equipment. Onboard processing or supporting processing is particularly important if the UAV has good line of sight but poor cloud connectivity (e.g., when observing in a canyon).

[0049] In order to dynamically assign processing steps to different computing units, for example, to decide where to process data and how to upload it, an arbitrator or scheduler unit (in the sense of a policy controller) can be implemented on the UGV, UAV, companion device or base station.

[0050] In another embodiment, the system is configured to perform system processing, the system processing including performing SLAM processes associated with an unmanned ground vehicle and / or an unmanned aerial vehicle; providing a reference for UGV lidar data and / or UAV lidar data relative to a common coordinate system; and performing path planning to determine an additional trajectory for the unmanned ground vehicle and / or unmanned aerial vehicle to follow. The system includes a UGV computing unit that is specifically foreseen to be located on the unmanned ground vehicle and is configured to perform at least a portion of the system processing. Similarly, the system includes a UAV computing unit that is specifically foreseen to be located on the unmanned aerial vehicle and is configured to perform at least a portion of the system processing, and the system includes an external computing unit configured to perform at least a portion of the system processing.

[0051] The communication unit of the system is configured to provide mutual communication between the UGV computing unit, the UAV computing unit, and the external computing unit by using a cellular communication connection (e.g., 4G or 5G). The communication unit also provides mutual communication between the UGV computing unit and the UAV computing unit by using a local communication connection (e.g., WLAN or Bluetooth).

[0052] Here, the system further includes a workload selection module configured to monitor available bandwidth of the cellular communication connection and the local communication connection to perform dynamic changes in the assignment of at least a portion of system processing to the UGV computing unit, the UAV computing unit, and the external computing unit. The dynamic changes in the assignments are dependent on the available bandwidth of the cellular communication connection and the local communication connection, wherein a prioritization rule is implemented to minimize the available processing load of the UAV computing unit before minimizing the available processing load of the UGV computing unit, and to minimize the available processing load of the UGV computing unit before minimizing the available processing load of the external computing unit.

[0053] The dynamic change of the assignment may also depend on the availability of UAV and / or UGV battery power, i.e., wherein the workload of the UAV computing unit and / or UGV computing unit is selected based on which one has the most available battery power.

[0054] The dynamic change of the assignment may also depend on the requirements of the SLAM associated with the UGV lidar data and / or the UAV lidar data. For example, the dynamic change of the assignment may be based on the requirement of providing sufficient overlap of the sparse map with the known digital model. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Below, with reference to the working examples schematically shown in the accompanying drawings, the system according to different aspects of the present invention will be described or explained in more detail by way of example only. The same reference numerals are used to mark the same elements in the drawings. The embodiments are generally not shown to scale, and these embodiments should not be interpreted as limiting the present invention. Specifically,

[0056] Figure 1 : Example embodiments of an unmanned ground vehicle operating in conjunction with an unmanned aerial vehicle;

[0057] Figure 2 : An exemplary embodiment of a corresponding lidar device for an unmanned ground vehicle or an unmanned aerial vehicle;

[0058] Figure 3 : An exemplary embodiment of a reference unit including a UGV marker and a UAV marker;

[0059] Figure 4 : An exemplary workflow using a reference unit comprising a UGV marker and a UAV marker, wherein the unmanned ground vehicle comprises an additional marker that serves as a reference for the unmanned aerial vehicle;

[0060] Figure 5 : an exemplary embodiment of a marker, for example, one of a UGV marker, a UAV marker, and another marker provided on a UGV;

[0061] Figure 6 : An exemplary communication scheme between an unmanned ground vehicle, an unmanned aerial vehicle, a companion device, and cloud processing;

[0062] Figure 7 : Other exemplary communication schemes with dynamic allocation of processing steps to different computing units. DETAILED DESCRIPTION

[0063] Figure 1 An exemplary embodiment of an unmanned ground vehicle (UGV) 1 is depicted working in conjunction with an unmanned aerial vehicle (UAV) 2. Each of the UGV 1 and UAV 2 is equipped with a lidar device, referred to as UGV lidar device 3 and UAV lidar device 4, respectively.

[0064] Here, the robotic ground vehicle 1 is implemented as a four-legged robot. Such robots are typically used in unknown terrain with varying surface characteristics, such as debris and steep slopes. The ground robot 1 has sensors and processing capabilities that provide simultaneous localization and mapping, including: receiving sensory data representing the environment surrounding the autonomous ground robot 1 at its current location; generating a map of the environment using this sensory data; and determining a trajectory within the map of the environment that represents the path traversed by the ground robot 1.

[0065] Aircraft 2 is implemented as a quadrotor drone to allow for further versatility in exploring areas that are difficult or impossible to access by robotic ground vehicle 1. Similar to UGV 1, aircraft 2 has sensors and processing capabilities that provide simultaneous localization and mapping, including: receiving sensory data that provides a representation of the environment surrounding unmanned aerial vehicle 2 at its current location; using the sensory data to generate a map of the environment; and determining a trajectory of the path that aircraft 2 has traversed within the map of the environment.

[0066] Each of the UGV LiDAR device 3 and the UAV LiDAR device 4 has a field of view of 360 degrees around a so-called slow axis 5 and a so-called band field of view 6 of at least 130 degrees around a fast axis (see FIG. Figure 2 ). Each of the two lidar devices 3, 4 is configured to generate corresponding lidar data at a point acquisition rate of at least 300,000 points per second. For example, the UGV lidar device 3 and the UAV lidar device 4 are each implemented as a so-called two-axis laser scanner (see Figure 2), wherein, in the case of the UGV lidar device 3, the fast axis 5 is essentially vertically aligned, while in the case of the UAV lidar device 4, the fast axis 5 is essentially horizontally aligned.

[0067] The SLAM units of the UGV and UAV are each configured to receive corresponding lidar data as perception data, which, for example, provides improved field of view and viewing distance, and thus provides improved larger-scale path determination. This is particularly beneficial, for example, for exploring unknown terrain. Another benefit is the omnidirectional horizontal field of view about the slow axis 5 and the 130-degree zonal field of view about the fast axis 6. In the case of the UGV 1, this provides the ability to substantially simultaneously cover the front, rear, and ground, while in the case of the UAV 2, this provides the ability to substantially simultaneously cover the rear and ground.

[0068] For example, the lidar data generated by means of the UGV lidar device 3 and the UAV lidar device 4 can be combined to fill in the gaps of the complementary system data. Typically, the UGV lidar device 3 "sees" objects close to the ground and at a side perspective (facades, soffit, etc.) and is used for indoor detection (buildings, tunnels, etc.). The UAV lidar device 4 observes objects on the ground (upper facades, roofs, etc.) and is often used for outdoor detection (buildings, bridges, etc.). In the accompanying drawings, the UAV lidar device 4 and the UGV lidar device 3 are both exemplarily used to collaboratively measure, for example, a pipeline 7 at a power plant site, wherein the UAV lidar device 4 mainly observes the top of the pipeline 7, while the UGV lidar device 3 only observes the pipeline 7 from a side perspective.

[0069] The combination of UGV 1 and UAV 2 also allows for the definition of the scanning area of UGV 1 (or UGV LiDAR device 3) through exploratory flights of UAV 2 and UAV LiDAR device 4. Through exploratory flights, the area of interest to be surveyed by UGV LiDAR device 3 is defined. For example, UAV 2 provides an overview of the path being followed by UGV 1. Spatial anchoring (repositioning) allows for the matching of UGV LiDAR data and UAV LiDAR data, as well as trajectory alignment with the line-of-sight environment.

[0070] The exploration of the UAV also allows for an estimation of whether a specific measurement goal can be achieved within the constraints, for example, providing an improved estimate of whether the battery of UAV 2 or UGV 1 is sufficient to achieve the anticipated mission. Since the battery power of UAVs is generally limited, UGV 1 can also be configured to serve as a landing / docking station for UAV 2 and as a mobile charging station for UAV 2. In this way, for example, during periods when only detection by the UGV's lidar device 3 is required, such as when stepping in an indoor environment, the reach of UAV 2 can be extended by recharging. Similarly, large data downloads can preferably be performed while UAV 2 is docked on UGV 1.

[0071] Figure 2 An exemplary embodiment of a UGV lidar device 3 or a UAV lidar device 4, respectively, is shown in the form of a so-called two-axis laser scanner. The laser scanner comprises a base 8 and a support 9, which is mounted on the base 8 so as to be rotatable about a slow axis 5. Usually, the rotation of the support 9 about the slow axis 5 is also referred to as an azimuth rotation, regardless of whether the laser scanner or the slow axis 5 is aligned perfectly vertically.

[0072] The core of the laser scanner is an optical distance measuring unit 10, which is arranged in the support 9 and is configured to perform distance measurement by emitting a pulsed laser beam 11 (for example, wherein the pulsed laser beam comprises 1.5 million pulses per second) and detecting the return portion of the pulsed laser beam by means of a receiving unit comprising a light-sensitive sensor. Thus, pulse echoes are received from backscattered surface points of the environment, wherein the distance to said surface point can be derived based on the flight time, shape and / or phase of the emitted pulses.

[0073] The scanning movement of the laser beam 11 is performed by rotating the support 9 relative to the base 8 about the slow axis 5 and with the aid of a rotating body 12, which is rotatably mounted on the support 9 and rotates about a so-called fast axis 14 (horizontal axis). For example, both the transmitted laser beam 11 and the return portion of the laser beam are deflected by a reflective surface that is integral with or applied to the rotating body 12. Alternatively, the transmitted laser radiation comes from the side facing away from the reflective surface (i.e., from the inside of the rotating body 12) and is emitted into the environment via a channel region within the reflective surface.

[0074] To determine the emission direction of the ranging beam 11, many different angle determination units are known in the prior art. For example, the emission direction can be detected using an angular encoder configured to acquire angular data for detecting the absolute angular position and / or relative angular change of the support 9 or the absolute angular position and / or relative angular change of the rotating body 12, respectively. Another possibility is to determine the angular position of the support 9 or the rotating body 12, respectively, by detecting only full revolutions and using knowledge of a set rotational frequency.

[0075] The visualization of the data may be based on known data processing steps and / or display options, for example, wherein the acquired data is presented in the form of a 3D point cloud or wherein a 3D vector file model is generated.

[0076] The laser scanner is configured to ensure that its total field of view for measurement operation is 360 degrees in the azimuth direction defined by the rotation of the support 9 about the slow axis 5, and at least 130 degrees in the yaw direction defined by the rotation of the rotating body 12 about the fast axis 14. In other words, regardless of the azimuth angle of the support 9 about the slow axis 5, the laser beam 11 can cover a so-called zone field of view (vertical field of view in the drawings) extending in the yaw direction with an expansion angle of at least 130 degrees.

[0077] For example, the total field of view typically refers to the center reference point 13 of the laser scanner, which is defined by the intersection of the slow axis 5 and the fast axis 14 .

[0078] Figure 3 An embodiment of a reference unit 15 according to the present invention is shown exemplarily, comprising UGV markers 16A, 16B and a UAV marker 17 .

[0079] Here, the reference unit 15 is implemented in the shape of a cube. In a nominal installation, for example, where one of the cube's sides is precisely leveled, the reference unit 15 provides four (vertical) sides that can be used to provide UGV markers 16A, 16B, and one (horizontal) side that can be used to provide a UAV marker 17. For example, the assembly of the cube in its nominal installation is assisted by a bubble level.

[0080] Here, the UGV markers 16A, 16B and the UAV marker 17 include a visual code that provides spatial 3D information about the spatially fixed arrangement of the UGV markers 16A, 16B and the UAV marker 17 relative to each other. The spatial 3D information can be determined by reading the code using a visual pickup unit (e.g., a camera or a UGV lidar device and a UAV lidar device) provided on the UGV and UAV, respectively.

[0081] As the UGV and UGV LiDAR device 3 pass by the reference cube 15, the visual UGV marker 16B is automatically identified, and a collaborative measurement of the visual UGV marker 16B is performed by the UGV LiDAR device 3, thereby determining relative position data that provides relative position information of the visual UGV marker 16B relative to the UGV LiDAR device 3. Thus, the relative position and particular orientation of the moving UGV LiDAR device 3 relative to the identified visual UGV marker 16B is tracked so that it can be used to facilitate later detection of the UAV marker 17 by, for example, a launched UAV.

[0082] For example, at UAV startup, this relative position data and the spatial 3D information about the determined spatially fixed arrangement of the identified visual UGV marker 16B and the UAV marker 17 relative to each other are taken into account to perform automatic detection of the UAV marker 17 and to perform collaborative measurement of the UAV marker 17 by the UAV lidar device 4. The collaborative measurement of the identified visual UGV marker 16B and the collaborative measurement of the UAV marker 17 are then taken into account to provide referencing of the UGV lidar data and the UAV lidar data relative to a common coordinate system.

[0083] Figure 4 Another exemplary workflow using a reference unit 15 comprising a UGV marker and a UAV marker is depicted, wherein the unmanned ground vehicle includes an additional marker 18 that serves as a reference for combining the UAV lidar data and the UGV lidar data in a common coordinate system.

[0084] The further marker 18 is provided on the UGV and is used to provide a link between the UAV lidar device 4 and the UGV lidar device 3 along the mobile map building process. The UAV lidar device 4 is configured to automatically perform collaborative measurements of the further marker 18 so as to take into account the collaborative measurements of the further marker 18 to provide a reference of the UGV lidar data and the UAV lidar data relative to a common coordinate system. For example, where the positional relationships of different reference units within the environment are known, e.g. the absolute positions of the different reference units given in an external coordinate system, a larger baseline is provided to overcome inaccuracies in the collaborative measurements of the referenced starting marker and can, for example, be used for so-called closed loops of SLAM algorithms, which allows compensation for position drift when referencing ("stitching together") lidar data at different positions along the route of the UGV or AGV.

[0085] Some of the markers (e.g., UGV markers 16A, 16B, UAV marker 17, and one of the additional markers 18 provided on the UGV) may also include a reference value indication that provides position information, e.g., 3D coordinates, about a set pose of the marker in a common coordinate system or in an external coordinate system (e.g., a world coordinate system). The set pose is a 6DoF pose, i.e., the position and orientation of the marker, and indicates the desired 6DoF pose of the marker. Thus, when correctly placed in the environment, the marker can act as a so-called probe control point, e.g., for closing the loop of a SLAM process and / or as an absolute reference in a world coordinate system or a local site coordinate system.

[0086] For example, the system is configured to derive a set pose and take the set pose into account for referencing UGV lidar data and UAV lidar data in the common coordinate system (e.g., by determining the pose of the marker in the common coordinate system or in the world coordinate system and performing a comparison of the determined marker pose with the set pose).

[0087] Figure 5 Depicted are exemplary embodiments of markers 30, such as UGV markers 16A, 16B, UAV marker 17, and one of additional markers 18 disposed on a UGV (see FIG. Figure 3 and Figure 4 ). On the left side, the mark 30 is shown in a front view. On the right side, the mark 30 is shown in an angled view.

[0088] The marking comprises a visually detectable pattern, for example provided by areas of different reflectivity, different grey levels and / or different colours, the pattern comprising a circular feature 31 and an inner geometric feature 32 enclosed by the circular feature 31 .

[0089] For example, the system is configured to determine the 6DoF (six degrees of freedom) pose of the marker. The 6DoF pose is derived by determining the 3D orientation of the pattern (i.e., the 3D orientation of the pattern plane) and by determining the 3D position of the pattern. For example, at least three marker corners 33 are analyzed to provide angles for determining the pattern plane. The marker corners 33 can be determined using a camera on a UGV or UAV, respectively.

[0090] The circular feature 31 provides an improved determination of the 3D orientation of the pattern plane. For example, the system is configured to generate an intensity image of the pattern by scanning the pattern with a lidar measurement beam from a UGV lidar device or a UAV lidar device, respectively, where the intensity image is generated by detecting the intensity of the returning lidar measurement beam. By identifying an image of the circular feature within the intensity image of the pattern and running a plane fitting algorithm to fit an ellipse to the image of the circular feature, the 3D orientation of the pattern plane can be determined with improved accuracy. Additionally, the center of the ellipse can be determined and used as an aiming point for the lidar device to determine the 3D position of the pattern, thereby allowing the 6DoF pose of the pattern to be determined.

[0091] The 3D orientation of the pattern, in particular the 6DoF pose of the pattern, is then considered to provide a reference for the UGV lidar data and the UAV lidar data relative to a common coordinate system.

[0092] Figure 6 Depicted are exemplary communication schemes between an unmanned ground vehicle (e.g., a UGV lidar device 3), an unmanned aerial vehicle (e.g., a UAV lidar device 4), an accompanying device 19 (e.g., a tablet computer), and a data cloud 20 providing cloud processing.

[0093] For example, the operator's tablet computer 19 is locally connected to the UAV lidar device 4 and the UGV lidar device 3, for example by means of Bluetooth or WLAN, wherein the tablet computer allows mediating the control of both lidar devices 3, 4. The tablet computer 19 is also connected to a cloud processing unit 20.

[0094] In the event of a loss of connectivity to the tablet 19, the optional connection between the UAV LiDAR device 4 and the UGV LiDAR device 3 provides redundancy. Cloud connectivity for the tablet 19, the UAV LiDAR device 4, and the UGV LiDAR device 3 allows operation without a local connection and provides an additional fallback scenario. For example, cloud connectivity is established via a 4G / 5G uplink.

[0095] For example, this universal communication capability allows for dynamic distribution of processing and data storage, e.g., to balance desired data processing rates with battery life.

[0096] Figure 7An exemplary communication scheme with dynamic allocation of processing steps to different computing units is shown. Here, the unmanned ground vehicle includes an onboard computing unit 21 and a cellular communication uplink 22 to the cloud 20. Similarly, the UAV includes a cellular communication uplink to the cloud 20 (not shown). In the two bottom schematics, the system also includes a base station 23, which is positioned close to the UGV and is configured for relatively heavy computing (compared to the onboard computing unit 21). The base station 23 can also have a cellular communication uplink 22 to the cloud 20 (bottom left schematic), or data upload to the cloud 20 can be primarily performed through the UGV cellular communication uplink 22 (bottom right schematic), for example, where the UGV uplink 22 acts as a relay between the base station 23 and the cloud 20.

[0097] An onboard computing unit 21 of the UGV and a base station 23 are provided to minimize processing on the UAV, thereby conserving the battery life of the UAV.

[0098] In the upper left schematic, a local data connection is established (e.g., via WLAN) between the UAV lidar device 4 and the UGV's onboard computing unit 21 to download data from the UAV lidar device 4. The UGV's onboard computing unit, which has more payload capacity, is computing the results on-site and providing uplink functionality to the cloud computing service 20.

[0099] In the upper right schematic, a local connection is established (e.g., via WLAN) to upload data from the UGV onboard computing unit 21 and / or the UGV lidar device to a UAV cellular uplink (not shown), which provides the data to the cloud 20. This approach is used, for example, if the UAV has better line of sight or connectivity to the cloud 20 (e.g., when the UGV is navigating a canyon with limited or no connectivity).

[0100] In the lower left diagram, local connections are established to download data from the UAV lidar device 4, the UGV lidar device 3, and the UGV onboard computing unit 21 to the base station 23. The main processing payload is on the side of the base station 23 and the cloud 20, which have established a cellular data connection between them.

[0101] Similarly, in the lower right schematic, the main processing is on the base station 23 and cloud 20 side, but communications with the cloud 20 are routed through the UGV data uplink 22.

[0102] For example, an arbitrator or scheduler unit, for example located on the UGV or on the base station 23, is used to dynamically distribute processing to different processing units, for example, to distribute at least part of the following: calculating additional trajectories, calculating maps for the SLAM process, and referencing the UGV lidar data and the UAV lidar data for a common coordinate system. The arbitrator or scheduler unit can also define where and how data is uploaded to / downloaded from the cloud 20.

[0103] In particular, switching between onboard computing, cloud processing, processing for the LiDAR device, and processing for the companion device is performed dynamically based on the connectivity between the computing locations and the available power on the UGV and UAV. Typically, processing is moved, for example, from the UAV to (and also from the UGV to) the cloud, companion devices, and base stations whenever possible, because battery power and data storage in UAVs and UGVs (and devices located on them) are limited.

[0104] Although the present invention has been illustrated above, partly with reference to certain preferred embodiments, it must be understood that many modifications of these embodiments and combinations of different features are possible. These modifications all fall within the scope of the appended claims.

Claims

1. A system for providing 3D detection of an environment, wherein: The system includes a first laser radar device and a second laser radar device, wherein, One of the first lidar device and the second lidar device is a UGV lidar device (3), the UGV lidar device (3) being specifically foreseen for use in picture composition of an unmanned ground vehicle (1) and being configured to generate UGV lidar data to provide a collaborative scan of the environment in relation to the UGV lidar device (3), The other of the first lidar device and the second lidar device is a UAV lidar device (4), the UAV lidar device (4) being specifically foreseen for use in picture composition of an unmanned aerial vehicle (2) and being configured to generate UAV lidar data to provide a collaborative scan of the environment in relation to the UAV lidar device (4), and The system is configured to provide a reference of the UGV lidar data and the UAV lidar data relative to a common coordinate system for determining a 3D detection point cloud of the environment, It is characterized in that The system comprises a reference unit (15), the reference unit comprising a first marker and a second marker (16A, 16B, 17, 30), wherein the first marker and the second marker are in a spatially fixed arrangement relative to each other, and each of the first marker and the second marker is configured as a target for collaborative measurement of the respective marker by a lidar device (3, 4), wherein the system is configured to: performing automatic detection of the first markers (16A, 16B, 17, 30) and performing collaborative measurement of the first markers (16A, 16B, 17, 30) by the first laser radar device (3, 4) to determine relative position data, the relative position data providing relative position information of the first markers (16A, 16B, 17, 30) relative to the first laser radar device (3, 4), taking into account the relative position data and the spatial 3D information about the spatially fixed arrangement of the first and second markers (16A, 16B, 17, 30) relative to each other to perform an automatic detection of the second marker (16A, 16B, 17, 30) and to perform a collaborative measurement of the second marker (16A, 16B, 17, 30) by the second lidar device (3, 4), and The collaborative measurements of the first markers (16A, 16B, 17, 30) and the collaborative measurements of the second markers (16A, 16B, 17, 30) are considered to provide the reference of the UGV lidar data and the UAV lidar data relative to the common coordinate system.

2. The system according to claim 1, It is characterized in that One of the first marker and the second marker is a UGV marker (16A, 16B, 30), the UGV marker being configured to be spatially arranged in a nominal installation of the reference unit in such a way that the UGV lidar device (3) can perform collaborative measurements of the UGV marker (16A, 16B, 30), wherein the collaborative measurements of the UGV marker are performed from a side view field associated with the picture synthesis of the UGV lidar device (3) on the unmanned ground vehicle (1), The other of the first marker and the second marker is a UAV marker (17, 30), and the UAV marker is configured to: in the nominal installation of the reference unit, the UAV marker is spatially arranged in such a manner that the UAV lidar device (4) can perform collaborative measurements of the UAV marker (17, 30), wherein the collaborative measurements of the UAV marker (17, 30) are performed from a top-down field of view associated with the picture synthesis of the UAV lidar device (4) on the unmanned aerial vehicle (2).

3. The system according to claim 1, It is characterized in that The system is configured to access assignment data providing the spatial 3D information about the spatially fixed arrangement of the first and second markers (16A, 16B, 17, 30) relative to each other, and / or At least one of the first marker and the second marker comprises a visual code, which provides the spatial 3D information about the spatially fixed arrangement of the first marker and the second marker relative to each other, wherein the system is configured to determine the spatial 3D information about the spatially fixed arrangement of the first marker and the second marker relative to each other by using a visual pickup device (3, 4).

4. The system according to claim 3, It is characterized in that The visual code is a barcode.

5. The system according to claim 3, It is characterized in that The visual code is a matrix barcode.

6. The system according to any one of claims 1 to 5, It is characterized in that The collaborative scanning of the environment by the UGV laser radar device (3) is provided based on a UGV scanning pattern provided locally by the UGV laser radar device (3), wherein the UGV scanning pattern has a plurality of scanning directions associated with the UGV laser radar device (3), The collaborative scanning of the environment by the UAV LiDAR device (4) is provided based on a UAV scanning pattern provided locally by the UAV LiDAR device (4), wherein the UAV scanning pattern has a plurality of scanning directions associated with the UAV LiDAR device (4), and The UGV scanning pattern and the UAV scanning pattern provide the same local angular distribution of the multiple scanning directions, the same angular point resolution of each scanning direction, and the same range resolution.

7. The system according to any one of claims 1 to 5, It is characterized in that The UGV lidar device (3) and the UAV lidar device (4) are in each case implemented as a laser scanner, which is configured to generate lidar data by means of a rotation of a laser beam (11) about two rotation axes (5, 14), wherein The laser scanner comprises a rotating body (12) configured to rotate about one of the two rotation axes and to provide a variable deflection of the exit and return parts of the laser beam (11), thereby providing a rotation of the laser beam about the one of the two rotation axes, the one of the two rotation axes being a fast axis (14), The rotating body (12) rotates at least 50 Hz around the fast axis (14), the laser beam rotates at at least 0.5 Hz around the other of the two rotation axes, the other of the two rotation axes being the slow axis (5), The laser beam (11) is emitted as a pulsed laser beam, and For the rotation of the laser beam (11) about two axes (5, 14), the field of view about the fast axis (14) is 130 degrees, while the field of view about the slow axis (5) is 360 degrees.

8. The system according to claim 7, It is characterized in that The pulsed laser beam includes 1.5 million pulses per second, thereby providing a point acquisition rate for the lidar data of at least 300,000 points per second.

9. The system according to any one of claims 1 to 5, It is characterized in that The first marking and the second marking (16A, 16B, 17, 30) are provided on a common component such that the relative spatial arrangement of the first marking and the second marking is mechanically fixed.

10. The system according to claim 9, It is characterized in that The common component includes an alignment indicator that provides visual confirmation of the alignment of the common component relative to an external coordinate system or relative to cardinal directions to establish a nominal installation.

11. The system according to any one of claims 1 to 5, It is characterized in that At least one of the first marking and the second marking (16A, 16B, 17, 30) comprises a visually detectable pattern, The system is configured to determine the 3D orientation of the pattern by: determining geometric features (31, 32, 33) in an intensity image of the pattern, wherein the intensity image of the pattern is acquired by scanning the pattern with a lidar measurement beam (11) of the UGV lidar device (3) or the UAV lidar device (4) and detecting the intensity of the returned lidar measurement beam (11), and performing a plane fitting algorithm by analyzing the appearance of said geometrical features (31, 32, 33) in said intensity image of said pattern in order to determine the orientation of the pattern plane, and The system is configured to take into account the 3D orientation of the pattern for providing the reference of the UGV lidar data and the UAV lidar data relative to the common coordinate system.

12. The system according to claim 11, It is characterized in that The visually detectable pattern is provided by areas of different reflectivity, different grey levels and / or different colours.

13. The system according to claim 11, It is characterized in that The pattern comprises circular features (31), The system is configured to identify an image of the circular feature (31) within the intensity image of the pattern, and The plane fitting algorithm is configured to fit an ellipse to the image of the circular feature (31) and to determine the orientation of the pattern plane based on the fitting.

14. The system according to claim 13, It is characterized in that The center of the ellipse is determined, and targeting information is derived for targeting the center of the ellipse with the lidar measurement beam.

15. The system according to claim 13, It is characterized in that The pattern includes internal geometric features (32, 33).

16. The system according to claim 15, It is characterized in that The pattern comprises rectangular features surrounded by the circular features (31).

17. The system according to any one of claims 1 to 5, It is characterized in that The first marker and the second marker (16A, 16B, 17, 30) each include a visual indication of the direction of at least two of the three primary axes spanning the common coordinate system, wherein the system is configured to determine the directions of the three primary axes by using the UGV lidar device (3) and the UAV lidar device (4), and to take into account the directions of the three primary axes for providing the reference of the UGV lidar data and the UAV lidar data relative to the common coordinate system.

18. The system according to claim 17, It is characterized in that The first and second markings (16A, 16B, 17, 30) each include a visual indication of a direction across three of the three principal axes of the common coordinate system.

19. The system according to any one of claims 1 to 5, It is characterized in that The system is configured such that the collaborative measurement of the first marker (16A, 16B, 30) is performed by the UGV lidar device (3), and the collaborative measurement of the second marker (17, 30) is performed by the UAV lidar device (4), wherein the automatic detection of the second marker and the collaborative measurement of the second marker by the UAV lidar device (4) are performed at each takeoff and landing of the unmanned aerial vehicle (2).

20. The system according to claim 19, It is characterized in that The system is configured to continuously update the relative position data so that the relative position information provides continuously updated spatial information about the arrangement between the first marker (16A, 16B, 30) and the UGV lidar device (3).

21. The system according to any one of claims 1 to 5, It is characterized in that The system comprises, in addition to the first and second markers (16A, 16B, 17, 30), further markers (18, 30), which are specifically foreseen for use in image synthesis on the unmanned ground vehicle (1), The UAV lidar device (4) is configured to automatically perform collaborative measurements of the further markers (18, 30), and The system is configured to take into account the collaborative measurements of the further markers (18, 30) to provide the referencing of the UGV lidar data and the UAV lidar data relative to the common coordinate system.

22. The system according to any one of claims 1 to 5, It is characterized in that The system comprises a visual pickup device, which is configured to be arranged on the unmanned ground vehicle (1) or the unmanned aerial vehicle (2), The system is configured to generate a sparse map using the visual pickup device and perform localization of the UGV lidar data or the UAV lidar data in the sparse map.

23. The system according to claim 22, It is characterized in that The visual pickup device is a camera or one of the UGV laser radar device (3) or the UAV laser radar device (4).

24. The system according to claim 22, It is characterized in that The sparse map is generated by photogrammetric triangulation, and the positioning includes a first reference between the UGV lidar data and the UAV lidar data, and After the first referencing, performing a second referencing between the UGV lidar data and the UAV lidar data based on point cloud matching between the UGV lidar data and the UAV lidar data, The sparse map is referenced relative to a known digital model of the environment.

25. The system according to any one of claims 1 to 5, It is characterized in that The system includes a UGV simultaneous positioning and mapping unit and a UAV simultaneous positioning and mapping unit, wherein the UGV simultaneous positioning and mapping unit is a UGV SLAM unit and the UAV simultaneous positioning and mapping unit is a UAV SLAM unit, wherein: The UGV SLAM unit is configured to: receive the UGV lidar data as UGV perception data, the UGV perception data providing a representation of the surrounding environment of the UGV lidar device (3) at a current location; generate a UGV map of the environment using the UGV perception data; and determine a trajectory of a path that the UGV lidar device (3) has traversed within the UGV map of the environment, and The UAV SLAM unit is configured to: receive the UAV lidar data as UAV perception data, the UAV perception data providing a representation of the surrounding environment of the UAV lidar device (4) at a current position; generate a UAV map of the environment using the UAV perception data; and determine a trajectory of a path that the UAV lidar device (4) has traversed within the UAV map of the environment.

26. The system according to any one of claims 1 to 5, It is characterized in that The system is configured to perform system processing, the system processing comprising: performing SLAM processes associated with the unmanned ground vehicle (1) and / or the unmanned aerial vehicle (2); providing the reference of the UGV lidar data and / or the UAV lidar data relative to the common coordinate system; and performing path planning to determine a further trajectory to be followed by the unmanned ground vehicle (1) and / or the unmanned aerial vehicle (2), wherein the system comprises: a UGV computing unit (21), the UGV computing unit being specifically foreseen to be located on the unmanned ground vehicle (1) and configured to perform at least a portion of the system processing, a UAV computing unit, the UAV computing unit being specifically foreseen to be located on the unmanned aerial vehicle (2) and configured to perform at least a portion of the system processing, an external computing unit (23, 20) configured to perform at least a portion of the system processing, A communication unit, the communication unit being configured to: providing intercommunication between the UGV computing unit (21), the UAV computing unit, and the external computing unit (23, 20) by using a cellular communication connection, and providing mutual communication between the UGV computing unit (21) and the UAV computing unit by using a local communication connection, a workload selection module configured to monitor available bandwidth of the cellular communication connection and the local communication connection to perform a dynamic change in assignment of at least a portion of the system processing to the UGV computing unit (21), the UAV computing unit, and the external computing unit (23, 20), wherein the dynamic change in assignment is dependent on: the available bandwidth of the cellular communication connection and the local communication connection, and Prioritization rules for minimizing the available processing load of the UAV computing unit before minimizing the available processing load of the UGV computing unit (21), and minimizing the available processing load of the UGV computing unit (21) before minimizing the available processing load of the external computing units (23, 20).

27. The system according to claim 26, It is characterized in that The cellular communication connection is 4G or 5G, and the local communication connection is WLAN or Bluetooth.

Images