System, method and computer program product providing 3D measurement of an environment

Abstract

The invention relates to a system, method and computer program product providing 3D measurement of an environment. The invention further relates to a system providing 3D measurement of an environment by a mobile reality capture device configured to be carried and moved during generation of 3D measurement data. A motion state tracker is used to determine a motion pattern of the mobile reality capture device, and if the determined motion pattern corresponds to a defined movement of the mobile reality capture device, the system is configured to automatically perform acquisition of an expected motion pattern of the mobile reality capture device. A movement category is associated to an environment specific measurement movement of the mobile reality capture device, and based on this association, an expected (best practice) motion pattern of the mobile reality capture device is associated to a current movement category. By performing a comparison of the determined motion pattern and the expected motion pattern, feedback is provided related to the comparison of the determined motion pattern and the expected motion pattern.

Description

Technical Field

[0001] The present invention relates to a system for providing 3D measurement of an environment by means of a mobile reality capture device configured to be carried and moved during the generation of 3D measurement data. Background Technology

[0002] For example, 3D measurements of buildings and surrounding terrain are of interest to architects or craftsmen to quickly assess the actual condition of rooms or the construction progress of a building site, enabling them to plan subsequent work steps effectively and efficiently. Digital visualization of the actual condition (e.g., in the form of point clouds or vector file models), or augmented reality capabilities, allows for the examination of different or extended options for other steps, which can optionally be presented to employees or clients in an easily accessible manner.

[0003] For example, 3D measurements can be used for environmental mapping, such as generating floor and room plans of buildings, tunnel plans of underground facilities, or pipeline maps of factories.

[0004] The environment can be optically scanned, for example, using time-of-flight methods or photogrammetry, to measure the 3D coordinates of the scene. For instance, a laser scanner emitting a measurement laser beam, such as using pulsed electromagnetic radiation, can scan the environment, receiving echoes from backscattered surface points in the environment and obtaining the distance to those points, which is then correlated with the angular emission direction of the associated measurement laser beam. This generates a 3D point cloud. For example, distance measurements can be based on the time-of-flight, shape, and / or phase of the pulse.

[0005] For additional information, laser scanner data can be combined with camera data, for example, using RGB or infrared cameras, to provide high-resolution spectral information.

[0006] For example, the ranging module used in LiDAR (Light Detection and Ranging) scanners can detect changes in intensity but lacks color sensitivity. This is why 3D models generated purely from LiDAR modules, especially point clouds or vector file models, can only be displayed in grayscale. As a result, many details remain hidden from human observers due to the lack of color effects and the absence of color-supported depth effects. Therefore, "colored" 3D point clouds are typically generated by referencing the "grayscale" 3D point cloud from the LiDAR module with color data from a camera, making the display easier for the human eye.

[0007] The referencing and fusion of different data types (such as laser scanner data, camera data, and positioning data from the Global Navigation Satellite System) are becoming increasingly standardized.

[0008] Specifically, the reality capture device can be mobile and configured to simultaneously provide measurement and reference data. For example, at least the device's trajectory data (e.g., position and / or pose data) is provided with detection data (e.g., LiDAR module data and / or camera data), enabling the detection data from different positions and / or poses of the reality capture device to be combined into a common coordinate system. Typically, the reality capture device is configured to, for example, autonomously create a 3D map of the new environment using Simultaneous Localization and Mapping (SLAM) functionality.

[0009] The 3D model data can then be analyzed using feature recognition algorithms, for example, by using shape information provided by virtual object data from CAD models to automatically identify semantic and / or geometric features captured by the probe data. Such feature recognition (especially for identifying geometric primitives) is now widely used in 3D data analysis.

[0010] Specific problems with existing monitoring devices involve handling changes in environmental conditions, such as alterations in surface reflectivity. This requires lidar sensors to handle a large dynamic range, from the lowest to the highest measurable intensity level. Other problems involve variations in ambient light, which can saturate camera sensors or result in low signal-to-noise ratios in camera data.

[0011] For mobile (e.g., handheld or backpack-mounted) LiDAR-based reality capture devices (such as the Leica BLK2GO), a number of new challenges arise compared to stationary laser scanners. For example, the close proximity of the surveyor to the device results in shadows in the acquired point cloud, requiring the surveyor to learn specific gestures or movements when operating the device in different measurement scenarios to ensure smooth operation and sufficient data quality. These challenging measurement scenarios include walking close to walls, traversing long corridors, opening doors, or climbing or descending stairs.

[0012] For example, when walking close to a wall, the surveyor should keep the device away from the wall. This may require switching hands to hold the device. When opening doors or walking through long corridors, the device should be raised and held to one side so that it is not obstructed by the surveyor's body and more data points are received from behind the surveyor to improve the SLAM algorithm. This is necessary, for example, because for long corridors, the end of the corridor may be too far away, and leaving only the side wall for the SLAM process may not be sufficient. When opening a door, the environment changes abruptly, and many new data points are recorded as soon as the door opens, making localization difficult. When going down stairs, the device needs to be tilted downwards so that the laser scanner's measurement area still "reaches" the stairs / floor and covers the data points from there. The user should go up the stairs or tilt the device downwards far enough so that the device's sensors can see the stairs. Other problem areas could be (the list is not exhaustive) device drop detection, measurements during turns, walking too slowly or too fast, differences in operator skill levels, differences between left-handed and right-handed movement, etc.

[0013] Similar issues may arise when the mobile reality capture device is mounted on a robotic vehicle (e.g., an autonomous, ground-based or aerial vehicle). For example, the mobile reality capture device could be carried by a legged robot (e.g., a quadruped robot), which can handle obstacles such as stairs and is therefore generally able to move freely within a building. Another possibility is the use of aerial drones (e.g., quadcopter drones), which makes it even more versatile for measuring inaccessible areas, but often at the cost of less measurement time and / or sensor complexity due to limited payload capacity and battery power. Summary of the Invention

[0014] Therefore, the object of the present invention is to provide an improved reality capture using a mobile reality capture device, which is easier to use and allows for faster and more reliable capture of changing environments, particularly indoor and outdoor environments of buildings or industrial facilities.

[0015] Another objective is to provide improved reality capture using mobile reality capture devices that are less susceptible to changes in environmental conditions.

[0016] This invention relates to a system for providing 3D measurement of an environment, wherein the system includes a mobile reality capture device configured to be carried and moved during the generation of 3D measurement data. The mobile reality capture device includes a 3D measurement unit configured to provide the generation of 3D measurement data for performing spatial 3D measurement of the environment relative to the mobile reality capture device, wherein the 3D measurement unit is configured to provide spatial 3D measurement with a 360-degree field of view about a first device axis and a 120-degree field of view about a second device axis perpendicular to the first device axis.

[0017] For example, the 3D measurement unit is specifically implemented as a laser scanner, which is configured to perform a scanning movement of a measurement laser beam relative to two rotation axes during the movement of the mobile reality capture device, so as to provide the generation of 3D measurement data based on the scanning movement.

[0018] The system also includes an inertial measurement unit (IMU) comprising sensors with accelerometers and / or gyroscopes, and the IMU is configured to continuously generate IMU data relating to the posture and / or acceleration of the mobile reality capture device.

[0019] For example, a mobile reality capture device includes an inertial measurement unit configured to generate inertial data about the movement of the mobile reality capture device. Alternatively or additionally, the mobile reality capture device is configured to communicate with an accompanying device (e.g., a tablet or smartphone) attached to the mobile reality capture device, and to use data from the inertial measurement unit of the accompanying device to measure the motion pattern of the mobile reality capture device.

[0020] The simultaneous localization and mapping (SLAM) unit is configured to perform a SLAM process that includes generating a map of the environment and determining the trajectory of the mobile reality capture device within the map of the environment.

[0021] For example, the SLAM process utilizes at least one of the following: inertial data (so-called IMU-SLAM), visual data from a camera of a mobile reality capture device or an accompanying device attached to the mobile reality capture device (so-called visual SLAM: VSLAM), 3D measurement data based on lidar (so-called lidar SLAM), and GPS data (such as GNSS data and RTK data).

[0022] 3D measurement data can be generated in such a way that it is associated with positioning data that provides a reference for generating 3D measurement data at different locations relative to a common coordinate system. For example, 3D measurement data can be generated in such a way that data from different locations of the mobile reality capture device are referenced to each other by means of a positioning unit, such as utilizing a SLAM process. Alternatively or additionally, 3D measurement data from different locations of the mobile reality capture device are referenced to each other through post-processing, for example, by using a feature matching algorithm on the 3D measurement data.

[0023] The system also includes a motion state tracker configured to determine the motion pattern of the mobile reality capture device using motion data about its movement. For example, the motion data is provided from IMU data (e.g., raw or processed IMU data). Alternatively, the motion data is obtained from SLAM data from a SLAM unit.

[0024] If the determined motion pattern corresponds to a defined motion category associated with an environment-specific measured motion of the mobile reality capture device, the system is configured to automatically acquire the expected motion pattern of the mobile reality capture device for that environment-specific measured motion. The comparison between the determined motion pattern and the expected motion pattern is performed by the system to provide feedback on the comparison between the determined motion pattern and the expected motion pattern.

[0025] Specifically, the mobile reality capture device can be configured to take this feedback into account to automatically perform actions associated with the expected movement pattern. For example, the feedback can be considered for real-time user guidance, such as triggering commands instructing the user to specifically move the device (e.g., "keep the device more horizontal or more vertical"), or indicating target movement by drawing guidance with the augmented reality device to point the device at a marker or object. The feedback can also be used to initiate user training, for example, to trigger a warning to a supervisor of the measurement personnel, to propose a training procedure, or to trigger log events for backend storage for later use (e.g., creating a user story). Another possibility is to use specific movements to trigger specific functions of the mobile reality capture device, such as a specific movement that shuts down the mobile reality capture device.

[0026] For example, the determination of motion patterns, the acquisition of expected motion patterns, and the comparison of the determined motion patterns with the expected motion patterns are performed in real time. For instance, feedback is provided to the mobile reality capture device and / or accompanying devices for real-time guidance of the user of the mobile reality capture device or the robotic carrier. For example, the comparison is used to verify that the operator is operating the device as instructed. If the operator is not operating the device as instructed, feedback is provided to the operator, for example, by means of visual instructions, audio instructions, or instructions via at least one of an accompanying device (such as a tablet or smartphone) communicating with the system (e.g., with the mobile reality capture device), to adjust the movement of the mobile reality capture device.

[0027] Alternatively or otherwise, feedback may be provided to the training algorithm to improve the classification of motion patterns and / or the acquisition of expected motion patterns.

[0028] In one implementation, the system includes a database containing multiple defined motion patterns, each associated with an environment-specific measured motion of a motion reality capture device. For example, each defined motion pattern may be predefined or user-defined. The database is used for classifying identified motion patterns and / or obtaining expected motion patterns.

[0029] For example, the system is configured to establish a data connection with a remote server computer and provide motion data to the remote server computer. Then, the system is configured to detect typical behavior of the person carrying the mobile reality capture device based on the motion data and send corresponding motion data to the remote server computer, wherein the sent data is dedicated to updating predefined motion patterns stored on the remote server computer, and the system is configured to receive the updated predefined motion patterns from the remote server computer.

[0030] For example, environment-specific measurement movement is the movement of a motion reality capture device that is specifically anticipated to ensure the definition quality of the 3D measurement unit, such as defining coordinate measurement accuracy and / or measurement point density. For instance, environment-specific measurement movement includes at least one of the following: measurement movement while carrying the motion reality capture device during opening and / or closing a door; measurement movement while carrying the motion reality capture device while walking through a door; measurement movement while moving the motion reality capture device along a corridor or in a tunnel; and measurement movement while carrying the motion reality capture device while climbing or descending stairs.

[0031] In another embodiment, the anticipated motion pattern provides a nominal orientation or a sequence of nominal orientations of the mobile reality capture device with respect to three mutually perpendicular rotational axes (e.g., the tilt axis, pitch axis, and yaw axis of the mobile reality capture device).

[0032] In another embodiment, the anticipated motion pattern provides a nominal relative position change of the mobile reality capture device with respect to its current position about three mutually perpendicular spatial axes, or a sequence of nominal relative position changes, for example, wherein the three mutually perpendicular spatial axes are provided with reference to a gravity vector, and in particular, wherein one of the spatial axes is parallel to the gravity vector.

[0033] For example, the correspondence between the determined motion patterns and the defined motion categories and / or the acquisition of expected motion patterns are provided by machine learning (ML) algorithms, which include processing motion data by Kalman filters to estimate the attitude parameters of the motion reality capture device, particularly the velocity parameters.

[0034] For example, one or more machine learning models (such as decision trees, random forests, support vector machines (SVMs), or neural networks) are trained to classify identified motion patterns into different motion categories. These models can, for example, run on the computing unit of a mobile reality capture device, such as at the edge, i.e., on the mobile reality capture device itself. Alternatively, or additionally, the mobile reality capture device can be configured to establish a data connection with a separate computing unit, such as on a central server or on an accompanying device such as a smartphone or tablet, wherein at least a portion of the models runs on that separate computing unit. Multiple machine learning models can run in parallel to detect multiple motion states simultaneously, because states may overlap or consist of connected states (e.g., a combination of lateral movement and upward movement).

[0035] For example, motion data processing is performed in segments of at least 1.5-second time windows. Alternatively, motion data processing is performed in a rolling manner by continuously processing a time series of continuously generated motion data.

[0036] In another embodiment, the correspondence between the determined motion pattern and the defined movement category, and / or the acquisition of the expected motion pattern, is provided by considering a feature extraction step that provides the detection of signal features from a plurality of different signal features, each of which indicates one of a plurality of defined environment-specific measured movements. The feature extraction step is then used to estimate attitude parameters, particularly velocity parameters, for example, where the signal features are used as input to a classical machine learning algorithm.

[0037] For example, the feature extraction step is provided by a deep learning algorithm configured to independently learn signal features. Alternatively, for example, in the case of a classical machine learning algorithm, the feature extraction step is provided by using defined statistics to compute the motion data in the frequency and / or time domains.

[0038] Mobile reality capture devices can be used by a variety of different handlers (e.g., humans or robots) and can be combined with different accompanying devices, such as attaching a smartphone or tablet to the mobile reality capture device for user guidance. For example, the mobile reality capture device can be configured to be held by a user or mounted in a backpack. If mounted on a robot, a variety of different robot types can be used, such as wheeled or legged robots, or flying vehicles, where the device itself and / or attachments can be mounted in a variety of different spatial arrangements. This results in different geometries of setup, weight distribution, and / or moment of inertia. In particular, when using machine learning models, the system can be configured to cope with multiple different configurations and uses of the mobile reality capture device. For example, the system can be configured to implicitly learn different configurations and associated environment-specific measured motions through so-called transfer learning, or explicitly learn through a dedicated calibration function performed by the handler of the mobile reality capture device.

[0039] For example, the system is configured to analyze motion data to generate a motion model that takes into account parameters of the range of motion of the relative movement of the motion reality capture device as it is handled and aligned by a handler, and / or the weight distribution of the combination of the motion reality capture device with accompanying devices and / or the handler. For example, the center of mass and moment of inertia of the combination of the motion reality capture device with accompanying devices and / or the handler are determined. The motion model is then considered for at least one of the following: providing a correspondence between the determined motion pattern and a defined motion category, obtaining a desired motion pattern, and comparing the determined motion pattern with the desired motion pattern.

[0040] The generation of the mobile model can be based on the optimization of a general model with free parameters, wherein the general model is adapted to the corresponding mobile reality capture device (e.g., device configurations with different geometric device dimensions, device configurations with or without attached accompanying devices, different mounting configurations of accompanying devices, center of mass and moment of inertia), and users with different body sizes, arm lengths and leg lengths, etc.

[0041] In another embodiment, the mobile reality capture device includes a calibration function based on a set of predefined controlled movements of the mobile reality capture device performed by the person being carried, wherein motion data measured during the controlled movements is analyzed to generate a motion model. For example, a parameter of the range of motion provides information about the length of the boom assembly of the mobile reality capture device (e.g., the arm length of a human carrier).

[0042] Evaluating the current measurement situation primarily based on motion data offers the advantage that it is largely independent of changing environmental conditions. If available, the evaluation of the current measurement situation can be improved through so-called context-specific evaluation, where additional background information about the environment and the arrangement of the motion reality capture device within the environment is provided by, for example, the visual sensors of the motion reality capture device and / or accompanying devices. For example, context-specific evaluation includes detecting proximity to a wall based on distance measurements from a 3D measurement unit, or detecting movement through a door using an ML-based door detection algorithm based on image and / or LiDAR data.

[0043] In another embodiment, the mobile reality capture device is configured to acquire, for example, perceptual data from 3D measurement data and / or from sensors of a simultaneous localization and mapping unit, wherein the perceptual data provides visual recognition of spatial features of the environment and for evaluating the spatial arrangement of the mobile reality capture device relative to these spatial features. The system is configured to analyze the perceptual data to provide identification of environment-specific measurement conditions regarding the spatial arrangement of the mobile reality capture device relative to spatial features in the environment, and to consider these environment-specific measurement conditions to obtain the motion category of a determined motion pattern (e.g., identifying the existence of a defining motion category) and / or to obtain a desired motion pattern.

[0044] For example, the system is configured to access a database comprising a set of geometric and / or semantic classes of spatial features, each set having corresponding classification parameters for identifying the geometric and / or semantic classes via perceptual data. Each of the geometric and / or semantic classes is associated with at least one of the following: rules regarding the minimum and / or maximum distance between the mobile reality capture device and the corresponding spatial feature associated with that class; and rules regarding the nominal relative orientation of the mobile reality capture device relative to the corresponding spatial feature associated with that class. The system is configured to use the database to identify environment-specific measurement conditions and to consider these conditions for obtaining the motion category of a determined motion pattern (e.g., identifying and defining a motion category) and / or for obtaining an expected motion pattern.

[0045] For example, the system includes optimization algorithms that take into account information from motion state trackers and rules provided by a database in order to improve the acquisition of expected motion patterns.

[0046] In another embodiment, the system includes a machine learning-based object detection algorithm, specifically configured to identify spatial constellations, such as doors, corridors, and stairs, within the perceived data. The spatial constellations are associated with a predefined sequence of motion states of the mobile reality capture device, such as a sequence of relative orientations and / or distances between the mobile reality capture device and the spatial constellations. The system is configured to consider the identification of spatial constellations by the object detection algorithm to identify environmentally specific measurement conditions, wherein the predefined sequence of motion states is considered to obtain the expected motion pattern of the mobile reality capture device.

[0047] In another embodiment, the system is configured to access map-building data of a model providing the environment, such as a relocation map based on sparse point clouds, a building information model (BIM) or computer-aided design (CAD) model, or similar structured data, and to track the position of the mobile reality capture device within the model of the environment. The position of the mobile reality capture device is then taken into account to obtain the motion category of the determined motion pattern (e.g., to identify and define the motion category) and / or to obtain the expected motion pattern (26).

[0048] In another embodiment, the system includes the object detection algorithm described above and uses the position of the mobile reality capture device within the model of the environment to identify spatial constellations for identifying environment-specific measurement conditions. Thus, for example, by continuously tracking its position within the model of the environment, the system can eliminate false alarms by comparing the class of detected motion patterns with its geometrically surrounding environment. Alternatively, or additionally, the comparison of the determined motion pattern with the expected motion pattern is triggered by the identification of spatial constellations. For example, once a spatial constellation indicates "the door is closed and it is likely that passage is imminent," only whether the mobile reality capture device is tilted sideways and raised upwards is checked.

[0049] The present invention also relates to a method for 3D measurement of an environment using a mobile reality capture device, the mobile reality capture device including a 3D measurement unit configured to provide the generation of 3D measurement data for performing spatial 3D measurement of the environment relative to the mobile reality capture device, wherein the 3D measurement unit is configured to provide spatial 3D measurement with a 360-degree field of view about a first device axis and a 120-degree field of view about a second device axis perpendicular to the first device axis. The method includes the following steps: generating 3D measurement data using a 3D measurement unit during movement of the mobile reality capture device; generating IMU data related to the posture and / or acceleration of the mobile reality capture device; and performing a simultaneous localization and mapping (SMR) process, which includes generating a map of the environment and determining the trajectory of the mobile reality capture device within the map of the environment; determining the motion pattern of the mobile reality capture device using motion data regarding its movement; associating the determined motion pattern with a defined motion category associated with an environment-specific measurement motion of the mobile reality capture device; obtaining the expected motion pattern of the mobile reality capture device for the environment-specific measurement motion according to the defined motion category; performing a comparison between the determined motion pattern and the expected motion pattern; and providing feedback related to the comparison between the determined motion pattern and the expected motion pattern, particularly wherein the feedback is taken into account when performing an action associated with the expected motion pattern.

[0050] Implementation of the method may include additional steps necessary for implementing and / or operating the system as described above, particularly any of the processing, obtaining, using, and determining steps for the system as described above.

[0051] The present invention also relates to a computer program product comprising program code stored on a machine-readable medium, or embodied by electromagnetic waves including program code segments, wherein the program code includes computer-executable instructions for performing the methods described above when executed in a measurement system, particularly in the system described above. Attached Figure Description

[0052] Hereinafter, systems, methods, and computer program products according to different aspects of the invention will be described or illustrated in more detail by way of example, with reference to the schematic examples shown in the accompanying drawings. The same reference numerals are used to label the same elements in the drawings. The embodiments are generally not shown to scale, and these embodiments should not be construed as limiting the invention. Specifically,

[0053] Figure 1 This is an exemplary implementation of a mobile reality capture device;

[0054] Figure 2 This is an exemplary implementation of a laser scanner to be used within a mobile reality capture device;

[0055] Figure 3 This is an exemplary environmental measurement scenario where a user walks down the stairs;

[0056] Figure 4 This is another exemplary environmental-specific measurement scenario where a user walks through a long corridor;

[0057] Figure 5 This is another exemplary environmental-specific measurement scenario in which the user walks close to the wall;

[0058] Figure 6 This is another exemplary implementation of a mobile reality capture device, which includes an LED ring and an interface for attaching accompanying devices to provide feedback to the user;

[0059] Figure 7 This is an exemplary implementation of IMU data analysis to detect the behavior of a surveyor and provide the surveyor with feedback on best practices and their current behavior;

[0060] Figure 8 This is another exemplary implementation of IMU data analysis, which further includes the analysis of sensed data;

[0061] Figure 9 This is an exemplary implementation in which the mobile reality capture device is carried by a robot to measure the building. Detailed Implementation

[0062] Figure 1 It shows a laser scanner (see) Figure 2 (details in the text) and an exemplary embodiment of a mobile reality capture device 1 having a camera unit with multiple cameras 2.

[0063] The laser scanner has a cover 3 that is opaque to visible light but optically transmissive to the measuring laser beam. The cover 3, together with the base of the laser scanner, forms a closed housing that is fixed relative to the moving reality capture device, wherein all moving parts of the laser scanner are surrounded by the housing.

[0064] For example, the mobile reality capture device can be configured to require only a minimum number of controls integrated into the device. For instance, the device may have only a single integrated control unit 4, where individual measurement procedures and / or actions of the reality capture device can be triggered by different tap sequences of the control unit 4. Such measurement procedures or actions may include at least one of the following: enabling / disabling the laser scanner, starting a defined measurement process, or interrupting / cancelling and restarting the measurement process.

[0065] The mobile reality capture device can also be configured such that defined measurement procedures and actions are stored on the device, and / or new measurement procedures and actions can be defined by the user, for example, via an accompanying device such as a tablet computer.

[0066] For example, the mobile reality capture device also includes a light indicator 5, which is used, for instance, to indicate the device status in such a way that the status indication appears consistent in all azimuth directions around the vertical axis of the reality capture device. Furthermore, the light indicator 5 can be configured to provide guidance commands.

[0067] Figure 2 It shows Figure 1 The image shows a close-up of a laser scanner, including a base 6 and a support 7, which is rotatably mounted on the base 6 about a support rotation axis 8. Typically, the rotation of the support 7 about the support rotation axis 8 is also referred to as azimuth rotation, regardless of whether the laser scanner or the support rotation axis 8 is perfectly vertically aligned.

[0068] The core of the laser scanner is an optical ranging unit 9, which is located at the support 7 and configured to perform distance measurement by emitting transmitted radiation 10 (e.g., pulsed laser radiation) and detecting the reflected portion of the transmitted radiation by means of a receiving unit including a photosensitive sensor. Thus, pulse echoes are received from backscattering surface points in the environment, and the distance to these surface points can be obtained based on the flight time, shape, and / or phase of the emitted pulses.

[0069] In the illustrated embodiment, the scanning movement of the measuring laser beam about the two rotation axes 8 and 11 is achieved by rotating the support 7 relative to the base 6 about the support rotation axis 8 and by means of a rotating body 12, which is rotatably mounted on the support 7 and rotates about the beam rotation axis 11.

[0070] For example, both the transmitted radiation 10 and the returning portion of the transmitted radiation are deflected by means of a reflective surface 13 that is integral with or applied to the rotating body 12.

[0071] Alternatively, the transmitted radiation originates from one side of the reflective surface 13, i.e., from the interior of the rotating body 12, and is emitted into the environment (not shown) via a channel region within the reflective surface.

[0072] To determine the emission direction of the ranging beam 10, many different angle determination units are known in the prior art. For example, the emission direction can be detected by means of an angle encoder configured to acquire angle data used to detect the absolute angular position and / or relative angular change of the support 7 about the rotation axis 8 of the support, or the absolute angular position and / or relative angular change of the rotating body 12 about the beam rotation axis 11. Another possibility is to determine the angular position of the support 7 or the rotating body by detecting only the full rotation and by using knowledge of a set rotation frequency.

[0073] Data visualization can be based on well-known data processing steps and / or display options, such as presenting the acquired data in the form of 3D point clouds, 3D vector file models, voxels, or meshes.

[0074] One of the challenges when using the mobile reality capture device described above for map building / data collection is that the user needs to perform movements / actions that capture data with sufficient quality. Inexperienced users may take some time to master the data capture process. According to one aspect of the invention, the user is guided / taught to perform correct movements, for example, where suboptimal movements provide feedback and prompts to the user to adapt to the correct data capture process. For example, the feedback can be explicit movement instructions or instructions related to what the user is currently doing. Alternatively, or additionally, the correct movement can also be indicated.

[0075] Feedback can be provided in real time. Alternatively, or additionally, feedback can be obtained / improved in post-processing, which provides additional possibilities for defining or obtaining expected motion patterns and provides additional robustness for learning or classifying the identified motion patterns.

[0076] For example, user guidance can be provided by analyzing usage patterns from inertial data read from an inertial measurement unit (IMU) associated with the movement of the mobile reality capture device. For instance, the inertial data is provided by the mobile reality capture device's IMU. Alternatively or additionally, the mobile reality capture device can be configured to communicate with an accompanying device (e.g., a tablet or smartphone) attached to it, and the motion patterns of the mobile reality capture device can be measured using data from the accompanying device's IMU.

[0077] Figures 3 to 5 Different environment-specific measurement scenarios are shown regarding the spatial arrangement of the mobile reality capture device 1 relative to spatial features in the environment, where the user needs to follow certain environment-specific measurements of the mobile reality capture device 1 to ensure optimal data quality.

[0078] Figure 3 The scenario depicts a user descending stairs, i.e., an environment-specific measurement situation. In a nominally upright position, mobile reality capture devices typically have a limited field of view facing the ground. Therefore, in order to measure stairs, the user must either walk up the stairs or tilt the device downwards while walking down them so that the device's sensors (especially the LiDAR unit) can see the stairs.

[0079] Figure 4 The scenario depicts a user walking through a long corridor. Here, the motion reality capture device 1 should be raised so that it is not obstructed by the user's body 14, and more data points should be received from behind the user to improve the SLAM algorithm. This is necessary, for example, because for a long corridor, the end of the corridor may be too far away, and leaving only the side walls for SLAM may not be sufficient. In particular, if possible, the motion reality capture device 1 should be raised upwards and to one side to avoid obstructing the field of view 14 by the user's body. In this figure, lateral movement is limited because the motion reality capture device 1 should not be positioned too close to the wall (see below).

[0080] Figure 5 The scenario depicts a user walking close to a wall. To improve efficiency, for example, the user should keep the device in their hand away from the wall. This may require switching hands while holding the device.

[0081] For example, another environment-specific measurement scenario involves a door opening, where the environment undergoes a sudden change with many new data points as soon as the door opens, which can make positioning difficult. Here, it might be necessary to lift the motion reality capture device when the user pauses briefly under the door. Moreover, it might be recommended to hold the device away from the door (or turn it into the hand) and have the user walk sideways through the door.

[0082] According to one aspect of the invention, IMU data, as well as other possible visual data, is analyzed to detect whether the surveyor is following best practices, and these events (hand movements, device moving upwards and to one side, device tilting downwards, surveyor taking a short break, etc.) are detected using machine learning. By verifying whether the surveyor is operating the motion reality capture device as prescribed, information / instructions are provided to the user to help him operate the device correctly. For example, an event in an environment-specific measurement situation is associated with an event-specific measurement movement of the motion reality capture device. If the system has not yet detected movement when it detects the event, the user is instructed to follow that event-specific measurement movement. For example, if the system detects that the user is walking down stairs but has not tilted the device, the user is notified to do so. This also applies to other mentioned movements / gestures.

[0083] Figure 6 Another exemplary embodiment of the mobile reality capture device 1 is depicted, which includes an LED ring 15 and an interface 16 for attaching an accompanying device 17 (e.g., a smartphone). The accompanying device 17 and the mobile reality capture device 1 are configured to communicate with each other, for example, via a wired or wireless connection.

[0084] For example, the LED ring 15 includes a plurality of light indicators 18, wherein each of the light indicators is assigned to a different azimuth direction (for a nominal upright arrangement of the device). For example, the plurality of light indicators 18 includes six light indicators such that the light indicators 18 correspond to the "forward" direction (e.g., with...). Figure 1 The control components depicted are opposite (reverse), "rearward", "left front", "right front", "left rear", and "right rear".

[0085] For example, light indicators can provide, for instance, in real time, an indication of the expected measured movement / gesture to be performed by the motion reality capture device 1 to ensure good data quality of the lidar data. This indication can be given by assigning a predefined movement sequence and / or a relative direction to one of these light indicators, for example, where different information is provided by means of LED rings and / or the flashing and / or color coding of the light indicators.

[0086] Alternatively or additionally, instructions for the expected measurement movements / gestures of the mobile reality capture device 1 are provided via the accompanying device 17. For example, this provides additional or more detailed information, such as for inexperienced users learning the correct operation of the mobile reality capture device. The representation of the measurement data can be displayed on the accompanying device, for example, where specific colors are used to highlight different aspects of the measurement process, such as scan gaps, different point densities, or the measurement area being captured with acceptable / unacceptable movement.

[0087] Another possibility for providing indications of the expected measured movement / gesture could be an audible command or alarm, or a device for haptic feedback, such as a motion reality capture device that captures vibrations. In another modification, a laser pointer or some other projection device could be mounted on a gimbal to keep the device rotating and pointing to a specific area, for example, where more data is needed to guide and instruct the user.

[0088] The system may also include an augmented reality device (e.g., augmented reality glasses) configured to provide the user with instructions on the correct movement of the motion capture device using feedback from a comparison of the determined motion pattern with the expected motion pattern. Furthermore, the augmented reality device can be used to notify the user of specific problem areas in the environment that require special attention / special movement and to direct the user to those specific problem areas. For example, these problem areas could be provided by an object detection algorithm for identifying star clusters, as described above.

[0089] Figure 7 The illustration schematically depicts an implementation of IMU data analysis for detecting a measurer's behavior and providing feedback to that measurer regarding best practices and their current actions. For example, the data analysis may also include determining the degree of a user's skill, such as binary classification or percentage estimation for informational purposes only or for user skills recommended through subsequent training.

[0090] The motion state tracker 19 is fed inertial data from an IMU 20 associated with the movement of the mobile reality capture device. For example, the IMU 20 is part of the mobile reality capture device or part of an accompanying device attached to the mobile reality capture device. The motion state tracker 19 measures and monitors the motion pattern 21 of the mobile reality capture device, wherein a machine learning module 22, including one or more machine learning algorithms, is used to detect one or more environment-specific measurement conditions indicating the mobile reality capture device and one or more specific movement categories 23 corresponding to the environment-specific movement within the determined motion pattern 21.

[0091] For example, machine learning module 22 includes deep learning module 24, which also extracts signal features. Alternatively, or additionally, the machine learning module includes a so-called "classical" machine learning model, which includes the signal extraction process described above. Furthermore, machine learning module 22 may also include a comparator 25, since classifying motion pattern 21 into a specific movement category 23 involves comparing it with the expected motion pattern 26.

[0092] For example, in this figure, the identified motion pattern 21 involves a surveyor walking through a long corridor, resulting in movement at a roughly constant speed and height without any lateral changes. For simplicity, the identified motion pattern 21 is depicted as a time series of height changes z of the motion reality capture device (increasing time t to the right). Other motion categories involve other specific movements of the motion reality capture device while performing a particular measurement task, such as climbing or descending stairs, opening a door, measuring a single object (e.g., where the object is measured using a circular pattern to observe all sides of the object), high-resolution measurements of a single part (e.g., where multiple passes of the motion reality capture device are required), etc.

[0093] Based on the detected environment-specific measurement conditions, the machine learning module 22 obtains the expected motion pattern 26 of the mobile reality capture device, for example by taking into account the nominal environment-specific measurement motion associated with the current motion category 23, as well as other limiting factors that may be provided by IMU data and / or visual perception data.

[0094] For example, to learn (i.e., “train”) the machine learning module 22, expected motion patterns are recorded, where the record includes different device configurations and different users. This covers a large space of possible expected motion patterns. As discussed earlier, different device-accompanying configurations and different handlers (e.g., different people) may cause different motion patterns on the device, thus requiring calibration. The machine learning module learns a single model by learning the features of each motion pattern independently of the device configuration or user. Training is performed offline, and the model encoding the expected motion patterns is uploaded to the device. Instead of a single algorithm, multiple algorithms can be used, for example, one algorithm per expected motion pattern, because motion patterns may overlap. The expected motion pattern data is split into two datasets: a dataset that will be used to train the model, and a dataset that will be used to test this model and all other models.

[0095] The machine learning module may be available on the mobile reality capture device and / or on a separate processing unit, for example, where the separate processing unit uses the data from the mobile reality capture device in a streaming manner. For example, in one embodiment, the machine learning module is used to post-process the data on a separate computer. Here, for example, the recorded trajectory is visualized by different identified motion patterns. Additionally, events such as walking down / up stairs or opening a door can be highlighted in / on the trajectory.

[0096] Machine learning algorithms allow for semi-automatic calibration using transfer learning and retraining capabilities. Users can record data on their own expected motion patterns, for example, guided by visual feedback on the device or in a mobile app. This is accomplished by recording the user's motion patterns on the device and then performing retraining offline (e.g., at a computer that will load the recorded data and software or in the cloud) or directly on the device. In both cases, features are computed as described above, where the new training process is guided using a previously trained machine learning model and / or the previously recorded motion patterns. For example, the newly recorded data is automatically split into training and test datasets. The model is trained using the new training data and optionally the old training data. The new model can then be loaded onto the device. To ensure the quality of the model and evaluate its performance, automated testing can be invoked to verify, for example, by using motion patterns recorded from both the old and new test datasets (regression testing).

[0097] The expected motion pattern 26 is then provided to the comparator 25, which is configured to perform a comparison between the determined motion pattern 21 and the expected motion pattern 26. If the comparison shows that the user has performed the correct movement / gesture using the motion reality capture device, corresponding feedback 27 (so-called positive feedback) is provided to the user.

[0098] In another implementation, so-called “anomalies” are detected without defining expected motion patterns. Using unsupervised machine learning, “regular” or “nominal” movement behavior is learned by observing the measured motion pattern 21. Thus, expected motion patterns are implicitly acquired, and the comparator can warn the user and trigger an alarm when unexpected motion patterns occur. For example, such unexpected behavior could be any motion pattern that is not part of the regular measured behavior, such as a device falling or other sudden movement.

[0099] In the scenario depicted in the figure, the comparison (including the analysis of the complete identified motion pattern 21 so far) shows that the user has not yet performed the best practices for measurement (e.g., raising the motion reality capture device upwards and to one side) while walking through a long corridor to avoid obstructing the field of view of the 3D measurement unit (e.g., the LiDAR unit) with their body. Here, corresponding feedback 27 is provided to the user, which informs and / or guides the user regarding the desired best practice movement. Later, once the comparator 25 detects the upward raising 28 and lateral movement of the motion reality capture device, the comparison again results in positive feedback.

[0100] Figure 8Another implementation of IMU data analysis is illustrated schematically, which further includes the analysis of sensing data 29, which is obtained, for example, from coordinate measurement data provided by a lidar unit and imaging data provided by a camera of a mobile reality capture device.

[0101] Perceptual data 29 provides visual recognition of spatial features of the environment and is used for evaluating the spatial arrangement of the mobile reality capture device relative to these features. For example, when walking in a long corridor, perceptual data allows for determining the distance to the corridor walls. Therefore, by analyzing perceptual data, which provides recognition of the scene "walking in a long corridor," and thereby provides an indication to move the mobile reality capture device upwards and to one side, the analysis of this perceptual data can indicate, based on the measured distance to the wall, which side the mobile reality capture device must move to (if any). In other words, perceptual data allows for improved context-specific evaluations that take into account specific environmental conditions.

[0102] In one implementation, the IMU 20 is used in conjunction with the results of on-device SLAM (not shown), rather than with “raw” data, such as information like position, velocity, and attitude determined by the SLAM algorithm. For example, the SLAM algorithm can process lidar data, visual data, and IMU data.

[0103] For example, rule-based methods, using machine learning (i.e., machine learning-based detection and classification) or a hybrid approach, can be used to identify best practice movements and gestures. For instance, a hybrid approach might include a decision tree with relevant scenarios and corresponding machine learning models, or conversely, a hybrid approach might use classification of processes using machine learning followed by a rule-based approach.

[0104] For example, rule-based approaches refer to using a fixed, more or less manually defined set of rules (so-called "conditions") to identify certain states. These conditions can be defined in a data-driven manner and / or can be partially adjusted manually by the user (e.g., fitting thresholds, fine-tuning the length of time intervals, limiting the occurrence of each time interval, etc.). For example, a rule could be an "if-then" condition and / or a simple threshold definition. In particular, rules can be obtained statistically.

[0105] Rule-based methods only allow for relatively simple rules, and are almost always manually defined. Machine learning, on the other hand, can identify and learn complex patterns or rules. Examples of ML-based algorithms that can be used include decision trees, random forests, support vector machines, or neural networks. These algorithms can be used for both signal classification and detection. For instance, during classification, the last measured value (e.g., the last two seconds) is used with a defined update frequency (e.g., 100 Hz) and the algorithm classifies the signal. If necessary, further steps, such as feature extraction, are performed before invoking the algorithm. Neural networks or other "deep learning" methods learn features independently. For classic algorithms, defined statistics can be computed using either the frequency or time domain. ML-based detectors work in a similar way, but process time series continuously, i.e., in a streaming manner.

[0106] Hybrid approaches combine rule-based methods with machine learning models. For example, simple states can be identified based on rules, while more complex states can be identified using ML methods.

[0107] Figure 9 Another exemplary embodiment is depicted, in which the mobile reality capture device 1 is carried by a robot 30 to measure the building 31. For example, the robot 30 is specifically implemented as a quadruped robot, wherein data from the 3D measurement unit can be used as perceptual data to move the robot autonomously.

[0108] For example, closed-loop control can be implemented to process motion state detection information and feedback from comparisons between determined motion patterns and expected motion patterns into robot control commands.

[0109] For example, similar to a user walking through an environment, a robot's specific movements and the resulting specific movements of the motion reality capture device are triggered by the system recognizing specific environmental measurement situations such as an upcoming door to pass through.

[0110] Robots can be configured to move with or without prior machine-readable information about the environment prior to the task. For example, in so-called exploration mode, active user controls are used quite frequently to move / command the robot. Alternatively, or additionally, robots can be configured to provide automated repetitive tasks, including task planning for robot navigation and real-time navigation improvements based on prior machine-readable location information such as location or relocation maps (e.g., sparse point clouds, building information models (BIM), or computer-aided design (CAD) models).

[0111] For example, the system can be configured to access map-building data that provides a model of the environment and track the position of a mobile reality capture device within that model. The position of the mobile reality capture device is then considered to define the movement category of the determined motion pattern and / or obtain the expected motion pattern.

[0112] Although the invention has been illustrated above, reference has been made in part to some preferred embodiments. It must be understood that many modifications and combinations of different features can be made to these embodiments. All such modifications fall within the scope of the appended claims.

Claims

1. A system for providing 3D measurement of an environment, the system comprising a motion reality capture device configured to be carried and moved during the generation of 3D measurement data, wherein, The system includes: A 3D measurement unit is disposed on the mobile reality capture device and configured to generate 3D measurement data for performing spatial 3D measurement of the environment relative to the mobile reality capture device. The 3D measurement unit is configured to provide the spatial 3D measurement with a 360-degree field of view about a first device axis and a 120-degree field of view about a second device axis perpendicular to the first device axis. An inertial measurement unit (IMU) includes sensors with accelerometers and / or gyroscopes, and is configured to continuously generate IMU data related to the posture and / or acceleration of the mobile reality capture device. Simultaneous localization and mapping (SMR) unit, configured to perform a simultaneous localization and mapping (SMR) process, the SMR process including generating a map of the environment and determining the trajectory of the mobile reality capture device within the map of the environment. Its features are, The system includes a motion state tracker configured to determine a motion pattern of the mobile reality capture device using motion data related to the movement of the mobile reality capture device. If the determined motion pattern corresponds to a defined motion category associated with an environment-specific measured movement of the mobile reality capture device, the system is configured to automatically obtain a desired motion pattern of the mobile reality capture device for that environment-specific measured movement and perform a comparison between the determined motion pattern and the desired motion pattern. The system is configured to provide feedback related to a comparison between a determined motion pattern and the expected motion pattern, wherein the motion reality capture device is configured to take the feedback into account to automatically execute an action associated with the expected motion pattern. The correspondence between the determined motion pattern and the defined motion category and / or the acquisition of the expected motion pattern is provided by a machine learning algorithm, which includes processing the motion data by a Kalman filter to estimate the attitude parameters of the motion reality capture device, wherein the attitude parameters include velocity parameters.

2. The system according to claim 1, wherein, The system includes a database containing multiple defined motion patterns, each defined motion pattern being associated with an environment-specific measured motion of the mobile reality capture device, wherein the database is used for the classification of determined motion patterns and / or the acquisition of the expected motion patterns.

3. The system according to claim 2, wherein, Each defined motion pattern is either predefined or user-defined.

4. The system according to claim 2, wherein, The system is configured to establish a data connection with a remote server computer and provide motion data to the remote server computer, wherein the system is configured to: The typical behavior of the person carrying the mobile reality capture device is detected based on the motion data, and corresponding motion data is sent to the remote server computer. This sent data is specifically used to update predefined motion patterns stored on the remote server computer. Receive updated predefined motion patterns from the remote server computer.

5. The system according to any one of claims 1 to 4, wherein, The expected motion pattern provides: The nominal orientation or a sequence of nominal orientations of the mobile reality capture device about its three mutually perpendicular rotational axes, and / or The nominal relative position change of the mobile reality capture device with respect to its current position about three mutually perpendicular spatial axes, or a series of nominal relative position changes.

6. The system according to any one of claims 1 to 4, wherein, The processing of the motion data is performed in segments for time windows of at least 1.5 seconds, or the processing of the motion data is performed in a rolling manner by continuously processing a time series of continuously generated motion data.

7. The system according to claim 1, wherein, The correspondence between the determined motion pattern and the defined motion category, and / or the acquisition of the expected motion pattern, is provided by considering a feature extraction step, which includes: This provides a method for detecting signal features from a plurality of different signal features, wherein each of the plurality of different signal features indicates a defined environment-specific measurement movement in a plurality of defined environment-specific measurement movements, and It was used for estimating the attitude parameters.

8. The system according to claim 7, wherein, The feature extraction step is provided by a deep learning algorithm configured to independently learn the signal features, or the feature extraction step is provided by calculating the motion data using defined statistics in the frequency or time domain of the motion data.

9. The system according to claim 4, wherein, The system is configured to analyze motion data to generate a motion model that takes into account parameters of the range of motion of the relative movement of the motion reality capture device as it is carried and aligned by the carrier, and / or the weight distribution of the combination of the motion reality capture device with accompanying devices and / or the carrier, wherein the motion model is considered for at least one of: providing a correspondence between a determined motion pattern and the defined motion category, obtaining the expected motion pattern, and comparing the determined motion pattern with the expected motion pattern.

10. The system according to claim 9, wherein, The center of mass and moment of inertia of the combination of the mobile reality capture device, the accompanying device, and / or the transporter are determined.

11. The system according to claim 9, wherein, The mobile reality capture device includes a calibration function based on a set of predefined controlled movements performed by the transporter, wherein motion data measured during the controlled movements is analyzed to generate the movement model.

12. The system according to claim 11, wherein, The range of motion parameter provides information about the length of the boom assembly used to move the mobile reality capture device.

13. The system according to claim 1, wherein, The mobile reality capture device is configured to obtain perception data from the 3D measurement data and / or from the sensors of the simultaneous localization and mapping unit, wherein the perception data is used for visual recognition of spatial features of the environment and for evaluating the spatial arrangement of the mobile reality capture device relative to the spatial features. The system is configured to analyze the perceived data to provide identification of environment-specific measurement scenarios regarding the spatial arrangement of the mobile reality capture device relative to spatial features in the environment, and Consider the specific environmental measurement conditions to obtain the movement category of the determined movement pattern and / or to obtain the expected movement pattern.

14. The system according to claim 13, wherein, The system is configured to access a database comprising a set of geometric and / or semantic classes of spatial features, the set of geometric and / or semantic classes having corresponding classification parameters for identifying the geometric and / or semantic classes through the perceptual data. Each of the geometry and / or semantic classes is associated with at least one of the following: ○ Rules regarding the minimum and / or maximum distances between the mobile reality capture device and the corresponding spatial features associated with that type, and ○ Rules regarding the nominal relative orientation of the mobile reality capture device relative to the corresponding spatial feature associated with that class. and The system is configured to use the database, which includes a set of geometric and / or semantic classes, to identify environment-specific measurement scenarios.

15. The system according to claim 13 or 14, wherein, The system includes a machine learning-based object detection algorithm, which is specifically configured to identify space constellations within the perceived data. The space constellation is associated with the following: the motion state of a predefined sequence of the mobile reality capture device, and The system is configured to consider identifying the space constellation through the object detection algorithm to recognize specific environmental measurement conditions, wherein the motion states of the predefined sequence are considered to obtain the expected motion pattern.

16. The system according to claim 15, wherein, The space constellation is associated with a sequence of relative orientations and / or distances between the mobile reality capture device and the space constellation.

17. The system according to claim 15, wherein, The system is configured to access map-building data that provides a model of the environment and to track the position of the mobile reality capture device within the model of the environment, wherein the position of the mobile reality capture device is taken into account to obtain the motion category of the determined motion pattern and / or to obtain the expected motion pattern.

18. The system according to claim 17, wherein, The location of the mobile reality capture device is used to identify the star cluster in order to recognize the specific measurement conditions of the environment.

19. The system according to any one of claims 1 to 4, wherein, The 3D measurement unit is specifically implemented as a laser scanner, which is configured to perform a scanning movement of a measurement laser beam relative to two rotation axes during the movement of the mobile reality capture device, so as to provide the generation of the 3D measurement data based on the scanning movement.

20. A method for 3D measurement of an environment using a mobile reality capture device, the mobile reality capture device including a 3D measurement unit configured to provide the generation of 3D measurement data for performing spatial 3D measurement of the environment relative to the mobile reality capture device, wherein... The 3D measurement unit is configured to provide the spatial 3D measurement with a 360-degree field of view around the axis of the first device and a 120-degree field of view around the axis of the second device perpendicular to the axis of the first device, wherein the method includes the following steps: During the movement of the mobile reality capture device, the 3D measurement data is generated using the 3D measurement unit. Generate IMU data related to the posture and / or acceleration of the mobile reality capture device, and The simultaneous localization and mapping (SMR) process is performed, which includes generating a map of the environment and determining the trajectory of the mobile reality capture device within the map of the environment. Its features are, The method includes the following steps: The motion pattern of the mobile reality capture device is determined by using motion data about its movement. The determined motion pattern is associated with a defined motion category that is linked to the environment-specific measured motion of the motion reality capture device. Based on the defined movement category, for the environment-specific measured movement, the expected motion pattern of the movement reality capture device is obtained. Perform a comparison between the determined motion pattern and the expected motion pattern, and Feedback is provided in relation to a comparison between the determined movement pattern and the expected movement pattern, wherein the feedback is taken into account to perform an action associated with the expected movement pattern. The correspondence between the determined motion pattern and the defined motion category and / or the acquisition of the expected motion pattern is provided by a machine learning algorithm, which includes processing the motion data by a Kalman filter to estimate the attitude parameters of the motion reality capture device, wherein the attitude parameters include velocity parameters.

21. A computer program product comprising program code stored on a machine-readable medium, or embodied by electromagnetic waves including segments of program code, wherein, The program code includes computer-executable instructions that, when executed in a measurement system or in a system according to any one of claims 1 to 19, perform the method of claim 20.

Images