FULLY AUTOMATIC POSITION AND ORIENTATION DETERMINATION PROCEDURE FOR A TERRESTRIAL LASER SCANNER

Description

[0001] The invention relates to a position and orientation determination method for a terrestrial surveying device with scanning functionality according to claim 1, to a surveying device of the same type according to claim 14 and to a computer program product of the same type according to claim 15.

[0002] 3D scanning is a highly effective technology for producing millions of spatial measurement points of objects within minutes or seconds. Typical measurement tasks include recording objects and their surfaces, such as industrial facilities, house facades, or historic buildings, as well as accident sites and crime scenes. Surveying devices with scanning functionality include, for example, laser scanners such as the Leica P20 or a total station or multi-station such as the Leica Multi Station 50, which are used to measure or create 3D coordinates of surfaces. To do this, they must be able to guide the measuring beam of a distance measuring device across a surface and simultaneously record the direction and distance to the measurement point. A so-called 3D point cloud is generated from the distance and the correlated directional information for each point using data processing.

[0003] In principle, such terrestrial scanners are designed to measure the distance to an object point using a rangefinder, usually an electro-optical and laser-based one. A directional deflection unit is also provided, which deflects the rangefinder's measuring beam in at least two independent spatial directions, allowing a spatial measurement area to be recorded.

[0004] The rangefinder can be designed, for example, according to the principles of time-of-flight (TOF), phase, waveform digitizer (WFD), or interferometric measurement. Fast and accurate scanners require a short measurement time combined with high measurement accuracy, for example, distance accuracy in the µm range or below with measurement times for individual points in the range of sub-microseconds to milliseconds. The measurement range extends from a few centimeters to several kilometers.

[0005] The deflection unit can be implemented in the form of a moving mirror or, alternatively, by other elements suitable for controlled angular deflection of optical radiation, such as rotatable prisms, movable light guides, deformable optical components, etc. The measurement is usually performed by determining distance and angles, i.e., in spherical coordinates, which can also be transformed into Cartesian coordinates for display and further processing.

[0006] In order to relate the measured object point coordinates to a known reference coordinate system, or in other words, to obtain absolute point coordinates, the location of the surveying device must be absolutely known. In principle, the station coordinates of the surveying device can be determined on-site as a so-called free stationing procedure by recording points that have already been absolutely referenced. For example, several target points of known position present in the measurement environment are recorded during the scan, which also allows the orientation or alignment of the device to be determined. However, such referencing usually requires at least partial manual execution, which is time-consuming and error-prone.

[0007] Alternatively, the live position of the surveying device can be determined using a GNSS receiver attached to the surveying device, using individual satellites as reference points. However, the disadvantage is the lower resolution, particularly with regard to height determination, compared to referencing using geodetic surveying. Furthermore, determining the scanner's orientation is only possible with multiple receivers, and this with even lower resolution. Above all, the process is dependent on the reception of GNSS signals, which is rarely or nonexistent, especially indoors or in narrow, winding measurement environments.

[0008] Image-based referencing methods are also known. EP 2199828 A2, for example, proposes a method within the framework of mobile scanning, with which the position of the surveying device relative to a (movement) reference system is determined from two 3D images of an environmental area taken at different angular positions. A disadvantage of this, however, is that creating at least two 3D images from different angular positions still requires a considerable amount of effort.

[0009] DE 102013110581 A1 describes a generic laser scanner that features an inertial measurement unit (IMU) for tracking a position change between two measurements in a measurement environment. Disadvantages of this solution are the need to integrate a correspondingly sensitive inertial measurement unit into the laser scanner, as well as the limitation that it can only track a continuous change between two locations, but not a determination of the current position in the measurement environment, for example, if a scan is to be performed again in the measurement environment after a longer period of time, such as hours, days, weeks, or longer.

[0010] US 2016 / 0187130 A1 discloses a method for precisely determining the position offset and orientation offset of a second stationing to a first stationing of a surveying device using image-based determined directions and a laser-optically measured distance to measurement environment points, which are imaged in both a second and a first environment image of the measurement environment.

[0011] US 2016 / 146604 A1 teaches a method for determining the position data of a surveying device by comparing a reference data set extracted from the data of an image of the surroundings of the surveying device position with position-referenced data sets and determining the position data based on the position reference of the selected position-referenced data set which has a comparatively significant degree of agreement with the reference data set.

[0012] US 2017 / 301132 A1 concerns a method for rendering 3D laser scan data with so-called cube maps.

[0013] US 2018 / 158200 A1 discloses a point cloud registration method using a SLAM process.

[0014] The object of the present invention is therefore to provide a simplified, fully automatic position determination method for a terrestrial scanning surveying device.

[0015] According to the invention, this object is achieved by the characterizing features of the independent claims or by features of the dependent claims or by developing these solutions further.

[0016] The present invention relates to a fully automatic method for determining the current georeferenced position and orientation of a terrestrial surveying device with scanning functionality at the current location based on a set of stored georeferenced 3D scanned panoramic images, i.e., panoramic images that also represent 3D coordinate data by scanning. The 3D scanned panoramic images are stored, for example, in a computer memory of the surveying device itself or on an external storage medium, e.g., a cloud, from which they are retrieved by the surveying device, e.g., via Bluetooth, the Internet, or Wi-Fi. The method is preferably carried out on a computer of the surveying device itself; however, processing can also take place externally, e.g., on an external computer or in a cloud, from where the calculated data is transmitted to the surveying device.

[0017] The process involves taking a panoramic image from the current location using the surveying device, with a multitude of object points being depicted in the captured panoramic image. A panoramic image can be captured, for example, using a super wide-angle lens, created by panning the camera, or combined from several individual images taken in different directions.

[0018] Furthermore, image matching, for example, using feature or keypoint matching or by applying machine or deep learning, is used to identify at least one 3D scanned panoramic image with object points that match the captured panoramic image. Matching object points are optionally determined using feature matching algorithms, in particular using SIFT, SURF, ORB, or FAST algorithms.

[0019] The next step is to determine (e.g., retrieve or calculate) the respective georeferenced 3D coordinates of the matching object points based on the identified 3D scan panoramic image and to calculate, e.g., by means of backslicing, the current georeferenced position and orientation of the surveying device based on the positions of the matching object points in the recorded panoramic image and their determined georeferenced 3D coordinates.

[0020] Optionally, the stored 3D scanned panoramic images represent 3D point clouds embedded or linked in the respective 2D panoramic images, for example, color images with a "resolution" of 360 x 180 pixels. The 2D panoramic images can also be created as individual 2D images, which, when combined using stitching, create the panoramic view. Alternatively or additionally, the stored 3D scanned panoramic images are stored in the form of cube or spherical projections, e.g., with a color channel and a combined depth channel. The panoramic image captured at the current location, on the other hand, is preferably a "simple" 2D (digital camera) image, which enables comparatively simple, fast, and inexpensive image acquisition with the on-site surveying device.

[0021] The set of stored 3D scan panoramic images preferably comprises spatially linked images representing a coherent / unitary environmental area, where the environmental area is, for example, an industrial plant, a building, or a building complex. The set of 3D scan panoramic images thus represents a coherent collection of reference images, all of which relate to a specific environment or object, e.g., a factory, a building cluster, a building floor, or a contiguous site. The stored 3D panoramic images can be retrieved, for example, from a database—e.g., one stored in a cloud—which is specifically assigned to the current measurement environment or object.

[0022] Optionally, the method involves moving the surveying device from its current location along a path to another location. During the movement, a series of images is continuously recorded with the surveying device as part of a SLAM (Simultaneous Localization And Mapping) process, with the identified 3D scan panoramic image being used as one, in particular the first, image in the series of images. The SLAM process is completed at the next location by recording a final image in the series of images, recording a 3D point cloud at the next location, and registering the 3D point cloud based on the SLAM process relative to the identified 3D scan panoramic image. Thus, the point cloud generated by laser scanning at another station is registered with the 3D scan panoramic image using camera images linked by a SLAM algorithm.Using the 3D scan panoramic image as part of the SLAM process offers, among other advantages, that image points linked to 3D data are available from the outset.

[0023] In a further development of the procedure, the number of stored (and identified) 3D scan panoramic images to be used for location determination is adapted to the current measurement situation.

[0024] Optionally, the number of stored 3D scan panoramic images to be used for location determination can be set as part of the adjustment process, depending on a similarity measure between the captured panoramic image and the identified 3D scan panoramic image. The similarity measure can be based on a number of matching object points. For example, the similarity measure represents a measure of a positional difference between the current location and the location where the identified 3D scan panoramic image was captured.

[0025] As a further option, the number of stored 3D scan panoramic images to be used for location determination is set as part of the situational adaptation depending on the nature of the location's surroundings determined on the basis of the recorded panoramic image, in particular a minimum distance or average distance to object points.

[0026] Optionally, the number of stored 3D scan panoramic images to be used for location determination can be set as part of the adjustment, depending on a desired level of precision for the current location. The number of reference images on which the position or orientation determination is based is thus adjusted to a required level of accuracy.

[0027] The invention further relates to a terrestrial surveying device with scanning functionality with a control and evaluation unit which is designed in such a way that the method according to the invention can be carried out.

[0028] Furthermore, the invention relates to a computer program product stored on a machine-readable carrier, in particular a terrestrial surveying device with scanning functionality, or computer data signal embodied by an electromagnetic wave, with program code suitable for carrying out the method according to the invention.

[0029] The method according to the invention offers the advantage of a simple method with which the position and orientation of a scanning surveying device can be determined fully automatically, directly on site, in real time and even before the start of the actual scanning process, when re-stationing in a measurement environment that has already been at least partially scanned. For example, the method makes it possible to calculate one's own location using only a panoramic image taken on site, e.g., with a conventional digital camera of the scanner. This allows, among other things, staggering setups on site, even if measuring activity is interrupted between two setups. Precise location determination inside buildings or, for example, in tunnels is also advantageously possible.

[0030] The on-site effort can be kept to a minimum by "tapping into" a particularly rich data source with georeferenced 3D scanned panoramic images, which enables a robust and precise determination of the current position and orientation based on a simple live image, such as a digital camera panoramic image. In other words, the data "load" is shifted to the database to be consulted, and the laser scanner itself advantageously only needs to capture a "light" image of the measurement environment.

[0031] Due to the existing three-dimensional point data, a scaling is given or provided from the outset, which advantageously means that no additional effort is required to determine a scaling factor, e.g. by measuring, scanning or capturing an object of known size (e.g. a measuring rod).

[0032] The option of fully automatic adaptation of the method to the specific measurement situation, such as accuracy requirements, type of measurement environment, or proximity to a reference location, allows the method to be tailored to actual needs. This results in customized optimization of process effort and results without requiring user intervention. This allows, for example, the method to be kept as lean as possible.

[0033] The position and orientation determination method according to the invention as well as the scanner according to the invention are described in more detail below purely by way of example with reference to embodiments shown schematically in the drawing.

[0034] Show in detail Fig. 1 shows a terrestrial surveying device of the generic type, designed as a laser scanner, Fig. 2 shows an example of a 3D scan panoramic image according to the invention, Fig. 3 shows an example of application of the method according to the invention in a measuring environment, Fig. 4 shows a further example of application of the method according to the invention in a measuring environment, Fig. 5 shows an example of a further course of the method according to the invention, Fig. 6 shows a further example of a further course of the method according to the invention, Fig. 7 shows a schematic example of a further development of the method according to the invention, and Fig. 8 shows an example of a further development of the method according to the invention for registering a point cloud at a further location.

[0035] Figure 1shows a terrestrial surveying device of the generic type, which can be designed, for example, as a total station with scanning functionality or scanning module or, as shown, as a laser scanner 1, for recording (scanning) object surfaces from a stationing, with a position P and an alignment or orientation O, which can be determined by means of the method explained below.

[0036] The device 1 has a radiation source (not shown), for example intensity-modulated, e.g. pulsed, such as a laser source, and an optics (not shown) such that a measuring beam L can be emitted into free space onto a target object in an emission direction, wherein the emission direction defines a measurement axis and the respective direction of the emission or the measurement axis is measured in the internal reference system of the scanner 1 (i.e. relative to an internal zero direction) by one or more position / angle detectors (not shown). The optics is designed, for example, as a combined transmitting and receiving optics or has separate transmitting and receiving optics. Light pulses reflected from the target object are received by the measuring device 1 and detected by an optoelectronic detector (not shown).For example, up to one million or more light pulses per second and thus 98 sampling points can be recorded.

[0037] To scan the object, the measuring radiation L or emission direction is continuously pivoted and measured, and at least one measured value is recorded successively at short intervals for each object point, including in any case a distance value to the respective scan point in the internal reference system, so that a large number of scan points are generated which, as a three-dimensional point cloud, form a 3D image of the object or the measurement environment. Scan data is therefore generated for each object point, which contains at least angle, direction, and distance information. To measure the distance value, the surveying device 1 has an electronic control system (not shown) which includes an evaluation function for measuring the respective distance value, e.g. according to the time-of-flight principle (evaluation according to the time-of-flight method).

[0038] The pivoting is carried out by means of a beam deflector 3, as shown, for example, in that an attachment or upper part A of the measuring device 1 is rotated step by step or continuously relative to a base B - relatively slowly - about a first, vertical axis V, so that the measuring radiation L is pivoted horizontally and the plurality of emission directions differ from one another in their horizontal alignment, and in that a pivotable optical component, e.g. a pivoting or rotating mirror, is rotated relatively quickly about a horizontal axis H, so that the measuring radiation L is pivoted vertically and the plurality of emission directions additionally differ from one another in their vertical alignment. In this way, the object surface is scanned line by line, e.g. using a line grid.Scanning takes place within a predetermined angular range, the limits of which are defined by a horizontal and vertical tilt range. The angular range is 360° horizontally and, for example, 270° vertically, resulting in a spherical scan area that covers almost the entire surrounding area in all spatial directions. However, any other angular ranges are also possible. There are also implementations in which the vertical resolution is achieved not by an additional rotation axis, but by several simultaneously operating transmitting and receiving units that have a specific, constant angular offset in the vertical direction.

[0039] In addition, the laser scanner 1 has at least one image camera 2, with which optical images of the object or the measurement environment can be captured, preferably color images in RGB format. A (2D) panoramic image can be generated using the camera 2. A panoramic image can be generated, for example, by capturing and combining multiple images while pivoting the pivoting device 3 through 360°. Alternatively, the laser scanner 1 has several cameras 2 directed in different viewing directions, the respective images of which can be combined to form a panoramic image.

[0040] A georeferenced 3D scan panoramic image, which according to the invention serves as the basis for subsequent position and orientation determination during re-stationing, is created by combining the result of the 360° scan or the generated 3D point cloud and the camera-based 360° (2D) panoramic image of a georeferenced stationing, so that a 3D scan panoramic image or a textured 3D point cloud is created, whereby optionally, for example, a brightness or intensity value (e.g. a grayscale value) of the measuring radiation L itself is recorded during scanning and taken into account when creating the image. By linking the camera image data and the scan data with pinpoint accuracy, a three-dimensional image of the object is thus available, which also contains color information about the object, or a panoramic image which additionally contains depth information using the scan data.By linking the 3D scan panoramic image with spatial information, i.e. by providing a location reference, a georeference is also created.

[0041] The Figure 2 shows an example of a 3D scanned panoramic image 4. In the example, panoramic image 4 is saved as a so-called cube map, i.e., a cube representation with six partial images, for example, partial image C1 or D1. The six partial images represent the forward, backward, right, left, top, and bottom views. Alternative examples of image formats or such image maps are spherical or dome views or projections. The cube representation is preferably distortion-free, ensuring a geometrically correct representation of the scan and image data.

[0042] In the example, the 3D scan panoramic image 4 is divided into the color image portion C obtained by the camera (in the Figure 2The 3D scanned panoramic image 4 is divided into the 3D component D (above) and the 3D component D obtained by scanning (below). The respective cube surfaces belong together, for example, the cube surface C1 of the color component and the cube surface D1 of the 3D component. The 3D scanned panoramic image 4 is described by four image channels: red, green, blue (C), and depth (D). Such a 4-channel image format can be stored and processed with comparatively little effort.

[0043] Such a 3D scan panoramic image 4 thus represents an information-rich image of measurement objects or an information-rich panoramic view of a measurement environment, providing not only optical image data but also distance data linked to a location or point, which also allows for automatic and immediate scaling. Such a 3D scan panoramic image 4, or a set of such images, thus also represents a very "powerful" referencing library, which, as shown below, is advantageously used to determine the position and orientation of a scanning surveying device at an unknown, new location or during a new stationing.

[0044] Figure 3shows a measurement environment 100, e.g., a building complex or a factory site. Further examples of such a contiguous environmental area are industrial facilities such as refineries or factory halls, a demarcated terrain, or even a single building or a single floor. The building complex 100 is scanned from three locations S1-S3 as described above, with panoramic images being taken in each case using the camera of the laser scanner 1. By linking the respective scan data with the respective panoramic images, 3D scan panoramic images are generated as described, so that in the example, three 3D scan panoramic images 4a-4c are available, which are stored, for example, in a database.

[0045] The 3D scanned panoramic images 4a-4c are also georeferenced, i.e., referenced to an external coordinate system. The referenced coordinate system can be a local system, defined, for example, by the setup / stationing of the laser scanner 1, or a global, higher-level system, such as a geodetic coordinate system such as WGS 84. This georeferencing can be performed using methods known from the state of the art for a location S1-S3 by scanning several known reference objects in the measurement environment 100.

[0046] However, such referencing using known methods is complex and error-prone. Therefore, within the scope of the invention, a method is proposed that optimally utilizes the information obtained from the 3D scanned panoramic images and the georeference of existing locations (reference locations) to determine the position and orientation of the laser scanner 1 at a new location or upon re-stationing in a measurement environment 100. This is described with reference to the following figures.

[0047] Figure 4shows a bird's-eye view of a measurement environment 100. It shows the three reference locations S1-S3, to which the three stored georeferenced 3D scan panoramic images 4a-4c belong. These are panoramic images (360°), form an image reference set for the measurement environment 100, and can be retrieved from the laser scanner, for example, in a database via WLAN, etc. The laser scanner is now positioned at the new or to-be-determined location S, which will serve as the stationing point for a subsequent scan.

[0048] To determine the unknown position and orientation of the scanner at the current location S, a 360° panoramic image I is captured in this position and orientation using the device camera. The panoramic image I can be a monochrome image (1-channel) or, by performing an on-site scan S1 and combining the scan data with the camera image data, a 3D scanned panoramic image (RGBD image). As a further alternative, the current panoramic image is an image that—in contrast to the 3D scanned panoramic image—does not cover the full circle, but "only" a large viewing or image angle of at least 90° (e.g., 180°) or is at least in the form of a super wide-angle image with a typically 92°-122° diagonal image angle or similar. For this purpose, the camera 2 has, for example, a correspondingly wide field of view, e.g.,using a super wide-angle lens or fisheye lens, or the current image is composed of several individual images taken from different viewing directions.

[0049] In the example, however, it is a simple camera color image (to distinguish the 3D scan images 4a-4c from the simple camera image I, the 3D scan images 4a-4c in Figure 4 are provided with a graphic pattern indicating the scan points). Such a simple 2D camera image has the advantage of being simple and quick to create, while – due to the use of the 3D scan panoramic images 4a-4c as a reference – still providing sufficient information about the current setup for robust and sufficiently precise position and orientation determination for the subsequent process steps. In other words, the use of 3D scan panoramic images 4a-4 as a comparison / referencing basis – i.e., panoramic images with depth information – allows for the effort required to create data to be compared (on-site image or live image) and the requirements for the data to be referenced to be largely minimized.

[0050] Based on known image processing methods or computer vision algorithms, the current camera image I is matched with the set of stored 3D scan images 4a-4c.

[0051] As in Figure 5As shown, image matching can be used to identify one or more of the stored 3D scan images 4a-4c that have matching environmental object points P1, P2, P3. The matching of the 3D scan images 4a-4c with the panoramic image I can be based, for example, on "classic" feature or keypoint matching, in particular through the application of a bag of words approach. Alternatively or additionally, machine or deep learning is used, in particular to learn a latent representation of the image or the image features, e.g. an encoding. An image is the input for a neural network, which delivers a so-called encoding (vector with n elements) as output. Image similarity is determined based on the difference between the respective vectors / encoding. A match exists if the difference is small. Alternatively, image similarity can be determined by applying a so-calledsiamese network, in which the images are input and a similarity measure is provided as output.

[0052] At least one of the stored images 4a-4c is determined which is similar or has corresponding object points P1, P2, P3.

[0053] The identification of matching points P1, P2, P3 can be based on feature matching, e.g. by applying a SIFT, SURF, ORB and / or FAST algorithm and / or on a deep learning based feature encoding.

[0054] The aforementioned image encoding vectors or enlarged / reduced images (thumbnails) of the images 4a-4c or I used optionally form the image pool for matching instead of the "original" image files. Using downscaled or coarsened image representations has the advantage of a lower data volume, which enables faster data transfer and / or processing. This reduced data volume is utilized, for example, by storing the database of 3D scan panoramic images in the memory of the surveying device in the form of small image files, and the uncompressed reference images (only) on an external storage device (so that a relatively small amount of storage space on the device itself is sufficient), from which individual matched images (based on the small 3D scan panoramic images) are then downloaded in their original format / size.

[0055] The matching can also be performed as a multi-stage process, in which an initial selection is made from the stored 3D panoramic images 4a-c using such small data size images. Then, using the complete, uncompressed 3D panoramic images 4a-c of the selection, a further matching is performed, in which mismatches are eliminated. In other words, an initial coarse matching is performed with the entire set of reduced-size images to create a subset of preselected images, followed by a fine matching with the "original" images of this subset.

[0056] In the example after Figure 5two images 4a, 4b are identified from the three available images which have matching object points P1, P2, P3. If the image matching is carried out as described above as one option, e.g. via image encoding, and thus corresponding object points P1, P2, P3 have not already been determined by or for the image matching, corresponding object points P1, P2, P3 are determined - e.g. using feature matching - as an additional step (only) for the images 4a, 4b which have already been determined as matching. If the matches between images 4a, 4b and the recorded image I are found, e.g. using a Siamese network, matching points P1-P3 between the matched images 4a and I or image 4b ​​and I are identified based on feature detection and matching.

[0057] The matching object points P1-P3 are then used to calculate position and orientation at location S as described below.

[0058] Figure 6 schematically shows the current location S with the 2D panoramic image I recorded there, as well as the two reference locations S1 and S2, for which the reference 3D scan panoramic images 4a and 4b are available. Also shown are object points P1-P3 corresponding to the current image I, with the reference image 4a having three object points P1-P3 corresponding to the current image I, and the reference image 4b ​​having one point P1.

[0059] Since the reference images 4a-4c are 3D scan images, the 3D coordinates of all object points P1-P3 can be determined from the respective image data. For example, the horizontal and vertical angles are calculated from the position of a respective point P1-P3 in image 4a. Together with the distance value from the depth channel and taking into account the position and orientation of image 4a, the (georeferenced) 3D coordinates (X, Y, Z) of the respective point P1-P3 are then calculated (in the Figure 6 indicated by the arrows 5). Depending on the type of storage of the 3D scan panoramic images 4a, 4b or depending on their type, the respective georeferenced 3D point coordinates can alternatively be determined, for example, by looking up a table in which, for example, the 3D coordinates are assigned to a respective image point.

[0060] Based on the georeferenced 3D point coordinates and the image position of the respective (corresponding) points P1-P3 in the current image I, the position and orientation of the laser scanner are then calculated. This calculation is performed, for example, by backslicing, as shown in the Figure 6 indicated by arrows 6.

[0061] As in the Figure 6 As shown, in the example, the position and orientation of the laser scanner or location S are determined only based on the one reference S1, without using the also identified second 3D scan image 4b ​​of the further reference location S2. Optionally, deviating from the illustration, all identified 3D scan panoramic images 4a, 4b or the available corresponding points P1-P3 of all identified reference images 4a, 4b are used for the position or orientation calculation, for example, up to a defined maximum number.

[0062] As a further preferred option, the number of 3D scan panoramic images 4a, 4b or references S1, S2 used for location calculation is automatically adjusted depending on the situation. In the present example, the current location S is close to the reference location S1. In both matched images I, 4a, a comparatively large number of matching points is determined, e.g., one hundred or more points, especially since panoramic images are available from both locations S, S1. The large number of points, of which three P1-P3 are shown as examples, already allows for a robust and precise calculation of the current position and orientation, especially since points with significantly different 3D coordinates can be selected.

[0063] One advantage of automatically adapting the reference data set used for location determination to the circumstances or requirements is that the effort is optimally adjusted. In other words, this ensures that as much (process) effort is expended as necessary, while at the same time as little as possible (which is particularly advantageous for mobile devices such as laser scanners with limited electrical and computing capacity). Exactly as many 3D scan panoramic images are used as required for the measurement situation. For example, the process can be adapted to whether the scanner is positioned and oriented in narrow, winding spaces or in open, spacious halls or open terrain.

[0064] The Figure 7explains the procedure with the automatic adjustment of the number of stored 3D scan panoramic images to be used for location determination to the current measurement situation, as described above.

[0065] According to the method, in step 7, a panoramic image is captured at the current location to be determined, with the current position and orientation. In step 8, this captured panoramic image is matched with the set of stored 3D scan panoramic images 4. In the example, several of the stored 3D scan panoramic images are identified (step 9) that have matching points or image features to the captured image.

[0066] In step 10, the number of reference images to be used for the actual position determination is determined based on a given first criterion K1 as an adaptation to the specific measurement situation.

[0067] Criterion K1, for example, is a measure of the similarity of the current panoramic image with one or more of the stored panoramic images. If, for example, a high degree of similarity is determined, the number of reference images used for location determination is kept small, e.g., limited to one image. If, on the other hand, a low degree of similarity is determined, a comparatively large number of reference images is used, e.g., the maximum available number of matches.

[0068] The similarity measure is based, for example, on the number of corresponding object points in a respective reference image that match the captured image. Alternatively or additionally, the type of matching features is used as a criterion to determine similarity. For example, image descriptors that describe dominant lines in the image can be used. A measure of similarity can also be determined based on properties that describe the respective images as a whole or statistical image properties, e.g. grayscale or color histograms or gradients or functions of brightness or surface normals. Furthermore, the similarity measure K1 can be based on a deep-learning-based encoding of the image or on a similarity measure determined using a Siamese network.

[0069] Such a similarity measure K1—or, to put it the other way around, a difference measure—can also be a measure of the positional difference between the current location and the reference position. The similarity measure K1 is then specified in such a way that it can be used to at least roughly classify whether the current location is close to a (respective) reference location or not. In the case of proximity, the number of 3D scan images to be used can be limited; for example, only the image (or the associated identified 3D point coordinates) of the nearby reference location can be used to calculate the location. If, on the other hand, the similarity measure K1 determines that no nearby reference location is available, the number of reference images to be used is set to, for example, 3, 5, or 10.

[0070] After adapting the number of reference images in step 10, in step 11 the position and orientation of the current stationing is calculated as described above based on the 3D coordinates of the object points of the selected reference image(s) and their positions in the images and the process is terminated (step 13; for step 12 see below).

[0071] Another example of an adaptation criterion K1 is a property of the site environment, i.e. a criterion K1 that describes a condition of the measurement environment. For example, the distance to object points is checked (e.g., the shortest measured distance or an average distance), and for large distances, a larger number of reference images is set to be used than for short distances. The distances to the scan points or to scan objects can, for example, be an indicator of whether the measurement environment is open terrain or a wide room, where a relatively large number of identified images are possible or necessary for referencing; or whether it is a winding terrain or a narrow room, where few images are possible or necessary for referencing.Alternatively or additionally, the characteristics of the respective location environment can already be connoted in a respective 3D scan panoramic image and can be directly accessed, e.g. as metadata.

[0072] Distances to object points can also be related to the current location and, for example, taken into account after calculating the current position and orientation, so that an adaptation of the number of reference images may only take place after step 11 if it is determined based on criterion K1 that an optimal result with regard to the nature of the measurement environment is only possible with an increased number of reference images.

[0073] Such a regulation of the number of reference images, which is used for location calculation, can - as in the Figure 7shown - optionally also based on a further criterion K2. Such a second criterion is, for example, a measure of the precision of the position and / or orientation calculated in step 11. If it is determined that the accuracy does not meet criterion K2 (step 12), the system goes back to step 10 and increases the number of images to be used, thus adjusting it. With this option, at least a second 3D scan panoramic image is used if necessary. If the current location is determined with sufficient precision based on the use of another reference location or other 3D object point coordinates, the location determination process is terminated with step 13, so that, for example, a scan can then be carried out automatically from the now known location.

[0074] Figure 8shows a further development of the method according to the invention. In this development, a 3D scan panoramic image 4a identified for location S is used as a starting point to register a point cloud PC at a further location S' by means of a SLAM process (Simultaneous Localization And Mapping). Here, the laser scanner 1 is moved from the current location S, for which at least one 3D scan panoramic image 4a has been identified as described above, along a path (symbolized by arrow 15) to the further location S'. Along the path 15 to the end location S', a series of images 14 are recorded using the camera 2 of the surveying device 1, e.g., by continuous photography or in the form of a video. The 3D scan panoramic image 4a is integrated as part of the image series 14.The image data of the image series 14 are processed in a manner known per se using a SLAM algorithm in such a way that a spatial link is created between them, whereby - starting from the 3D scan panoramic image 4a as the starting image - the last image taken at the further location S' can finally be placed in spatial relation (position and orientation) to the starting image 4a.

[0075] As in Figure 8 As indicated by way of example, points P1-P3 of the 3D scan panoramic image 4a, which are also detected in the first subsequent camera image, are used for the SLAM process, as well as points V1-V3, which correspond in subsequent images, and finally points C1-C3.

[0076] These latter points C1-C3, in addition to correspondences within the image series 14, also have correspondences in the point cloud PC, which is recorded at location S'—in addition to the camera image—by means of a scan (symbolized by lines 16). Thus, the point cloud PC can ultimately also be spatially related to the 3D scanned panoramic image 4a or registered relative to it.

[0077] The fact that a 3D scan panoramic image 4a simultaneously and linked contains a camera image and a 3D point cloud is therefore advantageously used to register two point clouds by using the camera image in a SLAM process as the first "pillar" of a "bridge" (image series 14) with which the new point cloud PC is connected to the known 3D point cloud of the 3D scan panoramic image 4a created by some previous scan.

[0078] The correspondences between the images of image series 14, including the 3D scan panorama image 4a, are determined, for example, using feature matching. However, the assignment of points C1-C3 of the point cloud PC to the camera image of the last image series (camera image of location S') can also be performed more simply without image matching, if camera 2 is calibrated to the scan module of surveying device 1, which is usually the case.

[0079] It is understood that these figures only schematically depict possible embodiments. According to the invention, the various approaches can also be combined with each other and with prior art devices and methods.

Claims

1. Fully automatic method for determining the current, geo-referenced position and alignment of a terrestrial surveying device with scan functionality (1) at the current location (S) on the basis of a set of stored, geo-referenced 3D scan panoramic images (4, 4a-c), i.e. panoramic images that also represent 3D coordinate data by means of a scan and provide optical image data and distance data linked to the image data true to point, with • recording (7) of a panoramic image (I) with the surveying device from the current location (S), wherein a large number of object points are represented in the recorded panoramic image (I), • identifying (9) at least one 3D scan panoramic image (4a, b) with object points (P1, P2, P3) corresponding to the recorded panoramic image (I) by means of image matching (8) with the set of stored, georeferenced 3D scan panoramic images (4, 4a-c), • determining the respective geo-referenced 3D coordinates of the corresponding object points (P1, P2, P3) on the basis of the identified 3D scan panoramic image (4a, b) and • calculating (11) the current geo-referenced position and alignment of the surveying device (1) on the basis of the positions of the corresponding object points (P1, P2, P3) in the recorded panoramic image (I) and of their determined geo-referenced 3D coordinates.

2. Method according to Claim 1, characterized in that the stored 3D scan panoramic images (4, 4a-c) are each based on 2D panoramic images and associated 3D point clouds and / or the recorded panoramic image (I) is a 2D image.

3. Method according to Claim 1 or 2, characterized in that the stored 3D scan panoramic images (4, 4a-c) each comprise color channels (C) based on digital photography and a depth channel (D) based on laser scanning.

4. Method according to one of the preceding claims, characterized in that the stored 3D scan panoramic images (4, 4a-c) are present in the form of cubic or spherical projections.

5. Method according to one of the preceding claims, characterized in that the set of stored 3D scan panoramic images (4, 4a-c) comprises spatially interlinked images of a surrounding region (100) forming a unit, in particular an industrial installation, a building or a building complex.

6. Method according to one of the preceding claims, characterized in that in the context of the method • the surveying device (1) is moved from the current location (S) along a path (15) to a further location (S'), • during the movement (15) a series (14) of images is continuously recorded with the surveying device (1) in the context of a SLAM process, wherein the identified 3D scan panoramic image (4a, b) is employed as an image (particularly the first image) of the image series (14), • the image series (14) of the SLAM process is completed by recording a final image at the further location (S'), • a 3D point cloud (PC) is recorded at the further location (S') by means of the surveying device (1), • the 3D point cloud (PC) is registered on the basis of the SLAM process relative to the identified 3D scan panoramic image (4a, b).

7. Method according to one of the preceding claims, characterized in that the number of stored 3D scan panoramic images (4, 4a-c) to be employed for the location determination is adjusted (10) to the existing measurement situation.

8. Method according to Claim 7, characterized in that in the context of the adjustment, the number of stored 3D scan panoramic images (4, 4a-c) to be used for the location determination is set depending on a measure of the similarity (K1) between the recorded panoramic image (I) and the identified 3D scan panoramic image (4a, b), in particular wherein the measure of similarity (K1) is based on a number of corresponding object points (P1, P2, P3).

9. Method according to Claim 8, characterized in that the measure of similarity (K1) represents a measure for a difference in position between the current location (S) and the recording location (S1, S2) of the respective, identified 3D scan panoramic image (4a, b).

10. Method according to any one of Claims 7 to 9, characterized in that in the context of the adjustment, the number of stored 3D scan panoramic images (4, 4a-c) to be used for the location determination is set depending on a minimum distance or average distance to object points and / or characteristic of the environment of the current location (S) determined on the basis of the recorded panoramic image (I), which is connoted in a respective 3D scan panorama image as directly retrievable meta data.

11. Method according to one of Claims 7 to 10, characterized in that in the context of the adjustment, the number of stored 3D scan panoramic images (4, 4a-c) to be used for the location determination is set depending on a desired degree of precision (K2) for the position and / or alignment (12) that are to be determined.

12. Method according to one of the preceding claims, characterized in that the image matching (8) is based on machine learning and / or deep learning, and / or that in the context of the method corresponding object points are determined by means of feature matching algorithms, in particular on the basis of SIFT, SURF, ORB or FAST algorithms.

13. Method according to one of the preceding claims, characterized in that the calculation of the current location (S) takes place by means of re-sectioning.

14. Terrestrial surveying device (1) with scan functionality and with a control and evaluation unit that is configured in such a way that the method according to any one of Claims 1 to 13 can be carried out with it.

15. Computer program product that is stored on a machine-readable carrier, in particular of a terrestrial surveying device (1), or a computer data signal, embodied by an electromagnetic wave, with program code, wherein the computer program product or the program code are suitable for carrying out the method according to any one of Claims 1 to 13.