Method for aligning two map datasets
Patent Information
- Application Number
- DE102023209728
- Authority / Receiving Office
- DE · DE
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-10-05
- Publication Date
- 2025-04-10
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
The present invention relates to a method for matching two map datasets for determining navigation information for a mobile device which is to move or is intended to move in an environment, to a mobile device and to a system and a computer program for carrying out the method.BACKGROUND OF THE INVENTIONMobile devices such as at least partially automated moving vehicles or robots typically move in an environment such as a home, in a garden, in a factory hall or on the road, in the air or in the water. One of the basic problems of such a mobile device or else of other mobile devices is to navigate, and in particular also to orient, i.e. to know how the environment looks, that is to say in particular where obstacles or other objects are and where it is (absolutely) located. For this purpose, the mobile device can be equipped, for example, with various sensors, such as cameras, lidar sensors, radar sensors or else inertial sensors or GNSS sensors (generalization of GPS sensors, for coarse positioning), with the aid of which the environment and the movement of the mobile device are detected, for example, two- or three-dimensionally.Disclosure of the InventionAccording to the invention, a method for matching two map data sets, a mobile device and a system and a computer program for carrying out the method having the features of the independent patent claims are proposed. Advantageous embodiments are the subject matter of the dependent claims and of the following description.The invention is generally concerned with mobile devices that can move or at least move in an environment such as on a road or in a work area. Examples of such mobile devices (or also mobile working devices) are e.g. robots and / or drones and / or also semi-automated or (fully) automated vehicles (moving to land, water or in the air). Possible robots are, for example, household robots such as cleaning robots (e.g. in the form of vacuum robots and / or wiping robots), floor or road cleaning appliances, construction robots or lawn mower robots, but also other so-called service robots, as well as vehicles moving at least partially in an automated manner, e.g. passenger transport vehicles or goods transport vehicles (also so-called industrial transport vehicles, e.g. in warehouses, but generally also passenger and truck), but also aircraft such as drones or watercraft.Such a mobile device has, in particular, a control or regulating unit and a drive unit for moving the mobile device, such that the mobile device can be moved in the environment, for example along a movement path. For this purpose, navigation information can be determined, i.e., for example, specific instructions as to the direction in which the mobile device is to travel in order to follow the movement path. These can then be implemented via the open-loop or closed-loop control unit and the drive unit. In general, this can be referred to as the navigation of the mobile device.In addition, a mobile device can have one or more sensors, by means of which the environment or information in the environment and, if appropriate, also the mobile device itself can be detected. As mentioned, these can be, for example, cameras, lidar sensors, radar sensors, ultrasonic sensors or inertial measurement units (or inertial sensors) as well as a radome, with the aid of which the environment and the movement of the mobile device are detected, for example, two- or three-dimensionally. Depending on the type of mobile device, however, other or further sensors can also be provided.An aspect in the case of navigation is what is known as mapping, as is used, for example, in the creation of maps of the environment. Such maps can also be used as a sensor to some extent. In this context, scan matching or map alignment is also mentioned. A goal here is to determine an matching transformation between input data that can originate from arbitrary sensors, such as lane markings or radar echoes detected by a vehicle camera. This transformation is generally a central first step in the creation of a consolidated map from a plurality of partially overlapping sensor data (or a representation derived therefrom). In general, this is the matching of two map datasets.An example of this is also the scan matching of so-called point clouds from lidar sensors. In this case, in particular the entire point cloud measured or captured by the laser scanner is used; a point cloud is a set of points in the environment that are determined by means of the laser scanner or lidar sensor. In this case, each point can be assigned a distance from the mobile device or laser scanner, and also an orientation relative to a reference orientation of the mobile device or laser scanner. The point cloud usually corresponds to one or more "point lines" along the contour of the objects in the field of view; if necessary, however, the points can also only lie approximately on such a line.However, this point cloud alone does not yet allow a position and / or orientation of the mobile device in the environment to be determined. For this purpose, the point cloud or set of points is compared with a reference point cloud or reference set of points. A transformation can then be determined which best matches the point cloud (or set of points) with the reference point cloud (or reference set of points), i.e. matches both to one another. This transformation then corresponds to a position and / or orientation of the mobile device when detecting the point cloud relative to the reference point cloud or a coordinate system of the reference point cloud. If the reference point cloud is a map of the environment or at least a part thereof, the current position and / or orientation of the mobile device in the environment can be determined. The reference point cloud (or map of the environment) can be expanded, for example, by continuous or repeated addition of new point clouds or parts thereof.In this context, SLAM is also referred to. A method in robotics is referred to as SLAM (Simultaneous Localization and Mapping), in which a mobile device such as a robot can simultaneously generate a map of its environment and can estimate or must estimate its spatial position within this map. It thus serves, for example, to recognize obstacles and thus supports autonomous navigation.Based on such a SLAM graph, a map of the environment (environment map) in which the mobile device is moving may be determined or determined. With each new map data set with information about the environment and / or about the mobile device, which information is obtained from one or more sensors of the mobile device or is based thereon, the map (or the SLAM graph) can be expanded or updated.In this case, attempts are made to bring the two map data sets, i.e. an existing map data set and a new map data set, into agreement, at least within certain tolerances, in order to determine the movement or trajectory of the mobile device in this way.In principle, the mapping process on the one hand and the localization process on the other hand can also be decoupled from one another (i.e. the map is generated to which mobile devices are played, there is then localized in the map). Scan matching or the adaptation of two map data sets can be used in both cases, i.e. combined or decoupled mapping and localization processes.Within the scope of the present invention, a method for matching two map datasets for determining navigation information such as a map or a trajectory for a mobile device which is moving or is intended to move in an environment is now proposed. For this purpose, two map data sets are provided, each with environmental information. In this case, in the case of both of the two map datasets, the environmental information has been recorded by the mobile device and / or the environment by means of a sensor of the mobile device. However, the same sensor does not have to have been used for both map data sets. The two map data sets are provided as input feature data, or input feature data is determined based on the two map data sets.An adaptation, i.e. matching, of the two map data sets is then carried out. In general, there are various approaches here, both classical approaches and approaches based on machine learning (ML), as described, for example, in "C. Choy, W. Dong, and V. Koltun, "Deep global registration," in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pp. 2514-2523, 2020.". Many of these approaches use two steps: in the first step, what are known as feature descriptors are generated, as described, for example, in "C. Choy, J. Park, and V. Koltun, "Fully convolutional geometric features," in Proceedings of the IEEE / CVF International Conference on Computer Vision, pp. 8958-8966, 2019. Based on these feature descriptors, the matching transformation is then determined, often as a combination of classical and ML methods.The adaptation or matching of the two map datasets is therefore carried out in particular using a machine learning algorithm. In this case, a so-called convolutional neural network (CNN) is considered in particular as a machine learning algorithm. Output data is generated from the input feature data, via intermediate feature data (feature maps) in one or more intermediate levels (so-called hidden layers). The output data in this case comprise information about a transformative relationship (or simply a transformation) between the two map data sets. The output data may then be ready for use in determining the navigation information.Examples of suitable machine learning algorithms or CNN are those described in "C. Choy, W. Dong, and V. Koltun, "Deep global registration," in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pp. 2514-2523, 2020." or in "C. Choy, J. Park, and V. Koltun, "Fully convolutional geometric features," in Proceedings of the IEEE / CVF International Conference on Computer Vision, pp. 8958-8966, 2019.However, it has now been shown that the map data sets to be matched are frequently so-called sparse map data sets, i.e. map data sets which have data, for example, only at points in space or in a detection area. Another term for a sparse map data set is a sparse map data set. A specific example of this is map datasets that comprise images and / or positions of specific objects such as lane markings, road poles, or road signs. In the case of images (in this case, it is then the case of digital images), it is possible here, for example, to focus only on the specific objects such as road signs, while other regions are disregarded.Here, a so-called sparse convolution can then be used for the adaptation of the map data sets. This entails many advantages, for example computing effort and memory requirements are greatly reduced. However, sparse convolution is typically computed only at positions of the input data (input feature data) where the input tensor was different from zero to preserve sparsity. As a consequence, however, the receptive field, i.e. the detected area, of the neural networks or machine learning algorithms in general can "tear open" if the range of a sparse convolution is not sufficient to bridge a gap in the data. This is because the convolutional kernel is typically not large enough to bridge such a gap in the data. This can be compensated for by either significantly increasing the size of convolutional kernels or by using large radius pooling operations; all of them, however, are at the expense of drastic increases in computational complexity or loss of fine granular information.This problem is relevant and unsolved in particular in the context of the mentioned machine learning algorithms from "C. Choy, W. Dong, and V. Koltun, "Deep global registration," in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pp. 2514-2523, 2020." and "C. Choy, J. Park, and V. Koltun, "Fully convolutional geometric features," in Proceedings of the IEEE / CVF International Conference on Computer Vision, pp. 8958-8966, 2019, since global relationships are often more relevant than local ones for the alignment, i.e. the adaptation.Within the scope of the present invention, two map data sets are now adjusted, of which at least one, but preferably both, is a so-called sparse map data set.Wherein performing the adjustment comprises one or more adjustments, wherein the one or each of the adjustments comprises:. Based on feature data, using a global pooling operation, e.g., global average pooling, a global map dataset is determined. The feature data thereby comprises the input feature data or the intermediate feature data of the one or one of the plurality of intermediate levels. Based on the global map data set, the intermediate feature data of the one or another of the plurality of intermediate levels is then adjusted.In one embodiment, the adaptation of the intermediate feature data comprises at least one of the following procedures in each case: an addition of the global map data set and the intermediate feature data, a multiplication of the global map data set with the intermediate feature data, or a concatenatenation of the global map data set and the intermediate feature data. If a number of channels of the global map data set and the intermediate feature data deviate from one another, in particular in the case of addition or multiplication, an interposed convolution (or convolution) can be provided in each case.In one embodiment, performing the adjustment includes multiple adjustments. Here, different intermediate feature data may then be adjusted in different ones of the plurality of adjustment operations. Likewise, in different ones of the plurality of adjustment operations, different global map datasets may be determined based on different feature data.In this way, the problem of tearing open the receptive field can be solved without significantly increasing the computing effort. Global context (i.e. the global map data record) can be combined with local information (the sparse map data records or, there, the sparse, present data). This results in a better performance with almost the same computational complexity or comparable performance with reduced computational complexity.It is therefore proposed to combine methods of global pooling (in an advantageous embodiment also extended forms such as, for example, squeeze and excitation with network architectures which are used for calculating, for example, FCGF feature descriptors or correspondence evaluations calculated by means of DGR for the purpose of aligning (matching) two map datasets (for example in the form of point clouds) in order to prevent the receptive field of the network from being abnormally reduced by unfavourable input data.In summary, a network architecture as described, for example, in "C. Choy, W. Dong, and V. Koltun, "Deep global registration," in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pp. 2514-2523, 2020." or "C. Choy, J. Park, and V. Koltun, "Fully convolutional geometric features," in Proceedings of the IEEE / CVF International Conference on Computer Vision, pp. 8958-8966, 2019. This network architecture will use map data sets or point clouds that include at least one plane of sparse computations. This network architecture, i.e. generally the machine learning algorithm, can be trained with given data and given loss function, as described there, for example.However, the network architecture is now modified to employ global pooling to incorporate global context at, e.g., different locations of the network. For this purpose, any level of feature data (input feature map, i.e. e.g. the direct network input, i.e. the input feature data, or a hidden layer, i.e. the intermediate feature data) with a specific number of channels (e.g. C channels) is processed on a parallel path with global pooling (e.g. global average pooling) added to the architecture. In an advantageous embodiment, this can be effected, for example, by means of a squeeze and excitation block. The result of this block (a so-called 1_C Feature Map) is then combined with any further feature map (i.e. an intermediate level with intermediate feature data) of the network, for example by positional addition, multiplication or concatenatenation. If, o.B.d.A., the latter feature map (intermediate feature data) has a number of channels (e.g. D channels) different from C for the case of addition or multiplication, this number can be matched with an interposed convolution, e.g. a so-called 1×1 convolution.The modification of the network described can, as mentioned, be repeated as often as desired and for any desired layer combinations. Analogously to the illustration in "J. Hu, L. Shen, and G. Sun, "Squeeze-and-excitation networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018", arbitrary network architectures such as, for example. Thus, the invention can be modified to use an Accept- or ResNet-like architecture. Further modifications to the training framework, which concerns data or loss function, for example, are not necessary, however. At runtime, the added layers or levels can remain in the network. Due to the very small sizes of the globally pooled data (global map data set), the number of additional operations occurring is very small and extends the run time only insignificantly.A system according to the invention for data processing or a computing unit, e.g. a control device or a control unit of a mobile device, or a server or other computer, is configured, in particular by programming, to carry out a method according to the invention, e.g. in one of the described embodiments.The invention also relates to a mobile device which has such a system for data processing or which is configured to obtain navigation information which has been determined as described above. The mobile device also includes a sensor for sensing environmental information, and is configured to navigate based on the navigation information.The implementation of a method according to the invention in the form of a computer program or computer program product with program code for carrying out all method steps is also advantageous since this causes particularly low costs, in particular if an executing control device is also used for further tasks and is therefore present in any case. Finally, a machine-readable storage medium is provided with a computer program stored thereon, as described above. Suitable storage media or data carriers for providing the computer program are, in particular, magnetic, optical and electrical memories, such as hard disks, flash memories, EEPROMs, DVDs, among others. Download of a program via computer networks (Internet, intranet, etc.) is also possible. Such a download can be effected in a wired or wired or wireless manner (e.g. via a WLAN network, a 3G, 4G, 5G or 6G connection, etc.).Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawing.The invention is schematically illustrated in the drawing on the basis of an exemplary embodiment and is described below with reference to the drawing.Brief Description of the DrawingsFIG. 1 schematically shows a mobile device in an environment for explaining the invention. FIG. 2 schematically shows map data sets for explaining the invention. FIG. 3 schematically shows a sequence of a method for explaining the invention in one embodiment.Embodiment(s) of the InventionFIG. 1 schematically and by way of example illustrates a mobile device 100 in an environment 120 for explaining the invention. The environment 120 here comprises, by way of example, a road 140 with roadway markings 142. The mobile device 100 is, for example, a vehicle having a control or regulating unit 102 and a drive unit 104 (having wheels) for moving the vehicle 100, e.g. along a movement path 130, which here runs, for example, along the road 140 or a lane of the road.Furthermore, the vehicle 100 has, by way of example, a sensor 106 designed as a camera with a detection range. For the sake of better illustration, the detection range is selected here to be relatively small; in practice, however, the detection range can also be, for example, up to 180°. Likewise, further cameras can be provided. The sensor 106 can be used to capture the environment 120, i.e. images of the environment or, in general, environmental information can be generated or captured therewith.Furthermore, the vehicle 100 has a computing unit or a system 108 for data processing, e.g. a control device, by means of which data can be exchanged with a superordinate system 110, e.g. via an indicated radio connection. For example, motion paths (or, generally, navigation information) may be determined in the system 110, which are then transmitted to the system 108 in the vehicle 100, which is then intended to follow the latter. However, it can likewise be provided that a movement path (or generally navigation information) is determined in the system 108 itself or is otherwise obtained there. Instead of a movement path or the navigation information, however, the system 108 can also, for example, receive control information which has been determined on the basis of a movement path or the navigation information and according to which the control or regulating unit 102 can move the vehicle 100 via the drive unit 104 in order to follow a movement path. The movement path 130 is indicated here only by way of example.In order to determine the mentioned navigation information, the images recorded by means of the camera 106, or map datasets in general, can be used, in particular in other types of sensors. For example, an image can in this case respectively comprise or image the roadway markings or positions thereof, while further information in the image is neglected. This is then a so-called sparse map data set.In FIG. 2, two such map datasets or images are shown schematically for explanation. These map datasets or images may have been recorded by the camera of the vehicle from FIG. 1, for example. Both in the map data record 200 and in the map data record 210 roadway markings can be seen in each case, but at different positions within the image which represents the map data record. Apart from the roadway markings, however, there is no further information in the images, i.e. these are sparse map datasets.In the context of the navigation of the vehicle, the two map data sets 200 and 210 should now be matched to one another, i.e. matched, in order to thus find a transformative relationship (or transformation) which maps the one map data set to the other map data set. This transformative relationship then stands for a movement of the vehicle between the times at which the two map datasets or images have been acquired. This applies in particular to the case in which a plurality of observations originate from a vehicle. In the case of mapping and localization being decoupled, it may be the same or similar local observation made at very different times from multiple vehicles.It should be mentioned at this point that the two images or map data sets are shown only by way of example and for explanation. Instead of two images, an image and an already present map can also be matched, wherein the map is in turn based on images itself. After matching, the new image can then be used to expand the map. In this case, one of the map datasets would be the map, while the other of the map datasets would be a (new) image.FIG. 3 schematically shows a sequence of a method for explaining the invention in one embodiment. In a step 300, two map datasets 302, 304, each with environmental information, are provided. As mentioned, in at least one of the two map datasets, the environmental information has been captured by the mobile device and / or the environment by means of a sensor of mobile device. In addition, at least one of the two map data sets is a sparse map data set, as explained, for example, with reference to FIG. 2.In a step 310, the two map datasets are provided as input feature data 312, for a machine learning algorithm, e.g. a CNN. It is also conceivable that the input feature data is determined based on the two map data sets. As already mentioned above, a feature descriptors can be extracted here, for example. The input feature data represents the input data for the machine learning algorithm.In a step 320, an adaptation, i.e. matching, of the two map data sets is then carried out. This is done using a machine learning algorithm 330, e.g., a CNN, and based on sparse convolution. The input feature data 312 is therefore supplied to the CNN here as input data (network input). In this case, output data 336 are obtained, which then comprise information about a transformative relationship between the two map data sets.The CNN 330 may include multiple intermediate layers (so-called hidden layers). The input feature data (input feature map) can be processed in this case via different intermediate feature data (feature maps), for example 332, 334, in the intermediate levels to form the output data 340. Depending on the type of CNN, of course, more intermediate levels or intermediate feature data may also be present.In this case, an adaptation process 340 is now carried out, in which, step 342, a global map data record 344 is determined on the basis of feature data using global pooling. The feature data can comprise or be, for example, the input feature data or else, as shown here by way of example, the intermediate feature data of one of the plurality of intermediate planesIn a step 346, the intermediate feature data of another of the multiple intermediate levels, here, for example, the intermediate feature data 334, is then adapted based on the global map data set 344.In this case, there may be a plurality of these adaptation processes 340, wherein then in each case different feature data are used to obtain the respective global map data record, and also different intermediate feature data are adapted.As mentioned, the adaptation can comprise, for example, an addition, a multiplication or concatenatenation, optionally with interposed convolution, if necessary.This output data 336 is then provided, step 350, for use in determining the navigation information. In particular, navigation information 362 is thus determined, step 360, based on the output data, the navigation information comprising, for example, a map of the environment and / or a trajectory for the mobile device.FIG. 4 illustrates such an adaptation or matching again in a different view. Here, a sparse convolution 402 is carried out from an (here exemplarily 3D) input feature map 400, that is to say feature data (e.g. of the network input or in a hidden layer). Here, a kernel 404 is used, which is applied to the input feature map 400 in order to obtain an output feature map 406 (e.g. in a hidden layer).The input feature map 400 reveals the basic sparse map data set, which is represented by the dark cuboids (information) compared to the light cuboids (no information). Since the kernel 404 covers only two cuboids here in the example, there is a gap between the dark cuboids when applied to the input feature map 400, which the kernel cannot bridge. This has the result that there was no interaction between dark cuboids in the output feature map 406.Global pooling 410 is now used here, in which information is extracted from input feature map 400, global map data set 412, which represent an interaction between cuboids of input feature map 400. This is then combined with the output feature map 406 in a step 414, so that an interaction or a relationship between the dark cuboids (which, of course, represent the information) is then generated there, via the information present in the global map data record 412.References included in the specificationThis list of documents cited by the applicant has been produced in an automated manner and is only included for the better information of the reader. The list is not part of the German patent application or utility model application. The DPMA does not take any adhesion for any faults or omissions.Cited Non-Patent LiteratureC. Choy, W. Dong, and V. Kolun, "Deep global registration," in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pp. 2514-2523, 2020 [0015, 0017, 0020, 0027]C. Choy, J. Park, and V. Koltun, "Fully convolutional geometric features," in Proceedings of the IEEE / CVF International Conference on Computer Vision, pp. 8958-8966, 2019 [0015, 0017, 0020, 0027]J. Hu, L. Shen, and G. Sun, "Squeeze-and-excitation networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018
[0029]
Claims
A method for matching two map datasets (302, 304) for determining navigation information for a mobile device (100) moving or intended to move in an environment (120), comprising: providing (300) two map datasets (302, 304) each with environment information, wherein in the two map datasets the environment information has been captured by the mobile device (100) and / or the environment (120) by means of a sensor (106) of the mobile device (100), and wherein at least one of the two map datasets is a sparse map dataset; providing (310) the two map datasets as input feature data (312) or determining, based on the two map datasets, input feature data; performing (320) an adaptation of the two map datasets (302, 304) using a machine learning algorithm (330), In particular a convolutional neural network based on sparse convolution, wherein output data is generated from the input feature data (312) via intermediate feature data (332, 334) in one or more intermediate levels, wherein the output data (336) comprises information about a transformative relationship between the two sets of map data, wherein performing the adaptation comprises one or more adaptation operations (340), wherein the one or each of the adaptation operations comprises: - determining (342), based on feature data, using a global pooling operation, a global map dataset (344), wherein the feature data comprises the input feature data or the intermediate feature data of the one or one of the plurality of intermediate levels, and - adapting (346), based on the global map dataset (344), providing (350) the output data (336) for use in determining the navigation information.Method according to Claim 1, wherein the adaptation of the intermediate feature data comprises in each case at least one of the following procedures: - addition of the global map data set and the intermediate feature data, - multiplication of the global map data set with the intermediate feature data, - concatenatenation of the global map data set and the intermediate feature data.The method of claim 1 or 2, wherein adjusting the intermediate feature data when a number of channels of the global map dataset and the intermediate feature data differ from each other comprises an intermediate convolution, respectively.The method of any preceding claim, wherein performing the adjustment comprises a plurality of adjustment operations.The method of claim 4, wherein different intermediate feature data is adjusted in different ones of the plurality of adjustment operations.The method of claim 4 or 5, wherein different global map datasets are determined in different ones of the plurality of adjustment operations based on different feature data.The method of any preceding claim, wherein the sensor of the mobile device comprises one of the following sensors: a camera, a radar sensor, a lidar sensor, an ultrasonic sensor.Method according to one of the preceding claims, wherein the environmental information of at least one of the two map datasets comprises images and / or positions of at least one of the following objects: lane markings, road poles, traffic signs.Method according to one of the preceding claims, further comprising: determining (360), based on the output data, the navigation information (362), wherein the navigation information comprises in particular a map of the environment and / or a trajectory for the mobile device.A data processing system (108, 110) comprising means for carrying out the method according to any preceding claim.Mobile device (100) comprising a system (108) according to claim 10 and / or being configured to obtain navigation information determined according to a method according to claim 9, wherein the mobile device (100) comprises a sensor (106) for detecting environmental information and being configured to navigate based on the navigation information, preferably comprising a control or regulating unit and a drive unit for moving the mobile device according to the navigation information.Mobile device (100) according to claim 11, which is designed as an at least partially automatically moving vehicle, in particular as a passenger transport vehicle or as a goods transport vehicle, and / or as a robot, in particular as a household robot, e.g. cleaning robot, floor or road cleaning appliance or lawn mower robot, and / or as a drone.A computer program comprising instructions which, when the program is executed by a computer, cause the computer to perform the method steps of a method according to any one of claims 1 to 9 when executed on the computer.A computer readable storage medium having stored thereon the computer program of claim 13.