Method and system for locating a mobile device and mobile device

Abstract

Method for locating a mobile device (10, 20), in particular a vehicle, in an environment using a map (11) representing the environment by means of simultaneous positioning and mapping (SLAM), wherein the mobile device (10, 20) has an environment detection device (12) and a computing device (Sys1); Mobile device (10, 20), in particular a vehicle, which is designed to carry out a localization in an environment of the mobile device (10, 20) using a map (11) representing the environment by means of simultaneous positioning and mapping (SLAM); and system (50, 51, 52, 53) for locating a mobile device (10, 20) by means of simultaneous positioning and mapping (SLAM).

Description

The invention relates to methods and a system for locating a mobile device in an environment. The invention also relates to a corresponding mobile device. The localization is effected by means of a so-called simultaneous positioning and mapping ("Simultaneous Localization and Mapping", SLAM for short). This means that the so-called SLAM method is used, in which a map of the environment of the mobile device is simultaneously created and its spatial position, i.e. its position, is determined within this map or the environment.Maps enable mobile devices, e.g., vehicles, to be located independently in an environment and can be used for navigation. Environmental properties are used for this purpose. These maps may be in two or three dimensional form.With the SLAM method, an existing map can be used or created in parallel. These cards may require a large amount of memory. Offloading to a cloud may reduce storage requirements on the mobile device.Using the SLAM method, a mobile device, e.g. a vehicle, can create a map of its environment and estimate its current relative position in the environment on the basis of this map. This allows the mobile device to be localized in the respective environment. To create the map, the mobile device can acquire sensor data of the environment. Corresponding sensor data may be depth or RGB images, for example. The sensor data can be evaluated in order to identify objects or edges of objects. The relative position of the detected objects to the mobile device can be stored in a frame (data structure for the representation of objects). By continuously recording the environment around the mobile device and detecting objects, classification can be carried out and a map of the environment can be created. As the mobile device moves in the environment, the positions of the detected objects change according to the acquired sensor data. A relative position change of the mobile device can be estimated on the basis of this change. If the map of the environment has been recorded, the mobile device can also independently determine its position by comparing its current sensor data with the map.Overall, a localization method based on the SLAM method can be substantially divided into the following three processes: tracking or object recognition (feature extraction), mapping or map creation and loop closing or recognition of common features of the current image with the previously recorded map. The SLAM method accordingly comprises three modules: tracking module, mapping module and loop closing module. Tracking is responsible for ascertaining properties of the environment. The result are frames in which the environment is represented. The mapping is responsible for creating a map from these frames and / or providing the map. Loop closing is responsible for checking whether the system was already at the location.All three processes can be very computationally expensive and at the same time time time-critical, since, for example, the control of the mobile device can also depend thereon. In order to make the required computing capacity available, the mobile device may be equipped with a processor (central processing unit, CPU for short) and a graphics card having a high computing capacity. However, such processors and graphics cards are expensive, often oversized, and typically consume much power. This can lead to a shorter operating time of the devices. Another possibility is to offload the computational operations and thus the computational load from the mobile device.To meet latency requirements, edge computing or edge cloud computing based solutions may be used. In this case, computation-intensive processes are swapped out to the edge and the map is also stored there. A local map may remain on the local system, i.e., the mobile device.For example, DE 10 2022 100 454 A1 discloses a method for locating an autonomously operated means of transport by means of a SLAM method. Environment data recorded by sensors of the means of transport are dynamically transferred to an external computing device, which can be provided by an edge cloud server.Furthermore, DE 10 2021 130 382 A1 discloses a method for unloading sensor data of an autonomous vehicle to a computer external to the vehicle, which can be provided by an edge server.US 2022 / 0351553 A1 describes data storage and retrieval of data in a distributed system comprising vehicles equipped with various automated driving sensors and generating a large amount of sensor data. The sensor data includes information about the physical environment being mapped. The sensor data is to be swapped out of the vehicles and provided in a useful manner.Cloud computing provides computer and storage services from a centralized location in a communication network. Cloud computing is suitable for applications that are not time critical. Edge computing provides computer and storage services for applications that operate in real-time or near real-time. Edge computing pushes computing to the border of both wired and wireless networks to support real-time applications that are required to operate. For example, an edge computing architecture may employ data centers near cell towers or at regional levels rather than remotely in the cloud.Against this background, the object of the invention is to provide methods and apparatuses which, using the SLAM method, implement reliable and rapid localization of a mobile device depending on the situation.This object is achieved by a method having the features of claim 1, by a mobile device having the features of claim 7 and by a system having the features of claim 9. Further, particularly advantageous embodiments of the invention are disclosed in the respective dependent claims.It should be noted that the features listed individually in the claims can be combined with one another in any technically expedient manner (even beyond category limits, for example between method and apparatus) and reveal further configurations of the invention. The description additionally characterizes and specifies the invention, in particular in conjunction with the figures.It should be further noted that a conjunction "and / or" used herein, between two features and linking them to one another, should always be interpreted such that in a first configuration of the subject matter according to the invention only the first feature may be present, in a second configuration only the second feature may be present, and in a third configuration both the first and the second feature may be present.The invention relates to a method for locating a mobile device in an environment using a map representing the environment by means of simultaneous position determination and mapping (referred to herein as SLAM), wherein the mobile device has an environment detection device and a computing device.The mobile device can be, for example, a vehicle, in particular an autonomously operable vehicle, a mobile robot, a smartphone or the like. The environment detection device can comprise, for example, radar, ultrasonic, optical sensors such as a camera and the like. The computing device may include one or more CPUs (computer processors), GPUs (graphics processors), FPGAs (field programmable gate array), or the like.According to the invention, the method comprises the steps of:acquiring environmental data, in particular sensor data, of the environment of the mobile device by means of the environment acquisition device,locating the mobile device by processing the acquired environmental data;at least one environmental feature of the environment is determined from the acquired environmental data in a tracking process,providing map data describing the map in a mapping, said map data having environmental features of the environment represented by the map, anda position of the mobile device in the environment is determined in a loop closing by comparing the acquired environment data with the map data for matching the respective environment features,providing a further computing device (e.g. in the form of one or more CPUs, GPUs, FPGAs or the like),providing a communication link between the computing device of the mobile device and the further computing device,determining an available bandwidth of the communication link,determining an available computing capacity of the computing device of the mobile device,determining an available computing capacity of the further computing device via the communication link,dividing the tracking, mapping and / or loop closing between the computing device of the mobile device and the further computing device depending on the available bandwidth of the communication link, the available computing capacity of the computing device of the mobile device and the available computing capacity of the further computing device according to a predetermined assignment rule, andtransmitting processing data relating to the tracking, mapping and / or loop closing allocated to the further computing device to the further computing device.In other words, the environment data of the environment of the mobile device are first acquired by means of the environment acquisition device. That is, the mobile device can record or record its environment itself. The captured environmental data are then processed or evaluated. The processing of the environmental data is substantially divided into the three processes mentioned at the beginning, namely object recognition (tracking), map creation (mapping) and map-image comparison (loop closing).For object recognition, at least one environmental feature of the environment is determined from the captured environmental data. For this purpose, methods known per se, such as FAST (Features from Accelerated Segment Test), BRIEF (Binary Robust Independent Elementary Features), ORB (Oriented FAST and Rotated BRIEF) etc., can be used, among others. The environmental feature can be, for example, a predetermined image feature and / or an object and / or an edge. The environment describes a detectable region in the vicinity of the mobile device. Depending on the environment detection device used and environment topography, the detectable range can be a few metres to a few kilometres. For example, a range of two to three kilometers of perimeter around the mobile device may be detected as the environment. The environment data may map this environment. The environment data can be provided, for example, by a respective sensor of the environment detection device as sensor data or measurement data. If the environment detection device is configured as a camera, for example, the environment data can be present as image data, for example in the form of an RGB image or stereo image or depth image. Of course, depending on the configuration of the environment detection device, other forms of the environment data are also possible. The environment detection device can be configured, for example, additionally or alternatively as a radar system, lidar system or ultrasonic system. The evaluation of the environmental data for determining the respective environmental feature can be carried out, for example, using known object recognition algorithms or feature detection methods.For map creation, map data describing a map of the surroundings of the mobile device is provided. In this case, the map data includes environmental features of the environment represented by means of the map. The map may create the mobile device itself, for example, by cyclically acquiring the environment from the acquired environment data. Alternatively, the mobile device may load or retrieve the map data from an internal data store or an external data server. When using an external data server, the map data can be used, for example, by a plurality of different mobile devices. For example, a first mobile device may access the map data of a second mobile device.For map-image comparison, a position of the mobile device in the environment is finally determined by comparing the captured environment data with the map data for matching the respective environment features with one another. That is, it is checked whether and where there are environmental features matching the determined environmental features in the map. Common features of the map with the environment data are thus determined.In order to be able to offload the individual processing steps for processing the captured environmental data, the further computing device is provided. A plurality of, i.e. two or more, corresponding computing devices can be provided. By means of the respective computing device, the corresponding computing operations for the environmental data processing can be carried out if necessary. For this purpose, the corresponding processing data can be transmitted between the computing device of the mobile device and the further computing device. Processing data is understood to mean those data which arise during the processing or evaluation of the captured environmental data. These can be either data or data packets which are sent to the further computing device (data to be processed) for processing or data which are sent back from the further computing device after processing (processed data).For transmitting the processing data, a communication link is provided or established between the computing device of the mobile device and the further computing device. The communication connection can be configured, for example, wirelessly, but is not necessarily limited thereto. It can be present, for example, as a radio connection, such as a WLAN connection or mobile radio connection. To establish the communication connection, the respective computing devices can have a respective communication module with a corresponding radio interface. Wired communication connections can be provided, for example, via network cables in a LAN (local area network), via a communication bus formed on a printed circuit board on which the respective computing devices can be provided, or via a communication bus of a so-called system-on-a-chip (SoC) in which both the respective computing devices and the communication bus can be provided on-chip.In any case, the invention provides a communication link having a high capacity with respect to possible throughput, which enables a division of bandwidth-intensive applications between at least two computing devices even if the communication link is a wireless communication link such as a radio link.In order to realize reliable and rapid localization of the mobile device in the environment depending on the situation (i.e. dynamically), the currently available bandwidth of the communication connection, the currently available computing capacity of the computing device of the mobile device and the currently available computing capacity of the further computing device are determined.The available bandwidth specifies in particular how fast or what amount of data can be transmitted at a specific point in time per unit of time via the communication link between the computing device of the mobile device and the further computing device.The available computing capacity of the computing device of the mobile device can be directly determined by inquiring for its current load. The available computing capacity of the further computing device, on the other hand, is provided to the mobile device via the communication link and can be determined in this way. The available computing capacity of the further computing device can be shared with the mobile device, for example, via regularly transmitted synchronization messages, but also irregularly, for example under event control when a predetermined capacity threshold is exceeded, which is a preferred possibility, in particular in the case of wireless communication connections. In the case of wired communication connections in which both computing devices are provided on the same printed circuit board, for example, the current load of the further computing device can be directly called up, in particular if the further computing device is provided by the same operating system, for example, as a container or virtual machine. In different operating systems for the computing device of the mobile device and the further computing device, messages or so-called signals can be used to exchange the available computing capacity.Depending on the available bandwidth, the available computing capacity of the computing device of the mobile device and the available computing capacity of the further computing device, the steps for locating the mobile device, i.e. the steps for processing the environmental data (also referred to herein as processing steps), in particular tracking, mapping and / or loop closing, are divided between the computing device of the mobile device and the further computing device (dynamic load distribution) according to a predefined assignment rule. This means that all processing steps can be carried out overall with exactly one of the computing devices, or a part of the processing steps is carried out by the computing device of the mobile device and another part of the processing steps is carried out by the further computing device. For example, object recognition (tracking) in the environment data can be carried out by means of the mobile device, while the examination of the environment data for edges (mapping) is carried out by means of the further computing device. In the following, for the sake of simplicity, only the division or assignment of the processing steps is discussed. This formulation is also understood to mean the subdivision of the processing steps into substeps or individual arithmetic operations.The dynamic load distribution proposed according to the invention thus essentially comprises three basic variants of the SLAM method: in the first variant (also referred to herein as "full offload", FO), the three modules of the SLAM method mentioned at the beginning, i.e. tracking, mapping and loop closing, are swapped out to the further computing device. In the second variant (also referred to herein as "partial offload", PO), only the mapping and the loop closing are swapped out to the further computing device. This allows the use of more computationally powerful systems for computationally expensive mapping and loop closing. In the third variant, the complete calculation for locating the mobile device takes place on the mobile device itself (also referred to herein as "on device", OD)."Full offload" requires the lowest computing capacity on the mobile device or on the computing device provided by the mobile device, since sensor data generated by the mobile device are forwarded directly to the further computing device via a (e.g. wireless) communication connection in order to ascertain the properties of its environment. However, this requires a higher bandwidth in the communication channel of the communication connection.In contrast, "partial offload" requires more computing capacity on the mobile device or the computing device provided by the mobile device, since the tracking still remains on the mobile device and is carried out by the mobile device. In comparison with the full calculation on the mobile device ("on device"), however, this is already significantly reduced. The requirements for the communication connection, in particular its available bandwidth, are lower in the case of "partial offload".The division of the processing steps results in the advantage that complicated computing tasks or computing operations can be divided between the respective computing devices. If necessary, costly computing operations can thus be swapped out from the mobile device. In addition, the processing of the environment data can thereby be parallelized. Performance advantages can thereby be achieved by means of higher computing capacities. The localization of the mobile device can be carried out efficiently and reliably. By taking into account the available bandwidth and the available computational capacities of the respective computing devices, a qualified decision can also be made as to which computing operations should be swapped out at all. It can thus be ensured, for example, that the processing data are swapped out only if the data transmission can take place substantially within a predefined time period. Thus, depending on the situation, the localization of the mobile device can be carried out as quickly as possible, i.e. essentially in real time. Dynamically offloading the processing steps also has the advantage that less or less-efficient hardware can be used on the mobile device. This saves costs, making the mobile device more cost-effective to produce. In addition, the mobile device becomes lighter and consumes less power.Furthermore, the invention enables a higher map creation rate, a higher speed of recognition of objects or map changes, and improves the accuracy of localization. Furthermore, a probability of failure (e.g. due to so-called track loss) of the localization and the mapping is reduced. In addition, the invention achieves a high degree of flexibility for implementation in different application cases.Moreover, the accurate and reliable localization of the mobile device in the case of a vehicle may ensure a safe functionality of driving safety functions and enable autonomous driving. The resource consumption on the mobile devices can be adapted depending on the situation. As a result, computing capacity on the mobile device can also be made available to other relevant safety functions to be provided by the mobile device.In an advantageous embodiment, a maximum number of the environmental features determined from the captured environmental data are dynamically adjusted during tracking as a function of the available computing capacity of the mobile device and / or the available bandwidth of the communication connection and / or a capture rate of the environmental data (e.g. image rate of a camera) and / or a movement speed of the mobile device according to a predetermined further assignment rule. The more environmental features are determined in the tracking, the greater is the computing effort and the greater is the amount of data generated thereby, which is to be transmitted to the further computing device, if appropriate, via the communication connection. In this way, both the computing capacity required for tracking and also for downstream mapping and / or loop closing can be flexibly adapted to the current load of the respective computing device.Alternatively or additionally, the maximum number of environmental features to be determined can be reduced at a high linear movement speed of the mobile device, e.g. of a vehicle, in order to reduce the computing latency. As it travels along a curved path, e.g., cornering, the maximum count may be increased to detect more new characteristics of the environment, or decreased to achieve lower computational latency. The decision as to whether a decrease or increase is selected may be made dependent on the travel / turn speed, a number of the characteristics / features of the environment present, the computing capacity available, and / or environmental conditions present (e.g., lighting, fog, etc.). The speed of movement can be detected by sensors of the mobile device (e.g. vehicle sensors) as well as by visual odometry (in the tracking module).In further preferred embodiments, an information content of the captured environment data is reduced before tracking depending on the available bandwidth of the communication connection. For example, an image rate of a camera can be reduced and / or an image quality such as resolution and colors can be reduced or removed by means of a provided filter function. The filter function is preferably applied to the data stream generated by the environment detection device. If there is a high load on the computing device and / or the communication connection, the data stream can be reduced. The reduction may be determined by a function of the available bandwidth of the communication link and a detection rate of the environmental data (e.g., incoming frame rate).In further advantageous embodiments, an information content of the processing data transmitted to the further computing device is reduced by the further computing device before carrying out the allocated tracking, mapping and / or loop closing depending on the available computing capacity of the further computing device. For example, by means of a provided further filter function, an image rate of a camera, the images of which are contained in the transmitted processing data, can be reduced and / or an image quality such as resolution and colors can be reduced or removed. The further filter function can be applied by the further computing device to the received data stream of the processing data. If there is a high load on the further computing device, the data stream can be reduced.Yet further advantageous embodiments provide that the processing data are compressed before being transmitted to the further computing device and are decompressed after being transmitted from the further computing device before execution of the allocated tracking, mapping and / or loop closing. The compression can optionally take place losslessly (lossless) or lossyly (lossy). With the lossy compression, higher compression of the processing data can be achieved.In other advantageous embodiments, the further computing device is provided by means of the mobile device or external to the device by means of a further mobile device, a cloud server or an edge cloud server. In the first case, the two computing devices of the mobile device can be provided as hardware on a system-on-a-chip. Alternatively or additionally, the further computing device of the same mobile device can also be provided virtually, i.e. by software technology in a virtual environment of the control software (e.g. operating system) of the mobile device.The communication connection can be provided as a wired connection or a wireless radio connection. The radio connection is preferably designed according to the 5G mobile radio standard or WLAN standard IEEE 802.11ax, but is not necessarily limited thereto. The aforementioned radio standards can ensure that, in particular, a latency and reliability of the communication connection for transmitting the processing data is known.The further mobile device can be, for example, a (further) vehicle. Accordingly, the communication link may be direct vehicle-to-vehicle (V2V) communication. In addition to the processing data, information about the previously determined / already known topology of the environment can be transmitted to the further mobile device. In this way, in the case of insufficient available computing capacity on the mobile device (e.g. a vehicle), the computing capacity of another mobile device (e.g. of a further vehicle) can be used to perform the tracking, mapping and / or loop closing. As a result, the map which is calculated on the one mobile device can also be used on the other mobile device. In addition, map details can also be distributed further to still further mobile devices (e.g. further vehicles in the environment).When the further computing device is provided by means of the cloud server or edge cloud server, the advantage results that the SLAM method steps can be carried out in a particularly efficient decentralized manner. Edge cloud server is understood in the present case to mean a server device whose computing capacity is located as close as possible locally to the terminal generating the data, i.e. in the present case the mobile device (e.g. a vehicle). This avoids communication bottleneckes, since direct physical communication connections or robust wireless communication connections can be realized. In addition, the edge devices may concentrate computing resources on a smaller set of data, thereby avoiding extensive use of computing resources in a cloud.The term "edge" describes that the data processing takes place at the edge of a common network. That is, the external computing device and the mobile device may be part of a local or closed computing network or network, such as an enterprise network to a location. Thus, the time delay of the data transmission in the network, i.e. via the communication connection, can be avoided. In contrast, cloud servers mean a server device external, i.e., outside a local computer network. The processing by means of an edge cloud server additionally has the advantage that the map data, i.e. the map, is available in a central computing device. There, it can be used or improved, for example, by other mobile devices.The invention further relates to a mobile device, in particular a vehicle, which is designed to carry out a localization in a surrounding area of the mobile device using a map representing the surrounding area by means of simultaneous position determination and mapping (SLAM), having:an environment detection device configured to detect environment data of the environment,a computing device configured and configured to locate the mobile device by processing the acquired environmental data,in a tracking system, at least one environmental feature of the environment is determined from the acquired environmental data,acquiring map data describing the map in a mapping, said map data having environmental features of the environment represented by the map, anddetermining a position of the mobile device in the environment in a loop closing by comparing the acquired environment data with the map data for matching the respective environment features,a communication module for providing a communication connection between the computing device of the mobile device and a further computing device, anda control device which is designed,determining an available bandwidth of the communication connection,determining an available computing capacity of the computing device of the mobile device; anddetermining an available computing capacity of the further computing device via the communication link, andthe tracking, mapping and / or loop closing between the computing device of the mobile device and the further computing device is divided depending on the available bandwidth, the available computing capacity of the computing device of the mobile device and the available computing capacity of the further computing device according to a predetermined assignment rule, andthe communication module for transmitting processing data relating to tracking, mapping and / or loop closing assigned to the further computing device to the further computing device.It is to be understood that, with regard to device-related term definitions and the effects and advantages of device-related features, the disclosure of analogous definitions, effects and advantages of the method according to the invention can be fully resorted to and vice versa. Repetition of explanations of correspondingly identical features, the effects and advantages thereof can thus be dispensed with in favor of a more compact description, without such omissions having to be interpreted as a restriction for one of the disclosed subject matter of the invention.In advantageous embodiments, the mobile device is designed as a vehicle, in particular as an autonomously operable vehicle, mobile robot or smartphone.Yet another subject matter of the invention is a system for locating a mobile device, in particular a vehicle, comprising the mobile device according to one of the embodiments disclosed herein and a further computing device which is designed to carry out the tracking, mapping and / or loop closing assigned to it according to one embodiment of the method according to the invention disclosed herein.In advantageous embodiments, the further computing device is provided by means of the mobile device or external to the device by means of a further mobile device, in particular a further vehicle, a cloud server or an edge cloud server.Further features and advantages of the invention will become apparent from the following description of exemplary embodiments of the invention, which are to be understood as non-limiting and are explained in more detail below with reference to the drawings. In this drawing, the following are shown schematically: FIG. 1 shows four partial views (a)-(d) of a mobile device for carrying out a localization by means of simultaneous position determination and mapping (SLAM) and a system relating thereto, in each case according to different embodiments of the invention; FIG. 2 is a functional diagram of localization by means of simultaneous positioning and mapping (SLAM); FIG. 3 is a functional diagram for describing a method for locating a mobile device using simultaneous positioning and mapping (SLAM) according to an exemplary embodiment of the invention; FIG. 4 is a functional diagram for describing a method for locating a mobile device by means of simultaneous positioning and mapping (SLAM) according to a further exemplary embodiment of the invention; FIG. 5 shows a section of four exemplary embodiments of the method from FIG. 3 ; FIG. 6 shows a section of two exemplary embodiments of the method from FIG. 4 ; FIG. 7 shows a diagram for the dynamic selection of states shown in FIG. 6 according to a predetermined assignment rule; and FIG. 8 shows a diagram for dynamically limiting a number n F of environmental features to be determined according to a further assignment rule.In the different figures, parts which are equivalent in terms of their function are always provided with the same reference symbols, so that they are generally also described only once.FIG. 1 schematically illustrates, in four partial views (a), (b), (c) and (d), a mobile device 10 and 20 each for carrying out a localization by means of simultaneous position determination and mapping (SLAM) and a system 50, 51, 52, 53 in this respect, in each case according to different embodiments of the invention.In FIG. 1, the mobile devices 10 and 20 are each represented as vehicles, but are not necessarily limited thereto. For example, a mobile device may be a smartphone or a mobile robot.Each of the illustrated mobile devices 10 and 20 is configured to perform a localization in an environment of the respective mobile device 10 or 20 using a map 11 (see FIG. 2 ) illustrating the environment by means of simultaneous position determination and mapping (SLAM).The mobile devices 10, 20 each have an environment detection device 12 (cf. FIG. 2 ) which is designed and configured to detect environment data SD (cf. FIG. 2 ) of the environment, and a computing device Sys 1 which is designed and configured to localize the mobile device 10 or 20 by processing the detected environment data SD,in a tracking T (cf. FIG. 2 ), at least one environmental feature of the environment is to be determined from the captured environmental data SD,providing map data describing map 11 in a mapping M (cf. FIG. 2 ) which map data have environmental features of the environment represented by map 11, andin a loop closing LC (cf. FIG. 2 ), a position of the mobile device 10 or 20 in the environment can be determined by comparing the captured environment data with the map data for matching the respective environment features.The mobile devices 10, 20 also have a communication module (not shown) for providing a communication connection 13, 14 or 15 between the computing device Sys 1 of the mobile device 10, 20 and a further computing device Sys 2, and also a control device (not shown) which is designed to ascertain an available bandwidth of the communication connection 13, 14, 15, to ascertain an available computing capacity of the computing device Sys 1 of the mobile device 10 or 20 and to ascertain an available computing capacity of the further computing device Sys 2 via the communication connection 13, 14 or 15, and to ascertain the tracking T, mapping M and / or loop closing LC between the computing device Sys 1 of the mobile device 10 or 20 and the further computing device Sys 2 on the basis of the available bandwidth, the available computing capacity of the computing device Sys 1 of the mobile device 10 or 20 and the available computing capacity of the further computing device Sys 2 are divided according to a predetermined assignment rule and the communication module is controlled to transmit processing data relating to the tracking T, mapping M and / or loop closing LC assigned to the further computing device Sys 2 to the further computing device Sys 2.As is illustrated in FIG. 1, the communication link 13 and 15 for transmitting the processing data between the computing device Sys 1 of the mobile device 10 and the further computing device Sys 2 can be configured in a wireless manner, for example as a radio link based on a WLAN or mobile radio standard.In FIG. 1, the further computing device Sys 2 in view (a) is provided as an edge cloud server 60. In view (c), the further computing device Sys 2 is provided as computing device of a further mobile device 30, i.e. the communication connection 15 is a direct vehicle-to-vehicle communication connection (V 2V). The direct V2V communication link enables direct map sharing and cooperative mapping between the two mobile devices 10 and 30. the mobile device 30 can be designed and configured essentially like the mobile device 10, so that in this case the computing device Sys2 can execute essentially the same functions as the computing device Sys1 of the mobile device 10. However, the invention is not necessarily limited thereto, i.e. the mobile device 30 including its computing device Sys 2 does not necessarily have to be designed to carry out the method according to the invention.The edge cloud server 60 may be part of a server system, cloud system, or an edge cloud system (MEC).In view (b) of FIG. 1, the communication connection 14 is configured as a wired communication connection, since the two computing devices Sys 1 and Sys 2 are provided on the same mobile device 20, for example in the form of a system-on-a-chip. The division of the two computing devices Sys 1 and Sys 2 within the same mobile device 20 can comprise different hardware chips or a hardware chip with different virtualized systems (e.g. containers, virtual machines).View (d) of FIG. 1 shows a wireless communication link 13 on the one hand between the mobile device 10 and the edge cloud server 60 and a forwarding or distribution of the map 11 to at least one further mobile device 30' (e.g. vehicle) with a further computing device Sys2' which can be designed and set up substantially like the mobile device 10, but without being necessarily limited thereto, i.e. the mobile device 30' including its computing device Sys2' do not necessarily have to be designed for carrying out the method according to the invention.FIG. 2 illustrates a functional diagram of localization using simultaneous positioning and mapping (SLAM). The calculation of the entire SLAM method on a single computing device, e.g. computing device Sys 1, is referred to as an execution "on device" (OD) and declared as state S 0.After the environment data SD has been output by the environment detection device 12, e.g. a camera (optical, monocular, RGB-D, LIDAR, etc.), the environment features are extracted and classified in the tracking T. For this purpose, methods such as ORB, FAST, LETTER, etc., can be used, among others. The number of extracted or determined environmental features is referred to herein as n F.The positions of the determined and detected environmental features in an image are referred to as a frame. A decision criterion is used to decide whether a frame is declared as key frame KF. The decision can be made, for example, on the basis of a (relative or absolute) number of newly determined / found and classified environmental features. The key frame KF has sufficient new information to provide added value for the card 11 or the location. When a new keyframe KF is declared, it is forwarded to mapping M. The mobile device can be localized on the basis of the map 11 created in the mapping M. In addition, the loop closing LC is performed.FIG. 3 is a functional diagram for describing a method for locating a mobile device using simultaneous positioning and mapping (SLAM), referred to as full offload (FO), according to an embodiment of the invention. Broken-line rectangles in FIG. 3 identify optional method steps which, in combination with the other method steps shown, form further exemplary embodiments of the method according to the invention.The method serves for locating a mobile device, e.g. the mobile devices 10, 20, in an environment using the map 11 representing the environment by means of simultaneous position determination and mapping (SLAM), wherein the mobile device 10, 20 has the environment detection device 12 and the computing device Sys 1.The environment data SD of the environment of the mobile device 10, 20 is initially acquired by means of the environment acquisition device 12. The mobile device 10, 20 is localized by processing the captured environment data SD by ascertaining at least one environment feature of the environment from the captured environment data SD in tracking T, providing map data describing the map 11 in mapping M, which map data have environment features of the environment represented by the map 11, and determining a position of the mobile device 10, 20 in the environment in the loop closing LC by comparing the captured environment data with the map data for matching the respective environment features with one another.In the exemplary embodiment illustrated in FIG. 3, tracking T, mapping M and loop closing LC take place on the provided further computing device Sys 2, for which reason the situation shown is also referred to as "full offload" FO.Furthermore, it can be seen from FIG. 3 that a communication connection, for example a communication connection 13, 14 or 15 as shown in FIG. 1, is provided between the computing device Sys 1 of the mobile device 10, 20 and the further computing device Sys 2.An available bandwidth of the communication link 13, 14 or 15 is determined, an available computing capacity of the computing device Sys 1 of the mobile device 10, 20 and an available computing capacity of the further computing device Sys 2 via the communication link 13, 14 or 15.The tracking T, mapping M and / or loop closing LC are divided between the computing device Sys 1 of the mobile device 10, 20 and the further computing device Sys 2 depending on the available bandwidth, the available computing capacity of the computing device Sys 1 and the available computing capacity of the further computing device Sys 2 according to a predetermined assignment rule, wherein the division in the present case is carried out, for example, in such a way that the tracking T, mapping M and loop closing LC are carried out by the further computing device Sys 2. Accordingly, processing data relating to the allocated tracking T, mapping M and loop closing LC are transmitted to the further computing device Sys 2 via the communication link 13, 14 and 15, respectively.FIG. 3 also shows that an information content of the captured environmental data SD can be reduced before tracking T as a function of the available bandwidth of the communication link 13, 14 or 15 optionally by means of a correspondingly configured filter function A, as has already been explained in the general part of this description. The filter function A is provided by the computing device Sys 1.Alternatively or additionally, an information content of the processing data transmitted to the further computing device Sys 2 can be reduced by the further computing device Sys 2 before carrying out the allocated tracking T, mappings M and / or loop closings LC as a function of the available computing capacity of the further computing device Sys 2 by means of a correspondingly configured filter function B, as likewise explained in the general part of this description. The filter function B is provided by the further computing device Sys 2.Alternatively or additionally, the processing data can be compressed by the computing device Sys 1 before transmission to the further computing device Sys 2 by means of a compression function C and, after transmission from the further computing device Sys 2, can be decompressed by means of a decompression function D before execution of the allocated tracking T, mappings M and / or loop closings LC. The compression C can optionally be effected, for example depending on the currently available bandwidth of the communication link 13, 14 or 15, losslessly (lossless) or lossyly (lossy).FIG. 4 illustrates a functional diagram for describing a method of locating a mobile device, e.g., one of the mobile devices 10, 20 of FIG. 1, using simultaneous positioning and mapping (SLAM), in accordance with another embodiment of the invention, referred to as a "partial offload" (PO).The essential difference of the method shown in FIG. 4 from the exemplary embodiment shown in FIG. 3 can be seen in the fact that the tracking T is carried out on the computing device Sys 1 of the mobile device 10 or 20 and the mapping M and loop closing LC on the further computing device Sys 2.As in the exemplary embodiment from FIG. 3, a filter function A can also be applied selectively to the data stream of the captured environmental data SD on the part of the computing device Sys 1 and / or a filter function B can be applied to the data stream of the processing data transmitted via the communication link 13, 14 or 15 on the part of the further computing device Sys 2.If the processing data is compressed (lossless or lossy) by the computing device Sys 1 by means of the compression function C before transmission via the communication link 13, 14 or 15, the compression function C is applied by the computing device Sys 1 in this case after the tracking T to the processing data to be transmitted. The further computing device Sys 2 applies the decompression function D to the transmitted processing data after reception and supplies the decompressed processing data to the mapping M.Since in FIG. 4 the tracking T is carried out on the computing device Sys 1, this is provided with a local copy of the map 11 with the map data describing the environment. For this purpose, the created and / or updated card 11 or parts thereof provided on the further computing device Sys 2 can be transmitted to the computing device Sys 1 via the communication connection 13, 14 or 15.FIG. 5 illustrates a portion of four example embodiments of the method of FIG. 3 (full offload, FO) labeled with FO states S 1, S 2, S 3, and S 4, where the FO state S 3 is again divided into two FO substates S 3 I and S 3 II. In FIG. 5, the respective section only represents the steps after the capturing of the environment data SD (cf. FIG. 3 ) up to the tracking T.The first FO state S 1 relates to the case in which the captured environmental data SD (cf. FIG. 3 ) are transmitted from the computing device Sys 1 as processing data via the communication link 13, 14 or 15 to the further computing device Sys 2 substantially without further processing, which is symbolized in FIG. 5 by a transmission function S. The reception of this processing data by the further computing device Sys 2 is symbolized in FIG. 5 by a reception function R.The second FO state S 2 relates to the case in which the captured environmental data SD (cf. FIG. 3 ) is filtered by means of the filter function A before transmission to the further computing device Sys 2, as has already been described in detail elsewhere herein.The third FO state S 3 relates to the case in which the captured environmental data SD (cf. FIG. 3 ) is compressed by the computing device Sys 1 by means of the compression function C before transmission to the further computing device Sys 2. In this case, the FO state S 3 is divided into the two FO substates S 3 I and S 3 II wherein the first FO substate S 3 I is based on lossy (lossy) compression and the second FO substate S 3 II is a lossless (lossless) compression, which can be used optionally. The further computing device Sys 2 applies the decompression function D to the received processing data.The fourth FO state S 4 relates to the case in which the processing data received from the further computing device Sys 2 are filtered by means of the filter function B as described herein before they are fed to the tracking T.FIG. 6 illustrates a portion of two example embodiments of the method of FIG. 4 ("Partial Offload", PO) labeled with PO states S 5 and S 6, where the PO state S 6 is again divided into two PO substates S 6 I and S 6 II. In FIG. 6, the respective section only represents the steps from the tracking T of the captured environment data SD (cf. FIG. 3 ) by the computing device Sys 1 to the receipt of the transmitted processing data by the further computing device Sys 2.The first PO state S 5 relates to the case in which the captured environment data SD (cf. FIG. 3 ) is transmitted after tracking T from the computing device Sys 1 as processing data via the communication link 13, 14 or 15 to the further computing device Sys 2, which is symbolized in FIG. 6 by the transmission function S. The reception of this processing data by the further computing device Sys 2 is symbolized in FIG. 6 by the reception function R.The second PO state S 6 relates to the case in which the captured environmental data SD (cf. FIG. 3 ) is compressed by the computing device Sys 1 after tracking T by means of the compression function C before transmission to the further computing device Sys 2. In this case, the PO state S 6 is divided into the two PO substates S 6 I and S 6 II wherein the first PO substate S 6 I is based on lossy (lossy) compression and the second PO substate S 6 II is a lossless (lossless) compression, which can be selected selectively. The further computing device Sys 2 applies the decompression function D to the received processing data.The states S 1-S 6 illustrated in FIGS. 5 and 6 are explained below by way of example as to how the tracking T, mapping M and / or loop closing LC is divided between the computing device Sys 1 of the mobile device 10 or 20 and the further computing device Sys 2 in accordance with a predetermined assignment rule as a function of the available bandwidth, the available computing capacity of the computing device Sys 1 of the mobile device 10 or 20 and the available computing capacity of the further computing device Sys 2.The available computing capacity Rl V of one of the computing devices Sys 1 or Sys 2 can be detected by the respective computing device Sys 1, Sys 2 by querying the current relative processor load RL A. An index I RL can be used which defines the computing capacity of the respective computing device Sys 1, Sys 2. Computing devices having different computing capacities (e.g. different processor hardware) can thus be made comparable:The respective states S 1-S 6 can be selected depending on the available computing capacity on Sys 1 RL V( Sys 1) and on Sys 2 RL V( Sys 2) and the available bandwidth BW of the communication link 13, 14, 15.The required computing capacity RL A of the individual states S1-S6 on Sys1 can be ordered as follows:The required bandwidth BW req of the communication link 13, 14, 15 of the individual states on Sys1 can be ordered as follows:For the selection of the state which is the most suitable at the present time, i.e. for the optimum division of the processing steps between the computing devices Sys1 and Sys2, a matrix can be used which is shown below.Rl V ( Sys1) H / RTMdepending on the bandwidth,Depending on I RL-Bw( Sys2):Rl V ( Sys2) HFalling BW rel:S6 lS1S6 llS3 llS5[Increasing I RL-BW( Sys2)] Dependency, since here the priority is on the processing in Mapping M and Loop Closing LCRL V ( Sys1) H / RLS5S0Rl V ( Sys2) LS6 llS5S6 lS6 lS4[Increasing I RL-BW( Sys2)][Increasing I RL-BW( Sys2)]RL V ( Sys1) L / RLS1S2Rl V ( Sys2) HS2S3 lS4S3 llS3 l[Increasing I RL-BW( Sys1)]S3 llSys1 here, since the limitation here is[Increasing I RL-BW( Sys1)] Here Sys1, since the limitation is present herepresent is presentRl V ( Sys1) L / Y.S2S2Rl V ( Sys2) LS4S3 lS1S5 (optionally in combination with[Increasing I RL-BW( Sys2)]Adaptation of n F ( cf. 8)) [Increasing I RL-BW( Sys2)]The decision threshold of the states within the matrix fields I RL_BW( Sys) is given by the available computing capacity on the two computing devices Sys1, Sys2 and the available relative bandwidth BW rezThe relative bandwidth BW rez indicates the available bandwidth of the communication link, e.g. communication link 13, 14, 15, depending on the maximum bandwidth of the communication link.A weight of the available computing capacity is defined as ≅ and the weight of the available bandwidth is defined as β under the condition:FIG. 7 illustrates, by way of example, a diagram for the dynamic selection of the states S 5, S 6 I and S 6 II illustrated in FIG. 6 according to a predetermined assignment rule 70. The case shown applies to β> ≅ and represents the matrix field RL V( Sys1) H / RL V( Sys2) H & BW rez L. It can be seen in FIG. 7 that depending on the decision threshold I RL_BW( Sys2), the respective states S6 I, S6 II or. S5 can be selected.In addition, the number n F of environmental features to be extracted may be adjusted in the tracking T. n F may be reduced or increased depending on the available bandwidth of the communication link, the available computing capacity, or a detection rate (e.g., frame rate) of the environmental data. The dependency may be linear, exponential or logarithmic, for example.FIG. 8 illustrates a diagram for dynamically limiting the number n F of environmental features to be determined. It can be seen in FIG. 8 that the number of determined environmental features n F increases with an increasing number of captured environmental data SD (e.g. images of a camera) until it reaches a maximum value predetermined by a boundary line 71. The height of the boundary line 71 can be dynamically adjusted as a function of the available computing capacity of the computing device Sys 1 of the mobile device 10, 20 and / or the available bandwidth of the communication link 13, 14, 15 and / or a detection rate of the environmental data SD and / or a travel speed of the mobile device 10, 20, which is indicated by the double arrow 72 in FIG. 8.For the setting high (H) or low (L) of BW rez and RL V in the matrix illustrated above, threshold values may be selected depending on the total computing capacity and the available bandwidth of the communication link depending on, for example, a resolution of the environment data (e.g., camera images), a predetermined value for n F and a capture rate (e.g., frame rate) of the environment data. These values can be determined and established application-specific and / or empirically.For example, a required bandwidth threshold BW rez for the transmission of the processing data may be set according to the state S 5. This value corresponds to the threshold BW schw. If BW rez > BW schw, then BW rez holds H, otherwise BW rez holds L.Alternatively or additionally, a threshold value for performing the tracking T and transmitting the processing data RL A( S5, Sys1) can be established. This value corresponds to the processor load on Sys1. If RL V( Sys1)>RL A( S5, Sys1), then RL V( Sys1) holds H, otherwise RL V( Sys1) holds L.Alternatively or additionally, a value of the processor load for receiving the processing data and performing the mapping M and loop closing LC to Sys2 RL A( S5, Sys2) may be set. If RL V( Sys2)>RL A( S5, Sys2), then RL V( Sys2) holds H, otherwise RL V( Sys2) holds L.LIST OF REFERENCE CHARACTERS:10 Mobile device 11 card 12 environment detection device 13 communication connection 14 communication connection 15 communication connection 20 mobile device 30 mobile device 30' mobile device 50 system 51 system 52 system 53 system 60 edge cloud server 70 allocation rule 71 boundary line 72 dynamic adaptation of 71 A filter function A B filter function B C compression function D decompression function I RL_BW decision threshold n F number of determined environmental features n SD number of detected environmental data R processing data received S processing data transmitted SD detected environmental data Sys 1 computing device Sys 2 further computing device Sys 2' further computing device S 0 Od state S1 First FO state S2 Second FO state S3 Third FO state S3 I First FO substate of S3 S3 II Second FO substate of S3 S4 Fourth PO state S5 First PO state S6 Second PO state S6 I First PO substate of S6 S6 II Second PO substate of S6 KF key frame LC loop closing M mapping SLAM Simultaneous positioning and mapping t trackingReferences included in the specificationThis list of documents cited by the applicant has been produced in an automated manner and is only included for the better information of the reader. The list is not part of the German patent application or utility model application. The DPMA does not take any adhesion for any faults or omissions.Patent Literature citedDE 10 2022 100 454 A1

[0008] DE 10 2021 130 382 A1

[0009] US 2022 / 0351553 A1

[0010]

Claims

Method for locating a mobile device (10, 20), in particular a vehicle, in an environment using a map (11) representing the environment by means of simultaneous position determination and mapping (SLAM), wherein the mobile device (10, 20) has an environment detection device (12) and a computing device (Sys1), and the method has the following steps: - detection of environment data (SD) of the environment of the mobile device (10, 20) by means of the environment detection device (12), - locating the mobile device (10, 20) by processing the detected environment data (SD) by - at least one environment feature of the environment being determined from the detected environment data (SD) in a tracking (T), - map data describing the map (11) being provided in a mapping (M), the environmental features of the environment represented by the map, and - a position of the mobile device (10, 20) in the environment is determined in a loop closing (LC) by comparing the acquired environmental data (SD) with the map data for matching the respective environmental features with one another, - providing a further computing device (Sys2), - providing a communication link (13, 14, 15) between the computing device (Sys1) of the mobile device (10, 20) and the further computing device (Sys2), - determining an available bandwidth of the communication link (13, 14, 15), - determining an available computing capacity of the computing device (Sys1) of the mobile device (10, 20), - determining an available computing capacity of the further computing device (Sys2) via the communication link (13, 14, 15), dividing the tracking (T), mapping (M) and / or loop closing (LC) between the computing device (Sys1) of the mobile device (10, 20) and the further computing device (Sys2) as a function of the available bandwidth, the available computing capacity of the computing device (Sys1) of the mobile device (10, 20) and the available computing capacity of the further computing device (Sys2) in accordance with a predetermined assignment rule (70), and transmitting processing data relating to the tracking (T), mapping (M) and / or loop closing (LC) assigned to the further computing device (Sys2) to the further computing device (Sys2).Method according to Claim 1, in which a maximum number (n F) of the environmental features determined from the captured environmental data (SD) is dynamically adapted as a function of the available computing capacity of the computing device (Sys1) of the mobile device (10, 20) and / or the available bandwidth of the communication connection (13, 14, 15) and / or a capture rate of the environmental data (SD) and / or a movement speed of the mobile device (10, 20) in accordance with a predetermined further assignment specification (72).Method according to Claim 1 or 2, in which an information content of the captured environment data (SD) is reduced before tracking (T) as a function of the available bandwidth of the communication connection (13, 14, 15).Method according to one of the preceding claims, in which an information content of the processing data transmitted to the further computing device (Sys2) is reduced by the further computing device (Sys2) before carrying out the allocated tracking (T), mapping (M) and / or loop closing (LC) as a function of the available computing capacity of the further computing device (Sys2).Method according to one of the preceding claims, in which the processing data are compressed before transmission to the further computing device (Sys2) and are decompressed after transmission from the further computing device (Sys2) before execution of the allocated tracking (T), mapping (M) and / or loop closings (LC).Method according to one of the preceding claims, in which the further computing device (Sys2) is provided by means of the mobile device (20) or is provided externally to the device by means of a further mobile device (30), a cloud server or an edge cloud server (60).Mobile device (10, 20), in particular a vehicle, which is designed to carry out a localization in an environment of the mobile device (10, 20) using a map (11) representing the environment by means of simultaneous position determination and mapping (SLAM), having - an environment detection device (12) which is designed to detect environment data (SD) of the environment, - a computing device (Sys1) which is designed and designed to localize the mobile device (10, 20) by processing the detected environment data (SD), - to determine at least one environment feature of the environment from the detected environment data (SD) in a tracking (T), - to provide map data describing the map (11) in a mapping (M) which map data have environment features of the environment represented by the map, and - in a loop closing (LC), determining a position of the mobile device (10, 20) in the environment by comparing the acquired environment data (SD) with the map data for matching the respective environment features with one another, - a communication module for providing a communication link (13, 14, 15) between the computing device (Sys1) of the mobile device (10, 20) and a further computing device (Sys2), and - a control device which is designed to - determine an available bandwidth of the communication link (13, 14, 15), - determine an available computing capacity of the computing device (Sys1) of the mobile device (10, 20) and - determine an available computing capacity of the further computing device (Sys2) via the communication link (13, 14, 15), and - the tracking (T), Mapping (M) and / or loop closing (LC) between the computing device (Sys1) of the mobile device (10, 20) and the further computing device (Sys2) as a function of the available bandwidth, the available computing capacity of the computing device (Sys1) of the mobile device (10, 20) and the available computing capacity of the further computing device (Sys2) in accordance with a predetermined assignment specification (70), and - to actuate the communication module for transmitting processing data relating to the tracking (T), mapping (M) and / or loop closing (LC) assigned to the further computing device (Sys2) to the further computing device (Sys2).Mobile device according to Claim 7, which is designed as a vehicle, in particular as an autonomously operable vehicle, mobile robot or smartphone.System (50, 51, 52, 53) for locating a mobile device (10, 20), in particular a vehicle, comprising the mobile device (10, 20) according to Claim 7 or 8 and a further computing device (Sys2) which is designed to carry out the tracking (T), mapping (M) and / or loop closing (LC) assigned to it according to a method according to one of Claims 1 to 6.System according to Claim 9, in which the further computing device (Sys2) is provided by means of the mobile device (20) or is provided externally to the device by means of a further mobile device (30), in particular a further vehicle, of a cloud server or of an edge cloud server (60).

Images