Learnable localization using images

Abstract

An apparatus for generating a model for pose estimation in a system obtains training data for multiple positions. The training data includes one or more images captured by an image capture device and corresponding poses of the captured images. At least one data sample is generated for each of the captured images from the training data, wherein the data sample of the image assigns the image and at least one other image selected from the training data to corresponding poses. A neural network is trained using a data set consisting of the data samples to estimate the corresponding pose of the positioning image from the following: the positioning image; at least one additional image from the training data; and the corresponding pose of each additional image.

Description

Background Art

[0001] The present invention, in some embodiments thereof, relates to image localization in a region of interest, and more particularly, but not exclusively, to image localization in a region of interest using a neural network.

[0002] Mapping and localization have become increasingly important in many technology areas, such as robotics, advanced driver assistance systems (ADAS), and autonomous driving systems.

[0003] Typically, a map is created by first mapping a designated area / route by collecting sensor information about the designated area. A map is a collection of geometric information and other data that describes the properties of an area of ​​interest (eg, a route).

[0004] For localization purposes, the map is used together with the current sensor information to deduce the current position of the vehicle or system within the map-building area. One of the most important types of sensor input is optical sensors, such as digital cameras.

[0005] Current approaches extract features from images based on a sequence of captured images. The extracted features are processed to deduce scene geometry and scene structure in order to create a map of a specified area, which typically contains image features with corresponding three-dimensional (3D) positions, and pose nodes containing information about the position and orientation of the vehicle along its trajectory during map construction.

[0006] Localization can be performed by matching the current sensor readings with map data using image features (e.g., point or edge features), position data, and orientation data contained in the map. These methods are typically gradient-based optimization methods, such as simultaneous localization and mapping (SLAM).

[0007] SLAM techniques use point features (e.g., ORB2 SLAM) or line / edge features (e.g., LSD-SLAM) for map construction and localization. This may be an appropriate solution for scenes with high texture and limited moving objects. However, in scenes with limited texture and / or many or large moving objects, these methods may lead to inaccurate localization results.

[0008] Another problem encountered by these methods is that both the mapping time and the localization time of the scene must be similar. For example, this requirement cannot be met when the mapping is done in the winter (e.g., a snowy scene) and the localization is done in the summer. The scene structure may be too different from the point features or edge features to be recognized and matched. In addition, some applications have very specific conditions for scenes with repeating patterns in the scene (e.g., parking lots). It may be difficult to have a reliable mapping and localization system using only local information within the image (e.g., point and edge features).

[0009] Other methods for localization are based on unsupervised deep learning using convolutional neural networks (CNNs) for ranging. These methods do not take into account the sequence of pre-map construction and therefore cannot be directly applied to high-precision localization. In one method, self-motion, ranging poses, and corresponding depth maps are estimated from multiple views simultaneously in an unsupervised manner. The relative pose relative to the reference pose depends only on the corresponding image and the corresponding estimated depth map. In a similar method, the optical flow can also be estimated and used to calculate the relative pose. Only the image and the corresponding depth map and the optical flow field are input to the CNN. In another method, the focus is on estimating the scale parameters of the monocular camera trajectory.

[0010] PoseNet is a convolutional network for real-time six degrees of freedom (6-DOF) camera relocalization. PoseNet achieves localization by training a CNN to estimate the 6DOF pose using a single image as input. PoseNet has a large error due to the limited amount of information about the scene provided to the estimator / CNN. The larger the area covered by the localization, the larger the localization estimation error.

[0011] Other background technologies include:

[0012] [1] Raul Mur-Artal et al., “ORB2-SLAM: A Versatile and Accurate Monocular SLAM System,” IEEE Robotics, vol. 31, no. 5, October 2015, pp. 1147–1163.

[0013] [2] Jakob Engel et al., “LSD-SLAM: Large Scale Direct Monocular SLAM,” www.vision.in.tum.de / research / vslam / lsdslam.

[0014] [3] Tinghui Zhou et al., “Unsupervised learning of depth and ego-motion from videos,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[0015] [4] Thanuja Dharmasiri et al., “ENG: End-to-End Neural Geometry for Robust Depth and Pose Estimation using CNNs,” www.arxiv.org / abs / 1807.05705.

[0016] [5] Ruihao Li et al., “UnDeepVO: Monocular Odometry via Unsupervised Deep Learning,” www.arxiv.org / abs / 1709.06841.

[0017] [6] Alex Kendall et al., “PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization,” www.arxiv.org / pdf / 1505.07427.pdf. Summary of the invention

[0018] The present invention aims to provide an apparatus, system, computer program product and method for robust and accurate localization using deep learning to exploit information provided by images taken during map construction of an area of ​​interest.

[0019] Aspects of the invention include:

[0020] (a) training a neural network to estimate the pose changes between images; and

[0021] (b) Apply the trained neural network to estimate the pose of the localization image.

[0022] In order to exploit more image information during localization, the images (or a selected subset of keyframes) obtained during the map building process are stored. Optionally, a necessary amount of overlap of content and / or scenes is maintained between each consecutive keyframe.

[0023] During training, a neural network (e.g., a deep neural network) is trained to predict pose differences based on a data set including a plurality of data samples. Each data sample assigns a corresponding pose to a subset of images taken when mapping a region of interest. Optionally, the pose is represented relative to a pose of a reference image, as described below.

[0024] During the localization phase, a trained neural network estimates the pose of the localization image based on the localization image itself and stored images taken during map construction and their corresponding poses. The output of the neural network is an estimate of the pose of the localization image. When the pose estimate is made relative to a reference image, a transformation can be applied to convert the pose estimate to a map relative pose.

[0025] The types of neural networks that can be trained and used for localization include, but are not limited to:

[0026] (1) Deep neural network (DNN);

[0027] (2) Convolutional neural network (CNN);

[0028] (3) recurrent neural network (RNN); and

[0029] (4) Long short-term memory (LSTM) neural network.

[0030] Advantages of the present invention include:

[0031] (1) Datasets for training neural networks can be easily constructed, for example using Differential Global Positioning System (DGPS) and stereo images recorded from multiple loops along the same route.

[0032] (2) Once the neural network is trained, only one loop is needed to build the map.

[0033] (3) Relative pose estimation can use multi-layer neural networks to improve robustness. Neural networks can not only use point, line and simple edge features, but also use more general and higher-order features in image content. The image content can be determined from the data by the training process.

[0034] (4) Improved sensitivity to difficult lighting conditions, allowing it to be explicitly trained.

[0035] (5) Achieve improved results in less structured environments, which may not provide point, line, and simple edge features but can be explicitly trained.

[0036] (6) No tracking loss.

[0037] (7) By including dynamic objects in the training data, the neural network can be explicitly trained to handle dynamic objects. When the training data is large enough, the neural network will learn to discard dynamic objects.

[0038] (8) A neural network can be trained for the case of imperfect mapping by artificially adding noise to the mapped poses while maintaining the appropriate ground truth localization poses in the dataset.

[0039] (9) The trained neural network can be used in a variety of environments (e.g., cities, urban scenes, highways, etc.).

[0040] (10) If a certain area changes significantly, only the map needs to be rebuilt so that the neural network detects that the positioning image is similar to the map image, but it is not necessary to retrain the neural network. The above and other objects are achieved by the features of the independent claims. Further implementations are apparent from the dependent claims, the detailed description and the drawings.

[0041] According to a first aspect of the present invention, a device for generating a model for pose estimation in a system includes a processor circuit. The processor circuit obtains training data for multiple positions. The training data includes one or more images captured by an image capture device and the corresponding poses of the captured images, wherein the corresponding poses of the images are the corresponding positions and orientations of the image capture device during the capture of the images. The processing circuit generates at least one data sample for each of the captured images from the training data, wherein the data sample of the image is the image and at least one other image selected from the training data assigned to the corresponding pose. The processor circuit trains a neural network using a data set consisting of the data samples to estimate the corresponding pose of the positioning image from the following:

[0042] (a) Positioning image;

[0043] (b) at least one additional image from the training data; and

[0044] (c) The corresponding pose of each additional image.

[0045] According to a second aspect of the present invention, a method for generating a neural network for pose estimation in a system includes: obtaining training data for multiple positions. The training data includes one or more images captured by an image capture device and the corresponding poses of the captured images, wherein the corresponding pose of the image is the corresponding position and orientation of the image capture device during the capture of the image. The method also includes: generating at least one data sample for each of the captured images from the training data, and training a neural network using a data set consisting of the data samples, wherein the data sample of the image is to assign the image and at least one other image selected from the training data to the corresponding pose. The neural network is trained to estimate the corresponding pose of the positioning image from the following:

[0046] (a) Positioning image;

[0047] (b) at least one additional image from the training data; and

[0048] (c) The corresponding pose of each additional image.

[0049] Datasets can be automatically constructed and can include images obtained under many conditions, so that the trained neural network is robust to changes in lighting conditions, the presence of dynamic objects in the image, and other changes. In addition, supervised learning can be used to train the neural network, which can improve performance relative to unsupervised learning methods.

[0050] In combination with the first aspect and / or the second aspect, in a possible implementation manner, generating at least one data sample includes:

[0051] Select at least one reference image from the training data;

[0052] Selecting at least one neighboring image from the training data whose pose is close to at least one reference image; and

[0053] • Assigning at least one reference image and at least one neighboring image to a respective pose of each neighboring image relative to each reference image.

[0054] Basing data samples on a single reference image simplifies the construction of data samples as well as the processing of data samples during neural network training.

[0055] In combination with the first aspect and / or the second aspect, in a possible implementation manner, for at least one data sample in the data samples, assigning the image and at least one other image selected from the training data to corresponding poses includes:

[0056] Select multiple reference images from the training data;

[0057] Selecting at least one neighboring image from the training data based on proximity to the reference image;

[0058] Specify the corresponding pose of each neighboring image relative to each reference image.

[0059] Using multiple reference images increases the amount of information in the data sample, thereby increasing the amount of information provided to the neural network.

[0060] In combination with the first aspect and / or the second aspect, in a possible implementation, a distance between a corresponding pose of at least one neighboring image and a pose of at least one of the reference images is within a specified distance. Since distant images are automatically excluded, (one or more) neighboring images can be easily selected.

[0061] In combination with the first aspect and / or the second aspect, in a possible implementation, before training, an error is introduced into the training data by changing at least one pose in the data sample. Intentionally introducing an error into the training data improves the robustness of the neural network trained after performing map construction to changes in the region of interest.

[0062] In combination with the first aspect and / or the second aspect, in one possible implementation, during training, errors in pose estimation are tracked, and corresponding confidence values ​​of poses estimated by the neural network are calculated based on the tracked errors. The confidence values ​​increase the amount of information available for training the neural network, thereby improving the training process.

[0063] In combination with the first aspect and / or the second aspect, in a possible implementation manner, at least one of the data samples further includes a semantic segmentation calculated for at least one image in the data set for map construction.

[0064] In combination with the first aspect and / or the second aspect, in a possible implementation manner, at least one of the data samples further includes a depth map calculated for at least one image in the data set for map construction.

[0065] The semantic segmentation and / or depth map information included in the data samples is used as additional input to the neural network, thereby reducing the requirements for training data and improving the generalization of the neural network.

[0066] In combination with the first aspect and / or the second aspect, in one possible implementation, the neural network is a deep neural network (DNN). DNN is capable of modeling the complex and nonlinear relationships required for image analysis and recognition, which is conducive to using actual images as inputs to the neural network to derive pose information. According to a third aspect of the present invention, a device for image-based positioning of a system in a region of interest includes a memory and a processor circuit. The memory is used to store images captured by an image capture device of the system at multiple locations in the region of interest and the corresponding poses of the image capture device associated with the stored images at each of the locations, wherein the pose of the image includes the position and orientation of the capture device associated with the corresponding image. The processor circuit is used to:

[0067] Obtaining a positioning image captured by an image capturing device;

[0068] Retrieve a subset of stored images and corresponding poses from memory;

[0069] Provide the trained neural network with:

[0070] (a) Positioning image;

[0071] (b) a subset of images; and

[0072] (c) the corresponding pose of the retrieved image; and

[0073] Obtain at least one estimated pose of the localization image from the trained neural network.

[0074] According to a fourth aspect of the present invention, a method for image-based positioning of a system in a region of interest comprises:

[0075] Obtain positioning images taken by the system's camera;

[0076] Provide the trained neural network with:

[0077] (a) Positioning image;

[0078] (b) a subset of images selected from a plurality of images taken during construction of a map of the area of ​​interest; and

[0079] (c) the corresponding poses of a subset of images; and

[0080] Obtain at least one estimated pose of the localization image from the trained neural network.

[0081] Neural networks are able to provide highly accurate pose estimates based on the large amount of available information by using the map image itself as part of the input data to the neural network during the localization process.

[0082] In combination with the third aspect and the fourth aspect, in one possible implementation, a proximate position of the image capture device during the capture of the positioning image is determined, and a subset of the stored images is selected based on the proximate position. Using close images can help detect errors in pose estimation. Since the pose difference between the images in the subset and the positioning image should be small, a large difference between the predicted pose of the image and the known pose can indicate an error.

[0083] In combination with the third aspect and the fourth aspect, in one possible implementation, the neural network outputs a single estimated pose relative to the pose of a single reference image selected from the retrieved images. Outputting a single pose estimate for each positioning image can minimize the processing required by the neural network afterwards.

[0084] In combination with the third aspect and the fourth aspect, in a possible implementation, the neural network outputs a plurality of estimated poses, each of which is related to the pose of a different reference image selected from the retrieved image. A final estimated pose is calculated from the plurality of estimated poses. By calculating the final estimated pose from a plurality of intermediate estimates, the amount of information used to obtain the final pose estimate is increased, thereby improving its accuracy.

[0085] In combination with the third aspect and the fourth aspect, in one possible implementation, an estimation accuracy metric is calculated by comparing a pose change between multiple estimated poses of a positioning image with at least one pose change between reference images used to calculate the multiple estimated poses. The estimation accuracy metric ensures that inaccurate pose estimates are identified and is not dependent on users or external systems.

[0086] In combination with the third aspect and the fourth aspect, in a possible implementation, the trained neural network is a deep neural network (DNN). DNN can model the complex and nonlinear relationships required for image analysis and recognition, which is conducive to using the actual image as the input of the neural network to derive the pose information.

[0087] In combination with the third aspect and the fourth aspect, in a possible implementation manner, the trained neural network is further provided with a semantic segmentation for calculating at least one of the images provided to the trained neural network.

[0088] In combination with the third aspect and the fourth aspect, in a possible implementation manner, the trained neural network is further provided with a depth map calculated for at least one of the images provided to the trained neural network.

[0089] Semantic segmentation and / or depth map information is used as additional input to the neural network, thereby improving the accuracy of pose estimation.

[0090] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those known to those of ordinary skill in the art. Methods and materials similar or identical to those described herein can be used in the practice or testing of embodiments of the present invention, and exemplary methods and / or materials are described below. If there is a conflict, the patent specification including the definitions shall prevail. In addition, materials, methods and examples are only used for illustration and are not necessarily limited.

[0091] Other systems, methods, features and advantages of the present invention will be or become apparent to those skilled in the art after studying the following drawings and detailed description. It is intended that all such other systems, methods, features and advantages be included in this description, within the scope of the present invention, and protected by the accompanying claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Some embodiments of the present invention are described herein with reference to the accompanying drawings only as examples. Now with reference to the accompanying drawings in detail, it should be emphasized that the items shown are examples for the purpose of illustrative discussion of the embodiments of the present invention. Thus, it will be apparent to those skilled in the art how to practice the embodiments of the present invention based on the accompanying drawings.

[0093] In the attached picture:

[0094] Figure 1 Map construction and positioning using a neural network provided by an exemplary embodiment of the present invention are shown;

[0095] Figure 2 A simplified block diagram of a device for generating a model for posture estimation in a system provided by an embodiment of the present invention;

[0096] Figure 3 A simplified flow chart of a method for generating a neural network for pose estimation in a system provided by an embodiment of the present invention;

[0097] Figure 4 It shows the creation of a data sample having a single reference node and multiple neighboring nodes provided by an embodiment of the present invention;

[0098] Figure 5 A simplified flow chart of a method for creating a data sample having a single reference node and multiple neighboring nodes provided by an embodiment of the present invention;

[0099] Figure 6 It shows the creation of a data sample having multiple reference nodes and multiple neighboring nodes provided by an embodiment of the present invention;

[0100] Figure 7A simplified flow chart of a method for creating a data sample having multiple reference nodes and multiple neighboring nodes provided by an embodiment of the present invention;

[0101] Figure 8 A simplified flow chart of a method for creating data samples with confidence values ​​provided for an exemplary embodiment of the present invention;

[0102] Fig. 9 A simplified flow chart of a method for introducing errors into a data sample provided for an exemplary embodiment of the present invention;

[0103] Fig.10 A simplified block diagram of an apparatus for image-based positioning of a system in a region of interest provided by an embodiment of the present invention;

[0104] Fig.11 A simplified flow chart of a method for image-based positioning of a system in a region of interest provided by an embodiment of the present invention;

[0105] Fig.12 The present invention shows a pose estimation of a positioning pose relative to a single reference node provided by an exemplary embodiment of the present invention;

[0106] Fig.13 Shows Fig.12 An exemplary embodiment provides inputs and outputs of a trained neural network;

[0107] Fig.14 Pose estimation of positioning poses relative to multiple reference nodes is shown;

[0108] Fig.15 Shows Fig.14 An exemplary embodiment provides inputs and outputs of a trained neural network;

[0109] Fig.16 shows different estimates obtained for a single localization pose; and

[0110] Fig.17 A simplified flowchart of a method for determining an estimation accuracy metric for pose estimation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0111] The present invention, in some embodiments thereof, relates to image localization in a region of interest, and more particularly, but not exclusively, to image localization in a region of interest using a neural network.

[0112] The embodiments of the invention presented herein use a hybrid approach to use a trained neural network for localization. The neural network not only performs localization based on data extracted from images taken during map construction, but also uses the images themselves for localization. Images taken during the map construction process are used to train the neural network and are input to the trained neural network during localization.

[0113] Feeding stored map images into the neural network during localization increases the amount of image information available to the neural network, resulting in highly accurate and reliable localization estimates.

[0114] As used herein, the term "pose of an image" refers to the position and orientation of an image capture device when it captures the image (i.e., at the time the image was captured). The pose can be expressed as three positional coordinates and three angular coordinates that indicate the position and orientation associated with the image. It should be understood that the pose is relative to a coordinate system, and the pose can be transformed from one coordinate system to another by linear and / or angular transformations.

[0115] As used herein, the term "localization image" refers to an image whose pose is being estimated by a neural network.

[0116] As used herein, the term “map-relative pose” refers to a pose expressed in a map coordinate system (also referred to as a ground truth pose).

[0117] As used herein, the term “pose relative to a reference image” refers to a pose expressed in the coordinate system of the reference image.

[0118] Before explaining at least one embodiment of the present invention in detail, it should be understood that the present invention is not necessarily limited in its application to the details of the structure and arrangement of the components and / or methods set forth in the following description and / or illustrated in the drawings and / or examples. The present invention may have other embodiments or may be practiced or implemented in various ways.

[0119] The present invention may be a system, method and / or computer program product. The computer program product may include (one or more) computer-readable storage media having computer-readable program instructions for causing a processor to execute various aspects of the present invention.

[0120] A computer readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. For example, a computer readable storage medium may be, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or a raised structure in a groove on which instructions are recorded, and any suitable combination of the foregoing. Computer-readable storage media as used herein should not be understood as signals that are themselves transient, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (such as light pulses through optical cables), or electrical signals transmitted through wires.

[0121] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in the respective computing / processing device.

[0122] The computer-readable program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc. and conventional procedural programming languages ​​such as "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer or partially on the user's computer, or as a separate software package partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be established on an external computer (e.g., using the Internet provided by an Internet service provider). In some embodiments, electronic circuits including programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs) can utilize state information of computer-readable program instructions to execute computer-readable program instructions to personalize the electronic circuits so as to perform various aspects of the present invention.

[0123] Here, various aspects of the present invention are described in conjunction with the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products provided in the embodiments of the present invention. It should be understood that each frame of the flowchart and / or block diagram and the combination of frames in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0124] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to generate a machine, so that these instructions executed by the processor of the computer or other programmable data processing device create a way to implement the functions / actions specified by one or more boxes in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can instruct a computer, a programmable data processing device, and / or other equipment to operate in a specific manner, so that the computer-readable storage medium storing the instructions includes an article containing instructions, which are used to implement various aspects of the functions / actions specified by one or more boxes in the flowchart and / or block diagram.

[0125] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operating steps are performed on the computer, other programmable apparatus, or other device, thereby generating a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified by one or more blocks of the flowchart and / or block diagram.

[0126] The flow chart and block diagram in the figure illustrate the structure, function and operation of the possible implementation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each block in the flow chart or block diagram can represent a part of a module, a section or an instruction, which includes one or more executable instructions for implementing a specified (one or more) logical function. In some optional implementations, the functions indicated in the box may not be implemented in the order in the figure. For example, in fact, the two blocks shown in succession can be executed almost simultaneously, or sometimes, the blocks can be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart and the combination of boxes in the block diagram and / or flow chart can be executed by a system based on special hardware, which performs a specific function or action, or performs a combination of special hardware and computer instructions.

[0127] Reference now Figure 1 , Figure 1 Map construction and positioning using a neural network provided by an exemplary embodiment of the present invention are shown.

[0128] During map construction, a sequence of images is captured at multiple locations in the area of ​​interest (in this case, the vehicle's route). The map includes the captured images and their map-relative poses (i.e., the ground truth poses).

[0129] Some exemplary embodiments are described herein in a non-limiting manner for maps represented as pose graphs (referred to as map building sequences). The pose graph includes interconnected pose nodes that define one or more pose chains (e.g., following one or more trajectories of a vehicle). It should be understood that corresponding embodiments may be implemented for maps used to train neural networks and / or other representations of image and pose information that are input to neural networks during localization.

[0130] Optionally, a neural network is trained to estimate the pose of the localization image relative to the coordinate system of a selected reference node 10. The ground truth pose estimate of the localization image is calculated from the ground truth pose of the reference node 10 (known from map construction) and the estimated pose output by the neural network.

[0131] As used herein, the terms "pose node" and "node" refer to an element that has an associated image and pose, as well as information connecting to other pose nodes.

[0132] A pose node may include additional information, including but not limited to:

[0133] (1) Date and time the image was taken;

[0134] (2) the settings of the image capture equipment during image capture; and

[0135] (3) Weather conditions during image capture.

[0136] Optionally, the neural network is further trained to detect whether the localization image is likely to be from a location outside the region of interest being mapped, where accurate localization may not be possible based on the localization image.

[0137] A. Map Construction

[0138] During the map building process, an image capturing device captures images on an area of ​​interest (e.g., a route). A map building sequence is created from the captured images and their corresponding poses. In one example, a vehicle equipped with an image capturing device and position and orientation sensors travels along the route. The image capturing device captures images at a fixed rate and captures the pose at the time of each image.

[0139] As used herein, the term "image capture device" refers to any type of device capable of capturing an image, without limitation on how the image is captured or the hardware or type of the image capture device. Examples of image capture devices include, but are not limited to, cameras and image sensors.

[0140] One or more images may be captured at different times and / or different orientations (eg, stereo images). The types of images that may be captured include, but are not limited to, RGB images, chrominance-luminance images, and infrared images.

[0141] Optionally, the map is updated during the localization phase to maintain accuracy.

[0142] B. Training a Neural Network for Pose Estimation

[0143] Reference now Figure 2 , Figure 2 A simplified block diagram of an apparatus for generating a model for posture estimation in a system provided by an embodiment of the present invention. The apparatus 200 includes a processor circuit 210. The embodiment of the apparatus 200 may also include one or more of the following:

[0144] (a) memory 220; and

[0145] (b) Communication interface 230.

[0146] Processor circuit 210 performs processing operations for training a neural network.

[0147] The apparatus 200 obtains training data collected at multiple locations during the map construction process. The training data includes at least one image captured at each location and the corresponding position and posture of the captured image.

[0148] Optionally, the device includes at least one memory 220, and stores part or all of the captured images and their corresponding poses as training data in the memory 220. Alternatively or additionally, the device includes a communication interface 230, and obtains part or all of the training data from an external element (e.g., an external memory, a server, a network, etc.).

[0149] The apparatus 200 generates data samples from the training data. Each data sample is assigned to its corresponding pose by an image selected from the training data. Figures 4 to 9 An example of generating data samples is shown in FIG. 1 and described in more detail below.

[0150] The data samples are combined into a dataset for training a neural network. The neural network is trained to recognize the pose of the localization image from:

[0151] (a) Positioning image;

[0152] (b) one or more additional images as part of the training data; and

[0153] (c) The corresponding pose of each of the (one or more) additional images.

[0154] Reference now Figure 3 , Figure 3 A simplified flowchart of a method for generating a neural network for pose estimation in a system provided by an embodiment of the present invention.

[0155] At 310 , training data is obtained.

[0156] At least one data sample is generated for each of the images at 320. The data sample of an image is an assignment of the image and at least one other image selected from the training data to corresponding poses.

[0157] At 330, a data set for training a neural network is formed from the plurality of data samples.

[0158] At 340, a neural network is trained using the dataset to estimate the pose of the localization image from:

[0159] (a) Positioning image;

[0160] (b) one or more additional images from the training data; and

[0161] (c) The corresponding pose of each of the (one or more) additional images.

[0162] C. Generate data samples

[0163] In some embodiments of the present invention, the data sample is generated by selecting at least one reference image from the training data. At least one neighboring image is also selected from the training data.

[0164] The neighboring image has a corresponding pose that is close to the pose of at least one reference image. In some embodiments, the neighboring image is an image taken at a distance very close to the location where the reference image was taken. The neighboring nodes are not necessarily continuous within the map construction sequence, nor are they necessarily the nodes closest to the reference node. Optionally, the neighboring node is within a specified distance from one or more reference nodes.

[0165] Data samples are created by assigning a corresponding pose to each neighboring image relative to each reference image.

[0166] Optionally, the data sample also includes semantic segmentation (SS) information of at least one of its images. The semantic segmentation information classifies pixels according to the type of object depicted in the image (e.g., pixels of cars and pixels of vegetation, buildings, sky, roads, etc.).

[0167] Optionally, the data sample further includes depth map information of at least one image thereof. A depth map (DM) provides corresponding depth information for scene points in an image.

[0168] The data samples that make up the dataset used for training do not necessarily have the same structure or type. Possible differences between data samples include, but are not limited to:

[0169] (1) The total number of images;

[0170] (2) the number of reference images;

[0171] (3) the number of neighboring images;

[0172] (4) Whether the data sample includes semantic segmentation and / or depth map information; and

[0173] (5) Whether the map construction pose introduces errors (to simulate incorrect map construction).

[0174] One advantage of creating a training data set according to an embodiment of the present invention is that no manual labeling is required, which enables the training data set to be generated completely automatically.

[0175] In the non-limiting embodiments described below, pose nodes of a reference image are referred to as reference nodes, and pose nodes of neighboring images are referred to as neighboring nodes.

[0176] C.1 Data samples of a single reference node and multiple neighboring nodes

[0177] Optionally, at least one data sample has a single reference node and multiple neighboring nodes. Figure 4 and Figure 6 In the non-limiting example shown, the reference node is shown as part of a first map building loop, and the neighboring node is shown as part of a second map building loop. It should be noted that the reference node and the neighboring node are not necessarily selected from different map building loops, but may be selected in any order or combination. Further, a map building sequence is not necessarily defined as two map building loops.

[0178] Reference now Figure 4 , Figure 4 The data sample provided by the embodiment of the present invention is shown as a data sample having a single reference node M and neighboring nodes (G1, G2, ...). The reference node M is selected as the origin coordinate system.

[0179] In the first example, the data sample is assigned:

[0180] (I(M), I(G1), I(G2),…)→(C1, C2,…),

[0181] Among them, I(X) represents the image at the pose node X, and C1, C2, ... are the relative poses of the image represented by node M.

[0182] In a second example, semantic segmentation data and / or depth map construction data are available for one or more nodes, and the data samples are:

[0183] (I(M),I(G1),I(G2),…,[SS(M),SS(G1),…,DM(M)])→(C1,C2,…),

[0184] Among them, SS(X) represents the semantic segmentation information of the image at pose node X, and DM(X) represents the depth map information of the image at pose node X.

[0185] Reference now Figure 5 , Figure 5 A simplified flow chart of a method for creating a data sample with a single reference node and multiple neighboring nodes is provided for an embodiment of the present invention.

[0186] In 510, a reference node M is selected.

[0187] In 520, neighboring nodes G1, G2, ... are selected.

[0188] In 530, data samples are constructed for M, G1, G2, ... by assigning corresponding poses. Figure 5 In the example of , the data sample does not include SS and DM information. Other types of data samples may be constructed for the same image with SS and / or DM information and / or additional information as described below.

[0189] C.2 Data samples with multiple reference nodes and multiple neighboring nodes

[0190] Optionally, at least one data sample has multiple reference nodes (denoted as M k+n ) and multiple neighboring nodes (denoted as G j )) For each neighboring image, a corresponding pose relative to each of the reference images is specified. By including the additional information provided by multiple reference nodes, the available information can be better utilized when training the neural network.

[0191] Optionally, all reference nodes M k+n The pose information of the neighboring nodes is included in the data sample as information. This helps to provide a wider coverage of the scene information provided by the neighboring nodes.

[0192] Reference now Figure 6 , Figure 6 The present invention provides a method for creating a plurality of reference nodes M. k+n and multiple neighboring nodes G j The data sample.

[0193] In the first example, the data sample is assigned:

[0194] (I(M k+n ), I(G j ), M′ k+n )→(C k,j ),

[0195] Among them, I(M k+n ) represents the reference node X k ...X k+n The image at, and I(G j ) represents the neighboring node G j The image at M′ k+n is the reference node relative to the coordinate system chosen as the origin (in this example, M k )’s reference node pose, where:

[0196] M′ k+n =M k -1 M k+n ,

[0197] C k,j is the pose of each neighboring node relative to each reference node expressed in the origin coordinate system.

[0198] In a second example, semantic segmentation data and / or depth map construction data are available for one or more nodes, and the data samples are:

[0199] (I(M k+n ), I(G j ), M′ k+n , ..., [SS(M k+n ), SS(G j ), ..., DM (M k+n )])→(C k,j ).

[0200] Reference now Figure 7 , Figure 7 A simplified flow chart of a method for creating a data sample having multiple reference nodes and multiple neighboring nodes is provided for an embodiment of the present invention.

[0201] In 710, a plurality of reference nodes M are selected. k+n , and select one of the reference nodes M k As the origin of the coordinate system.

[0202] In 720, select the neighboring node G j .

[0203] In 730, the pose of the reference node is converted to M k Coordinate system.

[0204] In 740, by assigning the corresponding pose C k,j M k+n and G j Construct a data sample. Figure 7 In the example, the data sample includes SS and DM information, and the data sample is (I(M k+n ), I(G j ), M′ k+n , ..., [SS(M k+n ), SS(G j ), ..., DM (M k+n )])→(C k,j ). Other data samples may be constructed for the same image without SS and / or DM information.

[0205] C.3 Confidence Value

[0206] Optionally, pose estimation errors are tracked while training the neural network. These errors are used to calculate a confidence value w for the pose estimate output by the neural network. Further optionally, the confidence value is used as an additional input to training the neural network.

[0207] In the first example, data samples are generated for a single reference node and multiple neighboring nodes without SS or DM information (eg Figure 2 and Figure 3 The data sample can be expressed as:

[0208] (I(M), I(G1), I(G2),…) → (C1, C2,…, w1, w2…),

[0209] Among them, w x is the confidence value of the pose CX.

[0210] Reference now Figure 8 , Figure 8 A simplified flow chart of a method for creating data samples with confidence values ​​is provided for an exemplary embodiment of the present invention. Figure 8 In an embodiment of the present invention, data samples are used for multiple reference nodes and multiple neighboring nodes having SS and DM information.

[0211] In 810, multiple reference nodes M are selected. k+n , and select one of the reference nodes M k As the origin of the coordinate system.

[0212] In 820, select the neighboring node G j .

[0213] In 830, the pose of the reference node is converted to M k Coordinate system.

[0214] In 840, by assigning the corresponding pose C k,j M k+n and G j Construct a data sample. Figure 8 In the example, the data sample includes SS and DM information, and the data sample is (I(M k+n ), I(G j ), M′ k+n , ..., [SS(M k+n ), SS(G j ), ..., DM (M k+n )])→(C k,j ). Other data samples may be constructed for the same image without SS and / or DM information.

[0215] C.4 Simulating incorrect map construction

[0216] Optionally, one or more errors are introduced into the training data by changing at least one pose in at least one data sample. For example, smaller angles and / or positions may be added to the correct pose parameters. Adding (one or more) errors to the pose simulates inaccurate map construction, so that the neural network can be trained to maintain high accuracy even when the map construction used during localization is inaccurate. It should be noted that the accurate pose C may be used during training. kj , so that the neural network can be trained to correctly estimate the pose.

[0217] Reference now Fig. 9 , Fig. 9 A simplified flow chart of a method for introducing errors into data samples is provided for an exemplary embodiment of the present invention. Fig. 9 In an embodiment of the present invention, data samples are used for multiple reference nodes and multiple neighboring nodes having SS and DM information.

[0218] In 910, multiple reference nodes M are selected. k+n , and select one of the reference nodes M k As the origin of the coordinate system.

[0219] In 920, select the neighboring node G j .

[0220] In 930, the reference node's pose is converted to M using the error ε k Coordinate system, where

[0221] In 940, by assigning the corresponding pose C k,j M″ k+n and G j Construct a data sample. Fig. 9 In the example, the data sample includes SS and DM information, and the data sample is (I(M k+n ), I(G j ), M′ k+n , ..., [SS(M k+n ), SS(G j ), ..., DM (M k+n )])→(C k,j ). Other data samples may be constructed for the same image without SS and / or DM information and / or confidence values.

[0222] In the additional example, SS and DM information is not available, but confidence values ​​are available, and the data sample is:

[0223] (I(M k+n ), I(Gj ), M″ k+n )→(C k,j , w k,j ).

[0224] D. Use trained neural network for positioning

[0225] Reference now Fig.10 , Fig.10 A simplified block diagram of an apparatus for image-based localization of a system in a region of interest provided for an embodiment of the present invention. Localization is performed using a neural network that has been trained to perform pose estimation on localization images using stored images taken during map construction (referred to as map images) and their corresponding poses.

[0226] Optionally, the trained neural network is one of the following:

[0227] (1) Deep neural network (DNN);

[0228] (2) Convolutional neural network (CNN);

[0229] (3) recurrent neural network (RNN); and

[0230] (4) Long short-term memory (LSTM) neural network.

[0231] The apparatus 1000 includes a processor circuit 1010 and a memory 1020. Optionally, the apparatus 1000 further includes one or more of the following:

[0232] (a) image capture device 1030; and

[0233] (b) Communication interface 1040.

[0234] The processor circuit 1010 performs processing operations for performing positioning.

[0235] The memory 1020 stores images captured at multiple locations in the region of interest during map construction and the corresponding poses of the stored images. As described above, the pose of the image is the position and orientation of the image capture device that captured the stored image during image capture.

[0236] The apparatus 1000 acquires a positioning image captured by an image capturing device. Optionally, the apparatus 1000 includes an image capturing device 1030 for capturing the positioning image. Alternatively, the positioning image is acquired from an external image capturing device.

[0237] To perform localization, a subset of the stored map images is retrieved from memory 1020. The information is provided to the trained neural network to obtain a pose estimate for the localization image. The information includes:

[0238] (a) Positioning image;

[0239] (b) retrieved images; and

[0240] (c) The corresponding poses of the retrieved images.

[0241] Based on this input, the neural network outputs at least one pose estimate of the localization image (referred to as a localization pose estimate).

[0242] Optionally, further input to the trained neural network includes: semantic segmentation information calculated for at least one of the images and / or a depth map calculated for at least one of the images.

[0243] Optionally, a subset of the stored images retrieved from memory 1020 is selected based on the located proximate location of the system. Further optionally, the subset includes some or all of the stored images taken within a certain distance from the approximate location of the system.

[0244] For example, a simple GPS device providing a low accuracy estimate of position may be used and / or a nearby location of a system that determines positioning through location recognition and / or based on previous positioning pose estimates.

[0245] Optionally, one or more of the retrieved images are selected as reference images.

[0246] The output of the neural network is at least one positioning pose estimate, wherein the at least one positioning pose estimate is expressed relative to a reference node coordinate system and / or relative to a map coordinate system.

[0247] Optionally, the neural network outputs a single localization pose estimate, wherein the single localization pose estimate is represented by a pose relative to a single reference image.

[0248] Alternatively or additionally, the neural network outputs a plurality of localization pose estimates. Each localization pose estimate is represented by a pose relative to a different reference image. A final localization pose estimate is calculated from the plurality of pose estimates.

[0249] Optionally, the neural network is trained to jointly estimate the pose of multiple localization images, such as consecutive localization images. Since the neural network can be trained to ensure that vehicle dynamics constraints are met (e.g., a maximum vehicle speed is not exceeded in consecutive pose estimates), the joint estimation of multiple localization poses reduces the error.

[0250] Reference now Fig.11 , Fig.11 A simplified flow chart of a method for image-based positioning of a system in a region of interest is provided for an embodiment of the present invention.

[0251] At 1110 , a positioning image is acquired from an image capture device.

[0252] At 1120, input data is provided to the trained neural network. The input data includes:

[0253] (a) Positioning image;

[0254] (b) a subset of images selected from the stored images; and

[0255] (c) The corresponding poses of a subset of images.

[0256] At 1130, at least one pose estimate is obtained from the trained neural network.

[0257] D.1 Single reference node

[0258] Reference now Fig.12 , Fig.12 The exemplary embodiment of the present invention provides a method for determining a single reference node M. k Positioning pose L j Pose estimation of .

[0259] In order to estimate the positioning pose L j , retrieve multiple map construction pose nodes M from the memory k+n In this example, select one of the pose nodes M k As a reference node. Relative to M k The coordinate system represents all poses, thus restricting the pose estimates to a limited range.

[0260] Optionally, reference node M k is the pose node closest to the neighboring position of the system. Further optionally, by calculating M relative to the positioning image k+n By selecting the relative pose of n with the smallest distance from the positioning pose, the pose node closest to the positioning pose can be found.

[0261] Optionally, other pose nodes M k-1 、M k+1 and M k+2 Close to M k , for example, in the case of k Within a specified distance.

[0262] Optionally, jointly estimate multiple positioning poses (..., L j-3 , Lj-2 , L j-1 , L j ……), where C k,j is relative to the reference node M k The positioning image I(L j )’s pose estimation.

[0263] Reference now Fig.13 , Fig.13 Shows Fig.12 An exemplary embodiment provides the input and output of a trained neural network. The input of the neural network DNN includes:

[0264] (a) Pose node image I(M k+n );

[0265] (b) Relative to M k , M' k+n The corresponding pose of the pose node image;

[0266] (c) Positioning image I(L j ) (or a localization image if multiple localization poses are jointly estimated); and

[0267] (d) Optionally – semantic segmentation and / or depth map information.

[0268] The output of the neural network DNN is M k Positioning pose estimation of the coordinate system C k,j , which is converted into the map relative pose L j .

[0269] D.2 Multiple reference nodes

[0270] Reference now Fig.14 , Fig.14 The pose estimation of multiple reference nodes provided by an exemplary embodiment of the present invention is shown. Using multiple reference nodes can reduce the estimation error by providing more information to the neural network.

[0271] In order to estimate the positioning pose L j , select multiple reference nodes M k+m The corresponding image and pose of the reference node M are retrieved from the memory k+m The neural network calculates multiple pose estimates for each localization image, where each pose estimate is relative to a different reference node. Multiple pose estimates are combined to form the final pose estimate L j For example, L j It is calculated as the weighted average of multiple pose estimates, where the weight is the confidence value w k,j .

[0272] Optionally, multiple positioning poses are estimated jointly, ..., L j-3 , L j-2 , L j-1 , L j ..., where C k,j is relative to the reference node M k The positioning image I(L j )’s pose estimation.

[0273] Reference now Fig.15 , Fig.15 Shows Fig.14 An exemplary embodiment provides the input and output of a trained neural network. The input of the neural network DNN includes:

[0274] (a) Pose node M k+n and I(M k+n )

[0275] (b) Relative to multiple reference nodes M' k+n+m The pose node M k+n 's posture;

[0276] (c) Positioning image I(L j ) (or a localization image if multiple localization poses are jointly estimated); and

[0277] (d) Optionally – semantic segmentation and / or depth map information.

[0278] The output of the neural network DNN is M k+m Each positioning image I(L j ) of multiple pose estimates (C k+i,j ). Calculate the weighted average of multiple pose estimates to obtain a single map relative positioning pose estimate: L j =i m ω k+m,j M k+m C k+m,j , where ∑ m ω k+m,j =1.

[0279] D.3 Estimation accuracy measures

[0280] Optionally, an estimation accuracy metric is provided for the pose estimation. The estimation accuracy metric is based on comparing the difference between the ground truth pose estimates obtained for the same localization image of different reference nodes with the difference between the actual ground truth poses of these reference nodes (known from the map).

[0281] Reference now Fig.16 , Fig.16 It shows the single positioning pose Lj How to obtain different estimates. C k,j Is to M k When used as a reference node, the positioning image I(L j ) is the pose estimation, and C k+1,j Is to M k+1 When used as a reference node, the positioning image I(L j )’s pose estimation.

[0282] Reference now Fig.17 , Fig.17 A simplified flowchart of a method for determining an estimation accuracy metric for pose estimation provided by an embodiment of the present invention.

[0283] In 1710, node M k and M k+1 The change in the ground truth pose of is calculated as:

[0284] In 1720, L j The change in the ground truth pose estimate is calculated as:

[0285] In 1730, the difference is calculated as: E = E = |S k,k+1 -S′ k,k+1 |.

[0286] In 1740, a confidence value V is calculated from the difference E, where V is a measure of the accuracy of the pose estimation. The above method is used to manufacture an integrated circuit chip.

[0287] The flow chart and block diagram in the figure illustrate the structure, function and operation of the possible implementation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each block in the flow chart or block diagram can represent a module, section or part of the code, which includes one or more executable instructions for implementing a specified (one or more) logical function. It should also be noted that in some alternative embodiments, the functions indicated in the block may not be implemented in the order in the figure. For example, in fact, the two blocks shown in succession can be executed almost simultaneously, or sometimes, the blocks can be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart and the combination of boxes in the block diagram and / or flow chart can be performed by a system based on special hardware, which performs a specific function or action, or performs a combination of special hardware and computer instructions.

[0288] The description of various embodiments of the present invention is for illustrative purposes only, and these descriptions are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the present embodiments, the practical application or technical improvement of the technology available on the market, or to enable other persons skilled in the art to understand the embodiments disclosed herein.

[0289] It is expected that during the validity period of this application's mature patent, many related neural networks, neural network training techniques, image types, image capture devices, pose sensors, and memories will be developed, and the terms neural network, trained neural network, image, image capture device, pose sensor, and memory are intended to a priori include all of these new technologies.

[0290] The terms "include", "comprising", "having" and variations thereof mean "including but not limited to". This term encompasses the terms "consisting of" and "consisting essentially of.

[0291] The phrase "consisting essentially of means that the composition or method may include additional ingredients and / or steps, but only if the additional ingredients and / or steps do not materially alter the basic and novel characteristics of the claimed composition or method.

[0292] As used herein, the singular forms "a", "an" and "the" include plural references unless the context clearly dictates otherwise. For example, the term "a compound" or "at least one compound" may include a plurality of compounds, including mixtures thereof.

[0293] The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any “exemplary” embodiment is not necessarily to be construed as preferred or advantageous over other embodiments, and / or does not exclude the incorporation of features of other embodiments.

[0294] The word “optionally” is used herein to mean “provided in some embodiments and not provided in other embodiments.” Any particular embodiment of the present invention may include a number of “optional” features, unless such features are mutually incompatible.

[0295] Throughout this application, various embodiments of the present invention can be presented in range format. It should be understood that the description of the range format is only for convenience and brevity, and should not be interpreted as a fixed limitation on the scope of the present invention. Therefore, the description of the range should be considered to have specifically disclosed all possible sub-ranges and individual numerical values ​​within the range. For example, the description of the range from 1 to 6 should be considered to have specifically disclosed sub-ranges, such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, etc., and individual numbers within the range, such as 1, 2, 3, 4, 5 and 6. Regardless of the breadth of the scope, this applies.

[0296] When a numerical range is indicated herein, it is intended to include any number (fractional or integer) listed within the indicated range. The phrases "within the range of the first indicated number and the second indicated number" and "from the first indicated number to the second indicated number" are used interchangeably herein to include the first and second indicated numbers and all fractions and integers between the two.

[0297] It should be understood that certain features of the present invention described in the context of a separate embodiment for the sake of brevity may also be provided in combination in a single embodiment. Conversely, the various features of the present invention described in the context of a single embodiment for the sake of brevity may also be provided individually or in any suitable sub-combination or as any suitable other embodiment of the present invention. Certain features described in the context of various embodiments are not considered essential features of those embodiments, unless the embodiment described is invalid without these elements.

[0298] Although the present invention is described in conjunction with specific embodiments, it is apparent that those skilled in the art will appreciate many alternatives, modifications and variations. Accordingly, it is intended to cover all such alternatives, modifications and variations that fall within the spirit and broad scope of the appended claims.

[0299] All publications, patents and patent applications mentioned in this specification are hereby incorporated by reference in their entirety, and similarly, each individual publication, patent or patent application is specifically and individually incorporated herein. In addition, the citation or identification of any reference in this application shall not be construed as allowing such reference to take precedence over the present invention in the prior art. To the extent that section headings are used, they should not be construed as necessarily limiting.

Claims

1. A device for generating a model for pose estimation in a system, characterized in that: The apparatus comprises a processor circuit configured to: Acquire training data of multiple positions, the training data comprising at least one image captured by an image capturing device and a corresponding pose of the at least one image captured by the image capturing device, wherein the corresponding pose of the image comprises a corresponding position and orientation of the image capturing device during capturing of the image; generating at least one data sample for each of the captured images from the training data, wherein the data sample for an image comprises assigning the image and at least one other image selected from the training data to respective poses; A neural network is trained using a dataset including the data samples to estimate a corresponding pose of the localization image from: the positioning image; at least one additional image from the training data; and a respective pose of each of the at least one additional image; Generating at least one data sample comprises: selecting at least one reference image from the training data; Selecting at least one neighboring image from the training data having a pose close to the at least one reference image; and The at least one reference image and the at least one neighboring image are assigned a respective pose of each neighboring image relative to each reference image; the respective pose of the at least one neighboring image being within a specified distance from a pose of at least one of the reference images.

2. The device according to claim 1, characterized in that The at least one reference image comprises a single reference image, and the assigning specifies a respective pose of each neighboring image relative to the single reference image.

3. The device according to claim 1, characterized in that The at least one reference image comprises a plurality of reference images, and the assigning specifies a respective pose of each neighboring image relative to each of the reference images.

4. The device according to any one of claims 1 to 3, characterized in that The processor circuit is further configured to introduce an error into the training data by changing at least one pose in the data samples prior to the training.

5. The device according to any one of claims 1 to 3, characterized in that The processor circuit is further configured to track errors in the pose estimates during the training, and to calculate corresponding confidence values ​​for poses estimated by the neural network based on the tracked errors.

6. The device according to any one of claims 1 to 3, characterized in that At least one of the data samples further comprises a semantic segmentation computed for mapping at least one image in the dataset.

7. The device according to any one of claims 1 to 3, characterized in that At least one of the data samples further comprises a depth map computed for mapping at least one image in the data set.

8. The device according to any one of claims 1 to 3, characterized in that The neural network includes a deep neural network (DNN).

9. A method for generating a neural network for pose estimation in a system, characterized in that The method comprises: Acquire training data of multiple positions, the training data comprising at least one image captured by an image capturing device and a corresponding pose of the at least one image captured by the image capturing device, wherein the corresponding pose of the image comprises a corresponding position and orientation of the image capturing device during capturing of the image; generating at least one data sample for each of the captured images from the training data, wherein the data sample for an image comprises assigning the image and at least one other image selected from the training data to respective poses; A neural network is trained using a dataset including the data samples to estimate a corresponding pose of the localization image from: the positioning image; at least one additional image from the training data; and a respective pose of each of the at least one additional image; Generating at least one data sample comprises: selecting at least one reference image from the training data; Selecting at least one neighboring image from the training data having a pose close to the at least one reference image; and The at least one reference image and the at least one neighboring image are assigned a respective pose of each neighboring image relative to each reference image; the respective pose of the at least one neighboring image being within a specified distance from a pose of at least one of the reference images.

10. The method according to claim 9, characterized in that For at least one of the data samples, the allocating comprises: selecting a single reference image from the training data; selecting at least one neighboring image from the selected training data based on proximity to the reference image; and A respective pose of the at least one neighboring image relative to the single reference image is specified.

11. The method according to claim 9 or 10, characterized in that: For at least one of the data samples, the allocating comprises: selecting a plurality of reference images from the training data; selecting at least one neighboring image from the training data based on proximity to the reference image; and Specify the corresponding pose of each neighboring image relative to each reference image.

12. The method according to claim 9 or 10, characterized in that: The method also includes introducing an error into the training data by changing at least one pose in the data samples before the training.

13. The method according to claim 9 or 10, characterized in that: Also included is tracking errors in the pose estimates during the training, and calculating corresponding confidence values ​​for poses estimated by the neural network based on the tracked errors.

14. An apparatus for image-based positioning of a system in a region of interest, characterized in that include: a memory for storing images captured by an image capturing device of the system at a plurality of locations in a region of interest and a corresponding pose of the image capturing device associated with the stored image at each of the locations, wherein the pose of the image comprises a position and an orientation of the capturing device associated with the corresponding image; and A processor circuit for: Acquire a positioning image captured by an image capturing device; retrieving a subset of the stored images and corresponding poses from the memory; Provide the trained neural network with: the positioning image; the retrieved image; and the corresponding pose of the retrieved image; and At least one estimated pose of the positioning image is obtained from the trained neural network, wherein the trained neural network is trained by the method according to any one of claims 9 to 13.

15. The device according to claim 14, characterized in that The processor circuit is further configured to: determining a proximate position of the image capturing device during capturing of the locator image; and A subset of the stored images is selected based on the neighboring locations.

16. The device according to claim 14 or 15, characterized in that The neural network outputs a single estimated pose relative to the pose of a single reference image selected from the retrieved images.

17. The device according to claim 14 or 15, characterized in that The neural network outputs a plurality of estimated poses, each of the estimated poses being related to a pose of a different reference image selected from the retrieved images, and wherein the processor circuit is further configured to calculate a final estimated pose from the plurality of estimated poses.

18. The device according to claim 14 or 15, characterized in that The processor circuit is further configured to calculate an estimation accuracy metric by comparing a pose change between a plurality of estimated poses of the positioning image with at least one pose change between reference images used to calculate the plurality of estimated poses.

19. The device according to claim 14 or 15, characterized in that The trained neural network includes a deep neural network (DNN).

20. The device according to claim 14 or 15, characterized in that The trained neural network is also provided with a semantic segmentation calculated for at least one of the images provided to the trained neural network.

21. The device according to claim 14 or 15, characterized in that The trained neural network is also provided with a depth map calculated for at least one of the images provided to the trained neural network.

22. A method for image-based positioning of a system in a region of interest, characterized in that include: Acquire a positioning image captured by a camera of the system; Provide the trained neural network with: the positioning image; a subset of images selected from a plurality of images taken during construction of a map of the area of ​​interest; and corresponding poses associated with a subset of the images, wherein the pose of an image comprises a position and an orientation associated with an image capturing device when the image was captured; as well as At least one estimated pose of the positioning image is obtained from the trained neural network, wherein the trained neural network is trained by the method according to any one of claims 9 to 13.

23. The method according to claim 22, characterized in that The subset of images is selected based on proximity to the estimated location of the localization image.

24. The method according to claim 22 or 23, characterized in that The neural network outputs a single estimated pose relative to the pose of a single reference image selected from the subset of images.

25. The method according to claim 24, characterized in that The method further includes obtaining a proximate location of the image capturing device from a satellite navigation system, wherein the reference image is selected based on the proximate location.

26. The method according to claim 22 or 23, characterized in that The neural network outputs a plurality of estimated poses, each of the estimated poses being relative to a pose of a different reference image selected from the subset of images, and also includes calculating a final estimated pose from the plurality of estimated poses.

27. The method according to claim 22 or 23, characterized in that The trained neural network includes a deep neural network (DNN).

Images