Scale-aware monocular localization and mapping

Abstract

An image processing device is disclosed, comprising a processor for estimating the scale of an image feature by: processing multiple images of a scene by a first training model to identify features in the image and estimating the depth of the features in the image; processing the multiple images by a second training model to estimate the scale of the image; and estimating the scale of the feature by adjusting the estimated depth according to the estimated scale. A method for training an image processing model is also disclosed.

Description

Technical Field

[0001] The present invention is concerned with resolving the problem of scale ambiguity when projecting images captured by a digital camera. Background Art

[0002] A digital camera captures an image through light falling on a digital sensor, which converts the collected light into digital data, or raw data. This raw data is then processed to create a raster image formed as a field of pixels, where each pixel represents the brightness and color received by an area of ​​the digital sensor, representing a region of the captured image. This pixel field can then be displayed on a screen for viewing, thus achieving projection of the captured image. Both raw and processed data can be stored and transferred to another device, allowing for further reproduction of the captured image as needed and as required.

[0003] If the image is a scene (which could be an outdoor scene or an indoor scene), it typically includes many objects or features. In some cases, it may be desirable to be able to identify the objects that form part of the scene. For example, if an object such as a restaurant is part of the scene, it would be useful to identify the restaurant so that it can be found and its name, location, etc., provided to a device that is searching for restaurants in the area where the identified restaurant is located. Alternatively or additionally, identifying the object can facilitate adjusting and / or enhancing the appearance of the object when displayed. In the case of a single camera capturing an image, it is difficult to know the distance of the object from the camera. This is typically at least somewhat obvious to a person viewing the scene in real life. For example, if the object is a car and it appears to be approximately one meter tall (approximately three feet), a person who knows that a typical car is approximately 1.3 meters (approximately four feet) tall would infer that the car must be approximately 10 meters (30 feet) away. This is because the person would rule out the possibility that it is the height of a house (which would be further away) and would also rule out the possibility that it is the height of a dog (which would be closer). However, digital image processors do not possess human-level knowledge of what an object is and its approximate possible size, and therefore cannot infer how far away it is in this way. In other words, the scale of an object is ambiguous because its distance from the viewpoint is unknown. Furthermore, the appearance of an image projected by a digital camera also depends on the distance of the sensor plane from the camera's optical center—different cameras in the same location may capture scenes of different sizes. Therefore, one aspect of processing raw image data involves accounting for the ambiguity in the scale of objects in the captured image, thereby being able to determine their depth (i.e., their distance from the camera) so that the image can be accurately projected and enhanced.

[0004] Images captured by a single camera are called monocular vision. One way to determine the depth of objects forming part of an image is to move the camera while capturing the image. Therefore, a video sequence is captured over a period of time while the camera is moving, and the position of each frame in the captured sequence depends on the position the camera was moving to at the moment the frame was captured. Processing such video sequences captured by a moving camera enables simultaneous localization and mapping (SLAM) up to a scale factor. This is because the principle of disparity can be exploited, whereby the same object appears to have different sizes depending on its distance from the camera. Therefore, if an image is acquired from two (or more) different spatial locations, points seen at different pixel locations in the two images can be triangulated. The relationship between the two images is described by their epipolar geometry, which can be used as part of the digital processing to create a pixel field. The difference in pixel position of a point in the scene is called disparity, which is inversely proportional to the distance (i.e., depth) of that point from the camera. If depth is encoded in an image, it is called a depth map. An image of equidistant depth is called a disparity map. The camera position can be localized while simultaneously mapping its surroundings, and can be updated using direct image input or feature keypoints for relocalization.

[0005] Monocular SLAM techniques often result in scale ambiguity, even after the triangulation processing described above. Existing digital image processors use algorithms to process the captured images. The processing part for determining the scale can be roughly divided into statistical-based algorithmic methods and learning-based algorithmic methods. The disadvantage of statistical-based methods is that they require a lot of processing power, which is disadvantageous in terms of the time spent and / or the size of the physical components. Therefore, neural networks have recently been used to create algorithms. Existing methods solve the scale ambiguity problem in three different ways. These methods use monocular depth estimation (1) or additional sensors (2) to estimate the depth, or keep the scale ambiguity (3). Each method is described in more detail below:

[0006] (1) Methods using monocular depth estimation solve the problem of height underconstraints and learn how to infer distances based on visual cues (e.g. a car is typically X meters long…) and rely solely on what they saw when they were trained. This is ill-posed, so the results are inaccurate and cannot be generalized to previously unseen scenes.

[0007] (2) Additional sensors, such as a second camera, an active depth sensor, or an inertial measurement unit (IMU), enable the system to directly recover scale-aware measurements, but this adds additional hardware and requires fusing information from different modalities. Furthermore, there is a calibration cost when determining the spatial displacement between the first camera and the additional sensors.

[0008] (3) Scale fuzzy methods are those that do not restore scale. The scaling parameter "s" is initially set randomly (usually to 1) and all subsequent measurements are referenced to this value. Measurements will only be in units of "s" multiplied by the measurement unit. In these scenarios, any virtual content cannot be placed in a metrically correct manner.

[0009] CN 107945265 A is an example of method (1) and describes a SLAM method based on an online learning depth prediction network, which involves the prediction of a full depth map and a very specific voting scheme. The method relies on knowledge about objects, which means that it can easily fail when faced with unknown objects or known objects that do not fit previously seen objects of that type. For example, if the method is familiar with trees such as apple trees, it knows that they are about 3 meters (10 feet) tall, but it will have difficulty identifying a redwood that may be about 100 meters (350 feet) tall as a tree. CN 106127739 A is an example of method (2) and describes a SLAM method that relies on monocular depth estimation, requiring a combination of a monocular camera and a depth sensor, as well as additional hardware sensors to estimate scale. CN 105825520 A is an example of method (3) and describes another monocular SLAM method that is able to create large-scale maps but is not scale-aware.

[0010] It is desirable to provide a technique that can provide accurate scale and depth prediction performance, thereby minimizing processing requirements. Summary of the Invention

[0011] According to a first aspect, an image processing device is provided, comprising a processor configured to estimate the scale of image features by: processing multiple images of a scene using a first trained model to identify features in the images and estimate the depth of the features in the images; processing the multiple images using a second trained model to estimate the scale of the images; and estimating the scale of the features by adjusting the estimated depth according to the estimated scale. This allows the depth of the scene and the scale of the features to be inferred without requiring additional hardware or knowledge of the scene features.

[0012] Some examples of features whose scale can be estimated by the device include pixels, distances between pixels, and image artifacts that collectively represent an object or a specific portion of an object. Depth can be the distance from the camera that captured the image. The camera can be the device, can include the device, or can be part of the device.

[0013] The multiple images may include images of a scene captured from different positions. This may allow the use of two images from a monocular mobile camera, or two images from two different cameras.

[0014] The first trained model and the second trained model can be used to operate independently of the distance between the locations. Therefore, the camera speed is not important.

[0015] The multiple images may include images of a scene captured at different times. This also includes images captured from a single mobile camera.

[0016] The multiple images can be frames from a common video stream. This allows for simultaneous localization of a moving camera and feature map construction.

[0017] In some implementations, the image processing device includes a camera, and the plurality of images are images captured by the camera. Thus, the processing can be performed by the camera. In other implementations, some or all of the processing can be performed by another device (e.g., a server or computer in the cloud).

[0018] The estimated depth can be a relative distance. This can be the relative distance between objects. Thus, objects can be correctly reconstructed relative to each other.

[0019] The estimated scale can be an absolute distance. This can be measured in actual distance units (e.g. meters, yards, etc.) from the camera that captured the image. In addition to being able to accurately reconstruct the object as part of the image, this information can also be used to determine what the object is.

[0020] The device can be configured to: run a first trained model on a downsampled image of a scene to form a dataset comprising depth estimates of features in the downsampled image; and process the dataset to upscale its resolution. This allows for a smaller amount of data to be stored and used for depth map calculation, for example in a mobile device with limited storage and / or processing power. However, the upscaling results in a better resolution of the resulting image.

[0021] According to a second aspect, a method for training an image processing model is provided, comprising: (a) receiving a plurality of pairs of images, each pair representing a common scene and the images in each pair being captured from different locations spaced apart by a distance; (b) for each pair of images: (1) receiving ground truth data, the ground truth data comprising a scale estimate of a region in the image; (2) processing the image through a first model to identify features in the image; (3) processing the image through a second model to estimate the relative depth of the features; (4) processing the image through a third model to estimate the absolute scale of the image; and (5) adjusting one or both of the second and third models based on the ground truth data. This can allow the generation of an algorithm that processes image data to resolve scale ambiguity of scene features, which can then be used without the need for an additional depth sensor or ground truth information.

[0022] The method may further comprise, for each pair of images, estimating the scale of a region in the images to form ground truth data by one of: (1) geometrically analyzing the images based on the distance between the locations at which the images were captured, and (2) estimating a transformation between the images based on data indicating the relative locations at which the images were captured and the relative orientations at which the images were captured. This allows the depth of objects estimated using stereo vision but having arbitrary scales to be corrected by an actual scaling factor.

[0023] In some implementations, the method includes, for each pair of images, warping one of the images to a common image frame with the other image in the pair; wherein steps (3) and (4) are performed on the warped image and the other image. This technique can be used to process two images of the same scene, such as those typically captured on the left and right sides of the scene, to scale features of the images.

[0024] In some implementations, step (2) of the method may include processing the images through the first model to identify features in the images and the location of each feature in the corresponding image. In some of these implementations, the method may include: for each pair of images, warping the location of the features identified in one of the images to an image plane that is common to the location of the features identified in the other image of the pair; wherein steps (3) and (4) of the method are performed on the warped feature locations and the feature locations of the other image. Correction to the common image plane reduces the amount of computation required because the search space for correspondences between the images is reduced to a single line.

[0025] According to a third aspect, there is provided an image processing model formed by any of the methods described herein.Such a model can be used to process further images that are not captured by the same camera as that used to generate the model.

[0026] According to a fourth aspect, there is provided an image processing device comprising a processor and a memory, the memory comprising instructions executable by the processor to implement an image processing model adapted by any of the methods described herein. Thus, such a device can be a standalone unit that can both capture images of a scene and process them for reconstruction while taking into account the scale of objects in the scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention will now be described, by way of example only, with reference to the accompanying drawings.

[0028] Figure 1 Illustrate the basic concept of monocular image capture.

[0029] Figure 2 An example of triangulating points using epipolar geometry is shown.

[0030] Figure 3 An overview of the stereo-based depth and scale estimation pipeline for training neural networks is shown.

[0031] Figure 4 An implementation of stereo-based depth and scale estimation is shown.

[0032] Figure 5 The feature tower extraction pipeline is shown.

[0033] Figure 6 (a)-(f) show the stages of using the upsampled disparity map.

[0034] Figure 7 (a)-(f) show the depth estimation refinement process based on scale correction.

[0035] Figure 8 (a)-(e) show the depth estimation output of the trained model.

[0036] Figure 9 Examples of methods for estimating the scale of image features of a scene and for correcting the depth of image features of a scene are shown.

[0037] Figure 10 An example of a method for training a model to estimate the scale of image features of a scene is shown.

[0038] Figure 11 An example of a system architecture including a camera for performing scale estimation is shown. DETAILED DESCRIPTION

[0039] Figure 1is a schematic diagram of monocular vision. An object 1 forms part of an image being captured. In this case, object 1 is a tree and an outdoor scene is being captured. A digital camera 2 has an aperture 4, approximately a pinhole, through which light can enter in order to record an image 6 on a digital sensor 8. The raw data recorded on the digital sensor can be processed so as to reconstruct the image in a format viewable by humans, either directly on the screen of the camera, and / or so that it can be transmitted to a separate device for further processing and / or viewing. As mentioned above, one difficulty with such processing is that the inverse mapping of the raw data back into an image is ambiguous with respect to the depth of an object (such as object 1), and therefore the scale of the object may not be known.

[0040] To resolve scale ambiguity, instead of a single image, if the camera 2 is moved while capturing multiple images as video frames, synchronized positioning of the cameras and accurate reconstruction mapping the captured data back to images can be achieved up to the scale factor. Figure 2 It is shown how two images of a point P of an object can be triangulated using this method. If two images of a point P are obtained from one camera located at two different positions, the point P will be seen in both images, but at different pixel positions. This is the so-called parallax effect. The epipolar geometry of these stereo images can be used to determine the parallax between two pixel positions. If the same technique is applied to other points of the image, a disparity map can be created. The values ​​of such a map will be inversely proportional to the depth of the points on the map, which is their distance from the camera. In Figure 2 In [1], the two capture points are labeled "left camera" and "right camera." These refer to two positions of a moving camera at two different times, although the principle remains the same if the two images were captured by two different cameras. The distance between the two camera positions is the baseline distance, and the shortest distance Z from the baseline to point P can be calculated with reference to the position seen in each image. Its position in each image depends on how far along the baseline the camera position is from the point on the baseline used to measure distance Z.

[0041] The stereo technique described above can be repeated for multiple pairs of images, thus processing a video sequence to enable simultaneous localization and mapping of the cameras. While this is useful for estimating distances between objects, it only works up to a certain scale factor, i.e., an arbitrary scale, since the baseline length is arbitrarily chosen. Another issue is that small errors accumulate over time and / or as the camera moves, causing drift in the scale.

[0042] Methods for training a neural network and running the trained model to estimate the scale of image features are described below. Apparatus and devices on which these methods can be implemented are also described. In some implementations, the methods are performed by components of a digital camera that has monocular vision but is moving to capture a video sequence.

[0043] The described neural network can perform stereo-based depth and scale estimation simultaneously. The method for training the network consists of combining the two. The scale ambiguity problem is a multi-task problem consisting of two tasks: (1) scale-ambiguous depth estimation and (2) scaling parameter estimation. The two tasks are based on the same data and their results are combined so that each task can gain information from the other. Task (1) is to estimate the depth map based on temporal monocular input with scale "s0" (i.e., images at different time instances). Task (2) is to estimate the scaling parameter "st" based on the same data and scale the output of task (1) accordingly. Therefore, the two tasks can be fused in an end-to-end image processing pipeline. At runtime, the depth estimation branch may be removed, thereby resolving the scale ambiguity. If the scale ambiguity is resolved, it means that the relative and / or absolute size of the objects in the scene being captured will be known by using their relative depth in the image or their actual depth (i.e., distance from the camera).

[0044] A pipelined approach to scale estimation will now be described.

[0045] Figure 3 Schematically shown is an overview of a processing pipeline that is used in some implementations as part of digital image processing of video captured by a monocular mobile camera in order to train a neural network to resolve scale ambiguity so that when run, it can compute the correct depth (i.e., actual unit distance from the camera) and scale (i.e., physical size) of features in the captured image. Spatiotemporal inputs (e.g., stereo images captured at different times) are fed into the pipeline. For training purposes, two cameras may be used, designated as left camera and right camera, as they may be typically positioned to the left and right of an image (e.g., a scene being captured). Two pairs of images are shown, it being understood that they are exemplary images of a number of video frames that may be captured by the two cameras. The left camera at time t ref A first image 10 of a scene is captured at time t n+1 The second image 12 is captured. The right camera captures the second image 12 at time t n A first image 14 of the same scene is taken at time t n+1A second image 16 is taken. These images are cropped and resized, and features are extracted from these images to generate feature maps. At this point, the process splits into two branches, which can actually be done simultaneously, although this is not necessary. The first branch (the lower branch in the figure, generally indicated by the reference numeral 18) performs task (1) discussed above (i.e., scale-blurred depth estimation). Therefore, the first branch estimates a scaled depth map. First, the feature map is warped to a common reference view - the output of this stage is H(d). Then, it is passed through a cost volume. Reference will be made to Figure 4 These steps are described in more detail. The output is a depth map at any scale. Therefore, this branch can probabilistically describe the corresponding image area and its depth. The second branch (the upper branch in the figure, generally represented by the reference numeral 20) performs task (2) discussed above (i.e., estimating the scaling parameter). This scaling parameter can then be used to scale the depth map output by the first branch to the correct value. In this way, the distance of the feature from the camera can be determined. These distances can be determined for objects in the image or for pixels in the image.

[0046] Figure 4 Shown is a reference Figure 3 A more detailed implementation of the training process is described in

[15] . Before passing the images of resolution w × h through the neural network, the two views (taken by the left and right cameras, respectively, as described above) are rectified to a common reference plane. Thus, using epipolar geometry, the search space for correspondences between the images is reduced to a horizontal line. In this way, ground truth data is obtained.

[0047] To implement the methods described in this article, Google's TensorFlow 1.4.1 and Python 2.7 libraries were used to create deep learning models based on the neural network processor implementation described in this article. Suitable hardware resources included an Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz running on a machine with 64GB of memory and an NVidia GeForce GTX1080 running on Ubuntu 16.04.5LTS (XenialXerus). Figure 4 The implementation of is marked as eight stages, each of which is described below.

[0048] Stage 1: Feature Extraction

[0049] Images of the left and right views—e.g. Figure 3 10, 12, 14, 16 shown in the image, but in Figure 4Represented as single boxes 22 (left image L) and 24 (right image R) in the figure, they are fed into two corresponding Siamese feature towers with shared weights during training. The left image 22 is fed into the first feature tower 26, and the right image 24 is fed into the second feature tower 28. To achieve shrinkage (downsampling), one convolutional layer of 3 ResNet blocks is used, and another convolutional layer utilizes BatchNorm and Leaky ReLu with parameter α=0.2. The latter can be repeated three times, and then 32-dimensional features are output for each pixel. The size of the feature map is h / n×w / n×f, n=8, where h is the height of the image, w is the width of the image, n is the ratio by which the image is shrunk (for example, by downsampling), and f is the feature depth (i.e., the number of features). This design follows the feature extraction proposed in [Zhang, Yinda et al. ActiveStereoNet, ECCV 2018].

[0050] Temporary turn Figure 5 , shows the implementation of feature towers 26 and 28. The numbers [C, K, S, D] represent the parameters used for the channels, kernel size, stride, and expansion of a specific convolutional layer or ResNetBlock layer. Back Figure 4 The output of stage 1 is a first feature map 30 of features of the left image generated by the first feature tower 26 and a second feature map 32 of features of the right image generated by the second feature tower 28.

[0051] After randomly sampling the scaling parameter s from the interval [0, s_max], the two stages 2 and 4 prepare the two branches (1) and (2) for scale estimation and depth estimation respectively (discussed above). Figure 3 This parameter is generated dynamically and saved for training purposes.

[0052] Stage 2: Feature Distortion

[0053] This stage is part of the second branch 20 that performs task (2) above, namely scale estimation. Figure 4 , the second image feature map 32 of the right view is warped onto the first feature map 30 of the left reference view using a differentiable homography with d = l and focal length l, so that the feature space difference can be calculated. The homography used is given by the following formula: where 1 corresponds to the left reference view during training, i corresponds to the right view, and where K i is the camera matrix (storing intrinsic camera parameters); R i is the rotation, t i is the translation of camera i relative to the reference camera 1; n1 represents the normalized z direction of the reference camera.

[0054] Figure 4 This stage is shown in boxes 34 and 36 in FIG.

[0055] Stage 3: Scale Estimation

[0056] This stage is the next part of the second branch 20 that performs task (2) above, i.e. scale estimation. A left feature map 30 and a right feature map 32 are used. The feature difference is calculated in feature space, where the dimension is h / n×w / n×f (box 38). The encoding stage 40 reduces this dimension to zero. The encoder gradually reduces the resolution to a zero-dimensional output (scalar). At each layer, the feature map resolution is reduced by half and the result is then passed through a fully connected layer connected to a single neuron. This parameter is called s0 (box 42) and represents the scale change relative to the normalized translation vector of the monocular SLAM system.

[0057] Stage 4: Differential Homography Warping

[0058] This stage forms part of the first branch 18 that performs task (1) above, i.e., scale-ambiguous depth estimation. The two feature maps (first left map 30 and second right map 32) are warped to different fronto-parallel planes in the reference frame of the left camera (block 44), i.e., the homography of the left feature map is the 3×3 identity matrix. For the right camera, the second feature map 32 is warped with a different depth hypothesis. The two warps are aggregated to form a single cost capacity in the next stage.

[0059] Stage 5: Statistical estimation of cost capacity

[0060] This stage forms the next part of the first branch 18 that performs task (1) above, i.e. scale-ambiguous depth estimation. At this stage, the left and right feature capacities are fused into a cost capacity 46. To be independent of the number of input views, a variance-based cost metric M is used as proposed in [Yao, Yao et al. MVSNet, ECCV 2018]. With respect to the feature capacities Vi, where V (italicized) is the dimensionality of the feature space (32 in this example) and N = 2 (the number of frames used), the cost capacity C is and the average cost capacity V i (dashed line) related.

[0061] Phase 6: Guide Refinement

[0062] This stage forms the next part of the first branch 18 that performs task (1) above, i.e., scale-ambiguous depth estimation. In order to maintain a low memory footprint and to be able to run on mobile devices (which impose limitations on processing power due to the desire to keep the size as small as possible), the resolution chosen at this stage is 1 / 8. However, in order to maintain the full resolution disparity map, an upsampling stage may be used in some implementations. One suitable upsampling procedure is the guided layered upsampling proposed in [Khamis, Sameh et al. StereoNet, ECCV 2018]. The selected depth is formed as a softmax-weighted linear combination of the depth values ​​of the cost capacity as follows:

[0063]

[0064] Where D is the maximum discretization in the depth direction.

[0065] This produces a 1 / 8 depth map, which is then passed through a refinement network to refine its residual after bilinear upsampling. The downscaled input image and the skip connections of the depth map of a specific resolution are processed through a small network consisting of convolution, BatchNorm, Leaky ReLu and two ResNet blocks. These skip connections are then cascaded to the processed upsampled disparity at a given stage, where the stages are hierarchically ordered as 1 / 8, 1 / 4, 1 / 2, 1 / 1 (i.e. full resolution). These processes are performed in Figure 4 denoted collectively by reference numeral 48. This passes four residual blocks and one convolution. At each individual scale, the supervision using the scaled ground truth is summarized by a loss.

[0066] Steering Figure 6 , shows the results of different upsampling stages from a coarse disparity map to full resolution. The brightness of the depth map color-codes the distance of the pixel from the camera. While at 1 / 8 resolution (Figure (a)) there is only limited detail, the layered upsampling refines the residual with the help of jump connections to the input image. Figure (b) shows 1 / 4 resolution and Figure (c) shows 1 / 2 resolution. Finally, at full resolution (Figure (d)), the granularity of the ground truth (Figure (e)) is recovered. This is particularly prominent in fine structures such as the leaves of the trees and the thin lampposts on the roadside. The original image (f) is shown for comparison.

[0067] Stage 7: Scaling and upsampling the depth map

[0068] As discussed above, the upsampled depth map of the lower branch 18 ( Figure 4 Stages 4-6 in [1]; Task (1) Depth Estimation) is incorrectly scaled by the wrong scaling parameter s. Stage 3 ( Figure 4The output of stages 2 and 3 in the

[15] task (2) scale estimation) can be used to scale the depth value to actual measurement units (e.g., metric units such as meters or non-metric units such as yards) so that it can be fully supervised by the ground truth depth map. The resulting depth estimate 50 is given by: s0*D_s, where scale s0 is the output 42 of the upper branch of the network ( Figure 4 Stages 2 and 3 in (2) scale estimation), D_s is the lower branch ( Figure 4 Stages 4-6 in [1] - Task (1) Depth Estimation) of incorrectly scaled depth maps.

[0069] Figure 7 This stage is shown, showing the estimated depth scaling for two different real scene input images from the SceneFlow dataset [Cordts et al. CVPR 2016]. For comparison, the input image is shown as image (a) and the ground truth depth is shown as image (b). The black areas in the ground truth image (b) are from invalid pixels. Images (c) and (e) show example outputs of scale estimates that are scaled incorrectly. The depth estimation network at this stage 7 can correct the incorrectly scaled depth (images (c) and (e)) into the final scale estimate (images (d) and (f) respectively).

[0070] Stage 8: Supervision

[0071] The entire pipeline is end-to-end trainable since all steps are fully differentiable. If the ground truth depth values ​​are given (via Figure 4 In some preferred implementations, in a (self-)supervised manner, a stereo method (e.g., stages 4-6 described above, but with known absolute camera displacements) can be used to estimate a depth map, which can be used to dynamically train the network. Once the entire pipeline is trained, the depth estimation output can be similar to Figure 8 An example is shown in . In this example, the brightness of the depth map color-codes the distance of the pixel from the camera. Images (a), (b), and (c) show the 1 / 8, 1 / 4, and 1 / 2 resolution stages, respectively. Image (d) shows the warped monocular depth at 1 / 1 resolution, and image (e) shows the original left image 22. Images (a)-(d) correspond to the guided refinement upscaling at stage 6. Therefore, supervision is performed at every scale.

[0072] The implemented loss function uses the generalized adaptive robust loss function of [Barron, Jonathan, A General and Adaptive Robust Loss Function, arXiv, 2019]

[0073]

[0074] Among them, α=1, c=2.

[0075] During inference, the classic SLAM pipeline can now be used to reconstruct and map the environment, while the network's fast real-time upper branch dynamically estimates scale. In this way, measurement units (such as metrically correct map construction and trajectory) are reconstructed, creating a fully trained model.

[0076] Figure 9 A method for estimating the scale of features in source images captured by a moving monocular camera is summarized. This method is performed using a neural network model trained in stages 1-8 described above. Thus, training can be performed using images captured from two cameras, but in this runtime method, a single moving monocular camera can be used because, as described above, dynamic depth correction can be performed. This can be a different camera from any of the cameras used during training. In step 901, multiple images of a captured scene are processed by a first trained model to identify features in the images and estimate the depths of these features in the images. These estimated depths can be relative distances because only an assumed scale factor (incorrect scaling parameter s) can be used, as discussed above in stage 7. This first trained model is performing the first task of scale-ambiguous depth estimation described above. In step 902, a second trained model processes multiple images to estimate the scale of the images. This second trained model is performing the second task of scale estimation described above. In the next step 903, the scale of the features is estimated by adjusting the estimated depths of step 901 according to the estimated scale of step 902. Thus, absolute depths in distances (e.g., meters, yards, etc.) can be determined.

[0077] Can be used with Figure 9 An optional step used in conjunction with the method is to use the first trained model on a downscaled image of the scene to form a set of data that includes estimates of the depth of features in the downscaled image. This data can then be processed to upsample its resolution, as discussed above with reference to stage 6.

[0078] Figure 10A method for training a model is shown, which generally corresponds to stages 1-8 discussed above. In step 1001, a plurality of pairs of images are received, each pair of images representing a common scene, the images in the pair being captured from different locations separated by a certain distance. The two images can be captured by different cameras, or by the same camera at two different locations. The remaining steps are performed for each pair of images. In step 1002, ground truth data is received comprising a scale estimate of a region in the image. As described above, this can be calculated using epipolar geometry applied to the principle of parallax, since the two images are acquired at different locations. In step 1003, the images are processed by a first model to identify features in the image. This can be done by referring to the above Figure 3 The feature tower pipeline described in step 1004 is performed. In step 1004, the image is processed by the second model to estimate the relative depth of the features. This can be done with reference to stages 4-6 discussed above. Therefore, the second model here is performing the first task of scale blurred depth estimation described above. In step 1005, the image is processed by the third model to estimate the absolute scale of the image. This can be done with reference to stage 3 discussed above. Therefore, the third model here is performing the second task of scale estimation described above. In step 1006, one or both of the second and third models are adjusted based on the ground truth data.

[0079] Figure 11 An example of an architecture including a camera that uses a model to perform scale estimation and depth resolution is shown. Camera 1101 is connected to a communication network 1112. Camera 1101 includes an image sensor 1102. Camera 1101 also includes memory 1103, a processor 1104, and a transceiver 1105. Memory 1103 stores code that can be executed by processor 1104 in a non-transitory form. In some implementations, the code may include the neural model described above. The algorithm may include code that is directly executable by the processor and / or parameters such as neural network weights derived from the training model described above, which are not directly executable instructions but are used to configure other executable code stored in memory 1103. Memory 1103 may also store raw and / or processed image data. Transceiver 1105 may be capable of sending and receiving data via one or both of a wired communication channel and a wireless communication channel. For example, it may support Ethernet, IEEE 802.11B, and / or cellular protocols such as 4G or 5G.

[0080] Such a camera 1101 typically includes some onboard processing capability. This may be provided by a processor 1104. The processor 1104 may also be used to implement other functions of the device.

[0081] Transceiver 1105 is capable of communicating with other entities 1110, 1111 via network 1112. Although two entities are shown, more entities may be provided. These entities may be physically remote from camera 1101. Network 1112 may be a publicly accessible network such as the Internet. Entities 1110, 1111 may be cloud-based. Entity 1110 may be a computing entity. Entity 1111 may be a command and control entity. These entities are logical entities. In practice, they may be provided by one or more physical devices (such as servers and data storage), and the functionality of two or more entities may be provided by a single physical device. Each physical device implementing an entity includes a processor and memory. These devices may also include a transceiver for sending data to and receiving data from transceiver 1105 of camera 1101. Their memories store code in a non-transient manner that can be executed by a processor to implement the corresponding entity in the manner described herein.

[0082] The command and control entity 1111 can train a model for estimating the scale and depth of source image features. This is typically a computationally intensive task, even if the resulting model can be efficiently described, so it may be efficient to perform algorithm development in the cloud, where substantial energy and computational resources may be available. This may be more practical than developing such a model on a typical camera, which may be limited in processor size and functionality.

[0083] In one implementation, once the algorithm has been developed in the cloud, the command and control entity 1111 can automatically form a corresponding model upon receiving the algorithm and transmit the model to the relevant camera device for execution. In this example, the method is executed by the processor 1104 at the camera 1101.

[0084] In another possible implementation, an image may be captured by a camera sensor 1102, and the image data may be sent to the cloud by a transceiver 1105 for processing, including scale estimation of scene features. The resulting target image may then be sent back to the camera 1101 via a network 1112.

[0085] Therefore, the method can be deployed in a variety of ways, such as in the cloud, on-device, or in dedicated hardware, or a combination thereof. As mentioned above, cloud facilities can perform training to develop new algorithms or improve existing ones. Depending on the computing power near the data corpus, training can be performed near the source data or in the cloud, for example using an inference engine. Scale estimation can also be performed on-camera, on dedicated hardware, or in the cloud.

[0086] As can be appreciated from the above, the implementation of the present invention is advantageous for a number of reasons. In particular, simultaneous training of tasks (1) and (2) provides the following advantages:

[0087] Fusing stereo depth and scale estimates in an end-to-end learning pipeline — This enables leveraging high-quality geometric SLAM methods and deriving meaningful scale from interdependent features, providing high-resolution and high-accuracy distance measurements.

[0088] Solution for mid-scale ambiguity in image-only monocular SLAM — In this pipeline, no additional sensors or other modalities such as IMU, LIDAR, active stereo, second camera, etc. are required at runtime.

[0089] Self-supervision via stereo SLAM — Additional sensors can be used during training to enable multi-view imaging. Therefore, no additional ground truth data is required as this can be generated dynamically using a second camera.

[0090] Creates a spatiotemporal pipeline that is independent of the number of input images—warping different images (in space and time) to a reference can be done using differentiable homography warps. Statistical analysis of these warped versions is independent of the number of input images, allowing the number of input images to be changed at inference time. This allows second, third, and other cameras to be used for both training and inference. At runtime, the pipeline can be used with both mono and stereo inputs in both the temporal and spatial domains.

[0091] Drift can be detected through statistical analysis of scaling parameters over time – Small errors accumulate in classic monocular SLAM methods (this is called drift), which is difficult to detect. A simple analysis of scaling parameters over time (which can be run in a separate thread from scaling and depth map creation) directly reveals drift issues. This can be used as a global optimization to force loop closure / adjust the depth map, which are both expensive ways to minimize drift in traditional systems.

[0092] Real-time applicability — The two tasks can be used independently (once trained), enabling real-time scale estimation for Task (2).

[0093] Applicants hereby disclose individually each individual feature described herein, as well as any combination of two or more such features, to the extent such feature or combination can be implemented according to the common general knowledge of a person skilled in the art based on the specification as a whole, regardless of whether such feature or combination of features solves any problem disclosed herein, and without limiting the scope of the claims. Applicants indicate that aspects of the present invention may include any such individual feature or combination of features. In view of the foregoing description, it will be apparent to those skilled in the art that various modifications may be made within the scope of the present invention.

Claims

1. An image processing device (1101, 1102), characterized in that: The method comprises a processor (1104) for estimating the scale of an image feature by: Processing multiple pairs of images of a scene through a first trained model to identify features in the images and estimate depths of the features in the images; Processing the plurality of pairs of images through a second training model to estimate scaling of the images; estimating a scale of the feature by adjusting the estimated depth according to the estimated scaling factor; Among them, the first training model includes: for each pair of images in the multiple pairs of images, respectively warping the feature maps of the two images in each pair of images to a common reference view; aggregating the warped feature maps into a single cost capacity, and using the single cost capacity to form an arbitrarily scaled depth map; the second training model includes: for each pair of images in the multiple pairs of images, warping the feature map of one image in each pair of images onto the feature map of another image; calculating feature differences in feature space; estimating scaling parameters by gradually reducing the resolution of the feature differences.

2. The image processing device (1101, 1102) according to claim 1, characterized in that: The plurality of pairs of images include images of the scene captured from different positions.

3. The image processing device (1101, 1102) according to claim 2, characterized in that: The first trained model and the second trained model are configured to operate independent of the distance between the locations.

4. The image processing device (1101, 1102) according to claim 2 or 3, characterized in that: The plurality of pairs of images include images of the scene captured at different times.

5. The image processing device (1101, 1102) according to any one of claims 1 to 3, characterized in that: The pairs of images are frames in a common video stream.

6. The image processing device (1101, 1102) according to any one of claims 1 to 3, characterized in that: A camera is included, wherein the plurality of pairs of images are images captured by the camera.

7. The image processing device (1101, 1102) according to any one of claims 1 to 3, characterized in that: The estimated depth is a relative distance.

8. The image processing device (1101, 1102) according to any one of claims 1 to 3, characterized in that: The scale of the estimates is absolute distance.

9. The image processing device (1101, 1102) according to any one of claims 1 to 3, characterized in that: The device is configured to: run the first training model on a downsampled image of the scene to form a data set including an estimate of the feature depth in the downsampled image; and The data set is processed to increase its resolution.

10. A method for training an image processing model, characterized in that: include: (a) receiving (1001) a plurality of pairs of images, each pair of images representing a common scene and the images in each pair of images are captured from different positions separated by a certain distance; (b) For each pair of images: (1) Receiving (1002) ground truth data, the ground truth data comprising a scale estimate of a region in the image; (2) processing (1003) the image using a first model to identify features in the image; (3) processing (1004) the image through a second model to estimate the relative depth of the feature; (4) processing (1005) the image through a third model to estimate the absolute scale of the image; (5) adjusting (1006) one or both of the second model and the third model based on the ground truth data; Among them, the second model includes: for each pair of images in the multiple pairs of images, warping the feature maps of the two images in each pair of images to a common reference view; aggregating the warped feature maps into a single cost capacity, and using the single cost capacity to form an arbitrarily scaled depth map; the third model includes: for each pair of images in the multiple pairs of images, warping the feature map of one image in each pair of images to the feature map of another image; calculating feature differences in feature space; estimating scaling parameters by gradually reducing the resolution of the feature differences.

11. The method according to claim 10, characterized in that The method includes: for each pair of images, estimating the scale of the region in the images to form the ground truth data by one of: (1) performing a geometric analysis of the images based on the distance between the locations at which the images were captured, and (2) estimating a transformation between the images based on data indicating the relative locations at which the images were captured and the relative directions at which the images were captured.

12. The method according to claim 10 or 11, characterized in that: include: For each pair of images, one of the images is warped to a common image frame with the other image in the pair; wherein steps (3) and (4) are performed on the warped image and the other image.

13. The method according to claim 10 or 11, characterized in that: The step (2) comprises processing the images through a first model to identify features in the images and the position of each feature in the corresponding image; the method comprises: for each pair of images, warping the position of the feature identified in one of the images to an image plane that is common to the position of the feature identified in the other image of the pair of images; wherein the steps (3) and (4) are performed on the warped feature position and the feature position of the other image.

14. An image processing device, characterized in that: The method comprises a processor and a memory, wherein the memory comprises instructions executable by the processor to implement an image processing model adjusted by the method according to any one of claims 10 to 13.

Images