Device for locating a vehicle and method for locating a vehicle

Abstract

A device for localizing a vehicle may include a machine learning model and a localizer. The machine learning model may be configured to receive a sequence of image sets. Each image set in the sequence of image sets may include at least a first image and a second image. The machine learning model may be configured to determine a respective depth map for each image set based on at least the first image and the second image of the image set, resulting in a sequence of depth maps. The machine learning model may be further configured to determine an optical flow based on at least one of the first image from the sequence of image sets and the second image from the sequence of image sets. The localizer may be configured to localize the vehicle based on the sequence of depth maps and the optical flow.

Description

[Technical Field]

[0001] Various embodiments relate to a device for localizing a vehicle and a method for localizing a vehicle. [Background technology]

[0002] Over the past decade, due to the increasingly remarkable performance of visual sensors in terms of image richness, price, and data volume, vision-based odometry, especially simultaneous localization and mapping (SLAM), has attracted more attention in the field of autonomous driving. Widely recognized SLAM techniques include parallel tracking and mapping (PTAM), large-scale direct monocular (LSD)-SLAM, and direct sparse odometry (DSO). These existing SLAM techniques can generally only work well in indoor or urban environments with obvious structural features. Their performance decreases over time in environments with complex terrain features, such as off-road environments, where environmental elements may be in a state of slight movement. For example, there may be vegetation moving due to wind, drifting clouds, and the changing texture of sandy roads due to passing vehicles. Therefore, these off-road environments may lack stable trackable points for existing SLAM techniques. Additionally, factors such as direct sunlight, vegetation occlusion, uneven roads, and sensor failure can further compound the difficulty of tracking objects in the environment.

[0003] In view of the above, there is a need for an improved method for locating a vehicle that can address at least some of the problems discussed above. Summary of the Invention

[0004] According to various embodiments, a device for localizing a vehicle is provided. The device may include a machine learning model and a localizer. The machine learning model may be configured to receive a sequence of image sets. Each image set in the sequence of image sets may include at least a first image and a second image. The machine learning model may be configured to determine a respective depth map for each image set based on at least the first image and the second image of the image set, resulting in a sequence of depth maps. The machine learning model may be further configured to determine an optical flow based on at least one of the first image from the sequence of image sets and the second image from the sequence of image sets. The localizer may be configured to localize the vehicle based on the sequence of depth maps and the optical flow.

[0005] According to various embodiments, a computer-implemented method for localizing a vehicle is provided. The method may include inputting a sequence of image sets into a machine learning model. Each image set in the sequence of image sets may include at least a first image and a second image. The method may further include determining, by the machine learning model, a respective depth map for each image set based on at least the first image and the second image of the image set, resulting in a sequence of depth maps. The method may further include determining, by the machine learning model, an optical flow based on at least one of the first image from the sequence of image sets and the second image of the sequence of image sets. The method may further include localizing the vehicle based on the sequence of depth maps and the optical flow.

[0006] Further features for advantageous embodiments are provided in the dependent claims.

[0007] In the drawings, like reference numbers generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments are described with reference to the following drawings: [Brief explanation of the drawings]

[0008] [Figure 1A] FIG. 1 is a simplified functional block diagram of a device for locating a vehicle, according to various embodiments. [Figure 1B] 1B is a simplified hardware block diagram of the device of FIG. 1A, in accordance with various embodiments. [Figure 2] 1A and 1B through block diagrams, according to various embodiments. [Figure 3A] FIG. 1C is a block diagram of one embodiment of a machine learning model of the device of FIGS. 1A and 1B performing operations in accordance with various embodiments. [Figure 3B] FIG. 1C is a block diagram of one embodiment of a machine learning model of the device of FIGS. 1A and 1B performing operations in accordance with various embodiments. [Figure 4] FIG. 1C is a block diagram of a pose estimator of the device of FIGS. 1A and 1B according to various embodiments. [Figure 5] FIG. 1C illustrates an example of input to the device of FIGS. 1A and 1B. [Figure 6] FIG. 1C shows an example of the output of the device of FIGS. 1A and 1B. [Figure 7] 1 is a flow diagram of a method for locating a vehicle, according to various embodiments. [Figure 8] FIG. 1 is a schematic block diagram of a vehicle, according to various embodiments. [Figure 9] FIG. 10 illustrates an example of how the location of feature points can differ in the left and right images of a stereo image pair. DETAILED DESCRIPTION OF THE INVENTION

[0009] Embodiments described below in the context of a device are equally valid for the respective method, and vice versa. Furthermore, it will be understood that the embodiments described below may be combined, e.g., parts of one embodiment may be combined with parts of another embodiment.

[0010] It will be understood that any property described herein for a particular device may also hold true for any device described herein. It will be understood that any property described herein for a particular method may also hold true for any method described herein. Furthermore, it will be understood that for any device or method described herein, not all of the components or steps described necessarily have to be present in the device or method, but that only some (but not all) of the components or steps may be present.

[0011] The term "coupled" (or "connected") in this specification may be understood as being electrically coupled or mechanically coupled, for example, attached or fixed, or simply in contact without any fixation, and it will be understood that both direct coupling or indirect coupling (in other words, coupling without direct contact) may be provided.

[0012] In this context, a device described in this description may include, for example, a memory used in processes performed in the device. The memory used in an embodiment may be a volatile memory, such as a DRAM (Dynamic Random Access Memory), or a non-volatile memory, such as a PROM (Programmable Read Only Memory), an EPROM (Erasable PROM), an EEPROM (Electrically Erasable PROM), or a flash memory, such as a floating gate memory, a charge trap memory, an MRAM (Magnetoresistive Random Access Memory), or a PCRAM (Phase Change Random Access Memory).

[0013] In order that the invention may be readily understood and put into practice, various embodiments will now be described by way of example and not by way of limitation, and with reference to the figures.

[0014] According to various embodiments, a method for localizing a vehicle may be provided. The method may include multi-camera collaboration to exploit the properties of panoramic vision and stereoscopic perception to improve localization accuracy in off-road environments. The method may be an improved simultaneous localization and mapping (SLAM) technique that progressively builds a consistent map of the environment, solving the problem of localizing a vehicle in an unknown environment. The method may have the ability to use uncalibrated or unrectified stereo cameras for three-dimensional (3D) environment reconstruction and vehicle localization. The method may enable estimation of scale from uncalibrated / unrectified 3D reconstruction. Because the method does not require the cameras to be calibrated, loosely coupled satellite stereo cameras may be used to capture images for SLAM. The cameras may be coupled to the vehicle using a non-rigid mounting structure. The method may be capable of accurate localization and mapping despite vibration and thermal effects experienced by the cameras. The map generated by the method may be of higher image quality due to reduced noise, and therefore small obstacles may be more effectively detected and identified in the map. Higher quality maps and faster location may also enable earlier detection of sudden traffic participants and changes in navigation routes.

[0015] According to various embodiments, the method may further include detecting two-dimensional (2D) or 3D objects in the generated map.

[0016] According to various embodiments, the method may further include tracking 2D or 3D objects in the generated map.

[0017] According to various embodiments, the method may further include reconstructing the 3D scene.

[0018] According to various embodiments, the method may further include 3D mapping and real-time detection of changes made to the environment.

[0019] According to various embodiments, the method may further include generating a 3D environment model in combination with other sensors, such as radar or laser sensors.

[0020] According to various embodiments, the method may be used to perform at least one of: detecting lost cargo, detecting objects, performing 3D road surface modeling, and performing augmented reality-based visualization.

[0021] According to various embodiments, device 100 may be configured to perform any one of the methods described above.

[0022] 1A shows a simplified functional block diagram of a device 100 for locating a vehicle, according to various embodiments. Device 100 may be configured to receive input 110 and generate output 120. The input may include a sequence of a set of images. Output 120 may include the location of the vehicle and may further include the trajectory of the vehicle. Device 100 may be capable of locating the vehicle based on the images.

[0023] The sequence of image sets in input 110 may be captured by sensors mounted on a vehicle. Each image set may include multiple images, and each image of the multiple images may be captured from a different position on the vehicle, and therefore the multiple images in each image set may be offset in at least one axis from each other. The multiple images may be captured by corresponding multiple sensors, respectively.

[0024] The device 100 may include a machine learning model 102. The machine learning model 102 may be configured to determine depth information for each image set based on multiple images in the image set. The machine learning model 102 may output a sequence of depth maps 112 based on a received sequence of image sets. Each depth map may be an image or image channel containing information related to the distance of an object's surface from a viewpoint. The viewpoint may be a vehicle, or more specifically, a sensor mounted on a vehicle. The machine learning model 102 may also be configured to determine an optical flow 114 based on the received sequence of image sets. The machine learning model 102 may determine multiple optical flows 114, and the number of optical flows may correspond to the number of images in each image set.

[0025] The device 100 may include a localizer 104. The localizer 104 may be configured to receive a sequence of depth maps 112 and an optical flow 114 from the machine learning model 102. The localizer 104 may be configured to generate an output 120 based on the received sequence of depth maps 112 and the optical flow 114.

[0026] In other words, the device 100 may include a machine learning model 102 and a localizer 104. The machine learning model 102 may be configured to receive a sequence of image sets. Each image set in the sequence of image sets may include at least a first image and a second image. The machine learning model 102 may be further configured to determine a respective depth map 112 for each image set based on at least the first image and the second image of the image set, resulting in the sequence of depth maps 112. The machine learning model 102 may be further configured to determine an optical flow 114 based on at least one of the first image from the sequence of image sets and the second image from the sequence of image sets. The localizer 104 may be configured to localize the vehicle based on the sequence of depth maps 112 and the optical flow 114.

[0027] By using depth information in combination with optical flow to determine vehicle location, device 100 may overcome challenges in matching vehicle location to dynamic scenes that include moving objects. Thus, device 100 may achieve improved localization accuracy.

[0028] 1B shows a simplified hardware block diagram of device 100 according to various embodiments. Device 100 may include at least one processor 130. Device 100 may further include multiple sensors 132. The at least one processor 130 may be configured to perform the functions of machine learning model 102 and locator 104. Device 100 may include at least one memory 134 that may store machine learning model 102 and locator 104. The at least one memory 134 may include a non-transitory computer-readable medium. The at least one processor 130, multiple sensors 132, and at least one memory 134 may be coupled to each other, for example, mechanically or electrically, via coupling lines 140.

[0029] According to one embodiment, which may be combined with any of the embodiments described above or any further embodiments described below, the localizer 104 may include a flow association module 202, a pose estimator 204, and a location module 206, which are further described with respect to FIG. 2. The flow association module 202 may be configured to generate a three-dimensional (3D) optical flow based on the sequence of depth maps 112 and the optical flow 114 determined by the machine learning model 102. The pose estimator 204 may be configured to determine vehicle motion parameters based on the 3D optical flow. The location module 206 may be configured to localize the vehicle based on the determined motion parameters. By combining the sequence of depth maps and the optical flow, the device 100 may generate a dense 3D optical flow that provides detailed information for accurate determination of vehicle motion parameters.

[0030] According to one embodiment, which may be combined with any of the above-described embodiments or any further embodiments described below, device 100 may further include a point cloud generator 208, further described with respect to FIG. 2 . The cloud generator may be configured to generate a 3D point cloud 250 based on three-dimensional optical flow. The generated 3D point cloud 250 may be a 3D reconstruction of the environment in which the vehicle is traveling. Because 3D point cloud 250 may be generated in real time, device 100 may provide the vehicle with previously unavailable environmental data, for example, previously unmapped terrain. Furthermore, 3D point cloud 250 may indicate the presence of dynamic objects, such as traffic participants, to the vehicle.

[0031] According to one embodiment, which may be combined with any of the above-described embodiments or any further embodiments described below, the point cloud generator 208 may be further configured to update the generated 3D point cloud 250 based on the determined motion parameters. This may enable the vehicle to continuously have information about its surroundings and thus avoid obstacles. The vehicle may also be able to collect environmental information over an area by performing a trajectory within the area, for example, to conduct a surveillance or exploration mission.

[0032] According to one embodiment, which may be combined with any of the above-described embodiments or any further embodiments described below, device 100 may be further configured to determine unrotated disparity for each image set. A method for determining unrotated disparity is described with respect to FIG. 9.

[0033] According to one embodiment, which may be combined with any of the above-described embodiments or any further embodiments described below, a pose of a camera capturing a sequence of image sets may be determined for each image frame based on the optical flow 114 and the sequence of depth maps 112. The 3D point cloud 250 may be updated based on the determined pose of the camera. The pose of the camera may be determined based on motion parameters.

[0034] According to one embodiment, which may be combined with any of the above-described embodiments or any further embodiments described below, the flow association module 202 may be configured to generate a 3D optical flow 220 by determining a depth flow for each pixel in a sequence of a set of images and concatenating the depth flow with the optical flow 114 determined by the machine learning model 102. The concatenated depth flow with the optical flow 114 may provide a compact data structure to be processed by the pose estimator 204. This may provide an efficient, i.e., computationally simple, approach to generating a dense 3D optical flow.

[0035] According to one embodiment, which may be combined with any of the above-described embodiments or any further embodiments described below, the flow association module 202 may be further configured to interpolate missing flow pixels between two adjacent image sets through bilinear filtering. Interpolating missing flow pixels may include mapping pixel locations to corresponding points on a text map, taking a weighted average of attributes, such as color and transparency, of four surrounding texels (i.e., texture elements), and applying the weighted average to the pixel. This may avoid gaps in the resulting 3D optical flow 220, thereby reconstructing the dense 3D point cloud 250.

[0036] According to one embodiment, which may be combined with any of the above-described embodiments or any further embodiments described below, the pose estimator 204 may include two branches of a convolution stack 420, a concatenation layer 404 connected to the two branches of the convolution stack, and two regressor stacks 408 connected to the concatenation layer 404. The pose estimator 204 is further described with respect to FIG. 4. The convolution stack 420 may extract features from the 3D optical flow 220. A first branch of the convolution stack 420 may extract feature information from the depth flow, i.e., along the Z direction. A second branch of the convolution stack 420 may extract feature information from the 2D flow, i.e., along the X and Y directions. The concatenation layer 404 may combine the extracted features to feed two regressor stacks 408. The regressor stacks 408 may then determine motion parameters based on the extracted features.

[0037] According to one embodiment, which may be combined with any above-described embodiment or any further embodiment described below, the pose estimator 204 may further comprise a squeeze layer 406 connected between the concatenation layer 404 and the two regressor stacks 408. The squeeze layer 406 may be configured to compress the output of the concatenation layer 404 into a lower-dimensional space, and thus the regressor stacks 408 may require fewer computational resources in determining the motion parameters.

[0038] According to one embodiment, which may be combined with any of the above-described embodiments or any further embodiments described below, the machine learning model 102 may include a feature extraction network 310 and a disparity computation network 320, further described with respect to FIGS. 3A and 3B . The feature extraction network 310 may be configured to extract features from images in a sequence of an image set to generate a feature map. The disparity computation network 320 may be configured to determine a two-dimensional offset (also referred to herein as a displacement) between images based on the generated feature map. The machine learning model 102 may thereby determine both optical flow and depth information using a single common set of neural networks, since both optical flow and depth information relate to two-dimensional offsets between images. Thus, the device 100 may be computationally efficient, and the machine learning model 102 of the device 100 may be trained in a shorter time and with fewer resources compared to training two separate machine learning models.

[0039] According to one embodiment, which may be combined with any of the above-described embodiments or any of the further embodiments described below, the machine learning model 102 may be trained through supervised training using scene flow stereo images. Training the machine learning model 102 may require, for example, 25,000 scene flow stereo images as training data. The machine learning model 102 may be fine-tuned using the stereo images. As an example, approximately 400 stereo images may be used for training. As an example, the stereo images may be obtained from a public training dataset, such as the KITTI dataset, Cityscapes, etc. The stereo images may also be synthetically generated based on ground truth images.

[0040] According to one embodiment, which may be combined with any of the above-described embodiments or any further embodiments described below, training of the machine learning model 102 may be based on, for example, standard backpropagation-based gradient descent. As an example of how the machine learning model 102 may be trained, a training dataset may be provided to the machine learning model 102, and the following training process may be performed.

[0041] Before training the machine learning model 102, the weights may be randomly initialized to numbers between 0.01 and 0.1, and the biases may be randomly initialized to numbers between 0.1 and 0.9.

[0042] A first observation of the dataset may then be loaded into an input layer of a neural network in the machine learning model 102, and output value(s) may be generated by forward propagation of the input values ​​of the input layer. The following loss function may then be used to calculate a loss using the output value(s):

[0043] TIFF2025538605000002.tif8170 where n represents the number of neurons in the output layer, y represents the real output value, TIFF2025538605000003.tif5170 represents the difference between the actual output and the predicted output.

[0044] The weights and biases may then be updated by an AdamOptimizer with a learning rate of 0.001. Other parameters of the AdamOptimizer may be set to default values, for example: beta_1=0.9 beta_2=0.999 eps=1e-08 weight_decay=0

[0045] The steps described above may be repeated with the next set of observations until all observations have been used for training, which may represent the first training epoch, and may be repeated until 10 epochs have been performed.

[0046] According to one embodiment, which may be combined with any of the above-described embodiments or any further embodiments described below, the feature extraction network 310 may include multiple neural network branches. For example, the neural network branches may include the first branch 350 and the second branch 360 shown in FIGS. 3A and 3B . Each neural network branch may include a respective convolution stack and a pooling module connected to the convolution stack. The convolution stack may include, for example, a CNN 312. The pooling module may include, for example, an SPP module 314. The multiple neural network branches may share the same weights. By having the neural network branches share the same weights, the feature extraction network 310 may be trained in a shorter time and with fewer resources than training the neural network branches individually.

[0047] According to one embodiment, which may be combined with any of the above-described embodiments or any of the further embodiments described below, the disparity computation network 320 may include a 3D convolutional neural network (CNN) 324 configured to generate three disparity maps based on the generated feature map 318. The feature extraction network 310 may extract features at different levels. To aggregate feature information along the disparity dimension as well as the spatial dimension, the 3D CNN 324 may be configured to perform cost volume regularization on the extracted features. The 3D CNN 324 may include an encoder-decoder architecture including multiple 3D convolution and 3D deconvolution layers with intermediate supervision. The 3D CNN 324 may have a stacked hourglass architecture including three hourglasses, thereby producing three disparity outputs. The architecture of the 3D CNN 324 may enable the 3D CNN 324 to generate accurate disparity outputs and predict optical flow. The filter size in the 3D CNN 324 may be 3*3.

[0048] According to one embodiment, which may be combined with any of the above-described embodiments or any further embodiments described below, the machine learning model 102 may be configured to determine a respective depth map 112 for each image set based on the determined 2D offsets between multiple images of the image set. The depth map 112 may provide information about the distance between objects in the images from the vehicle. This information may improve localization accuracy. Each pixel in the depth map may be determined based on the following equation: TIFF2025538605000004.tif8170Here, baseline refers to the horizontal distance between the viewpoints from which images in the same image set were captured, focal length refers to the distance between the lens and the camera sensor, and parallax refers to the 2D offset.

[0049] According to one embodiment, which may be combined with any of the above-described embodiments or any further embodiments described below, the machine learning model 102 may be configured to determine an optical flow 112 based on a determined 2D offset between at least one of a first image of the set of adjacent images and a second image of the set of adjacent images. The optical flow 112 may provide information regarding a change in position of the vehicle.

[0050] According to one embodiment, which may be combined with any of the above-described embodiments or any further embodiments described below, device 100 may further include multiple sensors 132 mountable on the vehicle. Multiple sensors 132 may be adapted to capture a sequence of image sets. Multiple sensors 132 may include, for example, a set of surround-view cameras. Multiple sensors 132 may provide redundancy, so device 100 may continue to receive multiple images when one sensor fails.

[0051] According to one embodiment, which may be combined with any of the above-described embodiments or any of the further embodiments described below, the plurality of sensors 132 may include a stereo camera. Each image set may include a pair of stereo images. A stereo camera includes two individual, but closely-located, sensors, such that the captured stereo images may provide depth information.

[0052] According to one embodiment, which may be combined with any of the above-described embodiments or any further embodiments described below, each first image is captured from a first position on the vehicle and each second image is captured from a second position on the vehicle, which may be different from the first position. The images are captured from different perspectives and, therefore, when combined, may provide depth information.

[0053] FIG. 2 illustrates the operation of device 100 according to various embodiments. Input 110 to device 100 may include a sequence of image sets. The sequence of image sets may include consecutively captured images. For example, the sequence of image sets may include image set 210a captured at t-1 and another image set 210b captured at t, where t represents time as a variable. In other words, image set 210b may be a subsequent frame of image set 210a. Each image set may include multiple images, each captured at a respective position. These positions may be offset from one another, so that a combination of the multiple images may provide depth information for an object depicted in the images. For example, image set 210a and image set 210b may each include a pair of stereo images. Image set 210a may include left image 212a and right image 214a. Image set 210b may include left image 212b and right image 214b.

[0054] A sequence of image sets 210a, 210b may be provided to the machine learning model 102. The machine learning model 102 may be trained to perform the dual functions of computing a depth map 112 and computing an optical flow 114 for at least two consecutive image sets. The machine learning model 102 may be trained to perform both functions using a common set of neural networks and using the same set of weights in that set of neural networks. The machine learning model 102 may be trained to identify features in the images and may be further configured to determine spatial offsets, also referred to herein as disparity data, of the features between images. The spatial offsets between images of the same image set may provide depth information of the features. The spatial offsets between images captured at the same location across a sequence of image sets, i.e., over time, may provide information about the optical flow. Correspondingly, the common set of neural networks may accomplish the dual functions of determining a depth map 112 and an optical flow 114.

[0055] The localizer 104 may receive the depth map 112 and the optical flow 114 from the machine learning model 102. The localizer 104 may include a flow association module 202, a pose estimator 204, and a location module 206. The optical flow 114 may include information of movement in two dimensions, i.e., two-dimensional (2D). The optical flow 114 may include a dense 2D optical flow. The flow association module 202 may generate a dense 3D optical flow 220 based on the optical flow 114 and the corresponding depth map 112.

[0056] The flow association module 202 may determine a depth flow, in other words, an optical flow along the depth axis (also referred to herein as the Z-axis), based on the sequence of depth maps 112 and the optical flow 114 provided by the machine learning model 102. The flow association module 202 may determine the depth flow according to equation (1) as follows: TIFF2025538605000005.tif6170In the above formula (1), TIFF2025538605000006.tif5170 Represents the depth flow between frame k and frame k+1 at pixel coordinates (x,y). TIFF2025538605000007.tif6170 represents the optical flow 114 on the XY image plane between frame k and frame k+1, and G k ∈s h×w represents the depth map 112 of frame k, where s represents the XY image plane, h represents the image height, and w represents the image width.

[0057] The flow association module 202 may generate a 3D optical flow 220 at each pixel coordinate by concatenating the 2D optical flow 114 and the depth flow. The 3D flow 220 may be determined according to equation (2) as follows: TIFF2025538605000008.tif6170In the above equation (2), TIFF2025538605000009.tif6170 represents the 3D flow at pixel coordinates (x,y), and C indicates the concatenation operation.

[0058] If the depth value in frame k+1 cannot be associated with the corresponding depth value in frame k, the flow association module 202 may interpolate the missing flow pixels between two adjacent frames through bilinear filtering. Inverse depth (i.e., disparity) may be more sensitive to the movement of the surroundings and objects close to the camera. Therefore, inverse depth is used instead of depth values. The difference between the coordinates of corresponding pixels in the left and right images is known as stereo correspondence or disparity, which is inversely proportional to the distance of the object from the camera. Therefore, disparity is sometimes referred to as inverse depth. The 3D optical flow 220 may be represented as a 3D motion vector.

[0059] The pose estimator 204 may receive the 3D optical flow 220 output by the flow association module 104. The pose estimator 204 may determine motion parameters 222 based on the 3D optical flow 220. The motion parameters 222 may include a six-degree-of-freedom (6DOF) relative pose transformation, including scale, between each pair of images. The location module 206 may determine a vehicle trajectory by accumulating the relative pose over time. The location module 206 may also localize the vehicle based on the accumulated relative pose. The output 120 of the location module may include at least one of a vehicle trajectory and a vehicle location.

[0060] According to various embodiments, the pose estimator 204 may include a neural network architecture, which is further described with respect to FIG.

[0061] Still referring to FIG. 2 , the device 100 may further include a point cloud generator 208. The point cloud generator 208 may generate a 3D point cloud 250 based on the 3D optical flow 220 and the image set 210 a, 210 b in the input 110. The 3D point cloud 250 may be generated based on the depth map and camera intrinsic parameters, such as focal length along x and y, camera principal point offset, and axis skew. The point cloud generator 208 may further update the 3D point cloud 250 based on the motion parameters 222. The point cloud generator 208 may also refine the 3D point cloud 250 to remove outliers and incorrect predictions, so that the 3D point cloud 250 may serve as an accurate dense 3D map. The point cloud generator 208 may remove outliers and incorrect predictions based on the probabilities of their data points.

[0062] Device 100 may simultaneously generate 3D point cloud 250 and output 120. Device 100 may further combine output 120 with 3D point cloud 250 to present vehicle movement in 3D point cloud 250. By generating both 3D point cloud 250 and vehicle location in parallel, device 100 may provide simultaneous localization and mapping (SLAM) capabilities.

[0063] 3A and 3B illustrate block diagrams of one embodiment of a machine learning model 102 performing operations according to various embodiments. Referring to FIG. 3A, the machine learning model 102 may be performing a stereo matching operation. The stereo matching operation may include determining an estimate of a pixel-by-pixel displacement map between input images. The input images may include multiple images of the same image set. For example, when the input 110 includes images captured by a stereo camera, the input images may include a left image 212b and a right image 214b.

[0064] In general, stereo images can be rectified or unrectified. A rectified stereo image is one in which the displacement of each pixel is constrained to a horizontal line. The displacement map is sometimes referred to herein as disparity. To obtain rectified stereo images, the sensors or cameras used to capture the images need to be precisely calibrated.

[0065] On the other hand, unrectified stereo images may exhibit both vertical and horizontal disparity. Vertical disparity may be defined as the vertical displacement between corresponding pixels in the left and right images. Horizontal disparity may be defined as the horizontal displacement between corresponding pixels in the left and right images. Obtaining rectified stereo images from sensors mounted on a vehicle is difficult because the sensors may shift or rotate in position over time due to vehicle movement and vibration. Therefore, input images captured by sensors mounted on a vehicle may be considered unrectified stereo images.

[0066] The machine learning model 102 may be trained to be robust to rotation, shift, vibration, and distortion of the input image. The machine learning model 102 may be trained to handle both horizontal (x-parallax) and vertical (y-parallax) displacements. In other words, the machine learning model 102 may be configured to determine both horizontal and vertical disparities of the input image. This makes the machine learning model 102 robust to vertical translation and rotation between cameras or sensors.

[0067] The machine learning model 102 may have a dual-branch neural network architecture including a first branch 350 and a second branch 360, such that each branch may be configured to determine disparity in a respective axis. The machine learning model 102 may include a feature extraction network 310 and a disparity calculation network 320. The feature extraction network 310 and the disparity calculation network 320 may each include components of the first branch 350 and the second branch 360.

[0068] The feature extraction network 310 may include a convolutional neural network (CNN) 312, a spatial pyramid pooling (SPP) module 314, and a convolutional layer 316 for each of the first branch 350 and the second branch 360. The CNN 312 may extract feature information from an input image. The CNN 312 may include three small convolution filters with a kernel size of 3 × 3 cascaded to construct a deeper network with the same receptive field. The CNN 312 may include conv1_x, conv2_x, conv3_x, and conv4_x layers, which form a basic residual block for learning simplex feature extraction. For conv3_x and conv4_x, dilated convolution may be applied to further expand the receptive field. The output feature map size may be ¼ × ¼ of the input image size. The SPP module 314 may then be applied to collect contextual information from the output feature maps. The SPP module 314 may learn relationships between objects and object subregions to incorporate hierarchical contextual information. The SPP module 314 may include four fixed-size average pooling blocks of sizes 64x64, 32x32, 16x16, and 8x8. The convolution layer 316 may be a 1x1 convolution layer to reduce feature dimensionality. The feature extraction network 310 may use bilinear interpolation to upsample the feature maps to the same size as the original feature maps. The size of the original feature maps may be 1 / 4 the size of the input image. The feature extraction network 310 may concatenate different levels of feature maps extracted by various convolution filters as a left SPP feature map 318 and a right SPP feature map 319.

[0069] The disparity computation network 320 may receive the left SPP feature map 318 and the right SPP feature map 319. The disparity computation network 320 may concatenate the left SPP feature map 318 and the right SPP feature map 319 into separate cost volumes 322 for x-displacement and y-displacement, respectively. Each cost volume may have four dimensions: height × width × disparity × feature size. The disparity computation network 320 may include a 3D-CNN 322 in each branch. The 3D-CNN 322 may include a stacked hourglass (encoder-decoder) architecture configured to generate the three disparity maps. The disparity computation network 320 may further include an upsampling module 326 and a regression module 328 in each branch. The upsampling module 326 may upsample the three disparity maps so that their resolutions match those of the input image size. The regression module 328 may apply regression to the upsampled disparity maps to calculate an output disparity map. The disparity computation network 320 may calculate the probability of each disparity based on the predicted cost via a softmax operation. The predicted disparity may be calculated as the sum of each disparity weighted by its probability. A smooth loss function may then be applied between the ground truth disparity and the predicted disparity. The smooth loss function may measure how close the prediction is to the ground truth disparity value. The smooth loss function may be a combination of l1 loss and l2 loss. This is used in deep neural networks because of its robustness and low sensitivity to outliers. The disparity computation network 320 then outputs the horizontal displacement 332 in the first branch 350 and the vertical displacement 334 in the second branch 360. The machine learning model 102 may then determine the depth map 112 based on the horizontal displacement 332 and the vertical displacement 334 using known stereo computation methods, such as semi-global matching.

[0070] 3B , the machine learning model 102 may be performing an optical flow calculation operation. The optical flow calculation operation may include predicting a displacement field for each pixel, so that for each pixel in a frame, the machine learning model 102 may estimate its corresponding pixel in the next frame. For the optical flow calculation operation, the input images used by the machine learning model 102 may be two consecutive images taken from the same position. In this example, the input images are a left image 212b captured at time=t and a left image 212a captured at time=t−1. The output of the machine learning model 102 for the optical flow calculation operation is an optical flow 114 including an x-direction displacement 342 and a y-direction displacement 344.

[0071] The first branch 350 may process the left image 212b, and the second branch 360 may process the earlier left image 212a. Similar to the stereo matching operation described with reference to FIG. 3A, the optical flow calculation operation may include extracting features using the CNN 312 of the feature extraction network 310. The SPP module 314 may collect contextual information from the output feature map generated by the CNN 312. The feature extraction network 310 generates a final SPP feature map 318 that is provided to the disparity calculation network 320. The difference from the stereo matching operation is that the generated SPP feature maps are an earlier SPP feature map 338a (for time = t-1) and a subsequent SPP feature map 338b (for time = t). The disparity calculation network 320 may concatenate the SPP feature maps 338a, 338b into separate cost volumes 322 for time = t-1 and time = t, respectively. Each cost volume may have four dimensions: height × width × disparity × feature size. The 3D-CNN 322 of each branch may generate three disparity maps based on its respective cost volume 322. The upsampling module 326 may upsample the three disparity maps. The regression module 328 may apply regression to the upsampled disparity maps to calculate an output disparity map. The disparity computation network 320 may calculate the probability of each disparity based on the predicted cost via a softmax operation. The predicted disparity may be calculated as the sum of each disparity weighted by its probability. A smooth loss function may then be applied between the ground truth disparity and the predicted disparity. The disparity computation network 320 then outputs the x-direction displacement 342 and the y-direction displacement 344 determined to occur between time = t-1 and time = t.

[0072] According to various embodiments, suitable deep learning models for machine learning model 102 may include, for example, PyramidStereoMatching and RAFTNet.

[0073] 4 shows a block diagram of the pose estimator 204 according to various embodiments. The pose estimator 204 may include a dual-stream architecture network consisting of two branches of a convolutional stack 402, followed by a concatenation layer 404, a squeeze layer 406, and two fully connected regressor stacks 408. The pose estimator 204 may receive as input a 3D optical flow 220. The 3D optical flow 220 may include a first data portion 420 that may include a 2D optical flow and a second data portion 422 that may include a depth flow. One branch of the convolutional stack 402 may receive 420, and the other branch of the convolutional stack 402 may receive 422.

[0074] The convolution stack 402 may include four layers, each consisting of a 3x3 filter and a stride of 2. The number of channels in the two branches of the convolution stack 402 may be 64, 128, 256, or 512. To preserve spatial geometry information, the pooling layer may be abandoned in these two CNN stacks, and instead, an attention layer may be added to capture features present in the image. The feature maps extracted by the two branches may be concatenated by the concatenation layer 404 and squeezed using a 1x1 filter in the squeeze layer 406. The squeeze layer 406 may embed the 3D feature map into a lower-dimensional space, thereby reducing the input dimension of the regressor stack 408. Each regressor stack 408 may include a three-layer fully connected network. The hidden layer of the regressor stack 408 may be set to size 128 with a ReLu activation function. One regressor stack 408 (referred to herein as a translation regressor) may output a translation 430 determined from the 3D optical flow 220. Another regressor stack 408 (referred to herein as a rotation regressor) may output a rotation 432 determined from the 3D optical flow 220. The output of the translation regressor may be 6 for a bivariate Gaussian loss, and the output of the rotation regressor may be 3, which may be trained through an L2 loss. To find correlations along the forward and left / right directions, a bivariate Gaussian probability distribution function may be used as a likelihood function. Once the pose estimator 204 is trained, the translation 430 and rotation 432 may be estimated from the pose estimator 204. The translation 430 and rotation 432 may be part of the motion parameters 220. The pose estimator 204 may be trained using sequences with ground truth, for example, 11 such sequences with ground truth.

[0075] The pose estimator 204 may add a new frame to the frame graph and add edges with its three nearest neighbors as measured by average optical flow. The pose estimator 204 may initialize the pose using a linear motion model. The pose estimator 204 may then apply several iterations of an update operator to update the keyframe pose and depth. The update operator may be configured to perform a PnP method. The first two poses may be fixed to remove gauge degrees of freedom, while all depths may be treated as free variables. After the new frame is tracked, a keyframe may be selected for removal. The pose estimator may determine the distance between pairs of frames by calculating the average optical flow magnitude and by removing redundant frames. If none of the frames are good candidates for removal, the oldest keyframe may be removed. After 10 frames, a reprojection error may be calculated to optimize the estimated pose between frames.

[0076] 5 shows an example of input 110 to device 100. Image 502 may be a left stereo image captured at time=t, and image 504 may be a right stereo image captured at time=t−1. Image 506 may be a left stereo image captured at time=t, and image 508 may be a right stereo image captured at time=t−1. The image frame captured at time=t may be referred to herein as the kth image frame, and the image frame captured at time=t may be referred to herein as the kth. FIG. 6 shows an example of output 120 of device 100. Image 602 shows a vehicle trajectory, and image 604 shows the vehicle trajectory displayed within a 3D point cloud.

[0077] FIG. 7 shows a flow diagram of a method 700 for locating a vehicle according to various embodiments. Method 700 may include processes 702, 704, 706, and 708. Process 702 may include inputting a sequence of image sets into a machine learning model, each image set in the sequence comprising at least a first image and a second image. The sequence of image sets may include, for example, image sets 210a and 210b. The machine learning model may be, for example, machine learning model 102. Process 704 may include determining, by the machine learning model, a respective depth map for each image set based on at least the first image and the second image of the image set, resulting in a sequence of depth maps. In one example, the image sets may include a pair of stereo images, where the first image may be, for example, left image 212a or 212b, and the second image may be, for example, right image 210a or 210b. The depth maps may include, for example, depth map 112. Process 706 may include determining, by a machine learning model, an optical flow based on at least one of a first image from the sequence of image sets and a second image from the sequence of image sets. The optical flow may be, for example, optical flow 114. Process 708 may include localizing the vehicle based on the sequence of depth maps and the optical flow. Various aspects described with respect to device 100 may be applicable to method 700.

[0078] According to various embodiments, a non-transitory computer-readable medium may be provided. The computer-readable medium may include instructions that, when executed by a processor, cause the processor to perform method 700. Various aspects described with respect to device 100 may be applicable to the computer-readable medium.

[0079] 8 shows a schematic block diagram of a vehicle 800 according to various embodiments. Vehicle 800 may include device 100 according to any of the above-described embodiments or with any of the further embodiments described below. Various aspects described with respect to device 100 and method 700 may be applicable to vehicle 800.

[0080] According to various embodiments, the method (700) for localizing a vehicle may further include determining a non-rotational disparity in each image set of the sequence of image sets. The depth map 112 may be determined based on the determined non-rotational disparity.

[0081] In a typical stereo camera setup, feature points in the left stereo image and the corresponding feature points in the right stereo image are aligned on the same x-axis, also called the horizontal axis. Due to vibration or mechanical setup, one of the stereo cameras may be subject to the following factors: roll, i.e., rotation angle (a), pitch, yaw, and translation (x t ,y t ) may be misaligned with the other stereo camera due to at least one of t refers to the movement of the image pixel along the horizontal axis, and y t refers to the displacement of an image pixel along the vertical axis. The machine learning model 102 can be configured to determine a rotated disparity for the misalignment described above. The rotated disparity map can include the actual disparity of every pixel between the uncalibrated left stereo image and the uncalibrated right stereo image.

[0082] Device 100 may further include a rotation correction module that corrects the rotated disparity map for relative rotations between images in the image set. The rotation correction module may generate an unrotated disparity map based on the rotated disparity map. The unrotated disparity map may include corrected disparities for every pixel between the uncalibrated left stereo image and the uncalibrated right stereo image, as if the misaligned stereo images had already been corrected back to their calibrated positions.

[0083] 9 shows an example of how the location of a feature point may differ in the left and right images in an image set. In this example, an image center 902 may be considered to be (0,0). A feature point 904 in the left image is denoted as (x2, y2). A corresponding position 906 of the feature point 904 in the rotated right image is denoted as (x'2, y'2). A line connecting the corresponding position 906 and the image center 902 may define a first hypotenuse 920, denoted herein as "h1," of a right triangle 910. The corresponding position 906 of the feature point 904 in the right image may be determined based on a rotational disparity map.

[0084] The rotational disparity map may include a rotational disparity value (dx2, dy2) for the feature point 904. A corresponding position 906 and a first hypotenuse 920 may be determined based on the feature point 904 and the rotational disparity value according to the following equations (1)-(3): x'2=x2+dx2 formula (1) y'2=y2+dy2Equation (2) TIFF2025538605000010.tif10170

[0085] The unrotated disparity value of the feature point 904 may be expressed as (dx2, dy2). The unrotated corresponding position 908 of the feature point 904 in the right image when the right image is adjusted to have a zero rotation angle with respect to the left image may be expressed as (x 2r ,y 2r )

[0086] The corresponding location 906, the unrotated corresponding location 908, and the image center 902 may define the vertices of an isosceles triangle 922. The distance between the corresponding location 906 and the image center 902 may be at least substantially equal to the distance between the unrotated corresponding location 908 and the image center 902, or in other words, equal to the first hypotenuse 920. A vertex angle 926 of the isosceles triangle 922 is denoted as a. The vertex angle 926 is sometimes referred to herein as the rotation angle 926.

[0087] The line connecting the corresponding location 906 and the non-rotated corresponding location 908 may define a second hypotenuse 924, denoted herein as "h2," of the other right triangle 912. The second hypotenuse 924 may be the base of the isosceles triangle 922. The second hypotenuse 924 may be determined according to equation (4). TIFF2025538605000011.tif7170

[0088] The base 928 of the other right triangle 412 is denoted as d. The length of the base 928 may be determined according to equation (5). TIFF2025538605000012.tif7170

[0089] An unrotated disparity value d2 may be determined based on the base 928 and the rotated disparity value dx2. The unrotated disparity value may be determined according to equation (6) below. d2=d+dx2 formula (6)

[0090] The rotation correction module may determine an unrotated disparity map based on the above calculations. The rotation correction module may determine a corresponding position 906 based on the rotated disparity values ​​in the rotated disparity map. The rotation correction module may determine a first hypotenuse 920 based on the corresponding position 906. The rotation correction module may determine a vertex angle 926. Determining the vertex angle 926 may include determining the direction of an epipolar line of the image set in the image plane without using 3D space. The vertex angle 926 may then be obtained by projecting the rotational disparity measurements toward the epipolar line direction. The rotation correction module may determine a second hypotenuse 924 based on the vertex angle 926 and the first hypotenuse 920. The rotation correction module may determine a base 928 based on the second hypotenuse 924 and the rotated disparity values. The rotation correction module 202 may determine an unrotated disparity value based on the base 928 and the rotated disparity values.

[0091] According to one embodiment, which may be combined with any of the above-described embodiments or any further embodiments described below, the rotation correction module may be configured to determine the direction of the epipolar line of the image set in the image plane without using 3D space. The rotation correction module may determine the epipolar line direction without prior knowledge of the intrinsic and extrinsic camera parameters. The rotation correction module may determine the epipolar line direction by processing the images in the image set on a region-by-region basis. In other words, the image may be segmented into multiple regions, and the epipolar line direction may be determined in each of the multiple regions.

[0092] The rotation correction module may be configured to check for infinity-negative disparity in vertical disparity. The actual distance represented in the image may be calculated by projecting the disparity through the epipolar line direction. The actual distance may be determined region by region, in other words, in each region of the plurality of regions.

[0093] According to one embodiment, which may be combined with any of the above-described embodiments or any of the further embodiments described below, the rotation correction module calculates the translation (x t ,y t ), and may be further configured to correct the right image relative to the left image based on the determined translation. t can be calculated using tracking and filtering of input images over time. t may be determined based on pixel information at the center of the y-disparity in the y-disparity array.

[0094] Although embodiments of the present invention have been shown and described in detail with respect to certain embodiments, it should be understood by those skilled in the art that various changes in form and detail can be made herein without departing from the spirit and scope of the invention as defined by the appended claims. The scope of the present invention is therefore indicated by the appended claims, and all changes that come within the meaning and range of equivalency of the claims are therefore intended to be embraced. It will be appreciated that common numerals used in related drawings refer to components that serve similar or the same purpose.

[0095] Those skilled in the art will appreciate that the terminology used herein is for the purpose of describing various embodiments only and is not intended to limit the invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly dictates otherwise. Furthermore, it will be understood that the terms "comprises" and / or "comprising," as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0096] It should be understood that the specific order or hierarchy of blocks in the disclosed processes / flowcharts is illustrative of example approaches. Based on design preferences, it should be understood that the specific order or hierarchy of blocks in the processes / flowcharts may be rearranged. Additionally, some blocks may be combined or omitted. The accompanying method claims present elements of the various blocks in an example order and are not limited to the specific order or hierarchy presented.

[0097] The foregoing description is provided to enable those skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Accordingly, the claims are not limited to the aspects set forth herein but are to be accorded the full scope consistent with the claim language, and in which reference to an element in the singular does not mean "only" unless so expressly stated, but rather "one or more." The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects. Unless expressly stated otherwise, the term "some" refers to one or more. Combinations include any combination of A, B, and / or C, and may include multiples of A, B, or C, such as "at least one of A, B, or C," "one or more of A, B, or C," "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, C, or any combination thereof." In particular, combinations may be A only, B only, C only, A and B, A and C, B and C, or A and B and C, where any such combination may include one or more members of A, B, or C. All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later become known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims.

Claims

1. A device (100) for locating a vehicle, said device (100) comprising: a machine learning model (102) configured to receive a sequence of image sets (210a, 210b), each image set in the sequence of image sets (210a, 210b) comprising at least a first image and a second image; the machine learning model (102) is further configured to determine a respective depth map (112) for each image set based on at least the first image and the second image of the image set, resulting in a sequence of depth maps (112); a machine learning model (102), wherein the machine learning model (102) is further configured to determine an optical flow (114) based on at least one of the first image from the sequence of a set of images and the second image from the sequence of a set of images; a localizer (104) configured to localize the vehicle based on the sequence of depth maps (112) and the optical flow (114); A device (100) comprising:

2. The location identifier (104) a flow association module (202) configured to generate a three-dimensional optical flow (220) based on the sequence of depth maps (112) and the optical flow (114) determined by the machine learning model (102); Equipped with the localizer (104) includes a pose estimator (204) configured to determine motion parameters (222) of the vehicle based on the 3D optical flow (220); a location module (206) configured to locate the vehicle based on the determined motion parameters (222); The device (100) of claim 1 further comprising:

3. a point cloud generator (208) configured to generate a three-dimensional point cloud (250) based on the three-dimensional optical flow (220); The device (100) of claim 2 further comprising:

4. 4. The device (100) of claim 3, wherein the point cloud generator (208) is further configured to update the generated three-dimensional point cloud (250) based on the determined motion parameters (222).

5. 5. The device (100) of claim 3 or 4, wherein the flow association module (202) is configured to generate the three-dimensional optical flow (220) by determining a depth flow for each pixel in the sequence of a set of images (210a, 210b) and concatenating the depth flow with the optical flow (114) determined by the machine learning model (102).

6. 6. The device (100) of claim 5, wherein the flow association module (202) is further configured to interpolate missing flow pixels between two adjacent image sets (210a, 210b) through bilinear filtering.

7. 7. The device of claim 3, wherein the pose estimator comprises two branches of a convolution stack, a concatenation layer connected to the two branches of the convolution stack, and two regressor stacks connected to the concatenation layer.

8. 8. The device of claim 7, wherein the pose estimator further comprises a squeeze layer connected between the concatenation layer and the two regressor stacks, the squeeze layer configured to compress the output of the concatenation layer into a lower dimensional space.

9. The machine learning model (102) a feature extraction network (310) configured to extract features from images in the sequence of image sets (210a, 210b) to generate a feature map; Equipped with 9. The device (100) of claim 1, wherein the machine learning model (102) further comprises a disparity calculation network (320) configured to determine a two-dimensional offset between the images based on the generated feature maps.

10. 10. The device of claim 9, wherein the feature extraction network comprises a plurality of neural network branches, each comprising a respective convolution stack and a pooling module connected to the convolution stack, and wherein the plurality of neural network branches share the same weights.

11. 11. The device (100) of claim 9 or 10, wherein the disparity computation network (320) comprises a 3D convolutional neural network (324) configured to generate three disparity maps based on the generated feature maps.

12. 12. The device (100) of claim 9, wherein the machine learning model (102) is configured to determine the respective depth map (112) for each image set (210a, 210b) based on the determined two-dimensional offset between the multiple images of the image set (210a, 210b).

13. 13. The device (100) of claim 9, wherein the machine learning model (102) is configured to determine the optical flow (114) based on the determined two-dimensional offset between at least one of the first image of the set of adjacent images (210a, 210b) and the second image of the set of adjacent images (210a, 210b).

14. a plurality of sensors (132) mountable on the vehicle, the plurality of sensors (132) adapted to capture the sequence of image sets (210a, 210b); Furthermore, The device (100) of any one of claims 1 to 13, wherein the plurality of sensors (132) preferably comprises stereo cameras, and preferably each image set (210a, 210b) comprises a pair of stereo images.

15. 1. A computer-implemented method (700) for locating a vehicle, the method (700) comprising: inputting a sequence of image sets (210a, 210b) into a machine learning model (102), each image set (210a, 210b) in the sequence comprising at least a first image and a second image; determining, for each image set (210a, 210b), a respective depth map (112) based on at least the first image and the second image of the image set (210a, 210b) by the machine learning model (102), to produce a sequence of depth maps (112); determining, by the machine learning model (102), an optical flow (114) based on at least one of the first image from the sequence of image sets (210a, 210b) and the second image of the sequence of image sets (210a, 210b); localizing the vehicle based on the sequence of depth maps (112) and the optical flow (114); A computer-implemented method (700) comprising:

Images