High precision localization of moving object on trajectory

Abstract

Techniques are provided for generating high-precision localization of a moving object on a trajectory. In one technique, a particular image associated with a moving object is identified. A set of candidate images is selected from a plurality of images used to train a neural network. For each candidate image in the set of candidate images: (1) generating an output from a neural network based on inputting a particular image and the each candidate image into the neural network; (2) determining a predicted position of the particular image based on the output and the position associated with each candidate image; and (3) adding the predicted position to the predicted position set. The predicted position set is aggregated to generate an aggregated position for the particular image.

Description

Technical Field

[0001] This disclosure relates to object localization, and more particularly, to obtaining high-precision localization in a familiar environment using machine learning. Background Technology

[0002] Various applications require a solution to determine the position of a moving object in a new environment using cameras mounted on it. Examples of such applications include autonomous navigation for drones, robots, and self-driving cars; vehicle trajectory analysis; augmented reality; and mapping in GPS-free environments.

[0003] Traditional methods use map building and location-based approaches to determine the location of moving objects. However, these methods are prone to errors in the map building and localization steps, leading to poor localization accuracy. These errors stem from two main areas: visual feature engineering and feature sparsity. Regarding the former, visual features (e.g., landscape elements) cannot perfectly describe all environments. If the correct image features are not selected, the accuracy of map building and localization will be affected. Regarding the latter (feature sparsity), even if appropriate visual features are selected, only a small number of features will exist in each image. This sparsity directly impacts localization accuracy.

[0004] The methods described in this section are permissible but not necessarily previously conceived or employed methods. Therefore, unless otherwise indicated, any method described in this section should not be considered prior art simply because it is included in this section. Summary of the Invention

[0005] In an embodiment, the first method includes: (1) identifying a specific image associated with a moving object; (2) selecting a set of candidate images from a plurality of images used to train a neural network; (3) for each candidate image in the set of candidate images: (i) generating an output from the neural network based on inputting the specific image and each candidate image into the neural network; (ii) determining a predicted location of the specific image based on the output and the location associated with each candidate image; (iii) adding the predicted location to a set of predicted locations; (4) aggregating the set of predicted locations to generate an aggregated location of the specific image; and (5) associating the aggregated location with the moving object.

[0006] In a related embodiment, the first method further includes: (a) for each of the plurality of images: (i) determining the location of each image; (ii) identifying a set of images whose locations are within a threshold distance from the location of each image; (iii) storing pairing data that associates the set of images with each image; and (b) training a neural network based on the pairing data associated with each of the plurality of images.

[0007] In a related embodiment of the third method (where the third method includes the first method), selecting a candidate image set includes: (a) determining the estimated location of a particular image; and (b) selecting a candidate image set based on a threshold distance between the estimated location of the particular image and the location of each candidate image in the candidate image set.

[0008] In a related embodiment of the third method, each of the plurality of images is associated with location data indicating the location of the object; and the third method further includes storing an index that indexes the plurality of images based on the location data associated with each of the plurality of images; wherein selecting a candidate image set includes identifying the candidate image set in the index using estimated locations.

[0009] In a related embodiment of the third method, determining the expected location of a particular image includes determining the estimated location based on one or more predicted locations of one or more images that precede the particular image in time.

[0010] In a related embodiment of the third method, selecting a set of candidate images based on a threshold distance includes: (a) selecting multiple candidate images based on a threshold distance; (b) determining the number of candidate images among the multiple candidate images; and (c) in response to determining that the number of candidate images is greater than a specific threshold number, selecting a set of candidate images whose corresponding positions are closest to the estimated positions.

[0011] In a related embodiment of the third method, selecting a candidate image set based on a threshold distance includes: (a) determining that no candidate image is within a threshold distance; and (b) in response to determining that no candidate image is within a threshold distance, increasing the threshold distance to a larger threshold distance, and including one or more candidate images within the larger threshold distance in the candidate image set.

[0012] In a relevant embodiment of the first method, the aggregation location includes an s value and a d value; the s value is a first distance from a starting point on a predefined path along a predefined path; and the d value is a second distance from a point on the predefined path indicated by the s value.

[0013] In a related embodiment of the first method, the output includes: (a) the displacement of the position of each candidate image; and (b) the displacement of the angle of each candidate image.

[0014] In a related embodiment of the first method, aggregating the predicted location set includes calculating the average value of the predicted location set.

[0015] In a related embodiment of the first method, the first method further includes: for each candidate image in the candidate image set, determining an estimated distance between the location of each candidate image and the location of a specific image, wherein aggregating the predicted location set includes calculating a weighted average of the predicted location set based on the estimated distances associated with each candidate image in the candidate image set. Attached Figure Description

[0016] In the attached diagram:

[0017] Figure 1 This is a block diagram depicting an example positioning system in the embodiments;

[0018] Figure 2 It is a diagram depicting how candidate images can be selected in the embodiments;

[0019] Figure 3 This is a flowchart depicting an example process for predicting the position of a moving object in an embodiment;

[0020] Figure 4 This is a block diagram illustrating a computer system on which embodiments of the present invention can be implemented;

[0021] Figure 5 It is a block diagram of a basic software system that can be used to control the operation of a computer system. Detailed Implementation

[0022] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the invention. However, it will be apparent to you that the invention can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the invention.

[0023] General Overview

[0024] A system and method for predicting the location of a moving object are provided. In one technique, a set of reference images with known locations is stored and used to train a neural network. Then, given an image without a location associated with it, a set of candidate images is selected from the set of reference images. For each candidate image in the candidate image set, that image and the given image are input into the neural network, which outputs a predicted location for the given image. The predicted locations are aggregated to generate a final predicted location of the moving object at the time corresponding to the given image.

[0025] The embodiments improve upon computer-related technologies, specifically localization techniques. Because the calculated position of the moving object is anchored to the exact location in the candidate image, drift does not occur in the localization system, a phenomenon typical of many Simultaneous Localization and Mapping (SLAM) systems. Another improvement is that, through these embodiments, the localization system can robustly maintain accurate positioning over long trajectories extending over long distances, such as several kilometers.

[0026] System Overview

[0027] Figure 1 This is a block diagram depicting an example localization system 100 in an embodiment. Localization system 100 includes an image database 110, a machine learning (ML) model trainer 120, a machine learning (ML) model 130, an input image data source 140, a candidate image selector 150, an ML model caller 160, a model output 162, and an aggregator 170. Each of the ML model trainer 120, candidate image selector 150, ML model caller 160, and aggregator 170 can be implemented using software, hardware, or any combination of software and hardware.

[0028] Image database 110 includes image sets (or “image collections”), each image set corresponding to a different video clip or video stream. For example, one or more moving objects, such as cars, airplanes, robots, drones, etc., have a digital camera that generates images (or video streams) and a position determination component that determines the position and yaw angle of the moving object at each time interval (such as every 50 milliseconds). If the moving object is moving in 3D space (e.g., a flying drone), then the position determination component also determines the pitch and roll of the moving object. (In the following text, wherever position is mentioned, yaw angle is included, although not necessarily specifically mentioned). The position determination component matches the generated images based on time to determine the time (or timestamp) of each image.

[0029] To generalize the ML model 130 to adapt to variations in the actual digital images used when calling the ML model 130, multiple image sets are required. Multiple image sets can be generated using the same moving object. For example, a drone can fly from point A to point B multiple times. Additionally or alternatively, multiple moving objects can be used to generate multiple image sets. For example, multiple race cars can drive from point A to point B once or multiple times. When using different multiple image sets, one or more moving objects can be instructed to begin generating images at slightly different locations and move laterally and / or vertically within a defined range. This will allow moving objects that will rely on the ML model 130 in the future to also be able to move within the same range and accurately determine their absolute position in three-dimensional or two-dimensional space.

[0030] In embodiments, one or more images in the image set are modified to simulate alternative weather and / or lighting conditions. For example, video footage shot during the day can be fed into a filter that darkens the images in the video footage, simulating what the recorded environment looks like at sunset or twilight. As another example, video footage shot during a clear blue day can be fed into a filter that adds simulated clouds, rain, and / or fog to the video footage. In these ways, if the recorded environment includes rain or poor lighting conditions during a positioning session of a moving object, then simulated video footage can be used to correctly position the moving object.

[0031] In an embodiment, the image set includes an image set corresponding to video clips of objects moving in different directions. For example, image database 110 stores (1) a first image set of one or more objects moving from point A to point B and (2) a second image set of one or more objects moving from point B to point A.

[0032] If the image database 110 includes image sets from different angles, then the number of image sets required to train the ML model 130 (and achieve a certain level of accuracy and usability) may be reduced compared to the number of image sets required if the image sets vary less.

[0033] Model training

[0034] ML model trainer 120 trains ML model 130 based on images in image database 110 using one or more machine learning techniques. ML model 130 can be any type of model, an example being an extended optical flow neural network. Examples of such neural networks include FlowNetS, FlowNet2C, and FlowNet2SD. Such neural networks are trained by displaying images in pairs to the neural network. As indicated herein, these images are captured from a known environment and are accompanied by camera position information (CPI). CPI includes yaw, pitch, and roll. CPI can be recorded using external sensors such as GPS (Global Positioning System), INS (Inertial Navigation System), or RTK (Real-Time Kinematics). Alternatively, CPI is generated offline using techniques such as DGPS (Differential GPS), PPK (Post-Processing Kinematics), or SFM (Structure of Motion Recovery). These images may have been captured at different times, under different lighting and weather conditions, and using different cameras.

[0035] In this embodiment, the architecture of the ML model 130 is based on a predefined or “off-the-shelf” neural network. For example, the number of layers and the number of neurons per layer, as well as at least some of the weights connecting the neurons, are predefined. However, during the training process, the ML model trainer 120 modifies these weights, such as through backpropagation. Furthermore, the last layer of the model architecture is changed to be able to predict multiple values, such as longitudinal distance (s), lateral distance (d), and the camera angle.

[0036] Each image used to train the ML model 130 is associated with a location. This location can be an absolute location in space, such as a GPS location with latitude and longitude coordinates. Alternatively, the location can be a relative location relative to a fixed geographic location, such as the starting position on a race track. Alternatively, the location can be a location whose measurements are only related to a known environment, such as the distance from the starting position along an arc and the distance from the arc, where the arc defines a known path of movement for a moving object determining its position. In addition to the location coordinates (whether (x, y) or (s, d)), each image is also associated with the angle of the camera relative to a given reference plane. This is important because the implementation predicts the angle of the moving object, which is used to describe the state of the moving object in its physical environment. Two moving objects may be in the same physical location (at different times) but pointing in very different directions. Therefore, the angle is important for calculating localization.

[0037] The training data for ML model 130 includes multiple training instances, each consisting of a pair of images (from image database 110) with known locations. The pair of images can be selected based on their distance relative to each other. For example, if two images are within a threshold distance of each other, then these two images are added as a pair for training purposes. Therefore, a single image can be included in many pairs used to train ML model 130.

[0038] In a relevant embodiment, the images in each pair of images used to train the ML model 130 come from different video clips that may have at least slightly different trajectories. For example, the first video clip may be along the left boundary of the track (or a recognized path of movement), the second video clip may be along the right boundary of the track, and the third video clip may be along the center of the track. Thus, given an image pair {v1, v2} used to train the ML model 130, v1 and v2 come from different video clips (and therefore have different trajectories) relative to each other.

[0039] ML model 130 is trained to "learn" the physical distance and angular differences between two images, where the position and angle of one image are known, and the position and angle of the other image are unknown. During training, since the positions of both images are known, and therefore the positional differences are known, ML model trainer 120 uses the output of a pre-trained version of ML model 130 to output a distance (i.e., a "guess") and compares that distance to the known differences. ML model trainer 120 uses the result of the comparison to modify the weights of ML model 130, such as the weights associated with different neurons in the neural network. The larger the comparison result, the larger the weight modification. Therefore, if the comparison result is 0, then ML model trainer 120 may not modify any weights.

[0040] Once trained, the ML model 130 processes image pairs and predicts the spatial correlation (or “displacement”) between the locations of the images taken from each pair. This spatial correlation can be in the form of (a) a 3D or 2D transformation or (b) a custom correlation, such as differences along a custom axis and angles of rotation. This spatial correlation is then converted into a relative position. Estimating relative positions tends to perform better than estimating absolute positions, which are often affected by drift. Example spatial correlations or displacements are sets of three values: s, d, and Where s is the longitudinal distance along the curve or predefined path, d is the lateral distance from the curve, and It's a perspective.

[0041] In a related embodiment, while the ML model 130 can output a displacement that is subsequently added to a known relative position to generate a predicted relative position, this predicted relative position can be converted into an absolute position, such as GPS coordinates. This conversion can be useful later when working with kinematic equations. For example, the ML model 130 is used to predict a relative position along a trajectory, which is then converted into GPS coordinates input to another model.

[0042] If the camera position of the first image in an image pair is known, then the camera position of the second image in the image pair can automatically become known, or at least be automatically predicted. This is done by comparing the position of (a) the "known image" (which is an image with a known position) with (b) the predicted spatial correlation (e.g., The location of an unknown image (which is an image with an unknown location) is obtained by combining various images (or "frames") with known locations. The location of the unknown image is calculated using various images (or "frames") with known locations, and the example achieves an accuracy of a few centimeters (<10) even at speeds up to 300 km / h (83 m / s) or higher and from images captured in segments of 30 FPS (moving more than 2.5 m per frame at maximum speed).

[0043] Reasoning time

[0044] Input image data source 140 is the source of input images used by ML model 130 to calculate the predicted location. Input image data source 140 may be a digital camera mounted on a moving object in the ongoing positioning session. Alternatively, input image data source 140 may be a storage device local to the candidate image selector 150, storing images streamed from the digital camera. Thus, while the candidate image selector 150 retrieves and processes the first digital image from input image data source 140, additional images are being stored (or uploaded) to input image data source 140, where the additional images originate from the same input data stream. For example, a wireless connection may exist between input image data source 140 and the digital camera that is generating the images stored in input image data source 140.

[0045] The candidate image selector 150 retrieves the input image from the input image data source 140 and determines the estimated location of the input image. The estimated location of the input image can be determined in one of several ways. If the input image is the first image in the video stream, then it can be assumed that the moving object begins at a starting position associated with location data in the same units as the location data associated with the training images used to train the ML model 130, such as GPS coordinates (an example of an absolute location) or {s,d} positions (an example of a relative location). If the starting position is unknown, but the first image is known to be located somewhere in a known area or space (e.g., a race track) through which the object has passed, then the first image can be compared with all images in an image library (e.g., image database 110), and the location of the most similar image can be used as the starting position.

[0046] If the input image is not the first image in the video stream, then the estimated position of the input image is based on the predicted position of one or more previous input images or each of the images received / processed before the current input image. The estimated position can then be an extrapolation of (one or more) previous predicted positions. For example, assuming v3 is on the same line as v1 and v2, the estimated position of v3 (the current input image) is determined using the predicted positions of v1 and v2. The estimated position can also be based on the velocity or rate determined for a moving object. For example, velocity can be calculated by dividing (a) the difference in predicted positions of the last two images by (b) the time difference between the last two images. The velocity of a moving object can be helpful in scenarios where the moving object is accelerating or decelerating. Accurate estimated positions of moving objects are useful when selecting a suitable set of candidate images to be input into the ML model 130.

[0047] Candidate image selection

[0048] Given the estimated location of an input image, the candidate image selector 150 selects a set of candidate images based on the estimated location. The candidate image set comes from the same image set used to train the ML model 130. Therefore, the candidate image set comes from the image database 110 and is an exact subset of the image set used to train the ML model 130. Thus, the images in the image database 110 not only serve as training images for training the ML model 130 but also as reference images. The selected set of candidate images is based on the known locations of the candidate images in this set. The closer the location of an image is to the estimated location, the more likely the candidate image selector 150 is to select that image as a candidate image.

[0049] The candidate image selector 150 may use a threshold distance value when selecting a set of candidate images. For example, if an image in the image database 110 is associated with a location within a threshold distance of the estimated location, then the candidate image selector 150 adds that image to the (initially empty) set of candidate images.

[0050] Alternatively or additionally, the candidate image selector 150 employs a threshold number such that it selects a number of candidate images equal to the threshold number. For example, if the threshold number is 7, then the candidate image selector 150 selects 7 images (in the image database 110) whose known locations are closest to the estimated locations of the input images.

[0051] As another example, candidate image selector 150 identifies (in image database 110) all images whose known locations are within a threshold distance of the estimated location of the input image. If the number of identified images exceeds the threshold, candidate image selector 150 removes one or more images from the identified images until the number of remaining images equals the threshold. The removed image may be the one whose known location is furthest from the estimated location compared to the locations of the remaining images. Alternatively, images may be randomly selected for removal.

[0052] If the number of images identified within a threshold distance is less than a threshold number (e.g., 1), then the candidate image selector 150 increases the threshold distance (at least for that input image, but not necessarily for subsequent input images) and then identifies zero or more additional images. If one or more additional images are identified due to the increase in the threshold distance, then the candidate image selector 150 determines whether the total number of identified images is equal to the threshold number. If so, then processing for the current input image stops. Otherwise, the threshold distance is increased again, and the process of identifying additional images is repeated. One reason for not increasing the search radius is that increasing the search radius forces the model to extrapolate (i.e., make predictions outside the radius where the model is trained to perform well), leading to larger errors.

[0053] In this embodiment, the images in the image database 110 are indexed based on their location data. The location data can indicate an absolute or relative location. The location data can be a single value or multiple values. Indexing images based on location data speeds up the candidate image selection process, so that the candidate image selector 150 does not have to scan every image in the image database 110 for each candidate image. For example, the location data can be indicated by GPS coordinates. As another example, the location data can include values ​​of variables s and d, where (1) s indicates the distance from the starting point along a predefined path, curve, or trajectory, and (2) d indicates the distance from a point s on the predefined path, where the line from s to d is perpendicular to the tangent of the predefined path at s. If the images in the image database 110 are indexed by s, then given an estimated location (s... e d e Candidate image selector 150 can use s e To identify its s in s e +T s and s e –T s One or more images within T, where T s It is the threshold distance value along a predefined path. If the images in the image database 110 are also indexed by d, then for the candidate image selector 150, it is based on s. e For each image identified, the candidate image selector 150 can determine whether the image has a value between d and d. e +T d and d e –T d The d values ​​between T, where T d It is the threshold horizontal distance. If d e If the object is near the edge or boundary of a predefined path, then T can be increased or decreased. d One possible approach for fast nearest neighbor search is a KD-tree, a binary tree where each node represents a k-dimensional point. Each non-leaf node in the binary tree acts as a hyperplane, dividing the space into two partitions. This hyperplane is perpendicular to a chosen axis, which is associated with one of the K dimensions.

[0054] Example candidate image selection processing

[0055] Figure 2 This is a diagram depicting how candidate images are selected in the embodiment. Lines 202 and 204 represent edges or boundaries that the moving object does not intend (or expects) to cross. Therefore, the moving object is to remain between lines 202 and 204 throughout the movement. The area between lines 202 and 204 represents a portion of the entire track that the moving object can pass through (or traverse).

[0056] Lines 210-230 represent the trajectories traversed by one or more moving objects in the past. A digital camera (attached to one or more moving objects) generates digital images during these corresponding movements. An ML model trainer 120 uses the generated digital images to train a neural network (e.g., ML model 130).

[0057] Objects 212 and 214 represent points on trajectory 210 from which digital images were generated. These images are associated with corresponding locations or positions on trajectory 210. Therefore, object 214 represents a time after the time represented by object 212. The digital images associated with objects 212 and 214 originate from the same video clip. Similarly, objects 222 and 224 represent points on trajectory 220 from which digital images were generated. These images are associated with corresponding locations or positions on trajectory 220. Therefore, object 224 represents a time after the time represented by object 222. The digital images associated with objects 222 and 224 originate from the same video clip, but from a different video clip than the one from which the digital images associated with objects 212 and 214 originate. Similarly, object 232 represents a point on trajectory 230 from which a digital image was generated. This image is associated with a specific location or position on trajectory 230 and originates from a different video clip than the previous two.

[0058] Object 240 represents the previous position of the moving object at time t-1, which may be a predicted position using the embodiments described herein. The value of t may be seconds, milliseconds, or some other unit of time. Therefore, the value of t-1 may not be literal t-1, but rather a value less than t that indicates a time prior to time t.

[0059] Point 242 represents the estimated position of the moving object at time t. Candidate image selector 150 (or another component of the positioning system 100) determines the estimated position based on the previous position and one or more other factors, such as the estimated velocity of the moving object and / or one or more other positions prior to time t-1. Search region 244 represents the area (around point 242) in which candidate image selector 150 searches for candidate images that have been used to train the neural network. In this example, candidate image selector 150 selects five digital images, namely, the five digital images corresponding to objects 212, 214, 222, 224, and 232.

[0060] Object 250 represents the actual position of a moving object at time t. Object 250 is a moving object whose position is predicted based on a digital image generated at time t. As described in more detail herein, the ML model caller 160 inputs this digital image (referred to as the "current" image) along with candidate images into a neural network (e.g., ML model 130), and the neural network outputs a predicted displacement used to calculate an absolute position (e.g., latitude and longitude coordinates) or a relative position (e.g., s and d values) on a predefined path. The ML model caller 160 repeats this process for each candidate image in the candidate image set, thereby producing a set of predicted positions. Thus, the current image is input into the neural network multiple times to obtain multiple predicted positions of object 250 at time t.

[0061] The final predicted location 252 is determined based on this set of predicted locations. This determination may involve performing one or more operations, such as aggregation operations, as described in more detail herein. In this example, the final predicted location 252 differs from the actual location of the moving object represented by object 250. This difference may be due to prediction errors.

[0062] Calling ML Model

[0063] After candidate image selector 150 selects a set of candidate images, ML model invoker 160 invokes ML model 130 by passing the candidate image and the input image as input. This invocation is repeated for each candidate image in the candidate image set. Candidate image selector 150 (or another component of positioning system 100) invokes or triggers ML model invoker 160. Thus, candidate image selector 150 can pass the input image and a single candidate image to ML model invoker 160. This passing can be performed while candidate image selector 150 is selecting candidate images given the input image. Alternatively, this passing can be performed only after candidate image selector 150 has finished selecting the set of candidate images.

[0064] As an alternative to passing a pair of images to the ML model caller 160 at a time (which would invoke the ML model 130 for each pair), the candidate image selector 150 can pass the input image and the entire set of candidate images selected by the candidate image selector 150 (i.e., given the input image) in a single call or transfer. In this scenario, the ML model caller 160 determines which image is the input image and which is part of the candidate image set. Subsequently, for each candidate image in the set, the ML model caller 160 invokes the ML model 130 by passing the candidate image and the input image.

[0065] Each invocation of ML model 130 produces an output (model output 162) based on which, given an input image, the predicted position of the moving object is ultimately determined. Therefore, if there are ten candidate images given an input image, ML model 130 is invoked ten times, generating ten model output instances, each including incremental or displacement information.

[0066] Therefore, each instance of model output 162 is a predicted displacement, which includes one or more values, such as increment / displacement position values ​​(which may include multiple values) and increment / displacement angles. To generate the (intermediate) predicted position of the input image, ML model invoker 160 (or another component of positioning system 100) adds the predicted displacements (which may be positive or negative) to the known positions of the input candidate images used to generate the predicted displacements. For each invocation, ML model 130 may output multiple values, each corresponding to a different component or dimension of the position data. For example, the output of a single invocation of ML model 130 is an s-value, a d-value, and a ψ-value. These three predicted displacements are then added to the corresponding known position values ​​of the input candidate images to compute the predicted position (comprising the three position values) associated with the input image.

[0067] polymerization

[0068] In this embodiment, aggregator 170 performs an aggregation operation on multiple predicted locations determined for the input image to generate final predicted locations. The aggregation operation can be a mean operation, a median operation, or a weighted average operation. If it is a weighted average operation, the weights applied to different predicted locations can vary depending on the distance between the corresponding candidate image and the estimated location of the input image. For example, if the location of the first candidate image is closer to the estimated location than the location of the second candidate image, then the location of the first candidate image is given a higher weight than the location of the second candidate image.

[0069] In this embodiment, the predicted increments / displacements from the ML model 130 are first aggregated. In this case, the aggregated increments / displacements will be the increments from the midpoints between all input candidate images. The absolute position of the midpoint can be calculated as the average of all absolute positions of the candidate images. The aggregated increments / displacements are then added to the absolute position of the midpoint. However, the embodiments described below are based on a method where aggregation occurs after the predicted increments are respectively added to the absolute positions of their corresponding candidate images.

[0070] In scenarios where the predicted location includes multiple values, multiple aggregation operations are then performed, one for each data component at each location. For example, one aggregation operation is performed on the predicted 's' value, another aggregation operation (of the same type) is performed on the predicted 'd' value, and so on. The value performs another aggregation operation.

[0071] In relevant embodiments, one or more predicted locations are not considered. For example, a predicted location may be associated with a confidence score output by the ML model 130 along with the predicted location. If the confidence score is below a certain threshold, the predicted location is not input into the aggregation operation.

[0072] The final predicted (and aggregated) position can be used in one or more ways. For example, positioning system 100 can display the final predicted position on a computer screen, indicating the location or position of a moving object on a track or in 3D space. The computer screen can be a handheld device containing control devices for controlling the movement of the moving object. Alternatively, the computer screen can be part of a computer monitor viewed by multiple people. As another example, positioning system 100 can transmit the final predicted position to an electronic receiver on the moving object, causing the moving object to make one or more adjustments based on the final predicted position, such as changing speed and / or changing direction. In this example, the goal of the computer software controlling the moving object could be to maintain a specific speed, get as close as possible to a predefined path (e.g., the center of the track), and avoid obstacles, such as the edges of the track and / or other moving objects. In the context of competitive racing, the final prediction can be used for further analysis. For example, if video clips of competitors in a racing competition are collected, these predictions may reveal details of their strategies, tactical choices, or driving styles of a particular driver, which could further help in defeating that driver in future races. Predictive data can also be used to analyze known vehicles / drivers to find new ways to optimize vehicle routes and / or speeds.

[0073] The following formula is one method for generating the final predicted location given an input image i:

[0074]

[0075] Where N is the number of candidate images in the candidate image set, s j These are candidate images from the candidate image set. The (unknown) position of input image i and candidate image s j The predicted displacement of s between (known) positions, and ψ is the final predicted position of the input image i. Similar formulas can be used for d and ψ, such as the following:

[0076]

[0077]

[0078] Example processing

[0079] Figure 3This is a flowchart depicting an example process 300 for predicting the position of a moving object in an embodiment. Process 300 can be performed by different components of the positioning system 100.

[0080] At box 310, a digital image is received. The digital image is received after training a neural network using one or more machine learning techniques. The entity performing processing 300 may be the same as or different from the entity used to train the neural network. The digital image may originate from a digital camera associated with (e.g., attached to) a moving object. The digital image may be received within milliseconds after the digital camera generates the digital image.

[0081] At box 320, a candidate image set is selected from the training image set used to train the neural network. Box 320 may involve determining the expected location of the received digital image, and then selecting the candidate image set based on a threshold distance between the expected location of the digital image and the absolute location of each candidate image in the candidate image set. Each training image is associated with a previously determined absolute location and associated with that training image.

[0082] If the number of candidate images initially selected is greater than a threshold number (e.g., ten), then a subset of these candidate images, such as the candidate image associated with the absolute position furthest from the expected position, can be removed. On the other hand, if the number of candidate images initially selected is less than a threshold number or zero, then the threshold distance can be increased (e.g., 20%) to identify one or more candidate images for selection.

[0083] The expected position of a received digital image can be determined in several ways, including based on the predicted position of each of one or more digital images received before the currently received digital image. For example, if the distance between the last two predicted positions is N, then the expected position of the received digital image could be the predicted position of the latest digital image on a line or arc that coincides with the last two predicted positions, plus N.

[0084] At box 330, a candidate image is selected from the candidate image set. Box 330 may involve randomly selecting a candidate image from the candidate image set. Alternatively, the candidate image set may first be sorted in order of nearest to farthest from the expected location of the received digital image, and the nearest candidate image (not yet processed in box 340) may be selected.

[0085] At box 340, the selected candidate image and the received digital image are input into a trained neural network that produces an output used to generate a predicted position of the moving object at the time corresponding to the received digital image. Box 340 may involve the trained neural network outputting a predicted displacement, which is added to the known position of the selected candidate image to calculate the predicted position.

[0086] At box 350, the predicted location is added to the predicted location set. Initially, this predicted location set is empty. However, for each subsequent 350 iterations with respect to a particular received digital image, another predicted location is added to the non-empty predicted location set.

[0087] At box 360, determine if there are more candidate images available in the candidate image set. If so, process 300 returns to box 330; otherwise, process 300 proceeds to box 370.

[0088] At box 370, the predicted positions in the set of predicted positions are aggregated to generate aggregated positions of the received digital image. Box 370 may involve averaging the predicted positions or calculating a weighted average of the predicted positions, wherein predicted positions associated with absolute positions closer to the expected position have a higher weight than predicted positions associated with absolute positions farther from the expected position.

[0089] At box 380, the aggregated location is associated with the moving object. Box 380 may involve storing the aggregated location along with time data indicating the current time or the time when the positioning system 100 received the digital image (received in box 310) or the time when the digital image was generated. Box 380 may also involve storing an object identifier that uniquely identifies the moving object.

[0090] After box 380, in response to the detection of another digital image to be processed, processing 300 can return to box 310.

[0091] advantage

[0092] Existing solutions to the localization problem involve using visual SLAM or offline map building and online localization-based methods. These methods rely on finding features in the input image / video, combining these features into a map, and finding the camera's current location on the map by associating observed features with features observed in the map.

[0093] The implementation is superior to current methods of this kind because it uses dense image data to locate moving objects, while previous methods used sparse feature data.

[0094] The embodiment also outperforms the current method because it provides an accuracy level down to a few centimeters at every step. Experiments show that the embodiment provides an accuracy of approximately 3–10 cm for vehicles moving at speeds up to 300 km / h (83 m / s), with the camera operating at an FPS range of 30 to 60 FPS (2.5–1.2 m of movement per frame), and may also capture movement outside this range. This level of accuracy at high speeds has been confirmed by over one hundred experiments using dozens of videos captured by a camera mounted on a car, which, as an example / test case, has been traveling at the aforementioned speeds on several closed-loop roads with known geometry. The current method provides high-accuracy localization only intermittently when there is a unique correlation between features in the current image and the map, and loses localization after a few seconds at such high speeds.

[0095] The implementation examples can be applied to different use cases or scenarios, each requiring a solution to determine the position of a moving object in a new environment using a camera mounted on it. Examples of such use cases include autonomous navigation for drones, robots, and self-driving cars; vehicle trajectory analysis; augmented reality; and surveying and mapping in GPS-free environments.

[0096] Hardware Overview

[0097] According to one embodiment, the techniques described herein are implemented by one or more dedicated computing devices. The dedicated computing device may be hardwired to execute the techniques, or may include digital electronic devices permanently programmed to execute the techniques, such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs), or may include one or more general-purpose hardware processors programmed to execute the techniques according to program instructions in firmware, memory, other storage devices, or combinations thereof. Such a dedicated computing device may also combine custom hardwired logic, ASICs, or FPGAs with custom programming to implement the techniques. The dedicated computing device may be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that combines hardwired and / or program logic to implement the techniques.

[0098] For example, Figure 4 This is a block diagram illustrating a computer system 400 on which embodiments of the present invention may be implemented. The computer system 400 includes a bus 402 or other communication mechanism for transmitting information and a hardware processor 404 coupled to the bus 402 for processing information. The hardware processor 404 may be, for example, a general-purpose microprocessor.

[0099] Computer system 400 also includes main memory 406, such as random access memory (RAM) or other dynamic storage devices, coupled to bus 402 for storing information and instructions to be executed by processor 404. Main memory 406 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 404. When such instructions are stored in non-transitory storage media accessible to processor 404, such instructions make computer system 400 a dedicated machine customized to perform the operations specified in the instructions.

[0100] The computer system 400 also includes a read-only memory (ROM) 408 or other static storage device coupled to the bus 402 for storing static information and instructions of the processor 404. A storage device 410, such as a disk, optical disk, or solid-state drive, is provided and coupled to the bus 402 for storing information and instructions.

[0101] Computer system 400 may be coupled via bus 402 to a display 412, such as a cathode ray tube (CRT), for displaying information to a computer user. Input device 414, including alphanumeric keys and other keys, is coupled to bus 402 for transmitting information and command selections to processor 404. Another type of user input device is a cursor control device 416, such as a mouse, trackball, or arrow keys, for transmitting directional information and command selections to processor 404 and for controlling cursor movement on display 412. Such input devices typically have two degrees of freedom on two axes (a first axis (e.g., x) and a second axis (e.g., y)) to allow the device to specify a position in a plane.

[0102] Computer system 400 may implement the techniques described herein using custom hard-wired logic, one or more ASICs or FPGAs, firmware, and / or program logic, which, in combination with the computer system, enable computer system 400 to become a special-purpose machine or to program computer system 400 as a special-purpose machine. According to one embodiment, the techniques herein are executed by computer system 400 in response to processor 404 executing one or more sequences of one or more instructions contained in main memory 406. These instructions may be read into main memory 406 from another storage medium, such as storage device 410. Execution of the sequence of instructions contained in main memory 406 causes processor 404 to perform the processing steps described herein. In alternative embodiments, hard-wired circuitry may be used instead of software instructions or in combination with software instructions.

[0103] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that enable a machine to operate in a particular manner. Such storage media can include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs, magnetic disks, or solid-state drives, such as storage device 410. Volatile media include dynamic memory, such as main memory 406. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips, or magnetic tape cassettes.

[0104] Storage media differ from transmission media but can be used in conjunction with them. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including wires containing bus 402. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0105] Various forms of media can involve carrying one or more sequences of instructions to processor 404 for execution. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit them over a telephone line using a modem. A modem local to computer system 400 may receive data over the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal, and appropriate circuitry may place the data on bus 402. Bus 402 carries the data to main memory 406, from which processor 404 retrieves and executes the instructions. The instructions received by main memory 406 may optionally be stored on storage device 410 before or after execution by processor 404.

[0106] Computer system 400 also includes a communication interface 418 coupled to bus 402. Communication interface 418 provides bidirectional data communication coupled to network link 420, which is connected to local network 422. For example, communication interface 418 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem providing data communication connectivity to a corresponding type of telephone line. As another example, communication interface 418 may be a LAN card providing data communication connectivity to a compatible local area network (LAN). A wireless link may also be implemented. In any such implementation, communication interface 418 transmits and receives electrical, electromagnetic, or optical signals carrying streams of digital data representing various types of information.

[0107] Network link 420 typically provides data communication to other data devices via one or more networks. For example, network link 420 may provide a connection to host computer 424 or to data devices operated by Internet Service Provider (ISP) 426 via local network 422. ISP 426, in turn, provides data communication services via a global packet data communication network now commonly referred to as the “Internet” 428. Both local network 422 and Internet 428 use electrical, electromagnetic, or optical signals that carry digital data streams. Signals through various networks, as well as signals on network link 420 and through communication interface 418, are example forms of transmission media that carry digital data to or from computer system 400.

[0108] Computer system 400 can send messages and receive data, including program code, through one or more networks, network links 420, and communication interfaces 418. In the Internet example, server 430 can transmit requested code for the application through the Internet 428, ISP 426, local network 422, and communication interface 418.

[0109] The received code can be executed by processor 404 when it is received, and / or stored in storage device 410 or other non-volatile storage device for later execution.

[0110] According to some embodiments, a system including one or more computing devices has means for performing operations, including the operations recited in any method claim. According to other embodiments, a computer program product includes instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, including the operations recited in any method claim. According to still other embodiments, a computer-readable medium includes instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, including the operations recited in any method claim.

[0111] Software Overview

[0112] Figure 5 This is a block diagram of a basic software system 500 that can be used to control the operation of computer system 400. The software system 500 and its components, including their connections, relationships, and functions, are intended to be exemplary only and are not intended to limit the implementation of one or more of the example embodiments. Other software systems suitable for implementing one or more of the example embodiments may have different components, including components with different connections, relationships, and functions.

[0113] Software system 500 is provided to guide the operation of computer system 400. Software system 500, which may be stored on system memory (RAM) 406 and fixed storage device (e.g., hard disk or flash memory) 410, includes a kernel or operating system (OS) 510.

[0114] OS 510 manages the low-level aspects of computer operations, including managing process execution, memory allocation, file input and output (I / O), and device I / O. One or more applications, denoted as 502A, 502B, 502C…502N, can be “loaded” (e.g., transferred from fixed storage device 410 to memory 406) for execution by system 500. Applications or other software intended for use on computer system 400 can also be stored as downloadable computer-executable instruction sets, for example, for downloading and installation from Internet locations (e.g., web servers, app stores, or other online services).

[0115] Software system 500 includes a graphical user interface (GUI) 515 for receiving user commands and data graphically (e.g., "click" or "touch gestures"). These inputs can then be manipulated by system 500 according to instructions from operating system 510 and / or (one or more) applications 502. GUI 515 also displays the results of operations from OS 510 and (one or more) applications 502, to which the user can provide additional input or terminate the session (e.g., log off).

[0116] OS 510 can execute directly on the bare hardware 520 of computer system 400 (e.g., one or more processors 404). Alternatively, a hypervisor or virtual machine monitor (VMM) 530 can be inserted between the bare hardware 520 and OS 510. In this configuration, VMM 530 acts as a software "buffer" or virtualization layer between OS 510 and the bare hardware 520 of computer system 400.

[0117] VMM 530 instantiates and runs one or more virtual machine instances (“guest machines”). Each guest machine includes a “guest” operating system (such as OS 510) and one or more applications designed to run on the guest operating system, such as application(s)502. VMM 530 presents a virtual operating platform to the guest operating system and manages the execution of the guest operating system.

[0118] In some instances, VMM 530 can allow a guest operating system to run as if it were running directly on the bare hardware 520 of computer system 400. In these instances, the same version of the guest operating system configured to run directly on the bare hardware 520 can also run on VMM 530 without modification or reconfiguration. In other words, VMM 530 can provide full hardware and CPU virtualization to the guest operating system in some instances.

[0119] In other instances, the guest operating system can be specifically designed or configured to run on VMM 530 for improved efficiency. In these instances, the guest operating system is "aware" that it is running on the virtual machine monitor. In other words, VMM 530 can provide paravirtualization to the guest operating system in some instances.

[0120] Computer system processes include the allocation of hardware processor time, as well as the allocation of (physical and / or virtual) memory, memory allocation for storing instructions executed by the hardware processor, memory allocation for storing data generated by the execution of instructions by the hardware processor, and / or memory allocation for storing hardware processor state (e.g., register contents) between hardware processor time allocations when the computer system process is not running. Computer system processes run under the control of the operating system and can also run under the control of other programs executing on the computer system.

[0121] The basic computer hardware and software described above are intended to illustrate the basic underlying computer components that can be used to implement one or more example embodiments. However, the one or more example embodiments are not necessarily limited to any particular computing environment or computing device configuration. Instead, the one or more example embodiments can be implemented in any type of system architecture or processing environment that a person skilled in the art will understand from this disclosure as capable of supporting the features and functionality of the one or more example embodiments presented herein.

[0122] cloud computing

[0123] This article generally uses the term "cloud computing" to describe a computing model that enables on-demand access to a shared pool of computing resources, such as computer networks, servers, software applications, and services, and allows for the rapid provisioning and release of resources with minimal management effort or service provider interaction.

[0124] Cloud computing environments (sometimes called cloud environments or the cloud) can be implemented in various ways to best suit different requirements. For example, in a public cloud environment, the underlying computing infrastructure is owned by an organization that makes its cloud services available to other organizations or the general public. In contrast, private cloud environments are generally designed for use by a single organization or within a single organization. Community clouds are designed to be shared by several organizations within a community; while hybrid clouds include two or more types of clouds (e.g., private, community, or public) bound together by data and application portability.

[0125] Generally, cloud computing models enable some of the responsibilities that might have previously been provided by an organization's own IT department to be delivered as service layers within a cloud environment for use by consumers (inside or outside the organization, depending on the public / private nature of the cloud). Depending on the specific implementation, the precise definition of the components or features provided by or within each cloud service layer can vary, but common examples include: Software as a Service (SaaS), where consumers use software applications running on cloud infrastructure, while the SaaS provider manages or controls the underlying cloud infrastructure and applications; Platform as a Service (PaaS), where consumers can use software programming languages ​​and development tools supported by the PaaS provider to develop, deploy, and otherwise control their own applications, while the PaaS provider manages or controls other aspects of the cloud environment (i.e., everything in the runtime execution environment); and Infrastructure as a Service (IaaS), where consumers can deploy and run arbitrary software applications and / or provide processing, storage, networking, and other basic computing resources, while the IaaS provider manages or controls the underlying physical cloud infrastructure (i.e., everything below the operating system layer). Database as a Service (DBaaS) is a service in which consumers use database servers or database management systems running on cloud infrastructure, while DbaaS providers manage or control the underlying cloud infrastructure, applications, and servers, including one or more database servers.

[0126] In the foregoing description, embodiments have been described with reference to numerous specific details that vary depending on the implementation. Accordingly, the description and drawings should be considered illustrative rather than restrictive. The sole and exclusive reference to the scope of the invention, and the content that the applicant intends to define as the scope of the invention, is the literal and equivalent scope of the set of claims issued in this application, in the specific form of such claims, including any subsequent corrections.

Claims

1. A method comprising: Identify specific images associated with moving objects; Select a set of candidate images from a plurality of images used to train a neural network; For each candidate image in the candidate image set: The output from the neural network is generated by inputting a specific image and each of the candidate images into the neural network. Based on the output and the location associated with each candidate image, the predicted location of a specific image is determined; Add the predicted location to the predicted location set; Aggregate the predicted location set to generate aggregated locations for a specific image; Associate the aggregation location with the moving object; The method is performed by one or more computing devices.

2. The method of claim 1, further comprising: For each of the plurality of images: Determine the position of each image; Identify the set of images whose locations are within a threshold distance from the location of each of the images; Store paired data that associates the image set with each of the images; The neural network is trained based on paired data associated with each of the plurality of images.

3. The method of any one of claims 1-2, wherein selecting the candidate image set comprises: Determine the estimated location of a specific image; The candidate image set is selected based on a threshold distance between the estimated location of a specific image and the location of each candidate image in the candidate image set.

4. The method of claim 3, wherein: Each of the plurality of images is associated with location data indicating the location of the object; The method further includes storing an index that indexes the plurality of images based on location data associated with each of the plurality of images; The selection of the candidate image set includes identifying the candidate image set in the index using estimated locations.

5. The method of claim 3, wherein determining the expected location of a particular image comprises determining an estimated location based on one or more predicted locations of one or more images prior to the particular image.

6. The method of claim 3, wherein selecting the candidate image set based on a threshold distance comprises: Selecting multiple candidate images based on threshold distance; Determine the number of candidate images among the plurality of candidate images; In response to the determination that the number of candidate images is greater than a certain threshold, a set of candidate images whose corresponding positions are closest to the estimated positions is selected.

7. The method of claim 3, wherein selecting the candidate image set based on a threshold distance comprises: It was determined that no candidate image was within the threshold distance; In response to determining that no candidate image is within the threshold distance, the threshold distance is increased to a larger threshold distance, and one or more candidate images within the larger threshold distance are included in the candidate image set.

8. The method as described in any one of claims 1-7, wherein: The aggregation location includes the s-value and the d-value; The s value is the first distance from the starting point on the predefined path; The d value is the second distance from the point indicated by the s value on the predefined path.

9. The method of any one of claims 1-8, wherein the output includes (1) the displacement of the position of each candidate image and (2) the displacement of the angle of each candidate image.

10. The method of any one of claims 1-9, wherein aggregating the predicted location set includes calculating an average value of the predicted location set.

11. The method of any one of claims 1-9, further comprising: For each candidate image in the candidate image set, determine the estimated distance between the position of each candidate image and the position of the specific image; The aggregated set of predicted locations comprises a weighted average of the predicted locations calculated based on the estimated distances associated with each candidate image in the candidate image set.

12. One or more storage media storing instructions that, when executed by one or more computing devices, cause to perform the method as described in any one of claims 1-11.

13. A system comprising: One or more computing devices; One or more storage media storing instructions that, when executed by the one or more computing devices, cause the method as described in any one of claims 1-11 to be performed.

Images