Annotating a map

Abstract

The present invention relates to a computer-implemented method of annotating a map used to navigate an autonomous vehicle, AV, with a three-dimensional location of an object. The computer-implemented method comprises: receiving a plurality of images captured by a camera of the AV, each image of the plurality of images associated with a pose of the AV; identifying an object in at least some of the plurality of images; determining, at each pose of the AV, a two- dimensional position of the identified object; generating a hypothesis for a three-dimensional position of the object based on at least some of the two-dimensional positions of the object; and annotating the map with the object at the three-dimensional position of the hypothesis.

Description

ANNOTATING A MAPFIELD

[0001] The subject-matter of the present disclosure relates to annotating maps, more specifically computer-implemented methods of annotating a map used to navigate an autonomy vehicle, AV with a three-dimensional location of an object, and transitory, or non-transitory, computer readable media.BACKGROUND

[0002] Autonomous vehicles, AVs, oftentimes require a map of an environment in order to navigate. The map is typically annotated with objects included in the environment. Annotation is typically done manually. Such manual annotation takes a lot of time meaning that an autonomy stack used to navigate the AV will take a long time to update when operating in new environments.

[0003] It is an aim of the present invention to address such problems and improve on the prior art.SUMMARY

[0004] According to an aspect of the present disclosure, there is provided a computer- implemented method of annotating a map used to navigate an autonomous vehicle, AV, with a three-dimensional location of an object, the computer-implemented method comprising: receiving a plurality of images captured by a camera of the AV, each image of the plurality of images associated with a pose of the AV; identifying an object in at least some of the plurality of images; determining, at each pose of the AV, a two-dimensional position of the identified object; generating a hypothesis for a three-dimensional position of the object based on at least some of the two-dimensional positions of the object; and annotating the map with the object at the three- dimensional position of the hypothesis. Annotating the map in this way provides a means of automatic annotation meaning that the AV is extendable to new operating regions more easily.

[0005] In an embodiment, the generating a hypothesis for a three-dimensional position of the object based on at least some of the two-dimensional positions of the object comprises: generating a hypothesis for a three-dimensional position of the object based on differences in directions between the pose of the AV and the two-dimensional position of the object for at least some of the poses of the AV.

[0006] In an embodiment, the computer-implemented method further comprises: generating a plurality of rays, wherein each ray includes a position of the AV at the pose and a direction from the position of the AV and the two-dimensional position of the identified object.

[0007] In an embodiment, the generating the plurality of rays, comprises: generating a detector score for the identified object; comparing the detector score to a detector score threshold; andgenerating, at each pose, the ray only when the detector score is greater than or equal to the detector score threshold.

[0008] In an embodiment, the generating a hypothesis for a three-dimensional position of the object based on differences in directions between the pose of the AV and the two-dimensional position of the object for at least some of the poses of the AV comprises, for each ray: determining a shortest distance to another ray; and generating a hypothesis for the three-dimensional position of the object at a point along the shortest distance.

[0009] In an embodiment, the computer-implemented method further comprises: comparing the shortest distance to a hypothesis intersection distance threshold; for any rays resulting in the shortest distance being less than the hypothesis intersection distance threshold, generating the hypothesis for the three-dimensional position of the object at a point along the shortest distance; and for any rays resulting in the shortest distance being greater than or equal to the hypothesis intersection distance threshold, not generating the hypothesis for the three-dimensional position of the object at a point along the shortest distance.

[0010] In an embodiment, the computer-implemented method further comprises: determining a pose distance between a pose of the ray and a pose of the other ray; comparing the pose distance to a maximum pose distance threshold; for any rays associated with a pose distance less than the pose distance threshold, generating the hypothesis for the three-dimensional position of the object at a point along the shortest distance; and for any rays associated with a pose distance greater than or equal to the pose distance threshold, not generating the hypothesis for the three- dimensional position of the object at a point along the shortest distance.

[0011] In an embodiment, the computer-implemented method further comprises selecting one or more of the hypotheses. The one or more selected hypotheses may be selected as the hypotheses for the object.

[0012] In an embodiment, selecting the one or more hypotheses may comprise: determining whether each ray supports each hypothesis by comparing a closest distance between the two- dimensional position of a ray and the respective hypothesis to a closeness threshold; determining that a ray supports a hypothesis if the closest distance is less than the closeness threshold; and determining that the ray supports the hypothesis if the closest distance is greater than or equal to the closeness threshold.

[0013] In an embodiment, the selecting the one or more hypotheses comprises: reducing a number of hypotheses based on numbers of rays that support each hypothesis.

[0014] In an embodiment, the reducing the number of hypotheses comprises using a greedy voting algorithm to vote for retaining one or more of the hypotheses based on a number of rays that support the hypothesis.

[0015] In an embodiment, the using the greedy voting algorithm comprises, iteratively: determining a number of two-dimensional positions of the rays supporting each hypothesis; identifying the hypothesis with a maximum number of supporting two dimensional positions; comparing the maximum number of supporting two-dimensional positions to a hypothesis threshold; discarding the hypothesis if the maximum number of supporting two-dimensional positions is less than the hypothesis threshold; if the maximum number of supporting two- dimensional positions is greater than or equal to the hypothesis threshold: retain the hypothesis; and set the two-dimensional positions supporting the hypothesis as not supporting any other hypothesis.

[0016] In an embodiment, the computer-implemented method further comprises: refining the three-dimensional position of the hypothesis using line intersection least squares on any rays that support the hypothesis.

[0017] In an embodiment, selecting the one or more hypotheses comprises: reducing a number of hypotheses using voxelisation.

[0018] In an embodiment, the voxelisation comprises: constructing a plurality of voxels; mapping each hypothesis positionally to a voxel of the plurality of voxels; and in each voxel, combining any hypotheses in the voxel to a single position.

[0019] In an embodiment, the determining, at each pose of the AV, a two-dimensional position of the identified object, comprises: constructing a bounding box around the detected object.

[0020] In an embodiment, the annotating the map with the object at the three-dimensional position of the hypothesis comprises: constructing, on the map, a three-dimensional bounding box having dimensions based on dimensions of the two-dimensional bounding box.

[0021] In an embodiment, the computer-implemented method further comprises: spatially sampling the plurality of images, and wherein the at least some of the plurality of images is the spatially sampled images.

[0022] According to an aspect of the present invention, there is provided a transitory, or non- transitory, computer-readable medium, having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform the computer- implemented method of any preceding aspect or embodiment.

[0023] According to an aspect of the present invention, there is provided a computer- implemented method of annotating a map used to navigate an autonomous vehicle, AV, with a three-dimensional location of an object, the computer-implemented method comprising: receiving a plurality of images captured by a camera of the AV, and a plurality of data points from a sensor having a different modality to the camera, wherein each image of the plurality of images and each data point is associated with a pose of the AV; identifying an object in the plurality of images; determining, at each pose of the AV, a two-dimensional position of the object wihtin the images;generating a hypothesis for a three-dimensional position of the object based on the two- dimensional positions and a depth obtained from the plurality of data points; and annotating the map with the object at the three-dimensional position of the hypothesis.

[0024] According to an aspect of the present invention, there is provided a computer- implemented method of annotating a map used to navigate an autonomous vehicle, AV, with a three-dimensional location of an object, the computer-implemented method comprising: receiving a plurality of images captured by a camera of the AV, wherein each image of the plurality of images is associated with a pose of the AV; identifying an object in the plurality of images; determining, at each pose of the AV, a two-dimensional position of the object using the images; generating a hypothesis for a three-dimensional position of the object based on the two- dimensional positions; and annotating the map with the object at the three-dimensional position of the hypothesis.BRIEF DESCRIPTION OF DRAWINGS

[0025] The subject-matter of the present disclosure is best described with reference to the accompanying figures, in which:

[0026] Figure 1 shows a schematic diagram of an AV according to one or more embodiments;

[0027] Figure 2 shows a flow chart showing a computer-implemented method of annotating a map with an object according to one or more embodiments;

[0028] Figure 3 shows a schematic diagram illustrating hypotheses and rays obtained as part of using the computer-implemented method of Figure 2; and

[0029] Figure 4 shows a flow chart summarising the computer-implemented method associated with the flow chart of Figure 2.DESCRIPTION OF EMBODIMENTS

[0030] At least some of the example embodiments described herein may be constructed, partially or wholly, using dedicated special-purpose hardware. Terms such as ‘component’, ‘module’ or ‘unit’ used herein may include, but are not limited to, a hardware device, such as circuitry in the form of discrete or integrated components, a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC), which performs certain tasks or provides the associated functionality. In some embodiments, the described elements may be configured to reside on a tangible, persistent, addressable storage medium and may be configured to execute on one or more processors. These functional elements may in some embodiments include, by way of example, components, such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data,databases, data structures, tables, arrays, and variables. Although the example embodiments have been described with reference to the components, modules and units discussed herein, such functional elements may be combined into fewer elements or separated into additional elements. Various combinations of optional features have been described herein, and it will be appreciated that described features may be combined in any suitable combination. In particular, the features of any one example embodiment may be combined with features of any other embodiment, as appropriate, except where such combinations are mutually exclusive. Throughout this specification, the term “comprising” or “comprises” means including the component(s) specified but not to the exclusion of the presence of others.

[0031] The embodiments described herein may be embodied as sets of instructions stored as electronic data in one or more storage media. Specifically, the instructions may be provided on a transitory or non-transitory computer-readable media. When executed by the processor, the processor is configured to perform the various methods described in the following embodiments. In this way, the methods may be computer-implemented methods. In particular, the processor and a storage including the instructions may be incorporated into a vehicle. The vehicle may be an autonomous vehicle (AV).

[0032] Whilst the following embodiments provide specific illustrative examples, those illustrative examples should not be taken as limiting, and the scope of protection is defined by the claims. Features from specific embodiments may be used in combination with features from other embodiments without extending the subject-matter beyond the content of the present disclosure.

[0033] With reference to Figure 1 , an AV 10 may include a plurality of sensors 12. The sensors 12 may be mounted on a roof of the AV 10. The sensors 12 may be communicatively connected to a computer 14. The computer 14 may be onboard the AV 10. The computer 14 may include a processor 16 and a memory 18. The memory may include the non-transitory computer-readable media described herein. The memory may also include an autonomy stack for operating the AV. Alternatively, the non-transitory computer-readable media may be located remotely and may be communicatively linked to the computer 14 via the cloud 20. The computer 14 may be communicatively linked to one or more actuators 22 for control thereof to move the AV 10. The actuators may include, for example, a motor, a braking system, a power steering system, etc.

[0034] The sensors 12 may include various sensor types. Examples of sensor types include LiDAR sensors, RADAR sensors, and cameras. Each sensor type may be referred to as a sensor modality. Each sensor type may record data associated with the sensor modality. For example, the LiDAR sensor may record LiDAR modality data.

[0035] The data may capture various scenes that the AV 10 encounters. For example, a scene may be a visible scene around the AV 10 and may include roads, buildings, weather, objects (e.g. other vehicles, pedestrians, animals, etc.), etc.

[0036] With reference to Figure 2, the computer-implemented method of annotating a map with an object starts at step 100. The term “computer-implemented method” may be used interchangeably with the term “method”.

[0037] The method includes receiving 102 a plurality of images captured by a camera of the AV, each image of the plurality of images associated with a pose of the AV. The method also includes receiving 104 data captured by a sensor of the AV at each pose of the AV. The sensor may be the LiDAR sensor, and the data may be a LiDAR point cloud.

[0038] The autonomy stack 106 generates 108 poses of the AV using the received images and the LiDAR point cloud data. An image may be captured at each pose of the AV. The poses may be relative poses.

[0039] The method may comprise spatially sampling 1 10 the plurality of images.

[0040] The method comprises identifying 112 objects and their 2D positions in at least some of the plurality of images. The term “at least some” may be used in this instance to mean the spatially sampled images. In other words, it is the spatially sampled images that are used for object identification. The algorithm used for object detection may be an algorithm such as you only look once (YOLO).

[0041] A first global parameter (GP1) may be input to this step. The first global parameter may be the camera configuration. The camera configuration may include details such as shutter speed, lens type, etc.

[0042] The determining the 2D position of the object is done at each pose of the AV.

[0043] The method may comprise generating 114, a plurality of rays. Each ray includes a position of the AV at the pose and a direction from the position of the AV and the two-dimensional position of the identified object. The position is obtained from the pose itself since pose is taken to mean the position and the orientation of the AV.

[0044] Aside from the two-dimensional positions of the objects, the inputs to the generating 114 the rays step may include a second global parameter GP2, a first input parameter IP1 , and the AV poses generated at step 108. The second global parameter GP2 may be the platform configuration, which may include details relating to the AV such as dimensions, positions of the sensors, dynamic configuration such as maximum speed, braking, etc. The first input parameter IP1 may be a minimum detector score.

[0045] The minimum detector score may be used as a detector score threshold. The step of generating 114, a plurality of rays, may comprise generating a detector score for the identified object. The detector score may be a probability relating to the confidence that the identification is a true positive, rather than a false positive, for example. In other cases, the detector score can be a probability relating to the semantic classification of the object. For example, the detector score may be a probability that the object has been correctly classified, e.g. as a dog, a person, avehicle, etc. The probability may be taken as the numerical value at a final layer of the YOLO algorithm for example, which may be between 0 (false) and 1 (true). The numerical value may be compared to a threshold value. In other embodiments, metrics such as precision and recall may be used.

[0046] The detector score may be compared to the detector score threshold. The ray for that pose may only be generated when the detector score is greater than or equal to the detector score threshold. In other words, a ray is generated when the object associated with it has a detector score greater than or equal to the detector score threshold, and no ray is generated when the detector score is less than the detector score threshold.

[0047] Next, the method checks whether LiDAR data is present. In other words, the method checks whether the LiDAR sensor is present and / or if it is operable and working correctly. As a result, the method checks whether there is a LiDAR point cloud available.

[0048] If the answer is no, in otherwords if the LiDAR data is not available, the method proceeds according to a first embodiment. A second embodiment is described below which differs to the first embodiment only in how to generate a hypothesis when LiDAR is available. Subsequent steps such as selecting hypothesis and hypothesis refinement are common to both embodiments.

[0049] In the first embodiment, where LiDAR is unavailable, the method comprises generating 118 a hypothesis for a three-dimensional position of the object based on at least some of the two- dimensional positions of the object. More specifically, the hypothesis is generated from the three- dimensional position of the object based on differences between the pose of the AV and the two- dimensional position of the object for at least some of the poses of the AV. The hypothesis may be generated from 2 or more two-dimensional position-pose pairs.

[0050] This is achieved using the rays. It is important to note that the rays are constructed in 3 dimensions. Therefore, they are unlikely to have a point of intersection. The point of intersection can thus be approximated using a shortest distance between two rays. For instance, for each ray, a shortest distance to another ray may be determined. The shortest distance may be a line. The hypothesis for the three-dimensional position of the object may be selected as a point along the shortest distance. The point may for example be a centre point of the line or may be any other point.

[0051] It is worth noting that not all rays should be compared. Doing so would be computationally expensive and also may lead to hypotheses that are poor approximations of the actual 3D position of the object. In order to reduce these negative effects, specific steps are provided below.

[0052] In another of these specific steps, the method may comprise determining a pose distance between an origin pose of the ray and an origin pose of the other ray. In other words, the pose distance is a distance between poses, or a distance between the AV’s position when a first ray is constructed and a distance between the AV’s position when a second ray is constructed.

[0053] The method may comprise comparing the pose distance to a maximum pose distance threshold. The maximum distance threshold may be a third input parameter, IP3. For any rays associated with a pose distance less than the pose distance threshold, the method may comprise generating the hypothesis for the three-dimensional position of the object at a point along the shortest distance. For any rays associated with a pose distance greater than or equal to the pose distance threshold, the method does not generate the hypothesis for the three-dimensional position of the object at a point along the shortest distance. It is beneficial not to use poses far from each other because they would likely view the object very differently, and in some cases the object may no long be visible, e.g. it may be out of sight or occluded.

[0054] In effect, these specific steps effectively filter out rays to be compared with to improve the hypothesis estimation accuracy.

[0055] Next, the method may comprise selecting 120 one or more of the hypotheses. The one or more hypotheses is selected by first determining whether each ray supports each hypothesis by comparing a closest distance between the respective ray and the respective hypothesis to a closeness threshold. The closeness threshold may be a fourth input parameter, IP4. Next the method comprises determining that a ray supports a hypothesis if the closest distance is less than the closeness threshold. If the closest distance is greater than or equal to the closeness threshold, the method comprises determining that the ray supports the hypothesis. The ray may be disregarded. The 3D hypothesis is projected into the 2D image and a distance in 2D is measured from that point to a centre of the detection (that generated the ray). In this way, the closest distance may actually be a hypothesis re-projection closeness.

[0056] Where this is a single hypothesis, that single hypothesis may be selected. Where there is a plurality of hypotheses, the selecting the hypotheses may comprise reducing a number of hypotheses based on number of rays that support each hypothesis. Selecting the hypotheses may also comprise reducing a number of hypotheses using voxelisation. the voxelisation comprises: constructing a plurality of voxels; mapping each hypothesis positionally to a voxel of the plurality of voxels; and in each voxel, combining any hypotheses in the voxel to a single position. The single position may for example be a centre of the voxel, an edge or corner of the voxel, or may be any other point of the voxel. This process may use a fifth input parameter, IP5, namely, a downsample voxel size.

[0057] The reducing the number of hypotheses may also comprise using a greedy voting algorithm to vote for retaining one or more of the hypotheses based on a number of rays that support the hypothesis. The greedy voting algorithm may be understood better with reference to the following pseudocode. The pseudocode can be divided into two portions

[0058] The first portion is as follows.

[0059] / / Returns 1 when Detection supports hypothesis. 0 otherwise

[0060] DetectionSupportsHypothesis(detection, hypothesis): return all(- Hypothesis is in front of the AV- Hypothesis and position of object are within max_distance_diff- Vehicle travelled between Hypothesis and detection are within 4*max_distance_diff- Hypothesis projects into detection in 2d within max_candidate_projection_diff)

[0061] / / Generates a matrix S where Sh,d = 1 when detection d supports hypothesis h (also called an inlier matrix); 0 otherwise.SupportMatrix(Detections, Hypotheses){S = [h * d] ZerosFor d in Detections:For h in Hypotheses:Sh,d = DetectionSupportsHypothesis(d,h) return S}

[0062] In this first portion, the max_distance_diff is the maximum distance threshold, IP3. The max_candidate_projection_diff is a sixth input parameter, IP6. S is a support matrix, h is a hypothesis, and d is detections, which may otherwise be understood as the ray leading to the two-dimensional position of the identified object. The hypotheses, h, are in rows of the support matrix, S, and the detections, d, are in the columns of the support matrix, S. This portion of the code words by initially setting the support matrix to 0, i.e. S is initially a matrix of zeros. Each element is given a value of 1 when the detection, or ray, supports the hypothesis.

[0063] A second portion of the pseudocode is provided below.

[0064] / / Greedily Vote on hypothesis. Return which hypotheses are kept and which detections are supporting them.GreedilySelectHypotheses(Detections, Hypotheses, min_votes):S = SupportMatrix(Detections, Hypotheses) Done = FalseSelectedHypotheses = []

[0065] Evidence = {} while(!Done){

[0066] / / Select the most popular hypothesis

[0067] / / Consider detections not used to support any other hypothesis

[0068] hypothesis, num_supports = row_with_max_sum(S) / / Greedily find best option If (num_supports < min_votes){ Done = True } Else {

[0069] evidence_detections = detections_counted_in_support(S, hypothesis)

[0070] EvidenceDetections[hypothesis] = evidence_detections SelectedHypotheses.add (hypothesis)S = set_used_detections_to_zero_for_all_hypotheses(S, hypothesis) }

[0071] } return SelectedHypotheses, Evidence

[0072] In this second portion, min_votes, is a minimum votes threshold.

[0073] The initial conditions of this code are that there are no selected hypotheses and there is no evidence. First, the values are summed to give a number of supports, num_supports, for each row. Next, a row with a maximum number of supports, row_with_max_sum(S), i.e. the row with the highest value, is determined. This row_with_max_sum is compared to the minimum votes threshold, min_votes. If it is less than the minimum votes threshold, the code ends and no hypotheses are selected. If it is greater than or equal to the minimum votes threshold, the code proceeds. In other words, the hypothesis associated with that row is selected and the number of detections, e.g. the number of 1s, is recorded as “evidence_detections”.

[0074] If any detections, i.e. rays, are used in support of the selected hypothesis, those detections are set to zero. In other words, those detections cannot support any other hypothesis. The code then loops, or iterates, using the same steps as above to identify if any other hypotheses meet the criteria for selection.

[0075] The term “greedy voting” may now be attributed because the first selected hypothesis is optimal at the time of selection and becomes the only hypothesis to be able to take support from its supporting detections.

[0076] This pseudocode may be summarised as the method comprising, iteratively: determining a number of rays supporting each hypothesis; identifying the hypothesis with a maximum number of supporting rays; comparing the maximum number of supporting rays to a hypothesis threshold; discarding the hypothesis if the maximum number of supporting rays is less than the hypothesis threshold; if the maximum number of supporting rays is greater than or equal to the hypothesis threshold: retain the hypothesis; and set the rays supporting the hypothesis as not supporting any other hypothesis.

[0077] The method may also comprise refining 122 the three-dimensional position of the hypothesis using line intersection least squares on any rays that support the hypothesis. In other words, if there is more than one ray supporting a hypothesis, least means squares isapplied to those hypotheses to obtain one position which can be used as the hypothesis of the three-dimensional position of the object. A ninth input parameter, IP9, is a switch to turn on or turn off the line intersection least squares.

[0078] The method may also comprise constructing a bounding box around the object. This bounding box construction may be part of the step of determining the two-dimensional position of the identified object at each pose of the AV.

[0079] The method then comprises annotating the map with the object at the three-dimensional position of the hypothesis. This can be achieved by constructing, on the map, a three- dimensional bounding box having dimensions based on dimensions of the two-dimensional bounding box.

[0080] According to the second embodiment, the step of generating a hypothesis for the three- dimensional position of the object based on at least some of the two-dimensional positions of the object is replaced with generating one or more hypotheses for a three-dimensional position of the object based on the two-dimensional positions of the object and a depth obtained from the data points. In other words, a plurality of hypotheses may be generated. For this embodiment, the sensor used for obtaining the data may be a LiDAR sensor and the data may be a LiDAR point cloud. The data used for each pose may be taken from LiDAR scans from a time period before and a time period after the pose, e.g. +0.1 seconds. These time periods are obtained from a seventh input parameter, IP7. The seventh input parameter, IP7, may be a LiDAR window, measured in seconds.

[0081] More specifically, the method involves registering the image and the data. In otherwords, the data is registered positionally with the point cloud. This is done for each sampled image. The method comprises identifying one or more points of the data within a bounding box surrounding one of the rays. The method also comprises determining the depth of the bounding box using the depths of the identified one or more points of the data. The depth of the one or more points may be obtained using an eighth input parameter, IP8. The eighth input parameter, IP8, may be a LiDAR hypothesis depth percentile. In otherwords, a distribution is constructed using the points. A predetermined percentile obtained from the eighth input parameter is used to obtain a value for the depth of the hypothesis.

[0082] The measured depth of the bounding box will disambiguate the three dimensional position of the object along the generated ray, thus providing a three dimensional point position.

[0083] With reference to Figure 3, a LiDAR scan is shown which shows rays 200, hypotheses 202, poses 204 of the AV and a three-dimensional bounding box 206 for the object at the hypothesis’ three-dimensional location, as annotated on the map.

[0084] With reference to Figure 4, a computer-implemented method of annotating a map used to navigate an autonomous vehicle, AV, with a three-dimensional location of an object may besummarised as including: receiving 300 a plurality of images captured by a camera of the AV, each image of the plurality of images associated with a pose of the AV; identifying 302 an object in at least some of the plurality of images; determining 304, at each pose of the AV, a two- dimensional position of the identified object; generating 306 a hypothesis for a three-dimensional position of the object based on at least some of the two-dimensional positions of the object; and annotating 308 the map with the object at the three-dimensional position of the hypothesis.

[0085] With reference to Figure 5, a computer-implemented method of annotating a map used to navigate an autonomous vehicle, AV, with a three-dimensional location of an object, may be summarised as comprising: receiving 400 a plurality of images captured by a camera of the AV, and a plurality of data from a sensor having a different modality to the camera, wherein each image of the plurality of images and each data is associated with a pose of the AV; identifying 402 an object in at least some of the plurality of images; determining 404 a two-dimensional position of the identified object at each corresponding pose; generating 406 one or more hypotheses for a three-dimensional position of the object based on the two dimensional positions of the object and a depth obtained from the data points; and annotating 408 the map with the object at the three-dimensional position of the hypothesis.

[0086] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments.

[0087] Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfil the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measured cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.

Claims

CLAIMS1 . A computer-implemented method of annotating a map used to navigate an autonomous vehicle, AV, with a three-dimensional location of an object, the computer-implemented method comprising: receiving a plurality of images captured by a camera of the AV, each image of the plurality of images associated with a pose of the AV; identifying an object in at least some of the plurality of images; determining, at each pose of the AV, a two-dimensional position of the identified object; generating a hypothesis for a three-dimensional position of the object based on at least some of the two-dimensional positions of the object; and annotating the map with the object at the three-dimensional position of the hypothesis.

2. The computer-implemented method of Claim 1 , wherein the generating a hypothesis for a three-dimensional position of the object based on at least some of the two-dimensional positions of the object comprises: generating a hypothesis for a three-dimensional position of the object based on differences in directions between the pose of the AV and the two-dimensional position of the object for at least some of the poses of the AV.

3. The computer-implemented method of Claim 2, further comprising: generating a plurality of rays, wherein each ray includes a position of the AV at the pose and a direction from the position of the AV and the two-dimensional position of the identified object.

4. The computer-implemented method of Claim 3, wherein the generating the plurality of rays, comprises: generating a detector score for the identified object; comparing the detector score to a detector score threshold; and generating, at each pose, the ray only when the detector score is greater than or equal to the detector score threshold.

5. The computer-implemented method of Claim 3 or Claim 4, wherein the generating a hypothesis for a three-dimensional position of the object based on differences in directionsbetween the pose of the AV and the two-dimensional position of the object for at least some of the poses of the AV comprises, for each ray: determining a shortest distance to another ray; and generating a hypothesis for the three-dimensional position of the object at a point along the shortest distance.

6. The computer-implemented method of Claim 5, further comprising: comparing the shortest distance to a hypothesis intersection distance threshold; for any rays resulting in the shortest distance being less than the hypothesis intersection distance threshold, generating the hypothesis for the three-dimensional position of the object at a point along the shortest distance; and for any rays resulting in the shortest distance being greater than or equal to the hypothesis intersection distance threshold, not generating the hypothesis for the three- dimensional position of the object at a point along the shortest distance.

7. The computer-implemented method of Claim 5 or Claim 6, further comprising: determining a pose distance between a pose of the ray and a pose of the other ray; comparing the pose distance to a maximum pose distance threshold; for any rays associated with a pose distance less than the pose distance threshold, generating the hypothesis for the three-dimensional position of the object at a point along the shortest distance; and for any rays associated with a pose distance greater than or equal to the pose distance threshold, not generating the hypothesis for the three-dimensional position of the object at a point along the shortest distance.

8. The computer-implemented method of any of Claims 5 to 7, further comprising selecting one or more of the hypotheses.

9. The computer-implemented method of Claim 8, wherein selecting one or more hypotheses comprises: determining whether each ray supports each hypothesis by comparing a closest distance between the two-dimensional position of a ray and the respective hypothesis to a closeness threshold; determining that a ray supports a hypothesis if the closest distance is less than the closeness threshold; anddetermining that the ray supports the hypothesis if the closest distance is greater than or equal to the closeness threshold.

10. The computer-implemented method of Claim 9, wherein the selecting the one or more hypotheses comprises: reducing a number of hypotheses based on numbers of rays that support each hypothesis.

11. The computer-implemented method of Claim 10, wherein the reducing the number of hypotheses comprises using a greedy voting algorithm to vote for retaining one or more of the hypotheses based on a number of rays that support the hypothesis.

12. The computer-implemented method of Claim 11 , wherein the using the greedy voting algorithm comprises, iteratively: determining a number of two-dimensional positions of the rays supporting each hypothesis; identifying the hypothesis with a maximum number of supporting two dimensional positions; comparing the maximum number of supporting two-dimensional positions to a hypothesis threshold; discarding the hypothesis if the maximum number of supporting two-dimensional positions is less than the hypothesis threshold; if the maximum number of supporting two-dimensional positions is greater than or equal to the hypothesis threshold: retain the hypothesis; and set the two-dimensional positions supporting the hypothesis as not supporting any other hypothesis.

13. The computer-implemented method of any of Claims 8 to 12, further comprising: refining the three-dimensional position of the hypothesis using line intersection least squares on any rays that support the hypothesis.

14. The computer-implemented method of any of claims 8 to 13, wherein selecting one or more of the hypotheses comprises: reducing a number of hypotheses using voxelisation.

15. The computer-implemented method of Claim 14, wherein the voxelisation comprises: constructing a plurality of voxels; mapping each hypothesis positionally to a voxel of the plurality of voxels; and in each voxel, combining any hypotheses in the voxel to a single position.

16. The computer-implemented method of any preceding claim, wherein the determining, at each pose of the AV, a two-dimensional position of the identified object, comprises: constructing a bounding box around the detected object.

17. The computer-implemented method of Claim 16, wherein the annotating the map with the object at the three-dimensional position of the hypothesis comprises: constructing, on the map, a three-dimensional bounding box having dimensions based on dimensions of the two-dimensional bounding box.

18. The computer-implemented method of any preceding claim, further comprising: spatially sampling the plurality of images, and wherein the at least some of the plurality of images is the spatially sampled images.

19. A transitory, or non-transitory, computer-readable medium, having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform the computer-implemented method of any preceding claim.

Images