Annotations on maps

Abstract

The present invention relates to a computer implementation method for annotating a map used to navigate an autonomous vehicle (AV) with the three-dimensional location of objects. A computer implementation method comprises the steps of: receiving a plurality of images captured by an AV camera, each of which is associated with the AV's orientation; identifying an object in at least some of the plurality of images; determining the two-dimensional position of the identified object in each orientation of the AV; generating a hypothesis for the object's three-dimensional position based on at least some of the object's two-dimensional positions; and annotating a map with the object at the hypothesized three-dimensional position.

Description

[Technical Field]

[0001] The subject of the present disclosure relates to annotating maps, and more specifically, to a computer-implemented method for annotating three-dimensional positions of objects on a map used for navigating an autonomous vehicle (AV), and to a transitory or non-transitory computer-readable medium. [Background Art]

[0002] Autonomous vehicles (AVs) often require maps of their environment for navigation. Maps are typically annotated with information about objects contained in the environment. Annotation is typically performed manually. Such manual annotation is time-consuming, which means that the autonomous stack used for navigating an AV takes a long time to update when operating in a new environment.

[0003] An object of the present invention is to address such problems and improve the prior art. [Summary of the Invention]

[0004] According to one aspect of the present disclosure, a computer implementation method is provided for annotating a map used to navigate an autonomous vehicle (AV) with respect to the three-dimensional location of an object, comprising the steps of: receiving a plurality of images captured by the AV's camera, each of the plurality of images being associated with the AV's pose; identifying an object in at least some of the plurality of images; determining the two-dimensional position (location) of the identified object in each pose of the AV; generating a hypothesis for the object's three-dimensional position based on at least some of the object's two-dimensional positions; and annotating the map with the object at the hypothetical three-dimensional position. Annotating the map in this manner provides an automated annotation means, meaning the AV can be more easily extended to new operating areas.

[0005] In one embodiment, the step of generating a hypothesis for the three-dimensional position of an object based on at least some of the two-dimensional positions of the object comprises the step of generating a hypothesis for the three-dimensional position of the object based on the directional difference between the orientation of the AV and the two-dimensional position of the object relative to at least some of the orientations of the AV.

[0006] In one embodiment, the computer implementation method comprises the step of generating a plurality of rays, each ray including the position of AV in orientation and the direction from the position of AV and the two-dimensional position of the identified object.

[0007] In one embodiment, the step of generating multiple rays includes the steps of generating a detector score for an identified object, comparing the detector score with a detector score threshold, and generating a ray only if the detector score is equal to or greater than the detector score threshold for each pose.

[0008] In one embodiment, the step of generating hypotheses for the three-dimensional position of an object based on the directional difference between the orientation of the AV and the two-dimensional position of the object relative to at least some of the orientations of the AV comprises, for each ray, the steps of determining the shortest distance to another ray and generating hypotheses for the three-dimensional position of the object at points along the shortest distance.

[0009] In one embodiment, the computer implementation method further comprises the steps of: comparing the shortest distance with a hypothetical intersection distance threshold; generating a hypothesis for the 3D position of an object at a point along the shortest distance for any ray that results in a shortest distance less than the hypothetical intersection distance threshold; and not generating a hypothesis for the 3D position of an object at a point along the shortest distance for any ray that results in a shortest distance greater than or equal to the hypothetical intersection distance threshold.

[0010] In one embodiment, the computer implementation method further comprises the steps of: determining the attitude distance between the attitude of one ray and the attitude of another ray; comparing the attitude distance with a maximum attitude distance threshold; generating a hypothesis for the 3D position of an object at a point along the shortest distance for any ray associated with an attitude distance less than the attitude distance threshold; and not generating a hypothesis for the 3D position of an object at a point along the shortest distance for any ray associated with an attitude distance greater than or equal to the attitude distance threshold.

[0011] In one embodiment, the computer implementation method further comprises the step of selecting one or more hypotheses. The selected one or more hypotheses may be selected as hypotheses for an object.

[0012] In one embodiment, the step of selecting one or more hypotheses includes determining whether each ray supports each hypothesis by comparing the closest distance between the ray's two-dimensional position and the corresponding hypothesis with a closeness threshold; determining that the ray supports the hypothesis if the closest distance is less than the closeness threshold; and determining that the ray supports the hypothesis if the closest distance is equal to or greater than the closeness threshold.

[0013] In one embodiment, the step of selecting one or more hypotheses comprises the step of reducing the number of hypotheses based on the number of rays that support each hypothesis.

[0014] In one embodiment, the step of reducing the number of hypotheses includes using a greedy voting algorithm to vote to retain one or more hypotheses based on the number of rays supporting each hypothesis.

[0015] In one embodiment, the steps of using a greedy voting algorithm include iteratively comprising: determining the number of 2D positions of rays that support each hypothesis; identifying a hypothesis that has the maximum number of 2D positions to support; comparing the maximum number of 2D positions to support a hypothesis threshold; discarding a hypothesis if the maximum number of 2D positions to support is less than the hypothesis threshold; retaining a hypothesis if the maximum number of 2D positions to support is equal to or greater than the hypothesis threshold; and setting the 2D positions that support a hypothesis as not supporting other hypotheses.

[0016] In one embodiment, the computer implementation method further comprises the step of refining the three-dimensional position of a hypothesis by using line intersection least squares for any ray supporting the hypothesis.

[0017] In one embodiment, the step of selecting one or more hypotheses comprises the step of reducing the number of hypotheses using voxelization.

[0018] In one embodiment, voxelization comprises the steps of constructing a plurality of voxels, positionally mapping each hypothesis to one of the voxels, and combining any hypothesis within each voxel at a single location.

[0019] In one embodiment, the step of determining the two-dimensional position of an identified object in each orientation of the AV includes the step of constructing a bounding box around the detected object.

[0020] In one embodiment, the step of annotating an object on a map at a hypothetical three-dimensional location comprises the step of constructing a three-dimensional bounding box on the map having dimensions based on the dimensions of a two-dimensional bounding box.

[0021] In one embodiment, the computer implementation method further comprises the step of spatially sampling a plurality of images, wherein at least some of the plurality of images are spatially sampled images.

[0022] According to one aspect of the present invention, a temporary or non-temporary computer-readable medium is provided which has instructions stored therein, and when such instructions are executed by one or more processors, causes one or more processors to execute a computer implementation method of any of the preceding aspects or embodiments.

[0023] According to one aspect of the present invention, a computer implementation method is provided for annotating the three-dimensional location of an object on a map used to navigate an autonomous vehicle (AV), comprising the steps of: receiving a plurality of images captured by a camera of the AV and a plurality of data points from sensors having different modalities to the camera, wherein each of the plurality of images and each data point is associated with the pose of the AV; identifying an object in the plurality of images; determining the two-dimensional position of the object in the image for each pose of the AV; generating a hypothesis for the three-dimensional position of the object based on the two-dimensional position and depth obtained from the plurality of data points; and annotating the map with the object at the hypothesized three-dimensional position.

[0024] According to one aspect of the present invention, a computer implementation method for annotating a map used to navigate an autonomous vehicle (AV) with respect to the three-dimensional location of an object, comprising the steps of: receiving a plurality of images captured by the AV's camera, each image of the plurality of images being associated with the AV's orientation; identifying an object in the plurality of images; determining the two-dimensional position of the object using the image for each orientation of the AV; generating a hypothesis for the three-dimensional position of the object based on the two-dimensional position; and annotating the map with respect to the object at the hypothesized three-dimensional position. [Brief explanation of the drawing]

[0025] The subject matter of this disclosure is best described with reference to the accompanying drawings. [Figure 1] Figure 1 shows a schematic diagram of an AV according to one or more embodiments. [Figure 2] Figure 2 shows a flowchart illustrating a computer implementation method for annotating objects on a map according to one or more embodiments. [Figure 3]FIG. 3 is a schematic diagram showing hypotheses and rays obtained as part of the use of the computer-implemented method of FIG. 2. [Figure 4] FIG. 4 is a flowchart summarizing the computer-implemented method associated with the flowchart of FIG. 2. [Figure 5] FIG. 5 shows a flowchart similar to that of FIG. 4. MODE FOR CARRYING OUT THE INVENTION

[0026] At least some of the exemplary embodiments described herein may be constructed in part or in whole using dedicated special-purpose hardware. Terms used herein, such as “component,” “module,” or “unit,” may include, but are not limited to, hardware devices such as circuits, field-programmable gate arrays (FPGAs), or application-specific integrated circuits (ASICs) in the form of discrete or integrated components that perform a particular task or provide associated functionality. In some embodiments, the described elements may be configured to reside on a tangible, persistent, and addressable storage medium and may be configured to run on one or more processors. These functional elements may, in some embodiments, include, for example, software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. While exemplary embodiments have been described with reference to the components, modules, and units discussed herein, such functional elements may be combined into fewer elements or separated into additional elements. Various combinations of optional features are described herein, and it will be understood that the features described may be combined in any preferred combination. In particular, features of any one exemplary embodiment may be combined with features of any other embodiment as appropriate, unless such combinations are mutually exclusive. Throughout this specification, the terms “comprising” or “comprises” include the specified components but do not exclude the existence of other components.

[0027] The embodiments described herein may be implemented as a set of instructions stored as electronic data in one or more storage media. Specifically, the instructions may be provided on a temporary or non-temporary computer-readable medium. When executed by a processor, the processor is configured to perform various methods described in the following embodiments. Thus, the methods may be computer implementation methods. In particular, the processor and the storage device containing the instructions may be incorporated into a vehicle. The vehicle may be an autonomous vehicle (AV).

[0028] The following embodiments provide specific exemplary examples, but these exemplary examples should not be taken (interpreted) as limiting, and the scope of protection is defined by the claims. Features from specific embodiments may be used in combination with features from other embodiments without extending the subject matter beyond the scope of this disclosure.

[0029] Referring to Figure 1, the AV10 may include a plurality of sensors 12. The sensors 12 may be mounted on the roof of the AV10. The sensors 12 may be communicatively connected to a computer 14. The computer 14 may be mounted on the AV10. The computer 14 may include a processor 16 and memory 18. The memory may include non-temporary computer-readable media as described herein. The memory may also include an autonomous stack for operating the AV. Alternatively, the non-temporary computer-readable media may be remotely located and communicatively linked to the computer 14 via a cloud 20. The computer 14 may be communicatively linked to one or more actuators 22, which control the movement of the AV10. The actuators may include, for example, motors, brake systems, power steering systems, etc.

[0030] Sensor 12 may include various sensor types. Examples of sensor types include LiDAR sensors, radar sensors, and cameras. Each sensor type may be called a sensor modality. Each sensor type may record data associated with the sensor modality. For example, a LiDAR sensor may record LiDAR modality data.

[0031] The data can capture various scenarios that AV10 encounters. For example, the scenarios could be the visible surroundings of AV10, including roads, buildings, weather, and objects (e.g., other vehicles, pedestrians, animals, etc.).

[0032] Referring to Figure 2, the computer implementation method for annotating objects on a map begins in step 100. The term "computer implementation method" can be used interchangeably with the term "method."

[0033] The method includes step 102 of receiving multiple images captured by the AV's camera, each image of which is associated with the AV's pose. The method also includes step 104 of receiving data captured by the AV's sensors at each of the AV's poses. The sensors may be LiDAR sensors, and the data may be LiDAR point clouds.

[0034] The autonomous stack 106 uses the received images and lidar point cloud data to generate the AV's attitude (108). Images may be captured for each attitude of the AV. The attitude may be a relative attitude.

[0035] This method may include a step 110 for spatially sampling multiple images.

[0036] This method comprises step 112 of identifying objects and their 2D (two-dimensional) locations in at least some of a set of images. The term "at least some" in this example (instance) may be used to mean spatially sampled images. In other words, spatially sampled images are used for object identification. The algorithm used for object detection may be an algorithm such as YOLO (you only look once).

[0037] A first global parameter (GP1) may be entered in this step. The first global parameter may be the camera configuration. The camera configuration may include details such as shutter speed and lens type.

[0038] The 2D position of an object is determined for each pose in AV.

[0039] This method may include step 114 of generating multiple rays. Each ray includes the position of AV in the orientation and the orientation from the position of AV and the 2D position of the identified object. Since the orientation is understood to mean the position and orientation of AV, the position is obtained from the orientation itself.

[0040] Apart from the object's two-dimensional position, the input to step 114 for ray generation may include a second global parameter GP2, a first input parameter IP1, and the AV attitude generated in step 108. The second global parameter GP2 may be a platform configuration, which may include dynamic configurations such as dimensions, sensor positions, and maximum speed, as well as AV details such as braking. The first input parameter IP1 may be the minimum detector score.

[0041] The minimum detector score can be used as a detector score threshold. Step 114, which generates multiple rays, may include generating a detector score for the identified objects. The detector score may be, for example, a probability related to confidence that the identification is true positive and not false positive. In other cases, the detector score may be a probability related to the semantic classification of an object. For example, the detector score may be the probability that the object was correctly classified as, for example, a dog, a person, a vehicle, etc. The probability may be represented as a numerical value in the final layer of the YOLO algorithm, which may be between 0 (false) and 1 (true). The numerical value may be compared to a threshold. In other embodiments, metrics such as precision and recall may be used.

[0042] The detector score can be compared to a detector score threshold. Rays for that pose can only be generated if the detector score is equal to or greater than the detector score threshold. In other words, a ray is generated if the object it is associated with has a detector score equal to or greater than the detector score threshold, and no ray is generated if the detector score is less than the detector score threshold.

[0043] Next, this method checks whether LiDAR data exists. In other words, this method checks whether a LiDAR sensor exists and / or whether the LiDAR sensor is operational and functioning correctly. As a result, this method checks whether there is an available LiDAR point cloud.

[0044] If the answer is no, in other words, if lidar data is not available, the method proceeds according to the first embodiment. Below, we describe a second embodiment that differs from the first embodiment only in the method of generating hypotheses when lidar is available. Subsequent steps such as hypothesis selection and hypothesis refinement are common to both embodiments.

[0045] In the first embodiment, if a LiDAR is not available, the method comprises step 118 of generating hypotheses for the object's three-dimensional position based on at least some of the object's two-dimensional positions. More specifically, the hypotheses are generated from the object's three-dimensional position based on the difference between the AV's pose and the object's two-dimensional position, for at least some of the AV's poses. The hypotheses may be generated from two or more pairs of two-dimensional position-poses.

[0046] This is achieved using rays. It is important to note that rays are constructed in three dimensions. Therefore, they are unlikely to have intersections. Thus, intersections can be approximated using the shortest distance between two rays. For example, for each ray, the shortest distance to another ray can be determined. The shortest distance can be a line. The hypothetical three-dimensional position of an object can be chosen as a point along the shortest distance. The point can be, for example, the center point of a line, or any other point.

[0047] It should be noted that not all rays should be compared. Doing so can be computationally expensive and may lead to the assumption that they are an inferior approximation of the object's actual 3D (three-dimensional) position. To mitigate these negative effects, specific steps are provided below.

[0048] In another step of these specific steps, the method may include determining the attitude distance between the origin attitude of one ray and the origin attitude of the other ray. In other words, the attitude distance is the distance between the attitudes, or the distance between the position of AV when the first ray is constructed and the position of AV when the second ray is constructed.

[0049] This method may include a step of comparing an attitude distance to a maximum attitude distance threshold. The maximum distance threshold may be a third input parameter (IP3). For any ray associated with an attitude distance less than the attitude distance threshold, this method may include a step of generating a hypothesis for the 3D position of the object at a point along the shortest distance. For any ray associated with an attitude distance greater than or equal to the attitude distance threshold, this method does not generate a hypothesis for the 3D position of the object at a point along the shortest distance. It is beneficial not to use attitudes that are far apart from each other, as they are likely to see the object very differently, and in some cases the object may no longer be visible, for example, because it is out of sight or occluded.

[0050] In practice, these specific steps effectively filter the rays being compared, improving the accuracy of hypothesis estimation.

[0051] Next, the method may include a step 120 for selecting one or more hypotheses. One or more hypotheses are selected by first determining whether each ray supports each hypothesis by comparing the nearest distance between each ray and each hypothesis with a proximity threshold. The proximity threshold may be a fourth input parameter (IP4). Next, the method includes a step of determining that a ray supports a hypothesis if the nearest distance is less than the proximity threshold. If the nearest distance is greater than or equal to the proximity threshold, the method includes a step of determining that a ray supports a hypothesis. The ray may be ignored. The 3D hypothesis is projected onto a 2D image, and the distance in 2D is measured from that point to the center of detection (which generated the ray). Thus, the nearest distance may actually be the proximity of the hypothesis reprojection.

[0052] If this is a single hypothesis, that single hypothesis may be selected. If there are multiple hypotheses, selecting a hypothesis may involve reducing the number of hypotheses based on the number of rays supporting each hypothesis. The step of selecting a hypothesis may also involve reducing the number of hypotheses using voxelization. Voxelization comprises the steps of constructing multiple voxels, positionally mapping each hypothesis to one of the multiple voxels, and, in each voxel, combining any hypotheses within that voxel into a single location. The single location may be, for example, the center of the voxel, an edge or corner of the voxel, or any other point on the voxel. This process may use a fifth input parameter (IP5), namely the downsample voxel size.

[0053] The step of reducing the number of hypotheses may also include the step of using a greedy voting algorithm to vote to retain one or more hypotheses based on the number of rays supporting each hypothesis. The greedy voting algorithm can be better understood by referring to the pseudocode below. The pseudocode can be divided into two parts.

[0054] The first part is as follows:

[0055] / / Returns 1 when Detection supports hypothesis. 0 otherwise

[0056] DetectionSupportsHypothesis(detection, hypothesis): return all( - Hypothesis is in front of the AV - Hypothesis and position of object are within max_distance_diff - Vehicle traveled between Hypothesis and detection are within 4*max_distance_diff - Hypothesis projects into detection in 2d within max_candidate_projection_diff )

[0057] / / Generates a matrix S where Sh,d = 1 when detection d supports hypothesis h (also called an inlier matrix); 0 otherwise. SupportMatrix(Detections, Hypotheses){ S = [h * d] Zeros For d in Detections: For h in Hypotheses: Sh,d = DetectionSupportsHypothesis(d,h) return S }

[0058] In this first part, max_distance_diff is the maximum distance threshold (IP3). max_candidate_projection_diff is the sixth input parameter (IP6). S is the support matrix, h is the hypothesis, and d is the detection, which can be understood as a ray guiding the 2D position of the identified object. The hypothesis h is in the rows of the support matrix S, and the detection d is in the columns of the support matrix S. This part of the codeword starts by setting the support matrix to 0, i.e., S is a matrix that is initially zero. Each element is given a value of 1 when the detection or ray supports the hypothesis.

[0059] The second part of the pseudocode is shown below.

[0060] / / Greedily Vote on hypothesis. Return which hypotheses are kept and which detections are supporting them. GreedilySelectHypotheses(Detections, Hypotheses, min_votes): S = SupportMatrix(Detections, Hypotheses) Done = False SelectedHypotheses = []

[0061] Evidence = {} while(!Done){

[0062] / / Select the most popular hypothesis

[0063] / / Consider detections not used to support any other hypothesis

[0064] hypothesis, num_supports = row_with_max_sum(S) / / Greedily find best option If (num_supports < min_votes) { Done = True} Else {

[0065] evidence_detections = detections_counted_in_support(S, hypothesis)

[0066] EvidenceDetections[hypothesis] = evidence_detections SelectedHypotheses.add (hypothesis) S = set_used_detections_to_zero_for_all_hypotheses(S, hypothesis) }

[0067] } return SelectedHypotheses, Evidence

[0068] In this second part, min_votes is the minimum voting threshold.

[0069] The initial condition for this code is that there are no selected hypotheses and no evidence. First, the values ​​are summed to give the number of supports (num_support) for each row. Next, the row with the maximum number of supports (row_with_max_sum(S)), i.e., the row with the highest value, is determined. This row (row_with_max_sum) is compared to the minimum voting threshold (min_votes). If it is less than the minimum voting threshold, the code terminates and no hypothesis is selected. If it is greater than or equal to the minimum voting threshold, the code proceeds. In other words, a hypothesis associated with that row is selected, and the number of detections, e.g., 1, is recorded as "evidence_detections".

[0070] If any detections, i.e., rays, are used to support the selected hypothesis, those detections are set to zero. In other words, these detections cannot support any other hypothesis. The code then loops or iterates using the same steps as above to identify whether any other hypothesis satisfies the selection criteria.

[0071] Here, the term "greedy voting" can be attributed because the first selected hypothesis is optimal at the time of selection and is the only hypothesis that can take support from its support detection.

[0072] This pseudocode can be summarized as an iterative method comprising determining the number of rays supporting each hypothesis, identifying the hypothesis with the maximum number of rays it supports, comparing the maximum number of rays it supports with a hypothesis threshold, discarding the hypothesis if the maximum number of rays it supports is less than the hypothesis threshold, retaining the hypothesis if the maximum number of rays it supports is equal to or greater than the hypothesis threshold, and setting rays that support a hypothesis as not supporting other hypotheses.

[0073] The method may also include step 122, which refines the 3D position of a hypothesis by using line intersection least squares for any ray supporting the hypothesis. That is, if there are multiple rays supporting the hypothesis, the least mean squares is applied to those hypotheses to obtain one position that can be used as a hypothesis for the 3D position of the object. The ninth input parameter (IP9) is a switch to turn line intersection least squares on or off.

[0074] This method may also include a step of constructing a bounding box around an object. This construction of the bounding box may be part of a step of determining the two-dimensional position of the identified object in each orientation of the AV.

[0075] The method then comprises the step of annotating a map with an object at a hypothetical three-dimensional location. This can be achieved by constructing a three-dimensional bounding box on the map, having dimensions based on the dimensions of a two-dimensional bounding box.

[0076] According to the second embodiment, the step of generating hypotheses for the object's 3D position based on at least some of the object's 2D positions is replaced by generating one or more hypotheses for the object's 3D position based on the object's 2D positions and depths obtained from data points. In other words, multiple hypotheses may be generated. In this embodiment, the sensor used to acquire data may be a lidar sensor, and the data may be a lidar point cloud. The data used for each attitude may be captured from a period prior to and after the attitude, for example, from a lidar scan starting from +0.1 seconds. These periods are obtained from a seventh input parameter (IP7), which may be a lidar window measured in seconds.

[0077] More specifically, the method involves registering images and data. In other words, data is registered positionally in a point cloud. This is done for each sampled image. The method comprises identifying one or more data points within a bounding box surrounding a ray. The method also comprises determining the depth of the bounding box using the depth of one or more identified data points. The depth of one or more points may be obtained using an eighth input parameter IP8. The eighth input parameter (IP8) may be a lidar hypothesis depth percentile. In other words, a distribution is constructed using points. A predetermined percentile obtained from the eighth input parameter is used to obtain a value for the hypothesis depth.

[0078] The measured depth of the bounding box clarifies the 3D position of the object along the generated ray, and thus provides the 3D point position.

[0079] Referring to Figure 3, a LiDAR scan is shown, showing Ray 200, Hypothesis 202, AV pose 204, and a 3D bounding box 206 for the object at the hypothetical 3D location, as annotated on the map.

[0080] Referring to Figure 4, a computer implementation method for annotating a map used to navigate an autonomous vehicle (AV) with the three-dimensional location of an object can be summarized as including: receiving a plurality of images captured by the AV's camera, each of the plurality of images being associated with the AV's pose; identifying an object in at least some of the plurality of images; determining the two-dimensional position of the identified object in each pose of the AV; generating a hypothesis for the three-dimensional position of the object based on at least some of the object's two-dimensional positions; and annotating the map with the object at the hypothesized three-dimensional position.

[0081] Referring to Figure 5, a computer implementation method for annotating the three-dimensional location of an object on a map used to navigate an autonomous vehicle (AV) can be summarized as comprising: step 400 receiving a plurality of images captured by the AV's camera and a plurality of data from sensors having different modalities to the camera, wherein each image and each data is associated with the AV's attitude; step 402 identifying an object in at least some of the plurality of images; step 404 determining the two-dimensional position of the object identified in each corresponding attitude; step 406 generating one or more hypotheses for the object's three-dimensional position based on the object's two-dimensional position and depth obtained from data points; and step 408 annotating the object on the map at the hypothesized three-dimensional position.

[0082] Although the present invention is illustrated and described in detail in the drawings and the foregoing description, such illustrations and descriptions should be considered illustrative or exemplary and not limiting, and the present invention is not limited to the disclosed embodiments.

[0083] Other variations of the disclosed embodiments can be understood and implemented by those skilled in the art in carrying out the claimed invention, based on the study of the drawings, this disclosure, and the appended claims. In the claims, the term “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude the plural case. A single processor or other unit may perform the functions of several items described in the claims. The mere fact that certain means are described in different dependent claims does not imply that combinations of these means cannot be used advantageously. No reference numeral in the claims should be construed as limiting the scope.

Claims

1. A computer implementation method for annotating a map used to navigate an autonomous vehicle (AV) with the three-dimensional location of an object, A step of receiving a plurality of images captured by the AV's camera, wherein each of the plurality of images is associated with the AV's posture, The steps include identifying the object in at least some of the plurality of images, and determining the two-dimensional position of the identified object in each pose of the AV. A step of generating a hypothesis for the three-dimensional position of the object based on at least some of the two-dimensional positions of the object, A computer implementation method comprising the step of annotating the object on a map at the three-dimensional location of the hypothesis.

2. A computer implementation method according to claim 1, the step of generating the hypothesis for the three-dimensional position of the object based on at least some of the two-dimensional positions of the object, A computer implementation method comprising the step of generating a hypothesis for the three-dimensional position of an object based on the directional difference between the orientation of the AV and the two-dimensional position of the object relative to at least some of the orientations of the AV.

3. A computer implementation method according to claim 2, A computer implementation method further comprising the step of generating a plurality of rays, each ray including the position of the AV in the orientation and the direction from the position of the AV and the two-dimensional position of the identified object.

4. The computer implementation method according to claim 3, wherein the step of generating the plurality of rays is: A step of generating a detector score for the identified object, A computer implementation method comprising the steps of: comparing the detector score with a detector score threshold; and generating the ray only when the detector score is equal to or greater than the detector score threshold in each posture.

5. A computer implementation method according to claim 3 or claim 4, wherein the step of generating the hypothesis for the three-dimensional position of the object based on the directional difference between the orientation of the AV and the two-dimensional position of the object with respect to at least some of the orientations of the AV is, for each ray, The step of determining the shortest distance to another ray, A computer implementation method comprising the step of generating a hypothesis for the three-dimensional position of the object at a point along the shortest distance.

6. A computer implementation according to claim 5, The steps include: comparing the shortest distance with a hypothetical intersection distance threshold; and for any ray that yields the shortest distance which is less than the hypothetical intersection distance threshold, generating a hypothesis for the three-dimensional position of the object at a point along the shortest distance. A computer implementation method further comprising the step of not generating the hypothesis for the three-dimensional position of the object at a point along the shortest distance for any ray that yields the shortest distance which is greater than or equal to the hypothesis intersection distance threshold.

7. A computer implementation method according to claim 5 or claim 6, A step of determining the attitude distance between the attitude of Ray and the attitude of the other Ray, The steps include comparing the attitude distance with a maximum attitude distance threshold, and for any ray associated with an attitude distance less than the attitude distance threshold, generating a hypothesis for the three-dimensional position of the object at a point along the shortest distance. A computer implementation method further comprising the step of not generating the hypothesis for the three-dimensional position of the object at a point along the shortest distance for any ray associated with an attitude distance greater than or equal to the attitude distance threshold.

8. A computer implementation method according to any one of claims 5 to 7, further comprising the step of selecting one or more of the aforementioned hypotheses.

9. The computer implementation method according to claim 8, wherein the step of selecting one or more hypotheses is: The steps include determining whether each ray supports each hypothesis by comparing the closest distance between the ray's two-dimensional position and the corresponding hypothesis with a proximity threshold, The step of determining that the ray supports the hypothesis if the closest distance is below the proximity threshold, A computer implementation method comprising the step of determining that the ray supports the hypothesis if the nearest distance is greater than or equal to a proximity threshold.

10. The computer implementation method according to claim 9, wherein the step of selecting one or more hypotheses is: A computer implementation method comprising the step of reducing the number of hypotheses based on the number of rays that support each hypothesis.

11. The computer implementation method according to claim 10, wherein the step of reducing the number of hypotheses is to use a greedy voting algorithm for voting to retain one or more hypotheses based on the number of rays that support the hypotheses.

12. The computer implementation method according to claim 11, wherein the step of using the greedy voting algorithm is: A step of determining the number of two-dimensional positions of the ray that support each hypothesis, The steps include identifying the hypothesis having the maximum number of supported two-dimensional positions, A step of comparing the maximum number of supported two-dimensional positions with a hypothetical threshold, If the maximum number of supported two-dimensional positions is less than the hypothesis threshold, the hypothesis is discarded. If the maximum number of supported two-dimensional positions is equal to or greater than the hypothesis threshold, the step of retaining the hypothesis, A computer implementation method comprising the step of iteratively setting the two-dimensional position that supports the aforementioned hypothesis as one that does not support any other hypothesis.

13. A computer implementation method according to any one of claims 8 to 12, A computer implementation method further comprising the step of refining the three-dimensional position of the hypothesis by using the least squares of line intersections for any ray supporting the hypothesis.

14. A computer implementation method according to any one of claims 8 to 13, wherein the step of selecting one or more hypotheses is: A computer implementation method comprising the step of reducing the number of hypotheses using voxelization.

15. A computer implementation method according to claim 14, wherein the voxelization is a step of constructing a plurality of voxels, A computer implementation method comprising the steps of: positionally mapping each hypothesis to one of the plurality of voxels; and combining any hypothesis within each voxel at a single position.

16. The computer implementation method according to any one of claims 1 to 15, wherein the step of determining the two-dimensional position of the identified object in each orientation of the AV comprises the step of constructing a bounding box around the detected object.

17. A computer implementation method according to claim 16, wherein the step of annotating an object on a map at the hypothetical three-dimensional location is: A computer implementation method comprising the steps of constructing a three-dimensional bounding box on a map, having dimensions based on the dimensions of the two-dimensional bounding box.

18. A computer implementation method according to any one of claims 1 to 17, The step of spatially sampling the plurality of images further comprises A computer implementation method wherein at least some of the aforementioned multiple images are spatially sampled images.

19. A temporary or non-temporary computer-readable medium having instructions stored therein, wherein, when the instructions are executed by one or more processors, the instructions cause the one or more processors to execute the computer implementation method described in any one of claims 1 to 18.

Images