Verifying traffic sign information using sensor-derived observations

Abstract

A computer-implemented method of verifying the accuracy of traffic sign information included in a digital map based on sensor-derived observations is proposed, wherein the digital map comprises a plurality of segments representative of a road network and the traffic sign information, the traffic sign information indicating, for each of a plurality of traffic signs, the type of traffic sign and the location along a segment of the digital map. The method comprises identifying first candidate corrections using a machine learning model and identifying second candidate corrections using a heuristic model. Recommended corrections are then determined based on the first and second candidates. The recommended corrections are used to update the machine learning model.

Description

[0001] VERIFYING TRAFFIC SIGN INFORMATION USING SENSOR-DERIVED OBSERVATIONS

[0002] Field

[0003] The present application relates to a computer-implemented method of verifying the accuracy of traffic sign information included in a digital map based on sensor-derived observations, as well as to a system performing such a method.

[0004] Background

[0005] The use of digital maps to represent a road network is well known, with applications in navigation, advanced driver assistance systems (ADAS), automated driving, and more. Typically, digital maps represent a road network using segments, wherein each segment can represent a portion of a road. Digital maps also usually include further information of many different types.

[0006] For instance, for many applications, it is useful for a digital map to include traffic sign information, indicating which traffic signs are present along a road. The traffic sign information generally indicates the type of traffic sign, such as a speed limit sign, a yield sign, a stop sign, a maximum height or weight sign, a no overtaking sign, etc. It is noted that some of these signs can be said to have both a type and a value: for instance, a sign indicating a speed limit of 50 km / hour could be seen as a sign of the type "speed limit" with value "50 km / hour", or could be seen as a sign of the type "speed limit of 50 km / hour". Generally, in the below, "type" can be understood as meaning "type and value" where appropriate.

[0007] The traffic sign information in a digital map generally also indicates the position of each traffic sign. This may be the actual position (longitude / latitude) of the traffic sign; however, in many digital maps, the position of the sign relating to a portion of the road network is projected onto the appropriate segment of the map. The position is then indicated in terms of the location along a segment of the digital map. It may also be that, in cases where a sign occurs on both sides of a road, these two signs are merged into a single traffic sign in the digital map.

[0008] The accuracy of traffic sign information in the map is very important. One method of verifying the accuracy of traffic sign information in a digital map is to use sensor-derived observations, i.e., observation based on sensor data captured by sensors of a plurality of vehicles traversing the road network. However, there are many issues in doing so. In particular, the sensor-derived observations tend to be numerous and noisy. Furthermore, considering the effort involved in map making, it is important to thoroughly validate required corrections to avoid overcorrections. Summary

[0009] Examples of the method described below may present an efficient way to verify traffic sign information included in a digital map, achieving a desired precision while minimizing the required human oversight and making advantageous use of various computational techniques.

[0010] The proposed method was developed for and optimized in particular for detecting likely superfluous signs, also referred to as "superfluous leads", i.e. traffic signs included in the digital map which are not observed, and are therefore likely not to be present in reality. Therefore, this is what is focused on in the below. However, the same techniques and principles could also be used to identify other errors in traffic sign information, including missing signs, signs of which the type and / or value is incorrect, incorrectly positioned signs, signs that are assigned to the wrong road segment, etc. Based on the above, a computer-implemented method of verifying the accuracy of traffic sign information included in a digital map based on sensor-derived observation is proposed. The digital map comprises a plurality of segments representative of a road network as well as the traffic sign information. The traffic sign information indicates, for each of a plurality of traffic signs, at least the type of traffic sign and the location along a segment of the digital map.

[0011] In a first step, the method includes identifying a first plurality of observed traffic signs based on first sensor data from sensors of a plurality of vehicles traversing the road network, comprising determining, for each observed traffic sign, at least a type and a position. The sensors, in this case, are typically cameras, though it is not excluded that sensor information from different types of sensors could be used. Traffic signs, including their type and optionally value, may be identified in an image or images registered by the cameras using a variety of image recognition techniques. The position of the traffic sign may be determined in a variety of ways, for instance based at least in part on the position of the vehicle at the time of registering the image, which may be established using a GNSS system or in any of the other known ways. Other factors could also be taken into account, such as the bearing of the vehicle, the position of the camera, the size of the sign in the image, signals from other sensors such as infrared / radar / lidar sensors etc. In this disclosure, "identifying" may comprise the analysis of the sensor data to determine which signs were observed and what the positions of these traffic signs is, but may also comprise receiving information about observed signs which was derived from the sensor data at another stage and / or by another entity. Furthermore, it may also comprise merging multiple observations, received from different vehicles and / or at different times, which are determined to likely relate to the same traffic sign.

[0012] The method includes obtaining an initial machine learning model configured to output candidate corrections based on traffic sign information included in the digital map corresponding to the type and position of at least one traffic sign and the type and position of observed traffic signs. It is noted that the machine learning model may be configured to take the (absolute) position and type of the traffic signs in the digital map and the (absolute) position and type of the observed traffic sign as input directly, but this and other information may also be pre-processed in some way before it is provided as input to the machine learning model, as will be explained in more detail below. Furthermore, the machine learning model may have additional inputs and / or outputs.

[0013] This initial machine learning model is then used to identify first candidate corrections based on the digital map and the first plurality of observed traffic signs.

[0014] It could be considered to stop here, and to use the first candidate corrections output by the machine learning model to update the digital map. Indeed, machine learning models are commonly used nowadays to deal with large sets of input, in particular to derive patterns.

[0015] A drawback of machine learning models, however, is that their performance requires training, which in turn requires substantial training sets. In the present case, the machine learning model - generally of the "decision tree" type - is typically one trained using so-called "supervised learning", wherein paired sets of inputs and outputs are provided, and the machine learning model is trained to reproduce a given output based on the corresponding input. For superfluous signs, this would consist at least of combinations of traffic sign information relating to a traffic sign already determined to be superfluous and information relating to observed traffic signs in the vicinity of this superfluous traffic sign. Furthermore, to improve training, training input could additionally be provided in the form of appropriately labelled combinations of traffic sign information relating to a correct traffic sign in the digital map and information relating to observed traffic signs in the vicinity of this correct traffic sign. To identify other types of errors, it could be that the training data includes more than just two ("superfluous / correct" or "yes / no") labels.

[0016] However, this means that training the machine learning model requires already having done some of the work of verifying the accuracy of the traffic sign information. This leads to an issue: if there is insufficient training data, the machine learning model will not be able to identify all errors; but to obtain sufficient data would require exactly the extensive amount of human labor that the use of a machine learning model was intended to obviate.

[0017] A further issue is that the performance of a machine learning model tends to degrade as the number of differences between training input and "real" input increases. The types and arrangement of traffic signs tends to differ quite strongly from county to country. Therefore, a machine learning model trained for a certain country may not be very accurate in assessing input from another country.

[0018] The presently proposed method therefore further includes associating each observed traffic sign of the first plurality of observed traffic signs to a segment of the digital map, and identifying second candidate corrections by comparing, for at least a subset of the segments of the digital map, traffic signs along the segment according to the digital map to observed traffic signs of the first plurality of observed traffic signs associated with the segment.

[0019] This method, which is also referred to in this document as the "heuristic model", is computationally much simpler than the machine learning model, and does not require any training, as it very simply looks at the traffic signs - both in the map and observed by sensors - on a segment-by-segment basis.

[0020] In particular, rather than taking the precise position of the observed traffic signs into account (and comparing this to the absolute position of traffic signs in the digital map), the method associates each observed traffic sign to a segment of the digital map. The data is then compared on a segment- by-segment basis. Preferably, identifying second candidate corrections comprises identifying traffic signs along a segment for which there is no traffic sign of the same type associated with the segment in the first plurality of observed traffic signs. Note that here, "same type" means "same type and value" for traffic signs which also have a value: as will be clear, a "speed limit 100 km / h" sign is not the same as a "speed limit 50 km / h" sign. Furthermore, note that other discrepancies can also be identified as candidate corrections, for instance if the number of signs of a certain type differs. Preferably, in the step of identifying second candidate corrections, only segments are considered for which the number of vehicles with sensors obtaining sensor data traversing the segment of the road network exceeds a predetermined threshold. This is because the absence of an observed sign can only be considered meaningful if there was actually a significant chance to observe the sign. Furthermore, it should be noted here that not every vehicle passing a sign is always capable of observing this sign, for instance due to obstruction by another vehicle. Similarly, in identifying the first candidate corrections, it can be decided not to assess traffic signs in the map which occur on such segments.

[0021] As a result of its simplicity, the accuracy of this heuristic model is limited. For instance, for longer segments, it may consider two traffic signs the same even if they are quite some distance apart, leading to a false negative. More worryingly, it may lead to false positives, i.e. correct traffic signs being flagged for correction, in particular if an observed traffic sign is associated to the wrong segment of the digital map. Any method which can lead to false positives requires manual checking, which is quite labor intensive. Furthermore, it is static, and therefore difficult to improve on without adding further assumptions to the model.

[0022] However, it has been found that a combination of the machine learning model and heuristic model can lead to good results. In particular, the presently proposed method therefore combines an initial machine learning model - with some initial training, but which does not need to have a high accuracy yet - with a simple heuristic model to generate further training input, which is then used to update the model. Additionally, the machine learning model, which looks at the absolute position and therefore does not require the error-prone association of an observed traffic sign to a segment, can be used to validate the heuristic model, in particular discovering false positives.

[0023] Specifically, the method includes determining, based on the first candidate corrections and second candidate corrections, recommended corrections to the digital map, and using at least some of the recommended corrections to further train the initial machine learning model and obtain an updated machine learning model.

[0024] While other types of machine learning models could be considered, it has been found that a model of the "decision tree" type can lead to satisfactory results. Identifying first candidate corrections preferably comprises comparing a traffic sign of the traffic sign information in the digital map to a predetermined number of nearest neighbor traffic signs of the first plurality of observed traffic signs, for instance to three nearest neighbor traffic signs.

[0025] Identifying first candidate corrections preferably comprises identifying traffic signs of the traffic sign information which, based on the first plurality of observed traffic signs, seem to be superfluous, in the sense that they do not clearly correspond to any of the observed traffic signs. In particular, in embodiments where a number of nearest neighbor traffic signs are considered, traffic signs may be identified which do not correspond to any of the nearest neighbors; on the other hand, traffic signs which correspond to the nearest neighbor but are different from the next few nearest neighbors are likely to be correct. Again, while the explanation here is for the case in which the identification of superfluous signs is focused on, the first candidates may also include (additionally or alternately) other potential errors, such as errors in position, missing signs, etc.

[0026] Considering a number n of nearest neighbors may be done by identifying the n observed traffic signs for which the distance to the traffic signs under consideration is the lowest. The distance, as well as preferably the relative orientation, can be used as input for the machine learning model. Other inputs could be used as well (alternately or additionally). For instance, the confidence of the observed traffic sign can be a useful input. This may be calculated by looking at the ratio between the number of times a sign was observed and the number of chances to observe the sign, i.e. the number of sensor-equipped vehicles which passed through the relevant part of the road network and were theoretically capable of observing the traffic sign. It is quite normal for this ratio to be less than 1, for instance due to obstruction of a camera by another vehicle or other impediments; however, if traffic sign is observed only rarely, this may indicate that the observed traffic sign may not correspond to an actual traffic sign. Rather than putting the ratio / estimated confidence into the machine learning model directly, it is also possible to provide the number of observations, or both the number of observations and the number of chances. The bearing information of the vehicle or vehicles observing the sign may also be relevant, in particular since some traffic signs may only be observable for certain bearings. Likewise, the side of the road on which the traffic sign was observed may be useful input; relatedly, it is noted that in some digital maps, the traffic sign information may also indicate which side of the road a traffic sign is deemed to be positioned at, and this information could also (alternately or additionally) be provided as input to the model. Finally, it is noted once more that the type of the traffic sign may be input in the form of a combination of type (e.g., "speed limit") and value (e.g. "60 km / h") where appropriate.

[0027] In preferred embodiments, the objective is to identify as many superfluous signs as possible while avoiding false positives, i.e., while minimizing the chance of unnecessarily deleting a traffic sign from the traffic sign information in the map.

[0028] Therefore, it is preferable to include in the recommended corrections only traffic signs corresponding both to a first candidate correction and to a second candidate correction, e.g., traffic signs identified as superfluous both by the machine learning model and the heuristic model. While this means fewer results will be returned (and some errors will be missed), this minimizes the chance of false positives.

[0029] Of course, while the heuristic model generally returns a binary answer (at least if only superfluous signs are considered: a traffic sign is either present along a segment or it is not), or at least a definite answer, this is not necessarily the case for the machine learning network, and in preferable embodiments the output of the machine learning network comprises, for each of the first candidate corrections, a confidence value.

[0030] Based on the confidence value, it is possible to identify not only useful further training data, but it is also possible to distinguish between corrections which should be implemented and corrections which should be validated first. In light of this, the method preferably further includes identifying traffic signs corresponding to a first candidate correction with a confidence value above a predetermined threshold, and preferably also corresponding to a second candidate correction, as high-confidence candidate corrections. For instance, a threshold of the order of 85% confidence can be used. These high-confidence candidate corrections are highly likely to correspond to actual superfluous traffic signs (or other errors the models are configured to identify) and therefore do not require further validation. The traffic sign information may therefore be corrected based on these high-confidence candidate corrections. This correction can be performed automatically / immediately, or, preferably, the high-confidence candidate corrections may be added to a set of accumulated recommended corrections, which are implemented at a later stage. This may be advantageous because it avoids modifying part of the input the machine learning model was trained on, namely the original / current digital map. Furthermore, once the corrections are implemented, the digital map contains fewer errors - while of course this is the eventual purpose, a side-effect of prematurely correcting the map means the process will produce less potential training data.

[0031] The high-confidence corrections are also advantageously used for further training the machine learning model. Additionally, the high-confidence corrections provide a useful tool to assess the accuracy of the machine learning model: by sampling the output and presenting such a sample to a user for validation, it can be determined how accurate the model is.

[0032] First candidate corrections with a confidence value below the predetermined threshold could be discarded, but doing so may make it challenging to identify a sufficient number of errors. The method therefore preferably further includes identifying traffic signs corresponding to a first candidate correction with a confidence value below the predetermined threshold, and preferably also corresponding to a second candidate correction, as lower-confidence candidate corrections. These lower-confidence candidate corrections are then presented to a user for validation to obtain validated candidate corrections and rejected candidate corrections. The validated candidate corrections and / or the rejected candidate corrections may then be used to further train the machine learning model. Furthermore, the validated candidate corrections may be added to the accumulated recommended corrections, together with the high-confidence candidate corrections. Note that a lower bound on the confidence may be used as well, i.e., selecting lower-confidence candidate corrections with a confidence below the predetermined threshold discussed above, but above a second predetermined threshold. An advantage of this is that this will lead to fewer candidates for validation (and thus less required human effort), more of which are likely to result in validated candidate corrections. A disadvantage may be that the rate of improvement of the machine learning network and / or the speed of accumulating recommended corrections may be lessened. A skilled person will be able to determine the appropriate threshold(s) for selecting the lower-confidence candidate corrections.

[0033] The method described above may be iterated, using further sensor data. This is not only to obtain additional recommended corrections, but also to take advantage of the fact that the updated machine learning model is expected to be able to more accurately identify candidates. Furthermore, by iterating the method, the machine learning model can be further improved until a desired accuracy is achieved.

[0034] For instance, after the method described above is first performed, the following further steps may be performed as often as necessary to achieve the desired result. First, a second plurality of observed traffic signs may be identified based on second sensor data from sensors of a plurality of vehicles traversing the road network, comprising determining, for each observed traffic sign, at least a type and a position.

[0035] Using the updated machine learning model - i.e. the most recently updated version of the model - third candidate corrections may be identified based on the digital map and the second plurality of observed traffic signs.

[0036] Each observed traffic sign of the second plurality of observed traffic signs may be associated to a segment of the digital map, and fourth candidate corrections may be identified by comparing, for at least a subset of the segments of the digital map, traffic signs along the segment according to the digital map to observed traffic signs of the second plurality of observed traffic signs associated with the segment.

[0037] Based on the third candidate corrections and fourth candidate corrections, further recommended corrections to the digital map may then be determined. These further recommended corrections may be added to the recommended corrections accumulated in earlier iterations of the method. At least some of the further recommended corrections are then used to further train the updated machine learning model and obtain a further updated machine learning model, and so on and so on. These steps can be iterated until the accuracy of the machine learning model exceeds a predetermined threshold, and / or until it seems the number of accumulated recommended corrections (e.g. superfluous signs) is estimated to have reached a desired level. For instance, if the number of additional merged candidates found per additional repetition of the method decreases to below a predetermined threshold, the result can be deemed sufficient. The digital map may then be updated based on the merged candidate corrections, i.e. included all accumulated recommended corrections identified as a result of the repeated performing of the steps. However, it is noted that corrections could also be performed earlier and / or more frequently.

[0038] For completeness, it is noted that it is preferred to perform the method in a similar manner each time, and that the additional optional details disclosed in the context of the first repetition of the steps are therefore also applicable to the later repetitions. For instance, any details relating to the identification of the first candidate corrections may also be considered application to the identification of the third candidate corrections.

[0039] One exception is that at some point, the machine learning model may be determined - for instance by sampling the output and presenting it to a user for validation - to be accurate enough that the use of the heuristic model no longer improves the result. In such a case, further iterations may omit the step of identifying fourth candidate corrections, and the recommended corrections may be determined based only on the third candidate corrections (and preferably the associated confidence).

[0040] There are various ways in which the initial machine learning model may be obtained. For instance, if the method was already performed in a certain country and / or at an earlier point in time, the eventual updated machine learning model obtained therefrom can be used as the initial machine learning model for another country and / or a significantly later time period.

[0041] The initial machine learning model may also be obtained by employing the heuristic model alone, and making use of user validation to ensure no false positives are included.

[0042] In particular, an initial plurality of observed traffic signs may be identified based on initial sensor data from sensors of a plurality of vehicles traversing the road network, comprising determining, for each observed traffic sign, at least a type and a position. Each observed traffic sign of the initial plurality of observed traffic signs may then be associated to a segment of the digital map. Initial candidate corrections may be identified by comparing, for at least a subset of the segments of the digital map, traffic signs along the segment according to the digital map to observed traffic signs of the initial plurality of observed traffic signs associated with the segment. These initial candidate corrections may then be presented to a user for validation, thus obtaining validated candidate corrections. These validated candidate corrections and / or the rejected candidate corrections may then be used to train the initial machine learning model. In particular, the model is trained to output the validated candidate corrections based on traffic sign information included in the digital map corresponding to the position and type of at least one traffic sign and the type and position of observed traffic signs of the initial plurality of observed traffic signs, and preferably also not to output the rejected candidate corrections.

[0043] The higher the number of validated and / or rejected correction candidates, the more accurate the initial machine learning network would be. In fact, with a high enough number of validated correction candidates, the machine learning network could be used by itself, without requiring the heuristic model. However, this manner of obtaining training data is quite labor intensive. It had been found that validating about 10.000 candidate corrections (or which generally about 50-75% leads to a validated candidate correction - note that this tends to be quite dependent on the type of sign) is sufficient to train an initial machine learning network such that it is accurate enough for the method to be initiated.

[0044] Brief description of the figures

[0045] The proposed method and system will be further explained at the hand of the figures, in which:

[0046] Fig. 1 is a flowchart corresponding to the proposed method; Fig. 2 is a block diagram further illustrating the proposed method, in particular the aspects relating to the training of the machine learning model;

[0047] Fig. 3 is a block diagram illustrating a more detailed embodiment of the method;

[0048] Fig. 4 shows an example process diagram using the proposed method;

[0049] Fig. 5A and 5B show examples of the heuristic model;

[0050] Fig. 6A, 6B and 6C illustrate further examples of the heuristic model;

[0051] Fig. 7 illustrates the machine learning model;

[0052] Fig. 8 shows the influence of certain features on the output of the machine learning model. Detailed description

[0053] Fig. 1 is a flowchart corresponding to the proposed method. In step SI, an initial machine learning (ML) model is obtained. In step S2, observed traffic signs are identified based on sensor data received from a plurality of vehicles traversing the road network. It is noted that step SI could also take place after the first time step S2 is performed; however, step SI in principle only needs to be performed once, while in general step S2 will be repeated, as additional sensor data is received. The observed traffic signs are compared to traffic signs in a digital map of which the accuracy is to be verified in two ways, which can be performed in parallel. In step S3, this is done using the machine learning model. The first time step S3 is performed, this is done using the initial machine learning model; however, subsequent times the step is performed, an updated version of the machine learning model will be used. The "heuristic model" is used in steps S4 and S5: in step S4 each observed traffic sign is associated with a segment of the digital map, and in step S5 a segment-by- segment comparison is performed. In step S6, recommended corrections are determined. In step S7, the machine learning model is updated based on at least some of the recommended corrections, as well as potentially other output from the ML model. Steps S2-S7 are then repeated, preferably based on new sensor data, until a sufficient number of corrections are identified. The recommended corrections are used in step S8 to update the map. While this may be done immediately or in several batches, as was outlined above it may be preferable to correct the digital map only once a sufficient number of corrections have been identified, and / or once the ML model has reached a desired level of accuracy. This is indicated by the dash-dotted line.

[0054] Steps S4 and S5 are indicated with dashed borders, and the lines from steps S2 to S4 to S5 to S6 are indicated with dotted lines. This is to indicate that after a certain number of iterations, the ML model will have improved to such an extent that the (static) heuristic model may unnecessarily limit the output recommended corrections. At this stage, in further iterations, steps S4 and S5 may be omitted.

[0055] Fig. 2 illustrates the various elements involved in the method. Sensor data 20 is obtained from vehicles 10, 11, 12, resulting in the identification of a plurality of observed signs 30. An observed sign may result from a plurality of sensor-derived observations by a plurality of vehicles. Each observed sign 30 has at least a type (optionally including a value for some types), and a position, generally expressed in terms of longitude or latitude. Furthermore, each observed sign 30 may be associated with additional information, in particular information relating to how it was obtained, for instance any of: a number of times it was observed, a number of times it could have been observed, a confidence value, a side of the road on which it was observed, a bearing of the vehicles observing the sign, etc.

[0056] The observed signs 30 are compared to traffic sign information 41 of digital map 40 in two ways. One is by using machine learning model 50, and inputting at least the position and type information of the observed signs 30 and the position and type information included in the traffic sign information 41 of digital map 40. Further information relating to the observed signs 30 and / or further elements of traffic sign information 41 can also be used. Some preprocessing may also be done, for instance to identify, for each traffic sign in digital map 40 a certain number of nearest neighbors among observed signs 30. The traffic sign information to be analyzed may also be filtered, for instance to exclude traffic signs in areas of the digital map 40 for which insufficient sensor data 20 was received - it is important to compare the traffic sign information to a meaningful amount of observed signs derived from sensor data to avoid errors. Machine learning model 50 then outputs first candidate corrections 51, and generally also outputs additional information, for instance a confidence associated with each first candidate correction 51. A candidate correction generally corresponds to a traffic sign in the digital map 40 which is incorrect in some way, for instance superfluous, and may additionally indicate what the traffic sign information should be corrected to. Generally, the term "candidate correction" in this document refers to information that is sufficient to update traffic sign information 41 such that the accuracy of digital map 40 is improved.

[0057] In parallel, each observed sign 30 is associated, in block 61, with a segment 42 of digital map 40 to allow for a segment-by-segment comparison in block 62. Preferably, only segments are considered which were traversed by at least a certain number of vehicles 10. This segment-by-segment comparison yields second candidate corrections 62.

[0058] Recommended corrections 70 are determined based on first candidate corrections 51 and second candidate corrections 62. For instance, the recommended corrections may correspond to candidate corrections which occur both as a first candidate correction 51 and as a second candidate correction 62. Other ways of determining recommended corrections from first candidate corrections 51 and second candidate corrections 62 can also be used, including methods which include some form of human verification. Aside from being used - either directly or at some later point in time - to correct digital map 40, the recommended corrections are then used to update / improve machine learning model 50. This is indicated by the dashed line in Fig. 2 - feedback to machine learning model 50 is also indicated using such lines in Fig. 3. The method may then be repeated based on new sensor data, until sufficient recommended corrections have been collected. In case the machine learning model 50 achieves a desired accuracy prior to sufficient recommended corrections having been collected - for instance if the accuracy of the machine leaning model 50 becomes higher than that of the heuristic model - blocks 60, 61 and 62 may be omitted in further repetitions, in which case recommended corrections 70 can be determined based on first candidates 51 alone. As in Fig. 1, this is indicated by the dashed outlines on these blocks, and the dotted arrow going to and from these blocks.

[0059] Fig. 3 illustrates a slightly more elaborate version of the method, which shows how the confidence value output by machine learning model 50 can be advantageously used. Repeated / similar elements which have already been discussed in the context of Fig. 2 are omitted here.

[0060] In the example shown in Fig. 3, first candidates 51 output by machine learning model 50 are subdivided into high-confidence first candidates 52 and lower-confidence first candidates 53 based on the confidence value associated with each first candidate being above or below a certain predetermined threshold.

[0061] For completeness, a further subdivision could also be envisaged, for instance assessing also traffic signs which are estimated as correct but with a confidence falling below a certain threshold. Furthermore, in some cases the first candidates may include different types of errors. Furthermore, it is not excluded that the predetermined threshold may be adjusted in later iterations of the method.

[0062] All high-confidence first candidates 52 could be identified as recommended corrections 70, or optionally - at least during a first number of iterations of the method - a high-confidence first candidate 52 which corresponds to a second candidate 62 can be identified as a recommended correction 70. Due to the high confidence, these recommended corrections 70 can be automatically added to a set of accumulated corrections 80, to be implemented directly or preferably at a later time. Furthermore, the recommended corrections 70 may be used to further train / update machine learning model 50.

[0063] Lower-confidence first candidates 53 are presented to a user for validation. Lower-confidence first candidates 53 which indeed indicate a required correction become validated candidates 54; lower- confidence candidates 53 which correspond to a sign which is correct in digital map 40 become rejected candidates 55. Note that some preliminary filtering could take place before presenting lower-confidence first candidates 53 to a user, for instance based on second candidates 62. Validated candidates 54 indicate required corrections, and are therefore added to accumulated corrections 80. Furthermore, validated candidates 54 and / or rejected candidates 55 can be used to update / further train machine learning model 50 - it is noted that this of course requires labeling the rejected candidates as corresponding to correct signs.

[0064] Fig. 4 illustrates how the method can be implemented in a map correction process. The timing indicated is of course a mere implementation choice. Generally, it is desirable to find the minimum amount of manual labeling which still leads to a rapid enough pace of improvement of the machine learning model and of the accumulation of required corrections. The process is ideally repeated periodically - for instance every six months or every year - to account for the fact that traffic signs may change in reality, and that the accuracy of the traffic sign information will generally decline over time as a result. Model training 111 may correspond to either step SI in Fig. 1 of obtaining the initial machine learning network based on an initial set of training data / ground truth 100, or, in later repetitions, to step S7 in Fig. 1 of updating the machine learning model. Block 112 corresponds to step S3 of using the machine learning model, and "ML based leads" 113 correspond to first candidates 51. Block 114 reflects the use of the heuristic model, i.e. steps S4 and S5 of Fig. 1. The leads are then fused at block 116, corresponding to step S6 in Fig. 1.

[0065] From fused leads 116, the ones that reach the automatic threshold removal are identified in block 130, and added to a collection of validated sign leads 140, which correspond to the accumulated corrections 80.

[0066] The elements requiring user input are represented in block 120. Specifically, fused leads 116 are sampled in block 121 and manually labelled and validated in block 122, for three reasons. The first is to assess the precision of the model, and thereby establish whether or not further iterations of the method are required. The second is to identify further "validated sign leads", i.e. to add to the set of accumulated corrections. The third is to obtain additional training data to obtain the method. For the first reason, the sampling may be from all fused leads, though it may be advantageous to select leads based on whether or not they meet the automatic removal threshold. For the second and third purpose, the sampling should include at least leads which do not meet this threshold, but it may be useful to include some leads which do meet the threshold as well. Validated leads, i.e. leads which a user has identified as corresponding to an error in the map, validated leads and / or rejected leads may be added to the set of "ground truth data" 100 to improve the machine learning model.

[0067] Fig. 5A and 5B illustrate the heuristic model and some of its drawbacks. Both figures show two segments, 401 and 402, of a digital map. In each case, the top figure shows the segments in the digital map and the traffic sign 410 according to the digital map, and the bottom figure shows an observed traffic sign 310, 311 in relation to the same segments 401, 402. The digital map is the same in both Fig. 5A and Fig. 5B, and indicates the presence of a traffic sign 410, specifically a 50 km / h speed limit sign, present near the start of segment 402. In the first scenario, shown in Fig. 5A, the observed traffic sign 310 is associated to the same segment, but is in fact at quite some distance from the traffic sign in the map: it is located more towards the end of segment 402. This means the heuristic model, in which a comparison is made segment by segment, will not identify this as a potential error - in other words, this will lead to a false negative. In the second scenario, shown in Fig. 5B, the observed traffic sign 311 is closer, in terms of absolute position, to traffic sign 410 in the map, and it seems plausible (taking into account possible measurement error) that the observed sign corresponds to the traffic sign in the map. However, since observed traffic sign 311 is mapped to segment 401, rather than segment 402, the heuristic model will flag this case as potential error, which may be a false positive.

[0068] Of course, the heuristic model could be refined and improved, for instance by subdividing the segments further. However, this will commensurately increase computational complexity and load, and will likely not avoid all errors, as will be explained at the hand of Figs. 6A-6C.

[0069] In both scenarios shown in Fig. 5A and Fig. 5B, it is clear that a method - for instance the proposed machine learning model - which looks at the absolute position, and specifically at the distance between a traffic sign 410 on the map and at least one observed traffic sign 310, 311, has at least the potential to be more accurate.

[0070] Figs. 6A-6C illustrate that the success of the heuristic model is highly dependent on the accuracy with which an observed traffic sign 312 is associated with a segment of the map 404, 405. The section of road which is represented here in an abstracted way corresponds to a point at which a single road segment 403 is connected, via node 420, to two continuing segments 404, 405, which deviate from each other at a fairly small angle. This is a scenario which occurs quite frequently, for instance at exits from highways. Furthermore, these scenarios often involve a plurality of traffic signs quite close together, for instance indicating a lowered speed at the exit with respect to the highway, and therefor are quite relevant for the presently proposed method.

[0071] Fig. 6A shows that digital map 40 includes traffic sign 411 present on segment 404. Fig. 6B and Fig. 6C indicate the same observed traffic sign 312, present between segments 404 and 405. In the scenario shown in Fig, 6B, this observed traffic sign 312 is mapped to segment 404. In the scenario shown in Fig. 6C, observed traffic sign 312 is mapped to segment 405. This means that the outcome of the heuristic model will be dependent on the mapping / associating: the scenario of Fig. 6B will not lead to the identification of a candidate correction / lead, while the scenario of Fig. 6C will not. This has the potential to lead both to false negative and false positives. The problem is further compounded by the fact that in the original creation of the digital map, a similar error may have been made, such that the positioning of traffic sign 411 on segment 404 was erroneous as well. Therefore, it is clear that the heuristic model is limited in its applicability, in particular due to the risk of false positives.

[0072] The general principle behind the machine learning model devised to supplement the heuristic model in accordance with the proposed method is illustrated in Fig. 7. A traffic sign 412 of the map is illustrated along a road 406, together with the nearest three observed signs, 313, 314 and 315, which are at respective distances DI, D2 and D3 from traffic sign 412. The positions themselves (of both traffic sign 412 and observed traffic signs 313, 314 and 315) could be used as input for the machine learning network, and / or the distances (potentially associated with angle information) could be used.

[0073] The influence of the various potential input features to the result, in particular to the identification of superfluous traffic signs, is illustrated in Fig. 8. By far the most influential factor has been found to be the DI (indicated as "eagle_distance_l"), i.e. the distance between traffic sign 412 and nearest observed traffic sign 313. In particular, a high distance is highly likely to indicate a potential error in the map, and therefore a "lead" / " candidate correction". Other features of which the impact is lesser but which lead to fairly clearly differentiated results are the traffic sign information in the map relating to the side of the road on which the traffic sign is deemed to be positioned ("genesis_position_left_side" and "genesis_position_right_side") as well as the corresponding information relating to all observed traffic signs, and in particular the nearest observed traffic sign ("eagle_left_side_l" and "eagle_right_side_l" - note that numerals 1, 2 and 3 correspond to the three nearest observed traffic signs ordered from closest to furthest). Interestingly, the distance to the third nearest observed sign has been observed as being more strongly predictive of the outcome than the second nearest observed sign - this may be because signs regularly occur in pairs, on either side of the road, as a result of which the second nearest observed sign could be the twin of the nearest observed sign or, as in Fig. 7, of the third nearest observed sign.

[0074] While the above focuses on the identification of superfluous signs, the disclosure is not limited thereto, as the disclosed principles and techniques are also applicable, with slight modifications where appropriate, to identifying other types of erroneous traffic sign information included in a digital map.

[0075] The figures show specific examples meant to illustrate the method, but should not be taken as limiting the disclosure, or the scope of the claims. A skilled person will be able to devise alternatives for many of the implementation choices discussed in this document.

Claims

CLAIMS1. Computer-implemented method of verifying the accuracy of traffic sign information included in a digital map based on sensor-derived observations, wherein the digital map comprises a plurality of segments representative of a road network and the traffic sign information, the traffic sign information indicating, for each of a plurality of traffic signs, the type of traffic sign and the location along a segment of the digital map, the method comprising: identifying a first plurality of observed traffic signs based on first sensor data from sensors of a plurality of vehicles traversing the road network, comprising determining, for each observed traffic sign, at least a type and a position; obtaining an initial machine learning model configured to output candidate corrections based on traffic sign information included in the digital map corresponding to the type and position of at least one traffic sign and the type and position of observed traffic signs; identifying first candidate corrections based on the digital map and the first plurality of observed traffic signs using the initial machine learning model; associating each observed traffic sign of the first plurality of observed traffic signs to a segment of the digital map; identifying second candidate corrections by comparing, for at least a subset of the segments of the digital map, traffic signs along the segment according to the digital map to observed traffic signs of the first plurality of observed traffic signs associated with the segment; determining, based on the first candidate corrections and second candidate corrections, recommended corrections to the digital map; using at least some of the recommended corrections to further train the initial machine learning model and obtain an updated machine learning model.

2. Computer-implemented method of claim 1, wherein identifying second candidate corrections comprises identifying traffic signs along a segment for which there is no traffic sign of the same type associated with the segment in the first plurality of observed traffic signs.

3. Computer-implemented method of claim 1 or 2, wherein, in the step of identifying second candidate corrections, only segments are considered for which the number of vehicles with sensorsobtaining the first sensor data traversing the segment of the road network exceeds a predetermined threshold.

4. Computer-implemented method of any of the previous claims, wherein identifying first candidate corrections comprises comparing a traffic sign of the traffic sign information in the digital map to a predetermined number of nearest neighbor traffic signs of the first plurality of observed traffic signs.

5. Computer-implemented method of any of the previous claims, wherein identifying first candidate corrections comprises identifying traffic signs of the traffic sign information which, based on the first plurality of observed traffic signs, seem to be superfluous.

6. Computer-implemented method of any of the previous claims, wherein the machine learning model is configured to further take into account at least one of the following features: the distance between the position of a traffic sign according to the digital map and the distance of at least one observed traffic sign; the confidence of the observed traffic sign, wherein the confidence is preferably based on the ratio between the number of times a sign was observed and the number of chances to observe the sign; the bearing information of the vehicle observing the sign; the side of the vehicle on which the traffic sign was observed; the side of the road on which the traffic sign is located in accordance with the traffic sign information in the digital map; the value of the traffic sign.

7. Computer-implemented method of any of the previous claims, wherein determining, based on the first candidate corrections and second candidate corrections, recommended corrections to the digital map comprises identifying traffic signs corresponding to a first candidate correction and to a second candidate correction.

8. Computer-implemented method of any of the previous claims, wherein the machine learning model is configured to output, for each of the first candidate corrections, a confidence value.

9. Computer-implemented method of claim 8, wherein determining, based on the first candidate corrections and second candidate corrections, recommended corrections to the digital map further comprising identifying traffic signs corresponding to a first candidate correction with a confidence value above a predetermined threshold, and optionally also corresponding to a second candidate correction, as high-confidence candidate corrections.

10. Computer-implemented method of claim 9, wherein the high-confidence candidate corrections are used to further train the machine learning model.

11. Computer-implemented method of claim 9 or 10, wherein the high-confidence candidate corrections are added to a set of accumulated recommended corrections.

12. Computer-implemented method of any of claims 8-11, wherein determining, based on the first candidate corrections and second candidate corrections, recommended corrections to the digital map comprises: identifying traffic signs corresponding to a first candidate correction with a confidence value below the predetermined threshold, and preferably also corresponding to a second candidate correction, as lower-confidence candidate corrections; presenting the lower-confidence candidate corrections to a user for validation to obtain validated candidate corrections and rejected candidate corrections.

13. Computer-implemented method of claim 11 and 12, further comprising adding the validated candidate corrections to the set of accumulated recommended corrections.

14. Computer-implemented method according to claim 12 or 13, wherein the validated candidate corrections and / or the rejected candidate corrections are used to further train the machine learning model.

15. Computer-implemented method of any of the previous claims, further comprising: identifying a second plurality of observed traffic signs based on second sensor data from sensors of a plurality of vehicles traversing the road network, comprising determining, for each observed traffic sign, at least a type and a position; identifying third candidate corrections based on the digital map and the second plurality of observed traffic signs using the updated machine learning model;associating each observed traffic sign of the second plurality of observed traffic signs to a segment of the digital map; identifying fourth candidate corrections by comparing, for at least a subset of the segments of the digital map, traffic signs along the segment according to the digital map to observed traffic signs of the second plurality of observed traffic signs associated with the segment; determining, based on the third candidate corrections and fourth candidate corrections, further recommended corrections to the digital map; using at least some of the further recommended corrections to further train the updated machine learning model and obtain a further updated machine learning model.

16. Computer-implemented method of claim 15 comprising repeating the steps until the accuracy of the machine learning model exceeds a predetermined threshold.

17. Computer-implemented method of any of the previous claims, further comprising updating the digital map based on the recommended corrections.

18. Computer-implemented method of any of the previous claims, wherein obtaining the initial machine learning model comprises: identifying an initial plurality of observed traffic signs based on initial sensor data from sensors of a plurality of vehicles traversing the road network, comprising determining, for each observed traffic sign, at least a type and a position; associating each observed traffic sign of the initial plurality of observed traffic signs to a segment of the digital map; identifying initial candidate corrections by comparing, for at least a subset of the segments of the digital map, traffic signs along the segment according to the digital map to observed traffic signs of the initial plurality of observed traffic signs associated with the segment; presenting the initial candidate corrections to a user for validation to obtain validated candidate corrections and rejected candidate corrections; training the initial machine learning model, using at least the validated candidate corrections and optionally also the rejected candidate corrections, to output the validated candidate corrections based on traffic sign information included in the digital map corresponding to the position and type of at least one traffic sign and the type and position of observed traffic signs of the initial plurality of observed traffic signs.

19. Machine learning network directly obtained as a result of performing the method of any of claims 1-17.

20. System comprising one or more processors configured to perform the method of any of claims 1-18.

Images