A method and apparatus for determining the quality of confidence in the existence of map features.

Abstract

This specification provides one or more embodiments of a method and apparatus for determining the quality of confidence in the existence of map features. The method includes: comparing intelligence data corresponding to a road to be evaluated with map data of a target high-precision map to obtain feature change information of each map feature in the target high-precision map; the intelligence data includes: real-world collected information of map features in the road to be evaluated; updating the confidence in the existence of map features corresponding to the target high-precision map based on the feature change information; obtaining quality result data of the confidence in the existence of map features based on the updated confidence in the existence of map features and the corresponding true value of map feature changes; the true value of map feature changes includes: real feature change information obtained based on the intelligence data and simulated feature change information of the road to be evaluated.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of mapping, and more particularly to a method and apparatus for determining the quality of confidence in the existence of map features. Background Technology

[0002] High-definition (HD) maps contain map features with an attribute called "existence confidence." This existence confidence expresses the real-world presence of map features. The real-world presence of map features may change; for example, a traffic pole on a road might be removed. Therefore, the existence confidence of that "traffic pole" map feature must be updated promptly to accurately reflect its real-world presence. Clearly, the accuracy of the existence confidence of map features is crucial for ensuring the data quality of HD map data; thus, it is necessary to conduct quality assessments of the existence confidence of map features.

[0003] In related technologies, the confidence level of the aforementioned existence is typically assessed based on real data, such as intelligence data corresponding to the road to be evaluated. However, the drawback of this approach is that real data may not capture all changes in map features in the real world, making it impossible to evaluate recall and resulting in insufficient evaluation metrics. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification provide a method and apparatus for determining the quality of confidence in the existence of map features.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a method for determining the quality of the confidence level of the existence of map features is provided, the method comprising:

[0007] Based on the comparison between the intelligence data corresponding to the road to be evaluated and the map data of the target high-precision map, the feature change information of each map element of the target high-precision map is obtained; the intelligence data includes: real-world collected information of the map elements in the road to be evaluated; the map data is the high-precision map data corresponding to the road to be evaluated;

[0008] Based on the change information of the elements, update the confidence level of the existence of the map elements corresponding to the target high-precision map;

[0009] Based on the updated confidence level of the map features and the corresponding ground truth values ​​of map feature changes, the quality result data of the confidence level is obtained; the ground truth values ​​of map feature changes include: real feature change information obtained based on the intelligence data and simulated feature change information of the road to be evaluated; the map data of the target high-precision map is obtained by modifying the initial high-precision map based on the simulated feature change information.

[0010] According to a second aspect of one or more embodiments of this specification, a quality determination apparatus for the confidence level of the existence of map features is provided, the apparatus comprising:

[0011] The data production module is used to compare the intelligence data corresponding to the road to be evaluated with the map data of the target high-precision map to obtain the feature change information of each map element of the target high-precision map; the intelligence data includes: real-world collected information of the map elements in the road to be evaluated; based on the feature change information, the confidence level of the existence of the map elements corresponding to the target high-precision map is updated; the map data is the high-precision map data corresponding to the road to be evaluated;

[0012] The quality measurement module is used to obtain the quality result data of the confidence level based on the updated confidence level of the map features and the corresponding true values ​​of map feature changes; the true values ​​of map feature changes include: real feature change information obtained based on the intelligence data and simulated feature change information of the road to be evaluated; the map data of the target high-precision map is obtained by modifying the initial high-precision map based on the simulated feature change information.

[0013] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising:

[0014] processor;

[0015] Memory used to store processor-executable instructions;

[0016] The processor executes the executable instructions to implement the method described in any embodiment of this specification.

[0017] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided having computer instructions stored thereon that, when executed by a processor, implement the methods described in any embodiment of this specification.

[0018] The method and apparatus for determining the confidence level of the existence of map features according to embodiments of this specification obtain true values ​​by simulating changes in map features. Based on these true values, the precision and recall of data products can be calculated, resulting in more comprehensive quality assessment. Furthermore, simulating changes in map features allows for an increase in the sample size used in the assessment, leading to more accurate and comprehensive quality results. The method for simulating changes in map features can also be flexibly configured according to assessment needs. The true values ​​used in this method can be used for multiple assessments, improving the efficiency of quality assessment. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in one or more embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in one or more embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a method for determining the quality of the confidence level of the existence of map features, provided in an exemplary embodiment.

[0021] Figure 2 This is a flowchart of a method for determining the quality of the confidence level of the existence of map features, provided in an exemplary embodiment.

[0022] Figure 3 This is a structural diagram of a quality determination apparatus for the confidence level of the existence of map features, provided in an exemplary embodiment. Detailed Implementation

[0023] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0024] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0025] High-precision maps include various map elements, such as, but not limited to, OBJ (Object, road component), high-precision lane dividers, etc. The OBJ can include, but is not limited to, various types of components as shown in the following examples:

[0026] 1) Ground traffic signs;

[0027] For example, text on the road, left and right turn signs, etc.

[0028] In the description of the embodiments in this specification, HDPSymbol_PG is used to represent this type of ground traffic sign.

[0029] 2) Traffic signs;

[0030] For example, signs prohibiting parking, U-turns, and pedestrian crossings.

[0031] In the description of the embodiments in this specification, for the sake of brevity, various traffic signs are referred to as HDPTrafficSign.

[0032] 3) Traffic lights;

[0033] For example, traffic lights at intersections.

[0034] In the description of the embodiments in this specification, various types of traffic lights are referred to as HDPTrafficLight.

[0035] 4) Rod-like components;

[0036] For example, road signs placed on roads.

[0037] In the description of the embodiments in this specification, various rod-like components are referred to as HDPPole.

[0038] In addition, the high-precision lane divider may include: single solid lines, lane lines such as guardrails, and road boundaries. In this embodiment, the above divider is represented by HDPDivider.

[0039] In the description of the embodiments in this specification, the confidence level of the existence of map features in a high-precision map can be referred to as a data product. This means that personnel responsible for data maintenance can periodically collect the status of map features in the real world and update the confidence level of the map features in the high-precision map in a timely manner to strive to maintain a relatively accurate confidence level and ensure the data quality of the high-precision map data. Therefore, the confidence level of the existence of map features can also be considered as data produced after the aforementioned comparison and updating processes, and thus referred to as a data product. This embodiment aims to evaluate the quality of this data product. Furthermore, the data product may include a set of confidence levels for multiple map features in the road to be evaluated. For example, if the road to be evaluated includes 20 map features, then the set of confidence levels for these 20 map features can be considered a data product.

[0040] The confidence level of existence is used to represent the reliability of the existence of map features in a high-definition map. For example, if the confidence level of the existence of a map feature is 10%, it means that the map feature is unlikely to exist in the real world. However, if the confidence level of the existence of a map feature is 98%, it means that there is a high probability that the feature exists in the real world. When the state of a map feature in the real world (e.g., a traffic sign) changes, such as from existing to non-existent, the confidence level of the map feature's existence can be refreshed through this data product, thereby achieving rapid updates to the high-definition map.

[0041] To ensure the quality of confidence-based data products, quality assessments are necessary to evaluate whether the results meet the needs of high-precision map users. For example, the assessment should evaluate whether the confidence level of map feature existence reflects the real-world presence of the feature, and whether the updates to the confidence level correspond to changes in the real-world map feature. This specification provides a method for determining the quality of confidence-based map feature existence. This method can be used to assess the quality of confidence-based map feature existence data products, aiming to provide a more comprehensive evaluation of the data product.

[0042] First, compared to existing technologies, the quality determination method in this specification uses simulated feature change information to generate true values ​​for map feature changes used in quality assessment. These true values ​​correspond to real-world map feature changes, such as the addition of a traffic pole on a real-world road or a change in the speed limit value of a speed limit sign on a real-world road. However, in this specification's embodiment, the true values ​​for map feature changes include not only real feature change information obtained from real intelligence data but also simulated road feature change information.

[0043] This method of simulating changes in map features and generating truth values ​​for evaluation:

[0044] On the one hand, the sample size used in the evaluation can be increased. For example, in the existing technology, the sample used in the evaluation is the change of map elements reflected by intelligence data. By increasing the simulated changes of map elements, the sample size used in the evaluation can be increased, which can make the quality result data obtained from the evaluation more accurate and sufficient.

[0045] On the other hand, since it uses simulated changes in map elements, it is possible to flexibly set how to simulate changes in map elements according to evaluation needs. For example, it is possible to flexibly select the type of map element to be simulated or flexibly set the type of change of map elements, which helps to conduct a more comprehensive evaluation of all map elements.

[0046] On the other hand, this method also generates a truth value for evaluation. The existence of the truth value enables the calculation of the "recall" evaluation metric, making the evaluation metric more comprehensive. Moreover, this truth value can be used for multiple evaluations, improving the efficiency of quality evaluation.

[0047] The following describes in detail the process of determining the quality of the confidence level of the existence of this map feature. OBJ will be used as an example in this description; however, it should be understood that the map feature involved in this method can be any feature other than OBJ. The description of this quality determination method is divided into two stages: the first stage is the creation of ground truth values ​​for the changes in the map feature used for evaluation; the second stage is the quality evaluation of the existing confidence level data product using the created ground truth values.

[0048] [Authenticity Creation]:

[0049] Figure 1 This is a truth-making flowchart in an exemplary embodiment of a method for determining the quality of confidence in the existence of map features, such as... Figure 1 As shown, the creation of this truth value may include the following processes:

[0050] In step 100, the intelligence data of the road to be evaluated is compared with the initial high-precision map to obtain the real element change information.

[0051] The roads to be evaluated can be designated road sections for evaluation. The intelligence data can include photos and trajectory information collected on the roads to be evaluated. For example, intelligence data can be collected on the roads to be evaluated using an information collection device, which simultaneously records the device's operating trajectory and corresponding photos. This allows the photos and their corresponding road locations to be obtained based on the trajectory. The collected photos can include various map elements on the evaluation road, thus obtaining real-world data collection information of the map elements on that road.

[0052] In addition, this intelligence data can usually be high-frequency intelligence data, that is, it can be collected at a high frequency. For example, for a certain road, multiple observation data (set to 5 times) can be collected within its historical period (such as within 2 weeks), including photos and tracks collected 5 times within these 2 weeks.

[0053] The intelligence data obtained in this step can be a compiled data package, which can be in the format of Table 1 below:

[0054] Table 1. Information on Intelligence Data

[0055] Serial Number Fields Parameter Description 1 task_ID Unique identifier ID within the table, data package ID 2 Info_ID The true sample sequence number is the same for the info_ID corresponding to the data of the same road segment. 3 name Road name where the data is located 4 Hroad_lists This data contains all high-precision road IDs, separated by commas. 5 City The city corresponding to this data 6 mesh The mesh corresponding to this data 7 ossaddress The OSS address where this data is stored. 8 Collecttime Data collection time, hours:minutes:seconds

[0056] The meanings of the fields in Table 1 above are explained as follows:

[0057] Taking the example of a road mentioned above that underwent five observations over two weeks, each observation yielded photos and trajectory information (i.e., intelligence data) of the road. Each observation's intelligence data can be represented by an "Info_ID," with different "Info_IDs" corresponding to different observations. The intelligence data from all five observations corresponds to a single "task_ID." Furthermore, this road may include multiple high-precision road IDs. The "mesh" corresponding to this data refers to a map sheet. A national high-precision map can be divided into multiple map sheets, each corresponding to a region. The map sheet containing this road is the mesh corresponding to the road mentioned above.

[0058] After obtaining the intelligence data (i.e., the aforementioned data package) for the road to be evaluated, this intelligence data can be compared with the initial high-precision map to obtain information on changes in actual features. This information on changes in actual features describes the changes in map features that have occurred in the intelligence data compared to the initial high-precision map. Since the high-precision map will be modified in subsequent steps, this embodiment refers to the high-precision map before modification as the initial high-precision map, and the high-precision map after modification in subsequent steps as the target high-precision map.

[0059] Furthermore, the collected intelligence data can include data from multiple road segments. Before comparing the intelligence data with the initial high-precision map, road segments that meet the consistency observation data requirements can be selected for quality assessment. Meeting the consistency observation requirement means that the existence status of the same map feature (OBJ) is consistent across multiple observation data sets. For example, in five observations, a certain traffic sign may not be visible in any of the collected photographs.

[0060] After selecting intelligence data that meets the consistency requirements, the intelligence data can be compared with the initial high-definition map to determine which map features have changed. These changes can then be recorded using the format shown in Table 2.

[0061] Table 2. Record of Map Feature Changes

[0062]

[0063]

[0064] As shown in Table 2 above, each HD_ID in this change record table corresponds to one map feature, meaning that one map feature can have one HD_ID corresponding to change information. If the intelligence data is compared with the initial high-precision map and three map features have changed, then there can be three change records as shown in Table 2. The HD_ID field of each change record is the feature ID of these three map features. Furthermore, the Info_id and Task_id to which the map feature belongs can also be recorded through "Socol_Info_id" and "Task_ids", indicating where the map feature originated.

[0065] In this embodiment, changes to map elements can include three types: addition, deletion, and attribute change. For example: addition could be "compared to the initial high-precision map, a traffic pole has been added to a certain road." Deletion could be "compared to the initial high-precision map, a traffic sign has been deleted from a certain road." Attribute change could be "compared to the initial high-precision map, the speed limit value of a speed limit sign has changed from s1 to s2."

[0066] Correspondingly, if the change type is addition, then `new_type` in Table 2 can record the specific type of the newly added map feature, such as adding an `HDPTrafficLight`. Similarly, if the change type is deletion, then `ori_type` in Table 2 can record the type of the deleted map feature, such as deleting an `HDPPole`. Furthermore, if the change type is attribute change, then `ori_type` can record the old attributes of the map feature before the change (such as the original speed limit), and `new_type` can record the new attributes of the map feature after the change (such as the new speed limit).

[0067] This step compares the intelligence data of the road to be evaluated with the initial high-precision map to obtain information on changes in actual features. This information describes the changes in map features that have occurred in the intelligence data compared to the initial high-precision map. For each map feature that has changed compared to the initial high-precision map, a change record table as shown in Table 2 can be recorded.

[0068] In step 102, simulated feature change information is generated, which describes the map feature changes that occur in the simulated intelligence data compared to the target high-precision map.

[0069] This step involves generating simulated feature change information, which describes the changes in map features that occur in the simulated intelligence data compared to the target high-precision map.

[0070] These simulated changes in map features can also be recorded in the format shown in Table 2.

[0071] For example, a new map feature can be simulated in the real world. For instance, by comparing intelligence data collected in the real world with the target high-definition map obtained after modifications in subsequent steps, it can be discovered that a traffic sign has been added to a certain road in the real world. When this map feature change information is recorded in Table 2, the TYPE type is "New", and the traffic sign is described in new_type.

[0072] For example, it's also possible to simulate the deletion of a map feature in the real world. For instance, by comparing real-world intelligence data with a high-resolution target map, it could be discovered that a traffic light has been removed from a certain road in the real world. Alternatively, it could simulate a change in the attributes of a map feature in the real world, such as a traffic line changing from a solid line to a dashed line.

[0073] The information on the changes in these simulated elements recorded in this step can be used as a basis for modifying the initial high-precision map in subsequent steps.

[0074] Furthermore, before generating simulated feature change information, it's necessary to first identify which map features to generate the simulated feature change information for. Specifically, based on the road to be evaluated, high-precision map data for the corresponding map sheet can be obtained. Map features other than the road to be evaluated are then deleted from the high-precision map data for that map sheet; that is, map features in areas without intelligence data are deleted. Simulated feature change information is then generated for the remaining map features in the high-precision map data for that map sheet. In one example, when generating simulated feature change information for the remaining map features, map features that have not changed from the intelligence data can be selected to generate the simulated feature change information.

[0075] For the high-resolution map resulting from the removal of map features in areas lacking intelligence data, list all map features. For any of the included map features of a particular type, select multiple subsets of map features, each subset containing at least one map feature of that type.

[0076] For example, a high-definition map can include map features of multiple feature types: HDPSymbol_PG, HDPTrafficLight, HDPTrafficSign, HDPPole, etc. For any feature type, such as HDPTrafficLight (traffic lights), there can be multiple traffic lights on the map. Assuming there are 20 traffic lights, they can be divided into multiple subsets of map features, for example, five traffic lights per subset. Thus, each subset includes multiple traffic lights. Of course, dividing them into three subsets, each containing five traffic lights, is just an example; actual implementation is not limited to this. A certain proportion of map features can be randomly selected to form a subset. Furthermore, not all map features in the high-definition map need to be included in the simulation; only a portion of the map features can be used.

[0077] For each map feature in any subset of map features, simulated feature change information is generated for that map feature; and the simulated feature change information generated for different subsets of map features has different types of feature changes. For example, in the example above, three subsets of map features are used to simulate different types of feature changes, one subset is used to simulate "addition", another subset is used to simulate "deletion", and yet another subset is used to simulate "attribute change".

[0078] Taking "deletion" as an example, for a subset of map features simulating deletion changes, simulated feature change information can be generated for each of the five traffic lights in that subset. This simulated feature change information describes the map feature changes that the simulated intelligence data would have compared to the high-definition map, i.e., the deletion of traffic lights. Furthermore, for map features in a subset simulating addition changes, the geometric location of the newly added map features in the high-definition map can be randomly generated.

[0079] In the example above, the generated simulated feature change information covers more types of feature changes, such as additions / deletions / attribute changes. Furthermore, this example also includes most map features from the high-precision map area corresponding to the intelligence data in the simulation, increasing the sample size for subsequent quality assessment and providing more data for subsequent indicator calculations, thus contributing to more accurate and comprehensive indicator calculations. The number of map features participating in the simulation and the types of map feature changes to be simulated can be flexibly set according to actual needs.

[0080] In step 104, based on the simulated feature change information, the corresponding map features in the initial high-precision map are modified to obtain the map data of the target high-precision map.

[0081] This step involves modifying map features in the initial high-definition map based on simulated feature change information. Table 3 below illustrates how to modify the initial high-definition map based on simulated feature change information:

[0082] Table 3 shows the map element operations corresponding to the change records.

[0083]

[0084] The following is an example, where the map feature being manipulated is referred to as the target map feature:

[0085] Example 1: In response to the simulated feature change information indicating the addition of a new target map feature, and the target map feature does not exist in the intelligence data, the target map feature is deleted from the initial high-precision map.

[0086] For example, suppose the desired simulated change is "the intelligence data contains one more traffic light compared to the target high-definition map." The target map element is this traffic light, which exists in both the intelligence data and the initial high-definition map. However, to achieve the simulated change, the initial high-definition map needs to be modified by deleting the traffic light from it. Only then, when the intelligence data is compared with the modified high-definition map, will the result show "the intelligence data contains one more traffic light compared to the target high-definition map." Referring to Table 3 above, the operation on the high-definition map involves deleting the traffic light element record. When the intelligence data is compared with the modified high-definition map, a new map element is output; this new map element is the newly added traffic light, and its existence has a high confidence level, for example, 98%.

[0087] Example 2: In response to the simulated feature change information indicating the deletion of the target map feature, and the target map feature does not exist in the intelligence data, the target map feature is added to the initial high-precision map.

[0088] For example, suppose the desired simulated change is that "the intelligence data is missing one traffic sign compared to the target high-definition map." The target map element is this traffic sign, which doesn't exist in the intelligence data and is also absent from the initial high-definition map. However, to achieve the simulated change, the initial high-definition map needs to be modified by adding the traffic sign. Only then, when the intelligence data is compared to the target high-definition map, will the statement "the intelligence data is missing one traffic sign compared to the high-definition map" be obtained. Referring to Table 3 above, the operation on the high-definition map involves adding a traffic sign element record. When the intelligence data is compared to the target high-definition map, the confidence level for the existence of this traffic sign is low, for example, 12%.

[0089] Example 3: In response to the simulated feature change information indicating a change in the feature attribute of the target map feature, and the feature attribute of the target map feature in the intelligence data is the new attribute after the change, then the feature attribute of the target map feature in the initial high-precision map is set to the old attribute before the change.

[0090] For example, suppose the desired simulated change is that "compared to the target high-definition map, the lane line of a certain lane in the intelligence data has changed from a solid line to a dashed line." The target map element is the lane line, and in the intelligence data, it has the new attribute "dashed line," and it is also a dashed line in the initial high-definition map. However, to achieve the simulated change, the initial high-definition map needs to be modified, setting the attribute of the lane line in the initial high-definition map to the old attribute "solid line." Only then, when the intelligence data is compared with the target high-definition map, can we obtain the result that "compared to the target high-definition map, the lane line in the intelligence data has changed from a solid line to a dashed line."

[0091] In Table 3 above, the element record is the information in the high-precision map data used to describe the map element. In this step, the corresponding map elements in the initial high-precision map can be modified based on the previously generated simulated element change information to obtain the map data of the target high-precision map.

[0092] In step 106, the true values ​​of map feature changes are obtained. The true values ​​of map feature changes include: real feature change information obtained based on the intelligence data and simulated feature change information of the road to be evaluated.

[0093] For example, after modifying the initial high-precision map, the reverse record of the modification operations made to the initial high-precision map can be exported to obtain the true values ​​of map feature changes. The modified high-precision map can then be stored in an online database for evaluation in subsequent steps.

[0094] Table 4 provides an example of a record table for the true values ​​of map feature changes:

[0095] Table 4 Record of True Values ​​of Map Feature Changes

[0096]

[0097]

[0098] As can be seen, the content format of the map feature change truth values ​​recorded in Table 4 above is basically the same as that in Table 2, except that some fields are missing. Other feature change types, feature types, feature IDs, etc., are the same.

[0099] Furthermore, it should be noted that for the attribute change type in Table 2, two truth values ​​can be generated. One truth value corresponds to the map feature before the attribute was modified, and the other truth value corresponds to the map feature after the attribute was modified. These two map features can correspond to the same location on the map.

[0100] As described above, the true values ​​of map feature changes have been obtained. These true values ​​consist of two parts: one part is the actual feature change information obtained based on the intelligence data, and the other part is the simulated feature change information of the road to be evaluated. Furthermore, the modified high-precision map has been stored in the online database for subsequent quality assessment. These true values ​​of map feature changes can also be stored and used multiple times.

[0101] Furthermore, the embodiments in this specification do not limit the timing of truth value generation, as long as it is generated before the truth value is used.

[0102] [Quality Assessment]

[0103] Figure 2 This is a flowchart of an exemplary embodiment of a method for determining the quality of the confidence level of the existence of map features. This method can be used to evaluate the quality of the aforementioned data products. Figure 2 As shown, the method may include the following processing:

[0104] In step 200, the intelligence data corresponding to the road to be evaluated is compared with the map data of the target high-precision map to obtain the feature change information of each map element of the target high-precision map, and the confidence of the existence of the map elements corresponding to the target high-precision map is updated accordingly.

[0105] In this step, a confidence data product for the existence of map features is created by comparing the intelligence data corresponding to the road to be evaluated with the map data of the target high-definition map previously stored in the online database. The intelligence data includes real-world data collected from the map features of the road to be evaluated. The target high-definition map data can be the high-definition map data corresponding to the road to be evaluated; that is, the map data compared with the intelligence data must also be data with corresponding intelligence data observations. For example, if the intelligence data includes real-world data collected from 20 map features of the road to be evaluated, then the map data of the target high-definition map compared in this step is also the data of those 20 map features corresponding to the road to be evaluated.

[0106] Specifically, the change information of each map element in the target high-definition map can be obtained by comparing the intelligence data corresponding to the road to be evaluated with the map data of the target high-definition map. For example, suppose that by comparing the intelligence data (specifically, the intelligence data obtained through image recognition of collected photos) with the map data of the target high-definition map, it is found that a traffic light on a certain road has been deleted. The confidence of the existence of the corresponding map element in the target high-definition map can be updated according to the change information. For example, the confidence of the existence of the traffic light can be modified from the original "98%" to "8%". In this way, the confidence of the existence of all map elements that have changed can be refreshed. For example, suppose the speed limit value of a speed limit sign has changed, then the existence confidence is modified from the original 98% to 20% (corresponding to the old speed limit value before the modification), and the existence confidence on the newly output speed limit sign at the same location (which has the modified new speed limit value) is modified to 98%.

[0107] In this step, the intelligence data collected on the road to be evaluated is compared with the map data of the road to be evaluated in the target high-precision map. Based on the changes in each map element in the road to be evaluated, the confidence level of each map element is updated, and a data product of the confidence level of the map elements corresponding to the road to be evaluated is obtained.

[0108] In step 202, the quality result data of the confidence level is obtained based on the updated confidence level of the map feature and the corresponding true value of the map feature change.

[0109] In this step, after refreshing the confidence level of map elements in the data product, the map elements can be divided into three categories: "suspected change," "unchanged," and "unknown," based on the changes in the confidence level and corresponding settings. The "suspected change" category can further include additions, deletions, and attribute changes. Map elements in the data product whose confidence levels have changed can be identified. The change information of these changed map elements is compared with the true values ​​of the changes to obtain the accuracy of the map element confidence level, which can also be referred to as the accuracy of the data product. Furthermore, the correct change information included in the change information is compared with the true values ​​of the changes to obtain the recall rate of the map element confidence level, which can also be referred to as the recall rate of the data product.

[0110] For example, consider a map element called "traffic sign." Initially, its existence confidence level is 98%. After updating the data product, the existence confidence level is 2%. The difference between the two existence confidence levels is "98 - 2 = 96%." This difference exceeds a certain threshold, so it's classified as a "suspected change." Based on the fact that the initial existence confidence level is higher than the updated existence confidence level, and combined with the specific numerical value of the existence confidence level, this suspected change can be determined as "deleted."

[0111] Then, the above archived results can be compared with the map feature change truth values. Specifically, we can first find the map features whose confidence levels have changed from the data product. For example, suppose there are 20 map feature changes. Based on the IDs of the 20 changed features, we can search the map feature change truth values. The truth values ​​record whether the actual situation of the map feature has changed, and record the change type as "added", "deleted" or "attribute change".

[0112] By using truth lookup, we can determine whether the changes to the aforementioned 20 elements have actually occurred. This involves not only comparing whether the element changes actually happened, but also comparing whether the change types are consistent. For example, if the data product determines that the change type of a certain map element is "added" based on the existing confidence level, but the truth value indicates that the change type of the same map element is "deleted," then the two are inconsistent.

[0113] If the truth table shows that 10 out of the 20 map features have changes consistent with the truth values, then the accuracy of this data product is 50%. Alternatively, if the truth values ​​indicate that 15 map features have changed compared to the high-definition map data, and this data product only found 10, then the recall rate of this data product is (10 / 15 = 66%).

[0114] This embodiment uses the "suspected change" category as an example to calculate the product recall and precision for this category. The calculation for other categories is similar and will not be detailed further. After calculating the precision and recall rates, the system can automatically determine whether the data product meets the product release requirements based on the requirements for these indicators.

[0115] The quality determination method for the confidence level of map feature existence in this embodiment obtains true values ​​by simulating changes in map features. This allows for the calculation of precision and recall metrics for data products based on the true values, resulting in a more comprehensive quality assessment. Furthermore, simulating map feature changes increases the sample size used in the assessment, leading to more accurate and complete quality results. The method for simulating map feature changes can also be flexibly configured according to assessment needs. The true values ​​used in this approach can be used for multiple assessments; for example, after a data product is updated, the true values ​​can be directly used for reassessment, improving the efficiency of quality assessment.

[0116] Figure 3 An exemplary apparatus for determining the confidence level of the existence of map features is provided, which can be applied to methods implementing any embodiment of this specification. For example... Figure 3 As shown, the device may include a data production module 31 and a quality measurement module 32.

[0117] The data production module 31 is used to compare the intelligence data corresponding to the road to be evaluated with the map data of the target high-precision map to obtain the feature change information of each map element of the target high-precision map; the intelligence data includes: real-world collection information of the map elements in the road to be evaluated; and to update the confidence level of the existence of the map elements corresponding to the target high-precision map according to the feature change information.

[0118] The quality measurement module 32 is used to obtain the quality result data of the confidence level based on the updated confidence level of the map features and the corresponding true values ​​of map feature changes; the true values ​​of map feature changes include: real feature change information obtained based on the intelligence data and simulated feature change information of the road to be evaluated; the map data of the target high-precision map is obtained by modifying the initial high-precision map based on the simulated feature change information.

[0119] In one example, the quality measurement module 32 is further configured to: compare the intelligence data of the road to be evaluated with the initial high-precision map to obtain the real feature change information, the real feature change information being used to describe the map feature changes that have occurred in the intelligence data compared to the initial high-precision map; generate simulated feature change information, the simulated feature change information being used to describe the simulated map feature changes that have occurred in the intelligence data compared to the target high-precision map; and modify the corresponding map features in the initial high-precision map according to the simulated feature change information to obtain the map data of the target high-precision map.

[0120] In one example, the quality measurement module 32, when used to generate simulated feature change information, includes: acquiring high-precision map data of the map sheet corresponding to the road to be evaluated; deleting map features other than the road to be evaluated from the high-precision map data of the map sheet; and generating simulated feature change information for the remaining map features in the high-precision map data of the map sheet.

[0121] In one example, the quality measurement module 32, when used to generate simulated feature change information, includes: for any type of map feature, selecting multiple subsets of map features, each subset including at least one map feature of that type; for each map feature in any subset, generating simulated feature change information corresponding to that map feature; and different subsets of map features correspond to different types of simulated feature change information with different types of feature changes.

[0122] In one example, when the quality measurement module 32 modifies the corresponding map element in the initial high-precision map according to the simulated element change information, it includes: in response to the simulated element change information indicating the deletion of the target map element, and the target map element not existing in the intelligence data, adding the target map element to the initial high-precision map.

[0123] In one example, when the quality measurement module 32 is used to modify the corresponding map elements in the initial high-precision map according to the simulated element change information, it includes: in response to the simulated element change information indicating a change in the element attributes of the target map element, and the element attributes of the target map element in the intelligence data being the new attributes after the change, setting the element attributes of the target map element in the initial high-precision map to be the old attributes before the change.

[0124] In one example, the quality measurement module 32, when obtaining quality result data of the confidence level based on the updated confidence level of the map feature and the corresponding true value of map feature change, includes: identifying map features with changed confidence levels in the map data corresponding to the road to be evaluated in the target high-precision map; comparing the change information of the changed map features with the true value of map feature change to obtain the accuracy of the confidence level of the map features; and comparing the correct change information included in the change information with the true value of map feature change to obtain the recall rate of the confidence level of the map features.

[0125] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0126] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0127] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0128] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0129] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0130] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0131] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0132] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0133] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A method for determining the quality of confidence in the existence of map features, characterized in that, The method includes: Based on the comparison between the intelligence data corresponding to the road to be evaluated and the map data of the target high-precision map, the feature change information of each map element of the target high-precision map is obtained. The intelligence data includes real-world collected information of the map elements in the road to be evaluated; the map data is the high-precision map data corresponding to the road to be evaluated. Based on the change information of the elements, update the confidence level of the existence of map elements in the target high-precision map; Based on the updated confidence level of the map features and the corresponding ground truth values ​​of map feature changes, the quality result data of the confidence level is obtained; the ground truth values ​​of map feature changes include: real feature change information obtained based on the intelligence data and simulated feature change information of the road to be evaluated; the map data of the target high-precision map is obtained by modifying the initial high-precision map based on the simulated feature change information.

2. The method according to claim 1, characterized in that, The method further includes: The intelligence data of the road to be evaluated is compared with the initial high-precision map to obtain the real feature change information. The real feature change information is used to describe the changes in map features that have occurred in the intelligence data compared with the initial high-precision map. Generate simulated feature change information, which describes the map feature changes that occur in the simulated intelligence data compared to the target high-precision map; Based on the simulated feature change information, the corresponding map features in the initial high-precision map are modified to obtain the map data of the target high-precision map.

3. The method according to claim 2, characterized in that, The generated simulated feature change information, which describes the changes in map features between the simulated intelligence data and the target high-precision map, includes: Obtain high-precision map data for the map sheet corresponding to the road to be evaluated; For the high-precision map data of the aforementioned map sheet, delete map elements other than the road to be evaluated; For the remaining map elements in the high-precision map data of the aforementioned map sheet, simulated element change information is generated.

4. The method according to claim 2, characterized in that, The generated simulated feature change information, which describes the changes in map features between the simulated intelligence data and the target high-precision map, includes: For any map feature of any feature type, select multiple subsets of map features, each subset of map features including at least one map feature of that feature type; For each map element in any subset of map elements, simulated feature change information corresponding to the map element is generated; and the simulated feature change information generated for different subsets of map elements has different types of feature changes.

5. The method according to claim 2, characterized in that, The step of modifying the corresponding map features in the initial high-precision map based on the simulated feature change information includes: In response to the simulated feature change information indicating the deletion of the target map feature, and the absence of the target map feature in the intelligence data, the target map feature is added to the initial high-precision map.

6. The method according to claim 2, characterized in that, The step of modifying the corresponding map features in the initial high-precision map based on the simulated feature change information includes: In response to the simulated feature change information indicating a change in the feature attributes of the target map feature, and the feature attributes of the target map feature in the intelligence data being the new attributes after the change, the feature attributes of the target map feature in the initial high-precision map are set to the old attributes before the change.

7. The method according to claim 1, characterized in that, The step of obtaining the quality result data of the confidence level based on the updated confidence level of the map features and the corresponding ground truth value of map feature changes includes: In the map data corresponding to the road to be evaluated in the target high-precision map, map elements with changed confidence levels are identified; The change information of the changed map features is compared with the true value of the changes in the map features to obtain the accuracy of the confidence level of the map features. Furthermore, the correct change information included in the change information is compared with the true value of the map element change to obtain the recall rate of the confidence level of the map element.

8. A device for determining the quality of confidence in the existence of map features, characterized in that, The device includes: The data production module is used to compare the intelligence data corresponding to the road to be evaluated with the map data of the target high-precision map to obtain the feature change information of each map element of the target high-precision map; the intelligence data includes: real-world collected information of the map elements in the road to be evaluated; based on the feature change information, the confidence level of the existence of the map elements corresponding to the target high-precision map is updated; the map data is the high-precision map data corresponding to the road to be evaluated; The quality measurement module is used to obtain the quality result data of the confidence level based on the updated confidence level of the map features and the corresponding true values ​​of map feature changes; the true values ​​of map feature changes include: real feature change information obtained based on the intelligence data and simulated feature change information of the road to be evaluated; the map data of the target high-precision map is obtained by modifying the initial high-precision map based on the simulated feature change information.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1-7 by executing the executable instructions.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-7.

Images