Semantic map updating method and device, vehicle and storage medium

Abstract

The invention provides a semantic map updating method and device, a vehicle and a storage medium, and the method comprises the steps: determining a target region which is a local region with the position of the vehicle as the center in a first semantic map; obtaining a semantic aerial view of the vehicle at the current position; registering the semantic aerial view with the target area based on multi-channel cross entropy loss and KL divergence; and determining whether to update the first semantic map based on the registered semantic aerial view and the target area. According to the method provided by the invention, the accuracy of semantic map updating can be improved.

Description

Technical Field

[0001] This application relates to the automotive field, and more particularly to a semantic map updating method, apparatus, vehicle, and storage medium. Background Technology

[0002] In related technologies, vehicle parking relies on maps. These maps can be of two types: geometrically based maps and semantically perceptual maps.

[0003] Understandably, semantic-aware methods can extract pixel-level semantic distribution information through deep segmentation networks, constructing more semantically robust perceptual representations that are more adaptable to unstructured changes.

[0004] When a vehicle parks using a semantically aware map, it can update the map if the parking environment changes. However, current semantically aware map update methods do not consider the interference caused by changes in the vehicle's perspective due to motion, resulting in low accuracy. Summary of the Invention

[0005] This application provides a semantic map updating method, apparatus, vehicle, and storage medium, which helps to improve the accuracy of semantic map updates.

[0006] In a first aspect, embodiments of this application provide a semantic map updating method, comprising: determining a target region, wherein the target region is a local region in a first semantic map centered on the location of a vehicle; obtaining a semantic bird's-eye view of the vehicle at its current location; registering the semantic bird's-eye view with the target region based on multi-channel cross-entropy loss and KL divergence; and determining whether to update the first semantic map based on the registered semantic bird's-eye view and the target region.

[0007] In one possible implementation, determining whether to update the first semantic map based on the registered semantic bird's-eye view and the target region includes: calculating the information entropy of the registered semantic bird's-eye view and the information entropy of the target region; determining candidate regions for change based on the information entropy of the registered semantic bird's-eye view and the information entropy of the target region; and determining whether to update the first semantic map based on the candidate regions for change.

[0008] In one possible implementation, the information entropy of the registered semantic bird's-eye view includes the information entropy of m grid cells in the registered semantic bird's-eye view, and the information entropy of the target region includes the information entropy of m grid cells in the target region, where m is a positive integer greater than 0. The step of determining the change candidate region based on the information entropy of the registered semantic bird's-eye view and the information entropy of the target region includes: determining the KL divergence of the i-th grid cell based on the information entropy of the i-th grid cell in the target region and the information entropy of the i-th grid cell in the registered semantic bird's-eye view, where the i-th grid cell is any one of the m grid cells; if the KL divergence of the i-th grid cell is greater than a first threshold, the i-th grid cell is determined as a change candidate region.

[0009] In one possible implementation, the information entropy of the registered semantic bird's-eye view includes the information entropy of m grid cells in the registered semantic bird's-eye view, and the information entropy of the target region includes the information entropy of m grid cells in the target region, where m is a positive integer greater than 0. The step of determining the change candidate region based on the information entropy of the registered semantic bird's-eye view and the information entropy of the target region includes: determining the JS divergence of the i-th grid cell based on the information entropy of the i-th grid cell in the target region and the information entropy of the i-th grid cell in the registered semantic bird's-eye view, where the i-th grid cell is any one of the m grid cells; if the JS divergence of the i-th grid cell is greater than a first threshold, the i-th grid cell is determined as a change candidate region.

[0010] In one possible implementation, the information entropy of the registered semantic bird's-eye view includes the information entropy of m grid cells in the registered semantic bird's-eye view, and the information entropy of the target region includes the information entropy of m grid cells in the target region, where m is a positive integer greater than 0. Determining the change candidate region based on the information entropy of the registered semantic bird's-eye view and the information entropy of the target region includes: determining the KL divergence and JS divergence of the i-th grid cell based on the information entropy of the i-th grid cell in the target region and the information entropy of the i-th grid cell in the registered semantic bird's-eye view, where the i-th grid cell is any one of the m grid cells; determining a semantic difference metric based on the KL divergence and JS divergence; and determining the i-th grid cell as a change candidate region if the semantic difference metric of the i-th grid cell is greater than a first threshold.

[0011] In one possible implementation, the semantic bird's-eye view of the vehicle at its current location includes the semantic bird's-eye view of the vehicle at time t1; the determination of the change candidate region based on the information entropy of the registered semantic bird's-eye view and the information entropy of the target region includes: determining the change candidate region based on the information entropy of the registered semantic bird's-eye view at time t1 and the information entropy of the target region.

[0012] In one possible implementation, the semantic bird's-eye view of the vehicle at its current location includes semantic bird's-eye views of the vehicle at multiple times. The step of determining the change candidate region based on the information entropy of the registered semantic bird's-eye view and the information entropy of the target region includes: determining a first change candidate region based on the information entropy of the registered semantic bird's-eye view at time t1 and the information entropy of the target region, where time t1 is the earliest time among the multiple times; and filtering the first change candidate region based on the information entropy of the registered semantic bird's-eye view at the multiple times and the information entropy of the target region to obtain a second change candidate region.

[0013] In one possible implementation, the step of filtering the first change candidate region based on the information entropy of the semantic bird's-eye view at the multiple times after registration and the information entropy of the target region to obtain the second change candidate region includes: determining the mean of the semantic difference metric of the i-th grid unit based on the semantic difference metric corresponding to the multiple times of the i-th grid unit; if the mean of the semantic difference metric of the i-th grid unit is greater than a second threshold, retaining the i-th grid unit in the first change candidate region; if the mean of the semantic difference metric of the i-th grid unit is less than or equal to the second threshold, deleting the i-th grid unit from the first change candidate region to obtain the second change candidate region; wherein, the second threshold is greater than the first threshold.

[0014] In one possible implementation, after filtering the first change candidate region based on the information entropy of the semantic bird's-eye view at the multiple time points after registration and the information entropy of the target region to obtain the second change candidate region, the method further includes: performing spatial consistency filtering and / or temporal consistency filtering on the second change candidate region; the spatial consistency filtering is used to filter the second change candidate region based on the changes of the grid cells around any detected grid cell, and the temporal consistency filtering is used to filter the second change candidate region based on the proportion of semantic bird's-eye view where the semantic difference metric in the time dimension is greater than the first threshold.

[0015] In one possible implementation, after determining whether to update the first semantic map based on the registered semantic bird's-eye view and the target region, the method further includes: If it is determined that the first semantic map should be updated, a difference map is generated. The difference map includes the change candidate region, and the change candidate region includes semantic difference information between the semantic bird's-eye view and the target region. The differential map is sent to the server, which then updates the first semantic map based on the differential map to obtain the second semantic map.

[0016] Secondly, embodiments of this application provide a semantic map updating apparatus, including one or more functional modules, which are used to perform the semantic map updating method as described in the first aspect.

[0017] Thirdly, embodiments of this application provide a vehicle, including: a processor and a memory, the memory being used to store a computer program; the processor being used to run the computer program to implement the semantic map update method as described in the first aspect.

[0018] Fourthly, embodiments of this application provide a readable storage medium storing a program that, when run on a vehicle, causes the vehicle to implement the semantic map update method as described in the first aspect.

[0019] Fifthly, embodiments of this application provide a program that, when run on a vehicle's processor, causes the vehicle to perform the semantic map update method as described in the first aspect.

[0020] In one possible design, the program in the fifth aspect can be stored wholly or partially on a storage medium packaged with the processor, or it can be stored wholly or partially on a memory not packaged with the processor. Attached Figure Description

[0021] Figure 1 A flowchart illustrating an embodiment of the semantic map update method provided in this application; Figure 2 A schematic diagram of the target area provided in the embodiments of this application; Figure 3 A flowchart illustrating another embodiment of the semantic map update method provided in this application; Figure 4 This is a schematic diagram of the semantic map updating device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the vehicle structure provided in an embodiment of this application. Detailed Implementation

[0022] In this embodiment of the application, unless otherwise stated, the character " / " indicates that the preceding and following objects are in an OR relationship. For example, A / B can represent A or B. "AND / OR" describes the relationship between the associated objects, indicating that three relationships can exist. For example, A AND / OR B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0023] It should be noted that the terms "first" and "second" used in the embodiments of this application are used only for distinguishing descriptive purposes and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated, nor should they be construed as indicating or implying order.

[0024] In the embodiments of this application, "at least one" means one or more, and "more than one" means two or more. Furthermore, "at least one of the following" or similar expressions refer to any combination of these items, which may include any combination of a single item or a plurality of items. For example, at least one of A, B, or C can represent: A, B, C, A and B, A and C, B and C, or A, B, and C. Each of A, B, and C can be an element itself or a set containing one or more elements.

[0025] In this application, terms such as "exemplary," "in some embodiments," and "in another embodiment" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the term "exemplary" is intended to present the concept in a concrete manner.

[0026] In the embodiments of this application, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction, their meanings are consistent. Similarly, in the embodiments of this application, "communication" and "transmission" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction, their meanings are consistent. For example, transmission can include sending and / or receiving, and can be a noun or a verb.

[0027] In the embodiments of this application, the term "equal to" can be used in conjunction with "greater than" to apply to technical solutions employing the condition of "greater than", and can also be used in conjunction with "less than" to apply to technical solutions employing the condition of "less than". It should be noted that when "equal to" is used with "greater than", it cannot be used with "less than"; and when "equal to" is used with "less than", it cannot be used with "greater than".

[0028] In related technologies, vehicle parking relies on maps. These maps can be of two types: geometrically based maps and semantically perceptual maps.

[0029] Understandably, semantic-aware methods can extract pixel-level semantic distribution information through deep segmentation networks, constructing more semantically robust perceptual representations that are more adaptable to unstructured changes.

[0030] When a vehicle parks using a semantically aware map, it can update the map if the parking environment changes. However, current semantically aware map update methods do not consider the interference caused by changes in the vehicle's perspective due to motion, resulting in low accuracy.

[0031] To address the aforementioned issues, this application provides a semantic map updating method that helps improve the accuracy of semantic map updates.

[0032] Figure 1 A flowchart illustrating an embodiment of the semantic map update method provided in this application includes the following steps: Step 101: Determine the target area, which is a local area in the first semantic map centered on the vehicle's location.

[0033] Specifically, when a vehicle is parked, it can trigger an update of the semantic map.

[0034] Understandably, when updating semantic maps, vehicles can first determine the target area.

[0035] The target area can be a local area centered on the vehicle's location in the first semantic map. The first semantic map can be a semantic map stored on a server or locally on the vehicle.

[0036] The method by which a vehicle determines a target area may include: retrieving a reference map area surrounding the vehicle's current location in a first semantic map based on the vehicle's current location. The vehicle's current location can be determined using Global Positioning System (GPS), Global Navigation Satellite System (GNSS), inertial navigation, or Simultaneous Localization and Mapping (SLAM). Then, a region centered on the vehicle's current location can be designated as the target area.

[0037] For example, a circular area with the vehicle's current location as the center and radius R can be used as the target area. Here, R can typically be set to 10 meters to 20 meters.

[0038] It is understood that the values ​​of R mentioned above are merely illustrative and do not constitute a limitation on the embodiments of this application. In some embodiments, R may also take other values.

[0039] Figure 2 An exemplary schematic diagram of the target area is shown.

[0040] refer to Figure 2 Map 200 is the first semantic map stored by vehicle 20, location 21 is the current location of vehicle 20, and region 201 is a local region selected with vehicle 20 as the center and R as the radius. This local region can be used as the target region.

[0041] Step 102: Obtain a bird's-eye view of the vehicle's current location.

[0042] Specifically, the vehicle can obtain a bird's-eye view (BEV) of its current location by using sensors installed on the vehicle body.

[0043] The bird's-eye view of the vehicle's current location is the same size as or comparable to the target area.

[0044] Once a bird's-eye view of the vehicle's current location is obtained, panoramic image semantic segmentation can be performed on the bird's-eye view to obtain a semantic bird's-eye view.

[0045] It is understood that the execution order of step 102 and step 101 can be partially sequential. For example, step 102 can be executed before step 101, or step 102 can be executed after step 101, or step 102 can be executed simultaneously with step 101. This application embodiment does not impose any special limitations on this.

[0046] In some alternative embodiments, the bird's-eye view of the vehicle's current location may be a frame of the bird's-eye view of the vehicle at a certain time (e.g., time t1) at the current location.

[0047] In some alternative embodiments, the bird's-eye view of the vehicle's current location can be a multi-frame bird's-eye view of the vehicle at multiple times (e.g., t1, t2, t3, etc.) at the current location.

[0048] Step 103: Register the semantic bird's-eye view with the target area.

[0049] Specifically, once the target region and semantic bird's-eye view are obtained, the semantic bird's-eye view can be registered with the target region.

[0050] Understandably, the purpose of registration is to align the semantic bird's-eye view with the target area.

[0051] The registration between the target region and the semantic bird's-eye view can be achieved using the following formula: ; ; in, The matching error is represented by C, the number of semantic channels (e.g., parking spaces, walls, obstacles), and Gref is used to represent the semantic map of the target region. For semantic bird's-eye view map The transformed bird's-eye view.

[0052] As can be seen from the above formula, in the registration process between the target region and the semantic bird's-eye view, the matching error, which includes multi-channel cross-entropy and KL divergence, is used as a reference factor, and the process is iterated continuously to find the optimal transformation. This is to complete the registration between the target area and the semantic bird's-eye view.

[0053] Among them, the best transformation It can be obtained by using the SLSQP optimization algorithm.

[0054] Understandably, compared to existing registration techniques, the registration based on multi-channel cross-entropy and KL divergence in this application can improve the accuracy of registration.

[0055] Step 104: Determine whether to update the semantic map based on the information entropy of the target area and the information entropy of the semantic bird's-eye view.

[0056] Specifically, after the target region and the semantic bird's-eye view are registered, the target region and the semantic bird's-eye view are in the same spatial reference system and the semantics are aligned.

[0057] The methods for determining whether to update the semantic map based on the information entropy of the target area and the information entropy of the semantic bird's-eye view include: determining whether the semantics of the target area have changed based on the information entropy of the target area and the information entropy of the semantic bird's-eye view, and determining whether to update the semantic map based on the semantic change results of the target area.

[0058] For example, if it is determined that the semantics of the target region has changed, the semantic map is updated. Or, If it is determined that the semantics of the target region have not changed, the semantic map will not be updated.

[0059] Next, taking a semantic bird's-eye view frame at time t1 as an example, the process of determining whether the semantics of the target region has changed will be illustrated.

[0060] First, it can be based on the optimal variable The registered semantic bird's-eye view is obtained by combining the semantic bird's-eye view with the semantic bird's-eye view. This registered semantic bird's-eye view can be calculated using the following formula: ; Where Gt is the registered semantic bird's-eye view and Scam is the semantic bird's-eye view.

[0061] Understandably, once the registered semantic bird's-eye view is obtained, it can be considered a semantic map of the same scale and coordinate system as the target region. Then, semantic distribution change analysis can be performed on each grid cell between the target region and the registered semantic bird's-eye view.

[0062] For example, for each grid cell i, both the target region and the registered semantic bird's-eye view can contain a C-dimensional semantic probability vector. It is understood that the target region and the registered semantic bird's-eye view can be semantically segmented into an equal number of grid cells; for example, the target region can be semantically segmented into m grid cells, and the registered semantic bird's-eye view can be semantically segmented into m grid cells, where m is a positive integer greater than 0.

[0063] The semantic probability vector of each grid cell i in the registered semantic bird's-eye view can be represented by the following formula: ; The semantic probability vector of each grid cell i in the target region can be represented by the following formula: ; Next, the information entropy of each can be calculated based on the semantic probability vector of the registered semantic bird's-eye view and the semantic probability vector of the target region, respectively.

[0064] The information entropy of each grid cell i in the registered semantic bird's-eye view can be obtained by the following formula: ; The information entropy of each grid cell i in the target region can be obtained by the following formula: ; Understandably, the higher the information entropy, the more uncertain the semantic distribution at that location, indicating the possibility of occlusion, change, or perceptual error.

[0065] In some optional embodiments, in order to accurately determine whether the semantics have undergone substantial changes, the embodiments of this application use KL divergence and JS divergence as comprehensive indicators. KL divergence can be used to characterize the information loss from the historical distribution to the current distribution. KL divergence takes into account directional changes, and JS divergence takes into account the overall dissimilarity of the distribution.

[0066] For example, the KL divergence of any grid i can be calculated using the following formula: ; The JS divergence of any grid i can be calculated using the following formula: ; By comprehensively considering the KL divergence and JS divergence of each grid cell i, a semantic difference metric can be obtained for each grid cell i. This semantic difference metric for each grid cell i can be represented by the following formula: ; As can be seen, the semantic difference metric Dtotal for each grid cell i i It further measures the degree of semantic change in the grid.

[0067] If Dtotal i If the value is greater than the first threshold, then the Dtotal value will be... i The corresponding grid cells are marked as candidate regions of change. Or, If Dtotal i If Dtotal is less than or equal to the first threshold, it can be considered that Dtotal i The corresponding grid cell has undergone semantic change.

[0068] In some alternative embodiments, KL divergence can also be used as a semantic difference metric.

[0069] For example, if Dkl i If it is greater than the first threshold, then the Dkl i The corresponding grid cells are labeled as candidate regions for change (i.e., candidate change grids). Or, If Dkl i If Dkl is less than or equal to the first threshold, it can be considered that the value is... i The corresponding grid cell has undergone semantic change.

[0070] In some alternative embodiments, JS divergence can also be used as a semantic difference metric.

[0071] For example, if Djs i If it exceeds the first threshold, then use the Djs. i The corresponding grid cells are marked as candidate regions of change. Or, If Djs i If the value is less than or equal to the first threshold, it can be considered that the Djs is... i The corresponding grid cell has undergone semantic change.

[0072] Step 105: Update the map.

[0073] In some optional embodiments, the vehicle may update the first semantic map based on the semantically changed region to obtain a second semantic map, and store the second semantic map locally on the vehicle.

[0074] In some optional embodiments, after updating the second semantic map, the vehicle can upload the second semantic map to the server, so that the server can update the map based on the second semantic map. This application does not impose any special limitations on this.

[0075] In some optional embodiments, a differential map can be generated based on semantically changed regions, which contains map data corresponding to the semantically changed regions. In scenarios where the server updates the semantic map, vehicles can upload only the differential map to the server, without uploading the entire map (e.g., the second semantic map), thereby reducing the amount of data uploaded and improving map update efficiency.

[0076] The construction method of the differential map may include: extracting the corresponding regions from the registered semantic bird's-eye view Gt based on the confirmed semantic change regions, calculating the difference between the two by combining the semantic distribution of grid cells at the same location in the target region, and thus constructing a semantic-level differential map. For example, a difference map can be characterized by the following formula: ; Among them, Gt R This represents a semantic distribution map within the identified semantic change regions in a bird's-eye view, Gref R This indicates the semantic distribution of the corresponding region within the target region.

[0077] In some optional embodiments, the differential map may also include the following metadata: change timestamps, vehicle location information, change type annotations (e.g., new obstacles / disappearance of parking spaces / change in traffic structure, etc.) and change confidence scores. The calculation method for the change confidence scores will be explained below and will not be repeated here.

[0078] In some optional embodiments, the differential map may also include the following metadata: corresponding area index (e.g., map title ID or center coordinates + size), spatiotemporal label + vehicle identification number, and status marker (e.g., temporary or long-term change).

[0079] Understandably, to ensure data compactness and efficient communication, differential maps can be encapsulated in compressed formats (such as sparse coding or RLE) and output using standardized protocols (such as JSON + image tiles).

[0080] In some alternative embodiments, the vehicle can package the differential map and the corresponding metadata and upload it to the server.

[0081] One method for uploading to the server is a combination of local caching and periodic uploading. For example, if communication conditions are limited, the differential map and metadata can be cached locally on the vehicle and then uploaded to the server when the network becomes available. This can improve bandwidth utilization and is suitable for parking scenarios with weak coverage or offline operation scenarios.

[0082] or, The upload to the server can be done via V2X / edge segment real-time upload. For example, if the vehicle supports vehicle-road cooperation (e.g., RSU, 5G MEC), the vehicle can upload differential maps and metadata to the edge server in real time. The edge server can perform preliminary review and multi-vehicle consistency comparison, and form a local dynamic map cache.

[0083] In some optional embodiments, after receiving the differential map and metadata uploaded by the vehicle, the server can perform corresponding processing.

[0084] The corresponding processing may include consistency verification, semantic fusion, and main graph update and distribution.

[0085] For example, consistency verification may include: the server receiving differential maps uploaded by multiple vehicles; if more than a preset number of vehicles observe the same change, it is considered reliable; if the changed area conflicts with the map editing record, manual verification can be performed.

[0086] Semantic fusion and master map updates may include: employing a confidence-weighted semantic fusion strategy to merge the differential maps into the master semantic map on the server; refining existing layers (e.g., obstacle layers, parking space layers); and updating version numbers and timestamps to record change history.

[0087] The distribution process can include: after the main semantic map on the server is updated, packaging the affected map titles (or regions of interest); and distributing the updated main map to subsequent vehicles or edge nodes entering the area, enabling dynamic map sharing and rapid adaptation.

[0088] In some optional embodiments, step 104 determines whether to update the semantic map by comparing a single frame of semantic bird's-eye view with the target region. Considering that semantic change detection in a single frame may be affected by sensor noise, transient occlusion, or model prediction errors, this application designs a multi-frame fusion verification mechanism to continuously observe and statistically analyze candidate change regions, thereby improving the robustness and confidence of change detection.

[0089] Figure 3 A flowchart illustrating another embodiment of the semantic map update method provided in this application is shown. Step 104 may include the following steps: Step 301: Determine the candidate region of change at time t1.

[0090] Specifically, the method for determining the candidate region of change at time t1 can be found in the relevant description in step 104 of the above embodiment, and will not be repeated here.

[0091] Step 302: Determine the semantic distribution changes of the candidate region after time t1 at multiple consecutive time points.

[0092] Specifically, the multiple consecutive times after time t1 can include n-1 times such as t2, t3, t4...tn, where n is a positive integer greater than 3.

[0093] By calculating the semantic difference metrics at multiple consecutive time points (e.g., the n time points t1-tn), the mean semantic difference metric of the candidate region of change can be obtained based on the semantic difference metric at these multiple consecutive time points.

[0094] For example, taking any grid cell in the candidate region of change as an example, the semantic difference metrics corresponding to n time points are as follows: .

[0095] Step 303: Update the candidate regions for changes based on the semantic distribution changes at multiple time points.

[0096] Specifically, after obtaining the semantic distribution changes at multiple time points, the mean of the semantic difference metric can be calculated based on the semantic difference metric corresponding to the multiple time points. The candidate regions of change are then updated according to the mean of the semantic difference metric. The mean of the semantic difference metric can be calculated using the following formula: ; Where T = {t1, t2, ..., tn}.

[0097] like If it is greater than the second threshold, then... The corresponding grid cells are marked as regions of high-confidence semantic change.

[0098] like If the value is less than or equal to the second threshold, then... The corresponding grid cell is marked as a low-confidence semantic change region. For example, this low-confidence semantic change region can be removed from the change candidate region.

[0099] Understandably, since the second threshold is greater than the first threshold, the above processing can filter out false alarms caused by noise in a single frame.

[0100] In some alternative embodiments, in addition to the multi-frame verification and updating of candidate regions described above, detection accuracy can be further improved by considering the spatial continuity and temporal consistency of the changed regions.

[0101] Taking spatial consistency as an example, the high-confidence semantic change region in step 303 can be mapped to a binary mask image.

[0102] Next, based on this mask image, morphological operations (e.g., dilation-erosion) can be used to perform connectivity analysis on any grid cell in the region of high confidence semantic change.

[0103] For any grid cell in a high-confidence semantic change region, if the grid cell is spatially isolated, for example, the 3 grid cells surrounding the grid cell... If there are no significant changes in the 3-neighborhood, the grid cell can be considered isolated and can be updated to a low-confidence semantic change region, meaning it can be removed from the high-confidence semantic change region. If the grid cell is not spatially isolated, for example, if the surrounding 3-neighborhoods are not significantly different... If at least one of the three neighborhoods has a significant change, the grid cell can be considered not isolated, and the grid cell can be retained in the region of high confidence semantic change.

[0104] It is understandable that changes in spatial connectivity are more likely to represent changes in the actual environmental structure (such as the addition of obstacles, parking space occupancy, etc.), and the accuracy of detection can be improved by considering spatial consistency.

[0105] Taking temporal consistency as an example, if any grid cell exhibits a brief high entropy difference in the time dimension, it may be an error caused by temporary changes (such as pedestrian occlusion, temporary parking, etc.), which needs to be excluded.

[0106] For example, for any grid cell in a high-confidence semantic change region, the number of frames greater than a first threshold among the frame counts at multiple times corresponding to that grid cell can be used as a temporal stability index. This temporal stability index can be calculated using the following formula: ; It should be noted that the above example is only used as an example to illustrate the calculation of time stability index using JS divergence, but it does not constitute a limitation of this application. In some embodiments, time stability index can also be calculated based on KL divergence or semantic difference metric. For specific methods, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0107] If the time stability index is less than the third threshold, the grid cell corresponding to that time stability index can be removed from the high-confidence semantic change region. Alternatively, If the time stability index is greater than or equal to the third threshold, the grid cell corresponding to the time stability index can be retained in the high-confidence semantic change region.

[0108] It is understood that the spatial consistency filtering and temporal consistency filtering described above can be used alone, or they can be used in combination. This application does not impose any special limitations on this.

[0109] Figure 4 This is a schematic diagram of the semantic map updating device provided in the embodiments of this application, as shown below. Figure 4 As shown, the semantic map update device 40 described above may include: a determining module 41, an acquiring module 42, a registration module 43, and an updating module 44; wherein, The determination module 41 is used to determine the target area, which is a local area in the first semantic map centered on the location of the vehicle; The acquisition module 42 is used to acquire a semantic bird's-eye view of the vehicle at its current location; Registration module 43 is used to register the semantic bird's-eye view with the target region based on multi-channel cross-entropy loss and KL divergence; The update module 44 is used to determine whether to update the first semantic map based on the registered semantic bird's-eye view and the target area.

[0110] In one possible implementation, the update module 44 is specifically used to calculate the information entropy of the registered semantic bird's-eye view and the information entropy of the target region; Based on the information entropy of the registered semantic bird's-eye view and the information entropy of the target region, candidate regions of change are determined. Whether to update the first semantic map is determined based on the changed candidate regions.

[0111] In one possible implementation, the information entropy of the registered semantic bird's-eye view includes the information entropy of m grid cells in the registered semantic bird's-eye view, and the information entropy of the target region includes the information entropy of m grid cells in the target region, where m is a positive integer greater than 0. The update module 44 is specifically used to determine the KL divergence of the i-th grid cell based on the information entropy of the i-th grid cell in the target region and the information entropy of the i-th grid cell in the registered semantic bird's-eye view, where the i-th grid cell is any one of the m grid cells. If the KL divergence of the i-th grid cell is greater than the first threshold, the i-th grid cell is determined as a candidate region for change.

[0112] In one possible implementation, the information entropy of the registered semantic bird's-eye view includes the information entropy of m grid cells in the registered semantic bird's-eye view, and the information entropy of the target region includes the information entropy of m grid cells in the target region, where m is a positive integer greater than 0. The update module 44 is specifically used to determine the JS divergence of the i-th grid cell based on the information entropy of the i-th grid cell in the target region and the information entropy of the i-th grid cell in the registered semantic bird's-eye view, where the i-th grid cell is any one of the m grid cells. If the JS divergence of the i-th grid cell is greater than the first threshold, the i-th grid cell is determined as a candidate region for change.

[0113] In one possible implementation, the information entropy of the registered semantic bird's-eye view includes the information entropy of m grid cells in the registered semantic bird's-eye view, and the information entropy of the target region includes the information entropy of m grid cells in the target region, where m is a positive integer greater than 0. The update module 44 is specifically used to determine the KL divergence and JS divergence of the i-th grid cell based on the information entropy of the i-th grid cell in the target region and the information entropy of the i-th grid cell in the registered semantic bird's-eye view, where the i-th grid cell is any one of the m grid cells. Based on the KL divergence and JS divergence, a semantic difference metric is determined; If the semantic difference metric of the i-th grid cell is greater than the first threshold, the i-th grid cell is determined as a candidate region for change.

[0114] In one possible implementation, the semantic bird's-eye view of the vehicle at its current location includes the semantic bird's-eye view of the vehicle at time t1; the update module 44 is further configured to determine the candidate regions for change based on the information entropy of the registered semantic bird's-eye view at time t1 and the information entropy of the target region.

[0115] In one possible implementation, the semantic bird's-eye view of the vehicle at its current location includes semantic bird's-eye views of the vehicle at multiple times. The update module 44 is further configured to determine a first change candidate region based on the information entropy of the registered semantic bird's-eye view at time t1 and the information entropy of the target region. Time t1 is the earliest time among the multiple times. Based on the information entropy of the semantic bird's-eye view at the multiple time points after registration and the information entropy of the target region, the first change candidate region is filtered to obtain the second change candidate region.

[0116] In one possible implementation, the update module 44 is further configured to determine the mean of the semantic difference metric of the i-th grid cell based on the semantic difference metric corresponding to the plurality of times of the i-th grid cell; If the average semantic difference metric of the i-th grid cell is greater than the second threshold, the i-th grid cell is retained in the first change candidate region; if the average semantic difference metric of the i-th grid cell is less than or equal to the second threshold, the i-th grid cell is deleted from the first change candidate region to obtain the second change candidate region. Wherein, the second threshold is greater than the first threshold.

[0117] In one possible implementation, the semantic map updating device 40 further includes: The filtering module is used to perform spatial consistency filtering and / or temporal consistency filtering on the second change candidate region; The spatial consistency filtering is used to filter the second change candidate region based on the changes in the grid cells around any detected grid cell, and the temporal consistency filtering is used to filter the second change candidate region based on the proportion of semantic bird's-eye view where the semantic difference metric in the time dimension is greater than the first threshold.

[0118] In one possible implementation, the semantic map updating device 40 further includes: The sending module is configured to generate a differential map if it is determined that the first semantic map needs to be updated. The differential map includes the change candidate region, and the change candidate region includes semantic difference information between the semantic bird's-eye view and the target region. The differential map is sent to the server, which then updates the first semantic map based on the differential map to obtain the second semantic map.

[0119] Figure 4 The semantic map update device 40 provided in the illustrated embodiment can be used to execute the technical solution of the method embodiment shown in this application. Its implementation principle and technical effect can be further referred to the relevant description in the method embodiment.

[0120] It should be understood that the division of the various modules in the semantic map update device 40 described above is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. For example, the detection module can be a separate processing element, or it can be integrated into a chip in the terminal device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0121] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). Alternatively, these modules can be integrated together as a system-on-a-chip (SOC).

[0122] Figure 5 This application provides a schematic diagram of the structure of a vehicle 500, which may include: at least one processor; and at least one memory communicatively connected to the processor. The memory stores program instructions executable by the processor, and the processor in the vehicle 500 can invoke these program instructions to perform actions executed in the memory access method provided in this application.

[0123] like Figure 5 As shown, vehicle 500 is represented in the form of a general-purpose computing device. The components of vehicle 500 may include, but are not limited to: one or more processors 510, memory 520, communication bus 540 connecting different system components (including memory 520 and processor 510), and communication interface 530.

[0124] The communication bus 540 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MAC) buses, Enhanced ISA buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses.

[0125] Vehicle 500 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by end devices, including volatile and non-volatile media, removable and non-removable media.

[0126] Memory 520 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The terminal device may further include other removable / non-removable, volatile / non-volatile computer system storage media. Although Figure 5 As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disc drive for reading and writing to a removable non-volatile optical disc (e.g., a compact disc read-only memory (CD-ROM), a digital video disc read-only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to the communication bus 540 via one or more data media interfaces. The memory 520 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0127] A program / utility having a set (at least one) of program modules can be stored in memory 520. Such program modules include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules typically perform the functions and / or methods described in the embodiments of this application.

[0128] Vehicle 500 can also communicate with one or more external devices (e.g., keyboard, pointing device, display, etc.), and with one or more devices that enable a user to interact with the terminal device, and / or with any device that enables the terminal device to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through communication interface 530. Furthermore, vehicle 500 can also communicate through a network adapter ( Figure 5 (Not shown) communicates with one or more networks (e.g., Local Area Network (LAN), Wide Area Network (WAN), and / or public networks, such as the Internet). The aforementioned network adapter can communicate with other modules of the terminal device via the communication bus 540. It should be understood that, although... Figure 5 As not shown in the diagram, other hardware and / or software modules can be used in conjunction with the vehicle 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, Redundant Arrays of Independent Drives (RAID) systems, tape drives, and data backup storage systems.

[0129] The processor 510 executes various functional applications and data processing by running programs stored in the memory 520, such as implementing the methods provided in the embodiments of this application.

[0130] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the vehicle 500. In other embodiments of this application, the vehicle 500 may also adopt different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0131] In the above embodiments, the processor may include, for example, a CPU, DSP, microcontroller, or digital signal processor, and may also include a GPU, embedded neural network processing unit (NPU), and image signal processor (ISP). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as an ASIC, or one or more integrated circuits for controlling the execution of the program in this application. Furthermore, the processor may have the function of operating one or more software programs, which may be stored in a storage medium.

[0132] This application also provides a readable storage medium storing a program that, when run on a system, causes the system to execute the method provided in the embodiments shown in this application.

[0133] This application also provides a program product, which includes a program that, when run on a system, causes the system to execute the method provided in the embodiments shown in this application.

[0134] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0135] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0136] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0137] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0138] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A semantic map update method, characterized in that, The method includes: Determine the target area, which is a local area in the first semantic map centered on the vehicle's location; Obtain a semantic bird's-eye view of the vehicle's current location; The semantic bird's-eye view is registered with the target region based on multi-channel cross-entropy loss and KL divergence; Based on the registered semantic bird's-eye view and the target area, determine whether to update the first semantic map.

2. The method according to claim 1, characterized in that, The step of determining whether to update the first semantic map based on the registered semantic bird's-eye view and the target region includes: Calculate the information entropy of the registered semantic bird's-eye view and the information entropy of the target region; Based on the information entropy of the registered semantic bird's-eye view and the information entropy of the target region, candidate regions of change are determined. Whether to update the first semantic map is determined based on the changed candidate regions.

3. The method according to claim 2, characterized in that, The information entropy of the registered semantic bird's-eye view includes the information entropy of m grid cells in the registered semantic bird's-eye view, and the information entropy of the target region includes the information entropy of m grid cells in the target region, where m is a positive integer greater than 0. Determining the candidate regions for change based on the information entropy of the registered semantic bird's-eye view and the information entropy of the target region includes: The KL divergence of the i-th grid cell is determined based on the information entropy of the i-th grid cell in the target region and the information entropy of the i-th grid cell in the registered semantic bird's-eye view. The i-th grid cell is any one of the m grid cells. If the KL divergence of the i-th grid cell is greater than the first threshold, the i-th grid cell is determined as a candidate region for change.

4. The method according to claim 2, characterized in that, The information entropy of the registered semantic bird's-eye view includes the information entropy of m grid cells in the registered semantic bird's-eye view, and the information entropy of the target region includes the information entropy of m grid cells in the target region, where m is a positive integer greater than 0. Determining the candidate regions for change based on the information entropy of the registered semantic bird's-eye view and the information entropy of the target region includes: The JS divergence of the i-th grid cell is determined based on the information entropy of the i-th grid cell in the target region and the information entropy of the i-th grid cell in the registered semantic bird's-eye view. The i-th grid cell is any one of the m grid cells. If the JS divergence of the i-th grid cell is greater than the first threshold, the i-th grid cell is determined as a candidate region for change.

5. The method according to claim 2, characterized in that, The information entropy of the registered semantic bird's-eye view includes the information entropy of m grid cells in the registered semantic bird's-eye view, and the information entropy of the target region includes the information entropy of m grid cells in the target region, where m is a positive integer greater than 0. Determining the candidate regions for change based on the information entropy of the registered semantic bird's-eye view and the information entropy of the target region includes: The KL divergence and JS divergence of the i-th grid cell are determined based on the information entropy of the i-th grid cell in the target region and the information entropy of the i-th grid cell in the registered semantic bird's-eye view. The i-th grid cell is any one of the m grid cells. Based on the KL divergence and JS divergence, a semantic difference metric is determined; If the semantic difference metric of the i-th grid cell is greater than the first threshold, the i-th grid cell is determined as a candidate region for change.

6. The method according to claim 5, characterized in that, The semantic bird's-eye view of the vehicle at its current location includes the semantic bird's-eye view of the vehicle at time t1; the determination of change candidate regions based on the information entropy of the registered semantic bird's-eye view and the information entropy of the target region includes: Based on the information entropy of the semantic bird's-eye view at time t1 after registration and the information entropy of the target region, candidate regions for change are determined.

7. The method according to claim 5, characterized in that, The semantic bird's-eye view of the vehicle at its current location includes semantic bird's-eye views of the vehicle at multiple times. The determination of candidate regions for change based on the information entropy of the registered semantic bird's-eye view and the information entropy of the target region includes: The first change candidate region is determined based on the information entropy of the semantic bird's-eye view at time t1 after registration and the information entropy of the target region, wherein time t1 is the earliest time among the plurality of times; Based on the information entropy of the semantic bird's-eye view at the multiple time points after registration and the information entropy of the target region, the first change candidate region is filtered to obtain the second change candidate region.

8. The method according to claim 7, characterized in that, The process of filtering the first change candidate region based on the information entropy of the registered semantic bird's-eye view at the multiple time points and the information entropy of the target region to obtain the second change candidate region includes: The mean of the semantic difference metric of the i-th grid cell is determined based on the semantic difference metric corresponding to the multiple times of the i-th grid cell; If the average semantic difference metric of the i-th grid cell is greater than the second threshold, the i-th grid cell is retained in the first change candidate region; if the average semantic difference metric of the i-th grid cell is less than or equal to the second threshold, the i-th grid cell is deleted from the first change candidate region to obtain the second change candidate region. Wherein, the second threshold is greater than the first threshold.

9. The method according to claim 7 or 8, characterized in that, After filtering the first change candidate region based on the information entropy of the semantic bird's-eye view at the multiple time points after registration and the information entropy of the target region to obtain the second change candidate region, the method further includes: Spatial consistency filtering and / or temporal consistency filtering are performed on the second change candidate region; The spatial consistency filtering is used to filter the second change candidate region based on the changes in the grid cells around any detected grid cell, and the temporal consistency filtering is used to filter the second change candidate region based on the proportion of semantic bird's-eye view where the semantic difference metric in the time dimension is greater than the first threshold.

10. The method according to claim 2, characterized in that, After determining whether to update the first semantic map based on the registered semantic bird's-eye view and the target area; If it is determined that the first semantic map should be updated, a difference map is generated. The difference map includes the change candidate region, and the change candidate region includes semantic difference information between the semantic bird's-eye view and the target region. The differential map is sent to the server, which then updates the first semantic map based on the differential map to obtain the second semantic map.

11. A semantic map updating device, characterized in that, The device includes: A determination module is used to determine a target area, which is a local area in the first semantic map centered on the vehicle's location. The acquisition module is used to acquire a semantic bird's-eye view of the vehicle at its current location; The registration module is used to register the semantic bird's-eye view with the target region based on multi-channel cross-entropy loss and KL divergence. An update module is used to determine whether to update the first semantic map based on the registered semantic bird's-eye view and the target area.

12. A vehicle, characterized in that, include: A processor and a memory, the memory being used to store a program; the processor being used to run the program to implement the semantic map update method as described in any one of claims 1-10.

13. A readable storage medium, characterized in that, The readable storage medium stores a program that, when run on a vehicle, implements the semantic map update method as described in any one of claims 1-10.

Images