Map element change detection method and system

Abstract

The embodiment of the invention provides a map element change detection method and system, and the method comprises the steps: determining a bird's-eye view angle feature based on a multi-view angle image shot at a target position; determining vector representation and semantics of map elements in a preset range around the target position based on the high-precision map; based on the vector representation and semantics of each map element, determining a feature tensor of the map element; and based on the aerial view angle features and the feature tensor of each map element, whether each map element around the target position is changed is determined by using a trained change discrimination model, and the change discrimination model is a machine learning model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for detecting changes in map elements. Background Art

[0002] Autonomous driving is a way of driving that is achieved through a processor-based automated system without human real-time operation. Autonomous vehicles rely on the collaboration of artificial intelligence, visual computing, detection radar, monitoring devices and positioning systems to enable the system to automatically and safely control vehicles without human real-time operation.

[0003] High-precision maps are an important part of the current mainstream autonomous driving system. They can be used to guide the autonomous driving vehicles to perform automatic and safe driving operations for the automated system. Autonomous driving vehicles can pre-plan their driving routes based on high-precision maps, but due to changes in actual traffic conditions (such as road repairs, new road signs, lane line changes, etc.), autonomous driving vehicles may not be able to drive safely and smoothly on the planned routes, and may even cause serious safety problems. Therefore, the detection of differences between actual road conditions and high-precision maps is of great significance to the autonomous driving system. Summary of the invention

[0004] One of the embodiments of the present specification provides a method for detecting changes in map elements, including determining a bird's-eye view feature based on a multi-view image taken at a target location; determining a vector representation and semantics of map elements within a preset range around the target location based on a high-precision map; determining a feature tensor of the map element based on the vector representation and semantics of each of the map elements; and determining whether each of the map elements around the target location has changed based on the bird's-eye view feature and the feature tensor of each of the map elements using a trained change discrimination model, wherein the change discrimination model is a machine learning model.

[0005] One of the embodiments of the present specification provides a map element change detection system, including: a perception feature determination module, used to determine a bird's-eye view feature based on a multi-view image taken at a target location; a map element determination module, used to determine the vector representation and semantics of map elements within a preset range around the target location based on a high-precision map; a map feature determination module, based on the vector representation and semantics of each map element, determines the feature tensor of the map element; a change detection module, based on the bird's-eye view feature and the feature tensor of each map element, using a trained change discrimination model to determine whether each map element around the target location has changed, wherein the change discrimination model is a machine learning model.

[0006] One of the embodiments of the present specification provides a map element change detection device, including a processor, wherein the processor is used to execute the above-mentioned map element change detection method. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0008] Figure 1 is a schematic diagram of a scene of a map element change detection system according to some embodiments of this specification;

[0009] Figure 2 is a schematic diagram of modules of a map element change detection system according to some embodiments of this specification;

[0010] Figure 3 is an exemplary flow chart of a map element change detection method according to some embodiments of this specification;

[0011] Figure 4 is a schematic diagram of a change discrimination model according to some embodiments of this specification;

[0012] Figure 5 It is an exemplary flowchart of a training method for a change discrimination model according to some embodiments of the present specification. DETAILED DESCRIPTION

[0013] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0014] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0015] Unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0016] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0017] Figure 1 It is a scene diagram of a map element change detection system according to some embodiments of this specification.

[0018] like Figure 1 As shown, the application scenario 100 of the map element change detection system may include a collection device 110 , a network 120 , a terminal 130 , a processing device 140 and a storage device 150 .

[0019] The acquisition device 110 is used to collect environmental information of the target area (such as a block or road section). In some embodiments, the acquisition device 110 may be one or more vehicles (such as an autonomous driving vehicle). In some embodiments, the acquisition device 110 may include a plurality of different types of sensing devices, such as a radar (Lidar), a temperature sensor, a humidity sensor, a GPS, a wind vane, etc., for collecting environmental features (such as point clouds, positions, temperature, humidity, visibility, wind direction, etc.). In some embodiments, the acquisition device 110 may be equipped with a plurality of image acquisition devices (such as image acquisition device 111-1...image acquisition device 111-n). The image acquisition device may be any suitable device such as a camera, a structured light camera, a depth camera, a laser camera, etc. that can collect multi-view image data of the target area.

[0020] The network 120 may include any suitable network that facilitates information and / or data exchange of the application scenario 100. In some embodiments, one or more components of the application scenario 100 (e.g., the acquisition device 110, the terminal 130, the processing device 140, or the storage device 150) may transmit information and / or data to one or more other components of the application scenario 100 via the network 120. For example, the processing device 140 may obtain multi-view image data of a target area from the acquisition device 110 via the network 120. In some embodiments, the network 120 may be any one or more of a wired network or a wireless network. In some embodiments, the network may be a point-to-point, shared, centralized, or other topological structure or a combination of multiple topological structures.

[0021] The terminal 130 may include a mobile device 130-1, a tablet computer 130-2, a laptop computer 130-3, etc., or any combination thereof. In some embodiments, the terminal 130 may interact with other components in the application scenario 100 through the network 120. In some embodiments, the terminal 130 may receive information and / or instructions input by a user (e.g., a system administrator), and send the received information and / or instructions to the collection device 110 or the processing device 140 via the network 120. For example, a user may input operating instructions (such as driving route planning instructions) to the collection device 110 through the terminal 130. In some embodiments, the terminal 130 may display a high-precision map, as well as map element change information generated by the processing device 140.

[0022] The processing device 140 may process data and / or information obtained from the acquisition device 110, the terminal 130, and the storage device 150. For example, the processing device 140 may obtain multi-view image data acquired by the acquisition device 110. For another example, the processing device 140 may generate map element change information by analyzing and processing the multi-view image data and the high-precision map corresponding to the relevant target area.

[0023] In some embodiments, the processing device 140 can be a single server or a server group. The server group can be centralized or distributed. In some embodiments, the processing device 140 can be local or remote. The processing device 140 can be directly connected to the medical scanning device 110, the terminal 130 and the storage device 150 to access the stored or acquired information and / or data. In some embodiments, the processing device 140 can be implemented on a cloud platform. By way of example only, the cloud platform can include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-layer cloud, etc. or any combination thereof.

[0024] The storage device 150 may store data and / or instructions. In some embodiments, the storage device 150 may store data obtained from the acquisition device 110, the terminal 130, and / or the processing device 140. For example, the storage device 150 may store high-precision maps corresponding to different areas, multi-view image data collected by the acquisition device 110, and the like. In some embodiments, the storage device 150 may store data and / or instructions used by the processing device 140 to execute the exemplary methods described in this specification. For example, the storage device 150 may store instructions for the processing device 140 to execute the methods shown in the various flow charts. In some embodiments, the storage device 150 may include a mass storage device, a removable storage device, a volatile read-write memory, a read-only memory (ROM), and the like, or any combination thereof. In some embodiments, the storage device 150 may be implemented on a cloud platform. In some embodiments, the storage device 150 may be part of the processing device 140.

[0025] The above description is for illustrative purposes only, and actual application scenarios may vary.

[0026] It should be noted that the application scenario 100 is provided for illustrative purposes only and is not intended to limit the scope of the present application. For those of ordinary skill in the art, various modifications or variations can be made according to the description of this specification. However, these variations and modifications will not deviate from the scope of the present application.

[0027] Figure 2 It is a module diagram of a map element change detection system according to some embodiments of this specification.

[0028] like Figure 2 As shown, the map element change detection system 200 (hereinafter referred to as the detection system) may include a perception feature determination module 210 , a map element determination module 220 , a map feature determination module 230 , and a change detection module 240 .

[0029] The perceptual feature determination module 210 may be configured to determine a bird's-eye view feature based on multi-view images captured at a target location.

[0030] In some embodiments, the multi-perspective image includes a multi-perspective image taken at a target position by image acquisition devices of multiple acquisition devices, and the perception feature determination module 210 is configured to further determine, for each acquisition device, multi-scale features corresponding to the multi-perspective image of the acquisition device, and determine the bird's-eye view features based on the multi-scale features corresponding to the multiple acquisition devices.

[0031] The map element determination module 220 can be used to determine the vector representation and semantics of map elements within a preset range around the target location based on the high-precision map.

[0032] The map feature determination module 230 may be configured to determine a feature tensor of each map element based on a vector representation and semantics of each map element.

[0033] In some embodiments, the map feature determination module 230 may also be used to: for each map element, determine the morphological attributes of the map element based on the vector representation of the map element; for each map element, determine the type of the map element based on the semantics of the map element; and determine the feature tensor of the map element based on the morphological attributes and type of the map element.

[0034] The change detection module 240 can be used to determine whether each map element around the target location has changed based on the bird's-eye view feature and the feature tensor of each map element using a trained change discrimination model, where the change discrimination model is a machine learning model.

[0035] In some embodiments, the change discrimination model includes a feature encoder for performing feature encoding on the bird's-eye view features and the feature tensors of each map element to obtain encoded feature information; an associated information extractor for determining the associated feature information between the multi-view images and the map elements based on the encoded feature information; and a change discriminator for determining whether each of the map elements around the target location has changed based on the associated feature information.

[0036] In some embodiments, the feature encoder includes: a first sub-encoder for encoding bird's-eye view features to generate bird's-eye view coding information; a second sub-encoder for encoding feature tensors of map elements to generate map element coding information.

[0037] In some embodiments, the context information extractor comprises a Transformer model.

[0038] In some embodiments, the multi-perspective images taken at the target location are determined based on multiple acquisition devices, and the multi-perspective images corresponding to each acquisition device are used to generate the change probability of each map element. The change detection module 240 is configured to further determine the weight of each acquisition device; based on the weight of each acquisition device and the change probability of each map element, determine whether each map element has changed.

[0039] In some embodiments, the detection system further includes a training module 250 .

[0040] The training module 250 can be used to obtain a sample multi-view image and a first sample high-precision map corresponding to the sample multi-view image; change at least one sample map element in the first sample high-precision map to obtain a second sample high-precision map; determine training samples and training labels based on the sample multi-view image and the second sample high-precision map, train an initial change discrimination model based on the training samples and the training labels, and generate a change discrimination model.

[0041] It should be noted that the functions described in the above modules can be implemented by the detection system after executing computer instructions (program code). In some embodiments, the detection system may include one or more processing devices (e.g., Figure 1 In some embodiments, the detection system may include a corresponding storage medium (such as a memory) or be connected to an external storage medium signal. The storage medium may store instructions for executing the map element change detection method disclosed in this specification. When the processing device reads these instructions, the processing device may be configured to execute the map element change detection method.

[0042] It should be noted that the above description of the detection system and its modules is only for the convenience of description, and this specification cannot be limited to the scope of the embodiments. It can be understood that for those skilled in the art, after understanding the principle of the system, it is possible to arbitrarily combine the modules or form a subsystem to connect with other modules without deviating from this principle. For example, each module can share a storage module, or each module can have its own storage module. Such variations are within the scope of protection of this specification. In some embodiments, the training module 250 and other modules can be implemented on different processing devices. For example, the training module 250 can be implemented on the processing device of the supplier of the machine learning model disclosed in this specification, while other modules can be implemented on the processing device of the user of the machine learning model.

[0043] Figure 3 It is an exemplary flow chart of a map element change detection method according to some embodiments of the present specification.

[0044] In some embodiments, process 300 may be performed by a detection system. Figure 3 As shown, process 300 includes the following steps.

[0045] Step 310 , determining a bird's-eye view feature based on the multi-view images taken at the target location.

[0046] The target position may be a position point or a position area (including multiple adjacent position points) when the multi-view image is taken. It may be the position where the acquisition device (such as the acquisition device 110) is located. The following description will be given using the acquisition device as an autonomous driving vehicle as an example. In some embodiments, the target position may be determined based on a position acquisition device (such as GPS) of the autonomous driving vehicle. The multi-view image includes multiple images of the surrounding environment of the target position taken from multiple different perspectives, including relevant information (such as shape, position, length, width, etc.) of geographic entities (such as traffic signs, lane lines) around the target position at a certain time (such as the current moment or the current time period).

[0047] In some embodiments, the detection system can collect multi-view images by controlling the multi-view camera on the autonomous driving vehicle during the driving process of the vehicle. In some embodiments, the multi-view camera can be a plurality of (e.g., 6) cameras surrounding the vehicle to respectively obtain multiple images of multiple viewpoints of the vehicle (e.g., front, rear, left front, right front, left rear, and right rear). In some embodiments, the multi-view image includes an image corresponding to each viewpoint. In some embodiments, the multi-view image includes multiple frames of images corresponding to each viewpoint (e.g., multiple frames of images continuously collected within a specific time period).

[0048] In some embodiments, the detection system may preprocess the multi-view images. The preprocessing may include, but is not limited to, denoising, rotation, scaling, and the like. When the multi-view images include multiple frames of images at each view, the preprocessing may also include image processing such as splicing, cropping, and integration of the corresponding multiple frames of image data at the same view (such as the front view).

[0049] Bird's Eye View (BEV) is similar to the perspective of a bird flying in the sky.

[0050] In some embodiments, the detection system may perform bird's-eye view conversion processing on the multi-view image to obtain a bird's-eye view image. The bird's-eye view conversion processing may use various methods such as perspective transformation to map the pixel coordinates in the multi-view image to a two-dimensional plane to generate a representation of the bird's-eye view image, and then extract features from the bird's-eye view image through a convolutional neural network (CNN) or other deep learning networks to generate bird's-eye view features corresponding to the multi-view image.

[0051] In some embodiments, the detection system may determine the multi-scale features corresponding to the multi-view images and convert the multi-scale features into bird's-eye view features. In some embodiments, after converting the multi-scale features into bird's-eye view features, the detection system may further flatten the bird's-eye view features before performing subsequent analysis. For example, the bird's-eye view features may be flattened into a one-dimensional vector.

[0052] Multi-scale features can represent the global and local information of multi-view images, and can more accurately and meticulously reflect the feature information of entities of different sizes and shapes in multi-view images.

[0053] The detection system can use various feasible methods to obtain multi-scale features. In some embodiments, the detection system can use a residual network and a feature pyramid network to extract features from a multi-view image and generate multi-scale features corresponding to the multi-view image. The residual network and the feature pyramid network can be trained feature extraction models.

[0054] In some embodiments, the detection system may perform bird's-eye view conversion on multi-scale features through a multi-layer perceptron (MLP) or a combined model such as a BEV network and a Transformer network (BEVFormer model) to generate bird's-eye view features.

[0055] In some embodiments of this specification, by first obtaining multi-scale features of multi-view images, and then determining bird's-eye view features based on the multi-scale features, the subsequent map element change detection is used. Compared with other perspectives, the bird's-eye view can extract global features and local features at the same time, avoiding the problem of feature information loss caused by object occlusion, making the subsequent map element change detection more accurate and comprehensive.

[0056] In some embodiments, in step 310, a multi-perspective image taken by an autonomous driving vehicle at a target location can be obtained, and the corresponding bird's-eye view features can be determined. Alternatively, in step 310, multi-perspective images taken by multiple autonomous driving vehicles at the same target location (or nearly the same target location) can be obtained. For each autonomous driving vehicle, the detection system can determine the multi-scale features corresponding to the multi-perspective image of the autonomous driving vehicle, thereby obtaining multiple groups of multi-scale features corresponding to multiple autonomous driving vehicles. The detection system can determine the bird's-eye view features based on the multiple groups of multi-scale features corresponding to the multiple autonomous driving vehicles.

[0057] Step 320: Determine the vector representation and semantics of map elements within a preset range around the target location based on the high-precision map.

[0058] High-precision maps refer to electronic maps with accuracy exceeding preset standards (such as reaching the centimeter level). High-precision maps can be used to provide decision support (such as obstacle avoidance, deceleration when turning, etc.) or navigation for autonomous vehicles driving in a certain area. They include rich road information (such as road type, lane line position, lane line type), environmental information (such as information on roadside infrastructure, obstacles, road signs and other environmental objects), and real-time dynamic information (such as traffic flow, red light status).

[0059] Map elements can be various objects in high-precision maps. For example, they include water systems (such as lakes, reservoirs, ditches), residential areas and facilities, and traffic objects. Exemplary traffic objects include crosswalks, lane lines, lane center lines, curves, various traffic signs (such as traffic lights, speed limit signs), road boundaries, etc.

[0060] In some embodiments, map elements may be represented by vector representations and corresponding semantics.

[0061] The vector representation of a map element may be a geometric object used to represent the map element, which is composed of one or more of a point, a line, a circle, an ellipse, a polygon, etc. For example, the vector representation of a lane line may be composed of several solid line segments and / or dashed line segments. For another example, the vector representation of a traffic light may be composed of a rectangle and three circles.

[0062] The vector representation of map elements can be used to represent the morphological attributes of map elements. For example, for lane lines, their morphological attributes include whether they are solid or dotted lines, length, starting point, end point, curvature of the curve, etc. The form of the vector representation of the lane line can be determined according to its morphological attributes, so that the vector representation can accurately represent the real lane line.

[0063] The semantics of a map element may be used to indicate the meaning (such as type) of the map element. For example, the semantics of a map element may indicate that the map element is a lane line.

[0064] In some embodiments, the high-precision map stores vector representations and semantics corresponding to each map element. The detection system can directly retrieve the vector representations and semantics of the map elements within a preset range around the target location from the high-precision map based on the coordinates of the target location. In some embodiments, the high-precision map stores point clouds and semantics corresponding to each map element. The detection system can retrieve the point clouds and semantics of the map elements within a preset range around the target location from the high-precision map based on the coordinates of the target location, and determine the corresponding vector representation based on the retrieved point clouds of the map elements. In some embodiments, the detection system can obtain the target high-precision map from a storage device (such as storage device 150, external storage device) and perform analysis and processing. Among them, the target high-precision map can be a high-precision map corresponding to the area where the autonomous driving vehicle is located (such as a block, road section, etc.).

[0065] The preset range may be a range having a specific size and shape around the target position. For example, the preset range may be a square area or a circular area centered on the target position.

[0066] In traditional methods, map elements are usually represented by data forms such as point clouds rather than vector representations. Compared with data forms such as point clouds, vector representations occupy much less data. Subsequent map change detection based on vector representations can reduce the required computing resources and improve computing efficiency.

[0067] Step 330 , determining a feature tensor of each map element based on the vector representation and semantics of the map element.

[0068] Feature tensors can be used to represent map features. Each map feature corresponds to a feature tensor.

[0069] In some embodiments, for each map element, the detection system may determine the morphological attributes of the map element based on the vector representation of the map element; determine the type of the map element based on the semantics of the map element; and then determine the feature tensor of the map element based on the morphological attributes and type of the map element.

[0070] In some embodiments, the feature tensor may be designed as a vector having three dimensions.

[0071] The value of the first dimension is used to indicate the type of map element, which can be determined based on the semantics of the map element. The value of the first dimension can be in the form of one or a combination of numerical values, symbols, etc., which can be pre-set based on the total number of types of map elements. For example, A1 represents the center line of the lane, A2 represents the deceleration warning line; B1 represents the traffic light sign, B2 represents the deceleration sign, B3 represents the left sharp turn sign, etc. In some embodiments, the first dimension can be set to the number of types of map elements, which represents the total number of types of map elements present in the target high-precision map, that is, the value of the first dimension of the feature tensor corresponding to each map element is equal.

[0072] The value of the second dimension is used to represent the number of the map element, which can be in the form of an integer. The number can be determined based on the total number of a certain type of map element. As an example only, in a map element set, the total number of lane centerlines is N, then the N map elements representing lane centerlines can be numbered from 0 to N-1. In other words, the value of the second dimension of the feature tensor corresponding to the map element representing the lane centerline can be a value within [0, N-1].

[0073] The number of each feature tensor, or the combination of type and number, can be used as a unique representation of the corresponding map element to distinguish different map elements and their corresponding feature tensors.

[0074] The value of the third dimension can be used to represent the morphological attributes of map elements. Morphological attributes may include position, length, width, color, shape, continuity (such as whether the lane line is a continuous solid line or a discontinuous dashed line), degree of curvature, etc. The value of each morphological attribute can be represented in various forms such as numbers and letters. In some embodiments, when there are multiple morphological attributes, the value of the third dimension can be an array (or vector) composed of the values ​​of multiple morphological attributes, or can be determined by concatenating the values ​​of multiple morphological attributes in a specific order (in this case, the value of the third dimension can be regarded as a string with a larger length).

[0075] In some embodiments, when there are multiple morphological attributes, the feature tensor may have more than three dimensions, with each morphological attribute having a corresponding feature tensor dimension.

[0076] In some embodiments of the present specification, feature tensors can be used to reduce the dimensionality of complex map elements, thereby reducing storage space and improving computational efficiency.

[0077] Step 340, based on the bird's-eye view feature and the feature tensor of each map element, use the trained change discrimination model to determine whether each map element around the target location has changed, and the change discrimination model is a machine learning model.

[0078] In some embodiments, the detection system may obtain the difference information between the actual geographic entity and each map element in the original high-precision map based on the comparative analysis of the multi-view image and the high-precision map, and then determine the change information of the map element. Among them, the change information of the map element may include but is not limited to the number and type of the changed map elements, the changed attributes of each map element, etc. The changed attributes of the map element include the addition, reduction, position change, morphological change, etc. of the map element.

[0079] In some embodiments, the detection system can use the trained change discrimination model to analyze the bird's-eye view features corresponding to the multi-view images and the feature tensors of the map elements in the target high-precision map to determine whether each map element has changed. For more information about the change discrimination model, see Figure 4 and its description.

[0080] In some embodiments, the autonomous driving vehicle may include multiple (e.g., 2, 4, etc.) autonomous driving vehicles, and the multi-view images corresponding to each autonomous driving vehicle are used to generate the change probability of each map element, wherein the change probability of each map element can be determined according to the change discrimination model.

[0081] The detection system can determine the weight of each autonomous driving vehicle, and determine whether each map element has changed based on the weight of each autonomous driving vehicle and the change probability of each map element.

[0082] It should be noted that multiple autonomous driving vehicles refer to vehicles that collect multi-view images of the same target location. The time periods for image collection by different vehicles can be the same (such as simultaneous collection) or different (such as sequential collection); the viewing angle settings of the multi-view cameras of different vehicles can also be the same or different.

[0083] In some embodiments, the weight of the vehicle is related to environmental features and vehicle acquisition features. Environmental features may include temperature, humidity, wind direction, wind speed, visibility, etc., which can be obtained based on various sensors of the autonomous driving vehicle (such as temperature sensors, wind vanes, air quality sensors, etc.). Vehicle acquisition features may be determined by the vehicle's driving speed, the camera's viewing angle combination, etc. It can be understood that different weather characteristics and vehicle acquisition features have different degrees of influence on the collected multi-view images, for example, the image clarity, the occlusion of physical entities, the contrast with the surrounding environment, etc., which have different degrees of influence on the subsequent map element change judgment results. For each autonomous driving vehicle, the detection system can estimate the quality of its collected multi-view images based on its vehicle acquisition features and / or the environmental features when the image is collected, and determine the weight of each autonomous driving vehicle based on the quality. For example, the weight and quality can be proportional.

[0084] In some embodiments, the detection system can execute process 300 respectively based on the multi-view images collected by each vehicle to obtain the change probability of each map element under the multi-view images collected by different vehicles. The detection system can use weighted summation and other methods to determine the final change probability of each map element. Exemplarily, the weights of vehicles V1, V2, and V3 are W1, W2, and W3 respectively. For map element M1, the change probabilities determined based on vehicles V1, V2, and V3 are F1, F2, and F3 respectively. Then the final change probability F of the map element M1 is F=W1×F1+W2×F2+W3×F3. When the value of the change probability F meets the preset conditions (such as being greater than the preset change probability threshold), it is determined that the map element has changed.

[0085] Some embodiments of the present specification take into account the impact of different environmental characteristics and vehicle acquisition characteristics on multi-view images, and conduct a comprehensive assessment of changes in map elements by setting weights of multiple vehicles, thereby preventing change detection results from being affected by specific environmental factors and vehicle acquisition parameters, thereby improving the accuracy of change detection.

[0086] In some embodiments of the present specification, by using a change identification model, change detection is performed based on the multi-view images taken by the multi-view camera of the autonomous driving vehicle and the map elements in the high-precision map. This can quickly and efficiently detect the differences between the real-time geographic environment information and the high-precision map, and automatically identify the changes of each map element, thereby providing a reliable and accurate basis for downstream map navigation and vehicle driving decisions.

[0087] In some embodiments, when the autonomous driving vehicle is traveling along a specific trajectory, its multi-view camera can be controlled to capture a set of multi-view images at regular intervals, and steps 320 to 340 described above can be executed for each set of multi-view images to determine whether the map elements corresponding to each position on the vehicle's driving trajectory have changed.

[0088] Figure 4 It is a schematic diagram of a change discrimination model according to some embodiments of this specification.

[0089] The change discrimination model refers to a model used to detect changes in map elements in a high-precision map. In some embodiments, the change discrimination model is a trained machine learning model.

[0090] In some embodiments, Figure 4 As shown, the change identification model 400 includes a feature encoder 420 , a related information extractor 430 , and a change identification unit 440 .

[0091] The feature encoder 420 may be used to perform feature encoding on the bird's-eye view feature 412 and the feature tensor 415 corresponding to each map element to obtain encoded feature information.

[0092] The bird's-eye view feature 412 can be determined based on the multi-view image 411, and the feature tensor 415 corresponding to each map element can be determined based on the high-precision map 414. For more information about multi-view images, bird's-eye view features, map elements, and feature tensors, see Figure 3 and its description.

[0093] The coded feature information includes bird's-eye view coded information 425 and map element coded information 426 .

[0094] In some embodiments, the feature encoder includes a first sub-encoder and a second sub-encoder, wherein the first sub-encoder can be used to encode the bird's-eye view feature to generate bird's-eye view encoding information, and the second sub-encoder can be used to encode the feature tensor of the map element to generate the map element encoding information.

[0095] The bird's-eye view coding information may refer to the position coding information of the bird's-eye view feature, which can be used to reflect the position information of each geographic entity in the bird's-eye view feature and the image feature information under the bird's-eye view. The map element coding information integrates the feature information of each map element in the high-precision map and can be used to reflect the global features of the entire high-precision map.

[0096] like Figure 4 As shown, the input of the first sub-encoder 421 includes the bird's-eye view feature 412 , and after being processed by the first sub-encoder 421 , the bird's-eye view coding information 425 is output.

[0097] In some embodiments, the first sub-encoder 421 may be an auto encoder (AE) model.

[0098] The input of the second sub-encoder 422 includes the feature tensor 415 corresponding to each map element. After being processed by the second sub-encoder 422, map element encoding information 426 is output.

[0099] In some embodiments, the second sub-encoder 422 may include a multi-layer perceptron (MLP) and a maximum pooling layer. The multi-layer perceptron and the maximum pooling layer may map the feature tensor 415 of each semantic element to a high-dimensional space and extract global information as map element encoding information 426.

[0100] The association information extractor 430 is used to determine the association feature information between the multi-view image and the map element based on the encoded feature information. For example, the association information can represent the difference between the multi-view image (ie, the real traffic conditions) and the map element (ie, the high-precision map).

[0101] In some embodiments, the context information extractor 430 includes a Transformer model, which can associate multi-view image information with map element information using a cross-attention mechanism. Figure 4 As shown, the associated information extractor 430 includes an encoder 431 and a decoder 432 .

[0102] The input of the encoder 431 includes the bird's-eye view coding information 425 , and the input of the decoder 432 includes the map element coding information 426 . After being processed by the encoder 431 and the decoder 432 , the associated information extractor 430 outputs associated feature information 435 .

[0103] In some embodiments, the encoder 431 includes multiple encoder layers, a layer normalization layer, and a residual connection layer, wherein each encoder layer includes a multi-head self-attention layer and a fully connected feed-forward network.

[0104] The multi-head self-attention layer in the encoder layer learns the positional relationship between different geographic entities in the multi-view image according to the bird's-eye view coding information 425 through the self-attention mechanism; the layer normalization layer can be used to standardize the output of each encoder layer; the residual connection layer allows information to be directly transferred across layers, which helps to alleviate the gradient disappearance problem. Among them, the encoder 431 outputs the Key and Value corresponding to each geographic entity by processing the bird's-eye view coding information 425, where the Key represents the location information of the geographic entity and the Value represents the image feature information corresponding to the geographic entity. Multiple Keys and Values ​​are used as inputs to the decoder 432 together with the map element coding information 426. Among them, the map element coding information 426 can be represented as a global query Query of a high-precision map, and the global query Query includes a query Query for each map element.

[0105] The decoder 432 includes multiple decoder layers, a normalization layer, and a residual connection layer. Each decoder layer includes a multi-head self-attention layer, a cross-attention layer, and a fully connected feedforward network.

[0106] The multi-head self-attention layer in the decoder layer learns the positional relationship between map elements according to the map element encoding information 426 through the self-attention mechanism. At the same time, the cross-attention layer performs association learning according to the position information of each geographic entity in the bird's-eye view encoding information 425 and the map element encoding information 426 representing the global information of the map element through the cross-attention mechanism to generate and output the representation of the associated feature information 435. Among them, the associated feature information 435 may include the association result of each geographic entity information and the map element information, which can be used to represent the difference information between the image feature information of the geographic entity and the morphological attribute information of the map element.

[0107] The change detector 440 may be used to determine whether each map element around the target location has changed based on the associated feature information 435. In some embodiments, the change detector 440 may be a deep neural network (DNN) model.

[0108] like Figure 4 As shown, the input of the change discriminator 440 includes the related feature information 435, and after being processed by the change discriminator 440, the change information 451 of the map element is output.

[0109] The map element change information 451 includes the judgment result of whether each map element has been changed, or the probability of each map element being changed (also called confidence).

[0110] In some embodiments, the output of the change discriminator 440 includes a second feature tensor corresponding to each map element. The second feature tensor is similar to the first feature tensor, and may also include a first dimension representing the type of the map element and a second dimension representing the number of the map element. The difference is that the third dimension represents the judgment result of whether a change has occurred or the probability of a change.

[0111] In some embodiments, the detection system may determine the map elements whose change probability is greater than a preset threshold (eg, 0.85) in the change information 451 of the map elements as target map elements that have changed.

[0112] In some embodiments, the detection system can generate warning information based on the map element change information 451 to prompt the system administrator; the map element change information 451 can also be sent to downstream application scenarios or tasks. For example, it can be sent to related autonomous driving vehicles to complete the update of driving decisions (such as slowing down, detouring, collision avoidance, etc. at appropriate times and operations related to the changed map elements).

[0113] In some embodiments, the change discrimination model can be obtained by jointly training the feature encoder, the associated information extractor, and the change discriminator. For more information about the training method of the change discrimination model, see Figure 5 and its description.

[0114] It should be understood that Figure 4 The related descriptions are for illustrative purposes only and are not intended to limit the scope of protection of this specification. For example, the feature encoder, the associated information extractor, and the change discriminator can be implemented with any type of machine learning component.

[0115] In some embodiments of this specification, the change discrimination model can be used to automatically analyze the correlation between multi-view images and high-precision maps, and achieve real-time and accurate change detection of map elements. Reduce the time and effort consumed by manual analysis. In some embodiments, by using a Transformer model based on a self-attention mechanism and a cross-attention mechanism, the extracted correlation feature information can have a higher accuracy, thereby improving the accuracy of the change detection results.

[0116] Figure 5 It is an exemplary flowchart of a training method for a change discrimination model according to some embodiments of the present specification.

[0117] In some embodiments, process 500 may be performed by a detection system. In some embodiments, process 500 may be performed by a supplier system that changes the discrimination model. Figure 5 As shown, process 500 includes the following steps.

[0118] Step 510, obtaining the sample multi-view image and the first sample high-precision map corresponding to the sample multi-view image.

[0119] The sample multi-view image refers to a historical multi-view image collected by one or more autonomous driving vehicles through a multi-view camera. For example, the sample multi-view image may be multi-view image data about a sample area. The sample area may be pre-set, for example, it may be urban area A, urban area B, etc.

[0120] It should be noted that, in some embodiments, the sample multi-perspective image can be a multi-perspective image captured at the current moment or at a time close to the current moment (i.e., the most recent), so that the real environment information corresponding to the sample multi-perspective image is consistent with the information of the currently known high-precision map, thereby reducing the difficulty of determining training samples and improving training efficiency.

[0121] The sample map elements contained in the first sample high-precision map are consistent with the real objects in the sample multi-view image. That is, the sample map elements are unchanged compared with the first sample high-precision map. In some embodiments, the first sample high-precision map can be generated based on the sample multi-view image, or based on multi-view images taken at a similar time to the sample multi-view image (the objects contained in these multi-view images are the same as those in the sample multi-view image).

[0122] Step 520, change at least one sample map element in the first sample high-precision map to obtain a second sample high-precision map.

[0123] The sample map elements refer to the map elements in the first sample high-precision map that are located within a preset range around the location of the sample multi-view image. The change operation on at least one sample map element may include deleting one or more sample map elements, adding one or more sample map elements, and changing the form (color, position, style, etc.) of one or more sample map elements.

[0124] The second sample high-precision map refers to a high-precision map after the sample map elements of the first sample high-precision map have been changed.

[0125] Step 530: Determine training samples and training labels based on the sample multi-view image and the second sample high-precision map.

[0126] It can be understood that compared with the sample multi-view image, the map elements of the second sample high-precision map have changed, so it can be used to simulate the situation where the geographical entities in the real environment are different from the map elements in the high-precision map.

[0127] The detection system can generate a sample bird's-eye view feature based on the sample multi-view image through the processing of step 310. At the same time, a sample feature tensor of the sample map element is generated based on the second sample high-precision map through the processing of step 330. Accordingly, the sample bird's-eye view feature and the sample feature tensor are used as training samples for the change discrimination model.

[0128] The training label can be determined based on the changes in the second sample high-precision map relative to the first sample high-precision map. Exemplarily, the training label can include gold standard change information (such as whether a change has occurred) for each sample map element.

[0129] In some embodiments, the detection system can obtain multiple groups of sample multi-view images and their corresponding first sample high-precision maps in step 510. For each group of sample multi-view images and the corresponding first sample high-precision map, steps 520 and 530 can be executed to obtain a pair of training samples and training labels. In one embodiment, for the first sample high-precision map corresponding to a group of multi-view images, multiple changes in different forms can be made in step 520 to obtain multiple second sample high-precision maps. In step 530, a pair of training samples and training labels can be determined based on the group of multi-view images and each second sample high-precision map. Since there are multiple second sample high-precision maps, multiple pairs of training samples and training labels can be generated. In this way, a large number of training samples can be quickly obtained, the difficulty of training can be reduced, and the training efficiency can be improved.

[0130] Step 540 , training an initial change discrimination model based on training samples and training labels to generate a change discrimination model.

[0131] In some embodiments, the detection system may construct an initial change discrimination model, which includes an initial feature encoder, an initial correlation feature extractor, and an initial change discriminator.

[0132] The initial feature encoder includes an initial first sub-encoder and an initial second sub-encoder; the initial associated information extractor is an initial Transformer model, which includes an initial encoder and an initial decoder. For more information about the change discrimination model, see Figure 4 And its description will not be repeated here.

[0133] During training, the detection system can input the sample bird's-eye view features into the initial first sub-encoder to obtain the sample bird's-eye view encoding information; input the sample feature tensor into the initial second sub-encoder to obtain the sample map element encoding information. Then, the detection system can further input the sample bird's-eye view encoding information and the map element encoding information into the initial encoder and initial decoder in the initial association information extractor respectively to obtain the sample association feature information; finally, the detection system can input the sample association feature information into the initial change discriminator to obtain the prediction result of the change information of the map element.

[0134] The detection system can determine the value of the loss function based on the prediction result of the change information of the map element and the training label. The detection system can update the parameters of the initial feature encoder, the initial associated information extractor, and the initial change discriminator based on the value of the loss function to obtain a trained feature encoder, an associated feature extractor, and a change discriminator, and then obtain a trained change discrimination model.

[0135] In some embodiments of the present specification, by changing the map elements in the high-precision map, it is possible to simulate the changes of physical entities in the real environment, thereby reducing the difficulty of obtaining training samples and improving the training efficiency; at the same time, the joint training of the feature encoder, the associated feature extractor and the change discriminator improves the training efficiency; on the other hand, the use of a supervised learning training algorithm based on training labels can improve the training accuracy and reduce the training difficulty.

[0136] It should be noted that the above description of the relevant process is only for example and explanation, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to the process under the guidance of this specification. However, these modifications and changes are still within the scope of this specification.

[0137] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

[0138] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.

[0139] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0140] Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0141] In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise specified, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of this specification are approximate values, in specific embodiments, the setting of such numerical values ​​is as accurate as possible within the feasible range.

[0142] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, etc., cited in this specification are hereby incorporated by reference in their entirety. Except for application history documents that are inconsistent with or conflicting with the contents of this specification, documents that limit the broadest scope of the claims of this specification (currently or later attached to this specification) are also excluded. It should be noted that if the descriptions, definitions, and / or use of terms in the materials attached to this specification are inconsistent or conflicting with the contents described in this specification, the descriptions, definitions, and / or use of terms in this specification shall prevail.

[0143] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A map element change detection method, characterized in that: include: Determine bird's-eye view features based on multi-view images taken at a target location; Determine the vector representation and semantics of map elements within a preset range around the target location based on the high-precision map; Determining a feature tensor of each map element based on the vector representation and semantics of each map element; Based on the bird's-eye view features and the feature tensor of each of the map elements, a trained change discrimination model is used to determine whether each of the map elements around the target location has changed, and the change discrimination model is a machine learning model.

2. The method according to claim 1, characterized in that Determining the feature tensor of each map element based on the vector representation and semantics of each map element includes: For each of the map elements, Determining the morphological attributes of the map elements based on the vector representation of the map elements; Determining the type of the map element based on the semantics of the map element; Based on the morphological attributes and types of the map elements, the feature tensor of the map elements is determined.

3. The method according to claim 1, characterized in that The change discrimination model includes: A feature encoder, used for performing feature encoding on the bird's-eye view feature and the feature tensor of each of the map elements to obtain encoded feature information; an association information extractor, configured to determine association feature information between the multi-view images and the map elements based on the coded feature information; and A change discriminator is used to determine whether each of the map elements around the target position has changed based on the associated feature information.

4. The method according to claim 3, characterized in that The feature encoder comprises: A first sub-encoder, used for encoding the bird's-eye view feature to generate bird's-eye view encoding information; The second sub-encoder is used to encode the feature tensor of the map element to generate map element encoding information.

5. The method according to claim 3, characterized in that: The associated information extractor includes a Transformer model.

6. The method according to claim 1, characterized in that The multi-view images taken at the target location are determined based on a plurality of acquisition devices, and the multi-view images corresponding to each acquisition device are respectively used to generate the change probability of each map element. The method further includes: Determining the weight of each of the acquisition devices; Based on the weight of each of the acquisition devices and the change probability of each of the map elements, it is determined whether each of the map elements has been changed.

7. The method according to claim 1, characterized in that The change discrimination model is trained and generated in the following way: Acquire a sample multi-view image and a first sample high-precision map corresponding to the sample multi-view image; Changing at least one sample map element in the first sample high-precision map to obtain a second sample high-precision map; Determine training samples and training labels based on the sample multi-view image and the second sample high-precision map; An initial change discrimination model is trained based on the training samples and the training labels to generate the change discrimination model.

8. The method according to claim 1, characterized in that The multi-view images include multi-view images captured at the target position by image acquisition devices of multiple acquisition devices, and the method further includes: For each of the acquisition devices, determining multi-scale features corresponding to the multi-view images of the acquisition device; The bird's-eye view feature is determined based on the multi-scale features corresponding to the multiple acquisition devices.

9. A map element change detection system, characterized in that: include: a perception feature determination module, for determining bird's-eye view features based on multi-view images taken at a target location; A map element determination module, used to determine the vector representation and semantics of map elements within a preset range around the target location based on a high-precision map; A map feature determination module, which determines a feature tensor of each map element based on the vector representation and semantics of each map element; The change detection module determines whether each of the map elements around the target location has changed based on the bird's-eye view features and the feature tensor of each of the map elements using a trained change discrimination model, wherein the change discrimination model is a machine learning model.

10. A map element change detection device, characterized in that: The method comprises a processor, wherein the processor is used to execute the map element change detection method according to any one of claims 1 to 8.

Images