Road map fusion

Abstract

The present disclosure provides a map fusion method, which includes receiving a source graph and a target graph, where the source graph represents a source map and the target graph represents a target map, and includes nodes and edges connecting the nodes. The method further includes: processing each of the source graph and the target graph in a graph convolutional layer to provide a graph convolutional layer output related to the source graph and the target graph; processing each of the graph convolutional layer outputs of the source graph and the target graph in a rectified linear layer to output a node feature map related to the source graph and the target graph, where the node feature map includes data representing the unique features of each node; and selecting multiple pairs of node representations from the node feature maps related to the source graph and the target graph, and concatenating the selected multiple pairs of node representations to output the selected and concatenated multiple pairs of node representations.

Description

Technical Field

[0001] The present disclosure relates to systems and methods for fusing road maps (generally referred to as "systems"). Background Art

[0002] Many applications, such as area exploration, location-based services, route planning, search for available parking spots, etc., are based on digital street maps. Due to various applications, there are also a large number of digital street maps that are different especially in terms of their coverage, recording time, map generation methods, attributes, etc. To utilize all the information included in different street maps, a technique called map fusion (also known as map merging, map combination, or map matching) is applied, which allows merging two or more maps into one map. However, due to the huge amount of information contained in numerous maps, map fusion methods can be time-consuming and resource-consuming, or inaccurate. Therefore, there is considerable interest in more efficient and accurate map fusion techniques. Summary of the Invention

[0003] A map fusion method includes receiving a source graph and a target graph, where the source graph represents a source map and the target graph represents a target map, and includes nodes and edges connecting the nodes. The method further includes: processing each of the source graph and the target graph in a graph convolutional layer to provide a graph convolutional layer output related to the source graph and the target graph; processing each of the graph convolutional layer outputs of the source graph and the target graph in a rectified linear layer to output a node feature map related to the source graph and the target graph, where the node feature map includes data representing unique features of each node; and selecting multiple pairs of node representations from the node feature map related to the source graph and the target graph, and concatenating the selected multiple pairs of node representations to output the selected and concatenated multiple pairs of node representations. The method further includes: processing the selected and aggregated multiple pairs of node representations in a fully connected layer to provide a fully connected layer output; performing softmax processing on the fully connected layer output to output the matching probabilities of the nodes in the node feature map related to the source graph and the target graph; and based on the matching probabilities of the nodes, determining whether to fuse the nodes in the source map with the corresponding nodes in the target graph.

[0004] After reviewing the following detailed description and the drawings (figures), other systems, methods, features, and advantages will be apparent or will become apparent to those skilled in the art. It is intended that all such additional systems, methods, features, and advantages be included within this specification, be within the scope of the present invention, and be protected by the appended claims. Description of the Drawings

[0005] The system can be better understood with reference to the following drawings and description.

[0006] Figure 1 It is a schematic diagram illustrating multiple different types of electronic maps to be fused.

[0007] Figure 2 It is a schematic diagram illustrating the fusion of a source map and a target map by matching intersections and streets.

[0008] Figure 3 It is a schematic diagram illustrating a 1:1 match between intersections in a source road network and a target road network.

[0009] Figure 4 It is a schematic diagram illustrating a 1:m match between intersections in a source road network and a target road network.

[0010] Figure 5 It is a schematic diagram illustrating an n:m match between intersections in a source road network and a target road network.

[0011] Figure 6 It is a schematic diagram illustrating an exemplary convolutional network or layer.

[0012] Figure 7 It is a schematic diagram illustrating another exemplary convolutional network or layer.

[0013] Figure 8 It is a schematic diagram illustrating an exemplary graph convolutional layer, where the node is examined for its adjacent nodes.

[0014] Figure 9 It is a schematic diagram illustrating a graph neural network architecture for map fusion using three exemplary graph convolutional layers.

[0015] Figure 10 It is a schematic diagram illustrating the learning of long short-term memory edge features. Detailed Description

[0016] As described above, digital street maps may differ especially in terms of their coverage, recording time, map generation method, and attributes, among other aspects. For example, the aspect "coverage" not only relates to the area captured by the map, but also to the granularity (e.g., some maps may focus only on highways, while other maps also include smaller streets). The aspect "recording time" is another important aspect, as the street network, for example, is constantly changing, such that maps capturing the same area at different times will likely be correspondingly different. It should be noted that the same map may also have different recording times for different areas. The aspect "map generation method" presents the fact that some maps are generated by mapping companies conducting mapping surveys in a rather consistent manner, while other maps (e.g., Open Street Map) are generated by many users through collaborative work, which may lead to less consistent modeling of the map. The aspect "attributes" takes into account that digital street maps typically include other information in addition to the pure road network. These attributes can be quite diverse and are important for supporting advanced applications beyond simple route selection. Examples of attributes are speed limits, traffic information, available parking spots, allowable gross vehicle weight, road conditions, etc.

[0017] See Figure 1 , the process of combining information from two or more maps 101 - 104 is referred to as map fusion, map conflation, map merging, or map matching. By combining different information from several maps 101 - 104 (e.g., digital street maps), it is possible to increase the coverage of the resulting map, improve the quality of the input maps, correct any errors the maps may contain, and compile a map with the combined attributes of the input maps. The basic principle of map fusion is to match the objects in the source map to the corresponding objects in the target map. The objects can be (for example) streets, intersections, or points of interest. Based on the matching of these objects, information (e.g., the geometry of a street, speed limit, or traffic information) can be transferred from one map to another.

[0018] As Figure 2 depicted, the road network can be represented as a graph G(V, E), i.e., as a set of nodes V (junctions, intersections) and edges E (road segments), where each edge E connects two nodes V. In the example Figure 2 shown and described, there are multiple edges 201 and nodes 203 in the target map to which multiple edges 202 and nodes 204 in the source map are mapped.

[0019] In the following and with reference to Figure 3 , Figure 4 and Figure 5, map fusion focuses on intersections (nodes), where each map is, for example, a road network. Map fusion is based on a source road network S and a target road network T, where it is assumed that the source road network S and the target road network T contain multiple pairs of matching (sv, tv) source nodes sv and target nodes tv, where sv ∈ S.V and tv ∈ T.V. If the source node sv and the target node tv represent the same physical intersection in the real world, then there is a matching pair (sv, tv). It should be noted that due to different modeling techniques, sometimes a 1:1 match between intersections (nodes) in the source road network and the target road network is not possible. In this case, 1:m, n:1, or n:m matching can be performed. Some map fusion methods attempt to match the streets or intersections of two input maps, where n represents the number of nodes to be matched in the source map, and m represents the number of nodes to be matched in the target map.

[0020] Figure 3 Depicts a 1:1 match of the source road network (map) and the target road network (map). In this example, the target road network (map) is graphically represented by node 301 at the intersection including edges 302, 303, and 304, and the source road network (map) is graphically represented by node 305 at the intersection including edges 306, 307, and 308. Figure 4 Depicts a 1:m (e.g., m = 3) match of the source road network (map) and the target road network (map). In this example, the target road network (map) is graphically represented by nodes 401, 405, and 408. Node 401 represents the intersection of edges 402, 403, and 404. Node 405 represents the intersection of edges 404, 406, and 407. Node 408 represents the intersection of edges 402, 407, and 409. The source road network (map) is graphically represented by node 410 at the intersection including edges 411, 412, and 413. Figure 5 Depicts an n:m (e.g., n = 2, m = 2) match of the source road network (map) and the target road network (map). In this example, the target road network (map) is graphically represented by nodes 501 and 505. Node 501 represents the intersection of edges 502, 503, and 504. Node 505 represents the intersection of edges 504, 506, and 507. The source road network (map) is graphically represented by nodes 508 and 512. Node 508 represents the intersection of edges 509, 510, and 511. Node 512 represents the intersection of edges 511, 513, and 514.

[0021] Exemplary algorithms for map fusion may include at least three processing stages. In a first processing stage called "candidate selection", for each object in the target map, a set of candidate objects t from the source map with support dist(s,t) < ε is selected, where ε represents a threshold, and dist() is typically a combined distance that depends on several aspects such as intersections, spatial distances, outgoing links, the structure of outgoing links, etc. and aspects such as streets, spatial distances, shape similarity, length, angle, etc.

[0022] In a second processing stage called "optimization", the distances from the first stage can be converted into scores (sometimes probabilities), which allow deciding whether two objects should match. Subsequently, these scores are repeatedly updated based on the structure of the neighbors of the matching pairs. The final result is a score matrix indicating the likelihood that these objects will match for each object pair (s,t).

[0023] In a third processing stage called "final selection", the final matching pairs are selected to ensure no contradictions remain. There are several examples of the above method applied to the merging process of databases from heterogeneous sources. The term merging is used to describe the procedure for integrating such different data, and the merging method plays an important role in the system in aspects such as updating databases, deriving new cartographic products, densifying digital elevation models, automatic feature extraction, etc. Each merging process can be classified based on (for example) the evaluation measures of each merging process and its main application problems. One way is to classify the merging process based on the matching criteria or the representation model. In an exemplary method, a heuristic probabilistic relaxation road network matching method is used to integrate the available and up-to-date information of multi-source data. This method starts with an initial probability matrix reflecting the dissimilarity of the shapes of the mapped objects, and then integrates the relative compatibility coefficients of adjacent candidate pairs to repeatedly update the initial probability matrix until the probability matrix is globally consistent. Finally, the initial 1:1 matching pairs are selected based on the probabilities calculated and refined according to the structural similarity of the selected matching pairs. Subsequently, a matching process is implemented to find m:n matching pairs. For example, the matching between OpenStreetMap network data and professional road network data has shown that our method is independent of the matching direction and successfully matches 1:0 (invalid) pairs, 1:1 pairs, and m:n pairs.

[0024] However, the above method may have one or more of the defects outlined below.

[0025] One defect is the unwanted parameter dependence, because most methods belonging to the above category may include several parameters that need to be highly tuned for the road networks to be fused. Finding the best settings can be difficult.

[0026] Another defect may occur during the adaptation to new scenarios. For example, when considering street pattern changes (e.g., in a new area), parameter dependencies are generally not transferable, which results in more errors or the need for dedicated parameters for each area individually.

[0027] Another defect may be that the system is unable to naturally report problematic situations. Most methods involve examining supposedly matching pairs by human annotators. However, in a production automation environment, this is highly undesirable.

[0028] Another defect may be slow performance. The runtime complexity of the above methods is typically O(n 3 ) or more, which makes them impractical for large-scale map fusion. One way to overcome at least some of these defects is referred to herein as "learning map fusion", where a trainable map fusion method is employed that learns from examples and adjusts its process accordingly.

[0029] Convolutions and convolutional layers are the main building blocks of convolutional neural networks. A convolution is simply the application of a filter to an input that results in an activation. Repeatedly applying the same filter to the input yields an activation map, which is called a feature map and indicates the location and intensity of detected features in the input such as an image, pattern, or figure. The benefit of convolutional neural networks lies in the ability to automatically learn in parallel a large number of filters that are specific to the training dataset under the constraints of a particular predictive modeling problem such as image classification. The result is highly specific features that can be detected anywhere in the input image, pattern, or figure. A convolutional neural network, or simply a CNN, applies filters to the input to produce a feature map that summarizes the presence of detected features in the input. The filters can be pre-determined, but convolutional neural networks allow the filters to be learned during training in the context of a particular prediction problem.

[0030] Filters smaller than the input are deliberately used because this allows the same filter to be applied multiple times at different points in the input. Specifically, the filter is systematically applied to each overlapping portion or patch of the filter size of the input data. This systematic application of the same filter across an image, pattern, or figure has the effect that if the filter is designed to detect a particular type of feature in the input, systematically applying the filter across the entire input image, pattern, or figure allows the filter to have the opportunity to find the feature anywhere in the image, pattern, or figure.

[0031] The first layer in a CNN is always a convolutional layer. A fully connected layer can be added at the end of the network. This layer essentially takes the input (the output of a convolutional or rectified linear unit (ReLU) or a pooling layer in front of it) and outputs a vector of N dimensions, where N is the number of classes from which the program has to choose. Each number in this N-dimensional vector represents the probability of a particular class. A CNN is a regularized version of a multi-layer perceptron. A multi-layer perceptron generally refers to a fully connected network, i.e., each neuron in one layer is connected to all the neurons in the next layer. The "fully connectedness" of these networks makes them prone to overfitting the data. Typical ways of regularization include adding a certain form of magnitude measure of the weights to the loss function. However, CNNs take a different approach to regularization: they exploit the hierarchical patterns in the data and use smaller and simpler patterns to assemble more complex patterns. Thus, in terms of the scale of connectivity and complexity, CNNs are at the lower end.

[0032] Figure 6 Illustrate an exemplary convolutional network or layer based on the above explanation. The convolutional network or layer receives a matrix of numbers representing the pixels of an image 601. For the purpose of illustration, the matrix here is a 5×5 matrix, but the matrix can also have any other size. Select one (or more) sub-matrices (a 3×3 matrix in this example), and then filter the sub-matrix through one (or more) filters 602 (and 603) to output one (or more) activation maps 604 (and 605), and the activation maps can be further processed by an activation layer 606. In Figure 7 In the alternative convolutional network or layer shown, each pixel in the image 601 collects a weighted average of the values of its neighbors and itself (using the filter weights of filter 701) to output an activation map 702, and the activation map can be further processed by an activation layer (not shown in Figure 7 )).

[0033] Refer to Figure 8 , the graph convolutional layer t + 1 can be described as follows: Each node v separately collects information from its neighbors N(v) based on the hidden states (i.e., features) and e vw of the node, its neighbors, and the edges between the node and its neighbors from the previous layer t

[0034]

[0035] Subsequently, update the hidden state of v according to the following formula:

[0036]

[0037] Message function M t and update function Ut is a learnable discriminative function. Nodes may have different numbers of neighbors, and neighbors may not have an order. Multiple layers or networks can be stacked to increase the receptive field.

[0038] An example of a graph convolutional layer that ignores edge features works as follows: Each node collects features from its neighbors w ∈ N(v). In the next step, each feature representation is processed by a single-layer fully connected neural network layer to obtain Then, the features of all neighbors of v are summed to obtain Subsequently, through the aggregated neighbor information and the features of node v the weighted average is calculated Finally, each node's feature representation calculated by the described method is processed by a ReLU unit.

[0039] Figure 9 Depicts the architecture of a Siamese graph convolutional network that allows the execution of a trainable map fusion method. Training is performed based on examples, where false positives and false negatives are added to the example-based training data. As Figure 9 shown, the input to this network is two street networks: a source street network 901 and a target street network 902. Each street network is represented by a corresponding graph, the graph having nodes representing intersections (the positions of which are considered features) of edges (potentially having geometries given by polylines). Each of the two graphs passes through a corresponding series of one or more graph convolutional layers 903, 904 (implemented by 128 graph convolutional filters), and ReLU layers 905, 906, 909, 910 therebetween. In the context of an artificial neural network, ReLU corresponds to a rectifier in the analog domain and provides an activation function defined as the positive part of its expression f(x) = x+ = max(0, x), where x is the input to the neuron. This is also called a ramp function and is similar to a half-wave rectifier in the analog domain. The corresponding graph convolutional filters update the features of the nodes by averaging the current features with the weighted features of neighboring nodes.

[0040] The output after a series of corresponding graph convolutional layers 903, 904, 907, 908, 911, 912 is the node feature maps 913, 914, where each node is represented by a 128-dimensional feature vector. Subsequently, in the selection and aggregation (e.g., concatenation) layer 915, multiple pairs of these node representations are selected and aggregated, and multiple pairs of these node representations are processed in two subsequent fully-connected layers 916, 917 and the subsequent softmax layer 918 to output matching probabilities. In mathematics, the softmax function (also known as softargmax or normalized exponential function) is a function that takes a vector of K real numbers as its input and normalizes the input into a probability distribution consisting of K probabilities. That is, before applying softmax, some vector components may be negative or greater than one and may not sum to 1, but after applying softmax, each component will be in the interval (0, 1) and the components will sum to 1, such that they can be interpreted as probabilities. Additionally, larger input components will correspond to larger probabilities. For example, Softmax is used in neural networks to map the unnormalized output of the network to a probability distribution over the predicted output classes.

[0041] In an exemplary method, the entire network can be trained end-to-end using labeled data. If the matches are not sufficiently represented in the training data, the matches can be over-sampled to achieve higher accuracy.

[0042] The above architecture can be a basis for other extensions such as increasing the depth and width of the network 1001, and is a basis for including edge geometry into the long short-term memory (LSTM) layer 1002, as Figure 10 depicted. LSTM is an artificial recurrent neural network (RNN) architecture used in the field of deep learning. Different from standard feedforward neural networks, LSTM has feedback connections that make it a "universal computer", that is, LSTM can compute anything that a Turing machine can do. It can not only process individual data points, but also process entire data sequences. A common LSTM cell consists of a cell, an input gate, an output gate, and a forget gate. The cell remembers values over an arbitrary time interval, and the three gates regulate the information flow into and out of the cell. In theory, classical RNNs can track any long-term dependencies in the input sequence. The problem with classical RNNs is essentially a computational (or practical) problem: when training classical RNNs using backpropagation, the gradients of backpropagation may "vanish" (i.e., they can tend to zero) or "explode" (i.e., they can tend to infinity) because of the calculations involving finite-precision numbers in the process. The RNN using LSTM cells partially solves the problem of vanishing gradients because LSTM cells also allow gradients to flow unchanged.

[0043] Edge feature e vwIt can also be learned by LSTM based on the basic geometry of the edge features in the map. Specifically, a sequence of support points from node v to node w can be used to learn these features, as Figure 10 depicted in

[0044] The methods described above can be encoded as instructions for execution by a processor and stored in a computer-readable medium such as a CD ROM, disk, flash memory, RAM or ROM, electromagnetic signal or other machine-readable medium. Alternatively or additionally, any type of logic can be utilized and any type of logic can be implemented as analog or digital logic using hardware such as one or more integrated circuits (including amplifiers, adders, delayers and filters) or one or more processors that execute instructions for amplification, addition, deferral and filtering; or any type of logic can be implemented in the form of software in an application programming interface (API) or a dynamic link library (DLL) as a function available in shared memory or defined as a local or remote procedure call; or any type of logic can be implemented as a combination of hardware and software.

[0045] The method can be implemented by software and / or firmware stored on or in a computer-readable medium, a machine-readable medium, a propagated signal medium and / or a signal-bearing medium. The medium can include any device that contains, stores, communicates, propagates or transports executable instructions for use by or in connection with an instruction-executable system, apparatus or device. The machine-readable medium can optionally be (but is not limited to) an electronic signal, a magnetic signal, an optical signal, an electromagnetic signal or an infrared signal or a semiconductor system, apparatus, device or propagation medium. A non-exhaustive list of examples of machine-readable media includes: magnetic or optical disks, volatile memory such as random access memory "RAM", read-only memory "ROM", erasable programmable read-only memory (i.e., EPROM) or flash memory and optical fibers. The machine-readable medium can also include a tangible medium on which executable instructions are printed, since logic can be stored electronically as an image or in another format (e.g., by optical scanning), followed by compilation and / or interpretation or otherwise processed. The processed medium can then be stored in a computer and / or machine memory.

[0046] The system may include additional or different logic and may be implemented in many different ways. The controller may be implemented as a microprocessor, a microcontroller, an application specific integrated circuit (ASIC), discrete logic, or a combination of other types of circuits or logic. Similarly, the memory may be DRAM, SRAM, flash memory, or other types of memory. Parameters (e.g., conditions and thresholds) and other data structures may be stored and managed separately, incorporated into a single memory or database, or organized logically and physically in many different ways. Programs and instruction sets may be part of a single program, separate programs, or distributed across several memories and processors.

[0047] The description of the embodiments has been presented for purposes of illustration and description. Appropriate modifications and changes to the embodiments may be made in light of the above description, or may be acquired through practice. For example, unless otherwise indicated, one or more of the described methods may be performed by suitable devices and / or combinations of devices. The described methods and associated actions may also be performed in various orders other than the order described in this application, in parallel, and / or simultaneously. The described system is exemplary in nature and may include additional elements and / or omit elements.

[0048] As used in this application, an element or step recited in the singular and preceded by the word "a" or "an" should be understood as not excluding a plurality of the elements or steps, unless such exclusion is specified. Additionally, the reference to "one embodiment" or "one example" of the present disclosure is not intended to be construed as excluding the existence of additional embodiments that also incorporate the recited features. The terms "first," "second," and "third," etc. are used only as labels and are not intended to impose numerical requirements or a particular positional order on their objects.

[0049] Although various embodiments of the invention have been described, those of ordinary skill in the art will appreciate that many embodiments and implementations are possible within the scope of the invention. Specifically, those skilled in the art will recognize the interchangeability of various features from different embodiments. Although these techniques and systems have been disclosed in the context of certain embodiments and examples, it will be understood that these techniques and systems may be extended beyond the specifically disclosed embodiments to other embodiments and / or uses and their obvious modifications.

Claims

1. A map fusion method, comprising: Receive a source graph and a target graph, where the source graph represents a source map and the target graph represents a target map, and includes nodes and edges connecting the nodes; Process each of the source graph and the target graph in a convolutional layer to provide a convolutional layer output related to the source graph and the target graph; Process each of the convolutional layer outputs of the source graph and the target graph in a rectified linear layer to output a node feature map related to the source graph and the target graph, the node feature map including data representing the features of each node; Select multiple pairs of node representations from the node feature maps related to the source graph and the target graph, and aggregate the selected multiple pairs of node representations to output the selected and aggregated multiple pairs of node representations; Process the selected and aggregated multiple pairs of node representations in a fully connected layer to provide a fully connected layer output; Perform softmax processing on the fully connected layer output to output the matching probabilities of the nodes in the node feature maps related to the source graph and the target graph; And Based on the matching probabilities of the nodes, determine whether to fuse the nodes in the source map with the corresponding nodes in the target graph.

2. The method according to claim 1, wherein in the node feature map, each node is represented by a node feature vector, and the node feature vector includes the data representing the features of each node.

3. The method according to claim 1 or 2, further comprising at least one additional convolutional layer and at least one additional rectified linear layer after the convolutional layer and the rectified linear layer for processing each of the source graph and the target graph to output a node feature map related to the source graph and the target graph.

4. The method according to claim 1, further comprising at least one additional fully connected layer after the fully connected layer for processing the selected and aggregated multiple pairs of node representations in the fully connected layer to provide a fully connected layer output.

5. The method according to claim 1, wherein the convolutional layer includes weights and the weights are self - learned.

6. The method according to claim 5, wherein the labeled data is used as training data to train the weights end - to - end.

7. The method according to claim 6, wherein, If the matching training data is insufficient, oversample the matching training data.

8. The method according to claim 1, further comprising a long short - term memory layer for processing each of the source graph and the target graph to output an edge feature map related to the source graph and the target graph, and the edge feature map includes data representing the features of each edge.

9. The method according to claim 8, wherein in the edge feature map, each edge is represented by an edge feature vector, and the edge feature vector includes data representing the features of each edge.

10. The method according to claim 8 or 9, wherein the edge features are learned from the basic geometry in the source graph and the target graph in the long short - term memory layer.

11. The method according to claim 10, wherein learning the edge features is based on one or more sequences of support points from one node to another node.

12. The method according to claim 1, further comprising processing the source graph and the target graph hierarchically, starting from the most stable node and moving down along the edges hierarchically.

13. A computer program product, the computer program product comprising instructions that, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1-12.

14. A computer, the computer comprising at least one processor and at least one memory, the computer being configured to execute the computer program according to claim 13.

Images