Enhanced semantic graph embedding for omnidirectional location recognition

Abstract

Enhanced Semantic Graph Embedding for Omnidirectional Location Recognition. A computer-implemented method for location recognition includes: obtaining information of an image identifying a first scene; identifying multiple pixel clusters in the image; generating a set of feature vectors associated with the pixel clusters; generating a graph of the scene; adding a first edge between a first node and a second node in response to determining that a first attribute associated with a first pixel cluster is similar to a second attribute associated with a second pixel cluster; generating a vector representation of the graph; calculating a measure of similarity between the vector representation of the graph and a reference vector representation associated with a second scene; and determining that the first scene and the second scene are associated with the same location in response to determining that the measure of similarity is less than a threshold.

Description

Technical Field

[0001] The embodiments described herein generally relate to methods and apparatus for place recognition, and methods and apparatus for training graph embedding networks for place recognition. Background Technology

[0002] Visual location recognition involves determining the position of a camera given its current view. Location recognition is an important problem in computer vision and robotics and has applications in a wide range of fields, including but not limited to autonomous driving and augmented reality.

[0003] Known methods for visual location identification formulate the problem as an image retrieval task. These methods typically involve extracting features from a captured image. The features of the captured image are then compared with features extracted from images in a reference dataset, where each image is associated with location-identifying information (e.g., GPS coordinates). In this method, the location associated with the captured image is obtained by determining, based on the extracted features, which image in the reference dataset is visually most similar to the captured image.

[0004] Visual feature-based methods for location identification often struggle to handle appearance variations between captured images and associated images in a reference dataset. These variations can be due to changes in lighting and appearance (e.g., due to annual seasonal variations), as well as viewpoint variations between the captured and reference images. Therefore, novel methods for visual location identification are needed. Attached Figure Description

[0005] The arrangement will be more fully understood and appreciated from the following detailed description, which is given by way of example only and in conjunction with the accompanying drawings, in which:

[0006] Figure 1A Robot 100 according to the example is shown;

[0007] Figure 1B An example use case for visual location recognition is shown;

[0008] Figure 2 A method for visual location recognition according to an embodiment is shown;

[0009] Figure 3 An implementation of instance segmentation using "Mask R-CNN" based on an example is shown;

[0010] Figure 4 The method for generating a set of detections is shown in the example.

[0011] Figure 5A An example of an equirectangular image is shown;

[0012] Figure 5B A visual representation of instance segmentation generated according to an embodiment is shown;

[0013] Figure 5C A visual representation of the depth information generated according to an embodiment is shown;

[0014] Figure 6A A 3D representation of a scene with superimposed nodes and edges according to an embodiment is shown;

[0015] Figure 6B A 2D visualization of the scene diagram based on the example is shown;

[0016] Figure 7 A graph embedding network according to an embodiment is shown;

[0017] Figure 8 A method performed by the propagation layer 703 of a graph embedding network 701 according to an embodiment is shown;

[0018] Figure 9 A method for training a graph embedding network according to an embodiment is shown;

[0019] Figure 10 The recall rates of “GraphVLADt” and “NetVLADt” for each validation type on the “OmniClevr” dataset are shown according to embodiments.

[0020] Figure 11 A system according to an embodiment is shown;

[0021] Figure 12 A schematic diagram of hardware that can be used to implement the method according to the embodiment is shown. Detailed Implementation

[0022] According to a first aspect, a computer-implemented method for location recognition is provided. The method includes: obtaining information about an image identifying a first scene; identifying a plurality of pixel clusters in the information identifying the image, the plurality of pixel clusters including: a first pixel cluster; and a second pixel cluster; generating a feature vector set from the information identifying the image, the feature vector set including: a first feature vector associated with the first pixel cluster; and a second feature vector associated with the second pixel cluster; generating a graph of the scene, the graph including: a first node representing the first pixel cluster, the first node being associated with the first feature vector; and a second node representing the second pixel cluster, the second node being associated with the second feature vector; adding a first edge between the first node and the second node in response to determining that a first attribute associated with the first pixel cluster is similar to a second attribute associated with the second pixel cluster; generating a vector representation of the graph; calculating a measure of similarity between the vector representation of the graph and a reference vector representation, wherein the reference vector representation is associated with a second scene; and determining that the first scene and the second scene are associated with the same location in response to determining that the measure of similarity is less than a threshold.

[0023] In this embodiment, a pixel cluster is a group of pixels.

[0024] In an embodiment, identifying multiple pixel clusters includes identifying multiple object instances, wherein a first pixel cluster corresponds to a first object instance, and a second pixel cluster corresponds to a second object instance.

[0025] In one embodiment, identifying multiple object instances in the information identifying the image includes using a machine learning model trained on an equal rectangular image to identify these multiple object instances.

[0026] In this embodiment, an object instance is the appearance of an object, and an object is a physical object.

[0027] In an embodiment, the feature vector is an n-dimensional vector representing the numerical features of a cluster of pixels (optionally representing an object).

[0028] In the embodiment, the first attribute is the 3D coordinate position of the first pixel cluster, the second attribute is the 3D coordinate position of the second pixel cluster, and the first attribute is similar to the second attribute when the distance between the 3D coordinate positions of the first pixel cluster and the 3D coordinate positions of the second pixel cluster is less than a distance threshold.

[0029] In this embodiment, determining the distance includes determining the Euclidean distance.

[0030] In this embodiment, the distance threshold is 3 meters.

[0031] In an embodiment, a first pixel cluster corresponds to a first object instance, a second pixel cluster corresponds to a second object instance, a first attribute is a first category label associated with the first object instance, a second attribute is a second category label associated with the second object instance, and wherein the first attribute is similar to the second attribute when the first category label is the same as the second category label (i.e., the first object instance and the second object instance involve objects of the same category).

[0032] In an embodiment, a first pixel cluster corresponds to a first object instance, a second pixel cluster corresponds to a second object instance, a first attribute is a first category label associated with the first object instance, a second attribute is a second category label associated with the second object instance, and wherein when the first category label and the second category label belong to a first group of categories, the first attribute is similar to the second attribute (e.g., when the first group of categories includes {"chair", "table"}, the first attribute "chair" is similar to the second attribute "table".

[0033] In an embodiment, the image comprises a plurality of pixels, and the information identifying the image includes the RGB value of each of the plurality of pixels.

[0034] In an embodiment, the method further includes: capturing information of an image identifying the first scene using a camera (optionally by taking a photograph using the camera).

[0035] In this embodiment, the image is a visual representation of a scene, the image of the scene is divided into multiple pixels, and the information identifying the image includes a value (e.g., an RGB value) associated with each pixel, wherein the value of each pixel is generated by taking a picture of the scene using a camera and obtaining the value by reading the sensor within the camera.

[0036] In an embodiment, the method further includes: in response to determining that the similarity metric is less than a threshold, outputting an indication that the location of the first scene is a location associated with the second scene.

[0037] In one embodiment, the method is performed by a robot, and the method further includes using an indication of the current location for indoor navigation.

[0038] In an embodiment, the diagram represents a plurality of nodes and optionally includes a plurality of edges connecting these nodes.

[0039] In this embodiment, the vector representation of a graph is a numerical vector that uses numbers to represent the features of the graph (i.e., nodes, edges, and spatial relationships between nodes).

[0040] In this embodiment, the graph is a graphical representation of the scene.

[0041] In an embodiment, the similarity between the vector representation of the graph and the reference vector representation is measured by the Euclidean distance between the two vectors.

[0042] In an embodiment, the method further includes obtaining a reference graph representation from a reference set, which includes a plurality of other reference graph representations.

[0043] In an embodiment, the method further includes calculating a measure of similarity between the vector representation of the graph and each reference graph representation in the reference set, and selecting the N reference graph representations with the highest similarity as matches. Optionally, N is 1 or 5.

[0044] In this embodiment, location refers to a position.

[0045] In this embodiment, the vector representation of the graph is the same as the graph embedding vector.

[0046] In an embodiment, the image includes a plurality of pixels, which include a first group of pixels associated with a first pixel cluster and a second group of pixels associated with a second pixel cluster, and generating a feature vector set includes: generating a third feature vector associated with a first pixel in the first group of pixels; generating a fourth feature vector associated with a second pixel in the first group of pixels; and generating a first feature vector by aggregating the third feature vector and the fourth feature vector.

[0047] In this embodiment, the first pixel and the second pixel are different pixels.

[0048] In an embodiment, aggregating the third feature vector and the fourth feature vector includes adding the third feature vector and the fourth feature vector.

[0049] In one embodiment, generating a vector representation of the graph includes: generating a first node vector, wherein the first node vector is associated with a first node; generating a second node vector, wherein the second node vector is associated with a second node; and generating a query graph representation based on the sum of the first node vector and the second node vector.

[0050] In an embodiment, the graph of the scene does not include edges connected to the first node; and wherein generating the first node vector includes setting the first node vector to be equal to the first feature vector.

[0051] In an embodiment, the second node in the graph is connected to the first node in the graph via a first edge, and generating the first node vector includes: setting the first node vector to be equal to a first feature vector; generating a message between the second node and the first node based on the first node vector and the second node vector; and updating the first node vector based on the value of the first node vector and the message.

[0052] In this embodiment, the message is a figurative message, that is, the message is a matrix of values.

[0053] In this embodiment, a machine learning model is used to generate the message.

[0054] In one embodiment, the first node vector is set to be equal to the first feature vector, the value of the first node vector is set for time t, and the first node vector is updated to set the value of the first node vector for time t+1.

[0055] In this embodiment, generating a message between the second node and the first node includes: generating a first set of gating weights using a first machine learning model based on the first node vector and the second node vector; gating the first node vector by multiplying it with the first set of gating weights to form a first gating vector; generating a second set of gating weights using a second machine learning model based on the first node vector and the second node vector; gating the second node vector by multiplying it with the second set of gating weights to form a second gating vector; and generating the message based on the sum of the first gating vector and the second gating vector. In this embodiment, the multiplication is element-wise multiplication.

[0056] In this embodiment, both the first and second machine learning models are trainable.

[0057] In an embodiment, the method further includes passing the first gating vector and the second gating vector through a sigmoid function and adding the results to generate the message.

[0058] In this embodiment, the first machine learning model and the second machine learning model each include a single linear layer.

[0059] In an embodiment, the scene graph further includes a third node representing a third pixel cluster, the third node being associated with a third feature vector and connected to a first node via a second edge; and the method further includes: generating a third node vector associated with a third node; and generating a second message between the third node and the first node based on the first node vector and the third node vector through the following processes: generating a third set of gating weights using a first machine learning model based on the first node vector and the third node vector; gating the first node vector by multiplying the first node vector with the third set of gating weights to form a third gating vector; generating a fourth set of gating weights using a second machine learning model based on the first node vector and the third node vector; gating the third node vector by multiplying the third node vector with the fourth set of gating weights to form a fourth gating vector; generating the second message based on the sum of the third gating vector and the fourth gating vector; and wherein: updating the first node vector based on the value of the first node vector and the message includes updating the first node vector based on the value of the first node vector and the sum of the message and the second message.

[0060] In an embodiment, updating the first node vector based on the value of the first node vector and the message includes: generating a fifth set of gating weights using a third machine learning model based on the first node vector and the message; gating the first node vector by multiplying the first node vector with the fifth set of gating weights to form a fifth gating vector; generating a sixth set of gating weights using a fourth machine learning model based on the first node vector and the message; gating the message by multiplying the message with the sixth set of gating weights to form a sixth gating vector; and updating the first node vector based on the sum of the fifth gating vector and the sixth gating vector.

[0061] In the embodiments, the third and fourth machine learning models each comprise a single linear layer.

[0062] In an embodiment, generating a query graph representation based on the sum of a first node vector and a second node vector includes: generating a seventh set of gating weights using a fifth machine learning model based on the first node vector; generating a first node representation of the first node vector using a sixth machine learning model based on the first node vector; multiplying the seventh set of gating weights by the first node representation of the first node vector to obtain a final first node representation; generating an eighth set of gating weights using the fifth machine learning model based on the second node vector; generating a second node representation of the second node vector using the sixth machine learning model based on the second node vector; multiplying the eighth set of gating weights by the second node representation of the second node vector to obtain a final second node representation; and generating a query graph representation by adding the final first node representation and the final second node representation. In an embodiment, the fifth machine learning model and the sixth machine learning model include a single linear layer.

[0063] In one embodiment, the fifth and sixth machine learning models implement row kernels such that the final first node representation and the final second node representation each comprise a matrix with a single column. In another embodiment, generating the query graph representation includes: determining the sum of the final first node representation and the final second node representation; and reducing the dimension of the sum to form the query graph representation.

[0064] In an embodiment, reducing the dimensionality of the sum includes using principal component analysis.

[0065] In this embodiment, a seventh machine learning model is used to perform principal component analysis.

[0066] In one embodiment, generating the first node vector further includes: incrementing a time value in response to updating the first node vector; determining whether the time value is less than a time threshold; and in response to determining that the time value is less than the time threshold: regenerating a message between the second node and the first node based on the first node vector and the second node vector; and updating the first node vector based on the value of the first node vector and the message.

[0067] In an embodiment, the method further includes, in response to determining that a time value equals a time threshold, generating a query graph representation based on the sum of a first node vector and a second node vector.

[0068] In this embodiment, the image is a rectangular image.

[0069] In this embodiment, the rectangular image is an image that spans 360 degrees horizontally and 180 degrees vertically.

[0070] In this embodiment, the rectangular image is a rectangular projection of the input image.

[0071] In this embodiment, the input image is an omnidirectional image.

[0072] In this embodiment, the omnidirectional image is obtained from an omnidirectional camera.

[0073] In an embodiment, identifying the plurality of pixel clusters in the information identifying the image includes: generating a first set of detections by performing instance segmentation on the image, the first set of detections including a first pixel cluster; rotating the image to form a rotated image; generating a second set of detections by performing instance segmentation on the rotated image, the second set of detections including a second pixel cluster; and combining the first set of detections and the second set of detections to form the plurality of pixel clusters.

[0074] In an embodiment, identifying the plurality of pixel clusters includes identifying a plurality of object instances, a first pixel cluster corresponding to a first object instance, and wherein a first set of detections includes a first objectness score associated with the first object instance (and equivalently with the first pixel cluster), and combining the first set of detections includes determining whether the first objectness score is greater than an objectness threshold, and in response to determining that the first objectness score is greater than the objectness threshold: adding the first pixel cluster to the plurality of pixel clusters.

[0075] In an embodiment, identifying the plurality of pixel clusters includes identifying a plurality of object instances, a first pixel cluster being a first object instance, and a first set of detections including a first bounding box and a first objectability score associated with the first object instance, and a second set of detections including a third object bounding box and a third objectability score associated with the first object instance, and combining the first set of detections and the second set of detections includes: forming a combined list including the first set of detections and the second set of detections; comparing the third objectability score with the first objectability score; adding the first object bounding box to a final list in response to determining that the first objectability score is higher than the third objectability score; generating an Intersection over Union (IoU) value based on the first object bounding box and the third object bounding box; and removing the third object instance from the combined list in response to determining that the IoU value is greater than an IoU threshold.

[0076] In one embodiment, the image is rotated 180 degrees around the vertical axis (i.e., in the horizontal plane).

[0077] In this embodiment, a Vector of Locally Aggregated Descriptors is used to represent the first feature vector.

[0078] In one embodiment, generating a feature vector set includes using a machine learning model to generate a first feature vector based on information from an image that identifies the first scene.

[0079] In one embodiment, the machine learning model includes a feature extraction layer and a VLAD layer, wherein the feature extraction layer is configured to extract features from an image, and the VLAD layer is configured to represent each feature as a local feature aggregation descriptor.

[0080] In this embodiment, the feature extraction layer includes a convolutional neural network (CNN).

[0081] In an embodiment, the first entry in the local feature aggregation descriptor indicates the vector difference (i.e., residual) between the cluster center and the first feature extracted by the feature extraction layer, where the cluster center represents a point in the feature space.

[0082] In an embodiment, the feature space includes every possible value of the feature.

[0083] In this embodiment, the cluster centers are trainable.

[0084] In an embodiment, determining that a first attribute associated with a first pixel cluster is similar to a second attribute associated with a second pixel cluster includes: determining a first set of 3D coordinates associated with the first pixel cluster; determining a second set of 3D coordinates associated with the second pixel cluster; calculating the distance between the first set of 3D coordinates and the second set of 3D coordinates; and determining that the first attribute is similar to the second attribute when the distance is less than a distance threshold.

[0085] In this embodiment, the distance is a Euclidean distance.

[0086] In an embodiment, determining the first set of 3D coordinates associated with the first pixel cluster includes: estimating the depth of the first pixel cluster; and generating the first set of 3D coordinates based on the depth and information identifying the image.

[0087] In an embodiment, the first pixel cluster is associated with a first object instance, and estimating the depth includes at least one of the following: 1) generating the depth using a machine learning model; 2) obtaining a depth measurement of the first object instance from a sensor and specifying that the depth is equal to the depth measurement; or 3) obtaining the depth from information identifying the image, wherein the information identifying the image includes a depth measurement associated with each pixel.

[0088] According to a second aspect, a non-transitory computer-readable medium is provided, comprising computer program instructions that, when executed by a processor, cause the processor to perform any of the methods described in the preceding items.

[0089] According to a third aspect, an apparatus configured to perform / implement the above-described method is provided.

[0090] According to a fourth aspect, a method is provided for training a graph matching network for location recognition, wherein the graph matching network includes a machine learning model. The computer-implemented method includes: obtaining a training triplet, the training triplet including a query graph, a positive graph, and a negative graph; using the machine learning model to generate a query vector representation of the query graph; using the machine learning model to generate a positive vector representation of the positive graph; using the machine learning model to generate a negative vector representation of the negative graph; determining a first similarity between the query vector representation and the positive vector representation; determining a second similarity between the query vector representation and the negative vector representation; generating a triplet loss based on the first and second similarities; and training the machine learning model based on the triplet loss.

[0091] In one embodiment, the query graph is associated with a first scene, the positive graph is associated with a second scene, and the negative graph is associated with a third scene, wherein the first scene and the second scene are associated with the same location, and the first scene and the third scene are associated with different locations.

[0092] In one embodiment, training a machine learning model based on triplet loss includes using backpropagation and gradient descent to determine the weights of the machine learning model.

[0093] In this embodiment, the triplet loss is calculated according to the following formula:

[0094] According to the fifth aspect, an apparatus is provided, which is configured to perform / implement the method described above for training a graph matching network for location identification.

[0095] According to a sixth aspect, a non-transitory computer-readable medium is provided comprising computer program instructions that, when executed by a processor, cause the processor to perform a method for training a graph matching network for location recognition.

[0096] According to a seventh aspect, an apparatus for location identification is provided, the apparatus being configured to: obtain information of an image identifying a first scene; identify a plurality of pixel clusters in the information identifying the image, the plurality of pixel clusters including: a first pixel cluster; and a second pixel cluster; generate a feature vector set from the information identifying the image, the feature vector set including: a first feature vector associated with the first pixel cluster; and a second feature vector associated with the second pixel cluster; generate a graph of the scene, the graph including: a first node representing the first pixel cluster, the first node being associated with the first feature vector; and a second node representing the second pixel cluster, the second node being associated with the second feature vector; add a first edge between the first node and the second node in response to determining that a first attribute associated with the first pixel cluster is similar to a second attribute associated with the second pixel cluster; generate a vector representation of the graph; calculate a measure of similarity between the vector representation of the graph and a reference vector representation, wherein the reference vector representation is associated with a second scene; and determine that the first scene and the second scene are associated with the same location in response to determining that the similarity measure is less than a threshold.

[0097] In an embodiment, the image includes a plurality of pixels, comprising a first group of pixels associated with a first pixel cluster and a second group of pixels associated with a second pixel cluster. The apparatus is further configured to, when generating a feature vector set,: generate a third feature vector associated with a first pixel in the first group of pixels; generate a fourth feature vector associated with a second pixel in the first group of pixels; and generate a first feature vector by aggregating the third and fourth feature vectors. In an embodiment, the apparatus is further configured to, when generating a vector representation of the graph,: generate a first node vector, wherein the first node vector is associated with a first node; generate a second node vector, wherein the second node vector is associated with a second node; and generate a query graph representation based on the sum of the first and second node vectors.

[0098] In an embodiment, the second node in the graph is connected to the first node in the graph via a first edge, and the device is further configured to, when generating the first node vector: set the first node vector to be equal to a first feature vector; generate a message between the second node and the first node based on the first node vector and the second node vector; and update the first node vector based on the value of the first node vector and the message.

[0099] In an embodiment, the device is further configured to, when generating a message between a second node and a first node: generate a first set of gating weights using a first machine learning model based on a first node vector and a second node vector; gating the first node vector by multiplying the first node vector with the first set of gating weights to form a first gating vector; generating a second set of gating weights using a second machine learning model based on the first node vector and the second node vector; gating the second node vector by multiplying the second node vector with the second set of gating weights to form a second gating vector; and generating the message based on the sum of the first gating vector and the second gating vector.

[0100] In an embodiment, the scene graph further includes a third node representing a third pixel cluster, the third node being associated with a third feature vector and connected to a first node via a second edge; and the apparatus is further configured to: generate a third node vector associated with a third node; and generate a second message between the third node and the first node based on the first node vector and the third node vector through the following processes: generating a third set of gating weights using a first machine learning model based on the first node vector and the third node vector; gating the first node vector by multiplying the first node vector with the third set of gating weights to form a third gating vector; generating a fourth set of gating weights using a second machine learning model based on the first node vector and the third node vector; gating the third node vector by multiplying the third node vector with the fourth set of gating weights to form a fourth gating vector; generating a second message based on the sum of the third gating vector and the fourth gating vector; and wherein: updating the first node vector based on the value of the first node vector and the message includes updating the first node vector based on the value of the first node vector and the sum of the message and the second message.

[0101] In an embodiment, the device is further configured to, when updating the first node vector based on the value of the first node vector and the message: generate a fifth set of gating weights using a third machine learning model based on the first node vector and the message; gating the first node vector by multiplying the first node vector by the fifth set of gating weights to form a fifth gating vector; generate a sixth set of gating weights using a fourth machine learning model based on the first node vector and the message; gating the message by multiplying the message by the sixth set of gating weights to form a sixth gating vector; and update the first node vector based on the sum of the fifth gating vector and the sixth gating vector.

[0102] In an embodiment, the apparatus is further configured to, when generating a query graph representation based on the sum of a first node vector and a second node vector: generate a seventh set of gating weights using a fifth machine learning model based on the first node vector; generate a first node representation of the first node vector using a sixth machine learning model based on the first node vector; multiply the seventh set of gating weights by the first node representation of the first node vector to obtain a final first node representation; generate an eighth set of gating weights using the fifth machine learning model based on the second node vector; generate a second node representation of the second node vector using the sixth machine learning model based on the second node vector; multiply the eighth set of gating weights by the second node representation of the second node vector to obtain a final second node representation; and generate a query graph representation by adding the final first node representation and the final second node representation. In an embodiment, the fifth and sixth machine learning models implement row kernels such that the final first node representation and the final second node representation each comprise a matrix with a single column.

[0103] In an embodiment, the apparatus is further configured to, when generating a query graph representation,: determine the sum of the final first node representation and the final second node representation; and reduce the dimension of the sum to form the query graph representation.

[0104] In an embodiment, the apparatus is further configured to, when generating a first node vector: increment a time value in response to updating the first node vector; determine whether the time value is less than a time threshold; and in response to determining that the time value is less than the time threshold: regenerate a message between the second node and the first node based on the first node vector and the second node vector; and update the first node vector based on the value of the first node vector and the message.

[0105] In this embodiment, the image is a rectangular image.

[0106] In an embodiment, the apparatus is further configured to, when identifying the plurality of pixel clusters in information identifying the image: generate a first set of detections by performing instance segmentation on the image, the first set of detections including a first pixel cluster; rotate the image to form a rotated image; generate a second set of detections by performing instance segmentation on the rotated image, the second set of detections including a second pixel cluster; and combine the first set of detections and the second set of detections to form the plurality of pixel clusters.

[0107] In this embodiment, a local feature aggregation descriptor is used to represent the first feature vector.

[0108] In an embodiment, the device is further configured to, when it is determined that a first attribute associated with a first pixel cluster is similar to a second attribute associated with a second pixel cluster: determine a first set of 3D coordinates associated with the first pixel cluster; determine a second set of 3D coordinates associated with the second pixel cluster; calculate the distance between the first set of 3D coordinates and the second set of 3D coordinates; and when the distance is less than a distance threshold, determine that the first attribute is similar to the second attribute.

[0109] In an embodiment, the device is further configured to, when determining a first set of 3D coordinates associated with a first pixel cluster: estimate the depth of the first pixel cluster; and generate the first set of 3D coordinates based on the depth and information identifying the image.

[0110] According to an eighth aspect, an apparatus is provided for training a graph matching network for location recognition, wherein the graph matching network includes a machine learning model, the apparatus being configured to: obtain a training triplets, the training triplets including: a query graph, a positive graph, and a negative graph; generate a query vector representation of the query graph using the machine learning model; generate a positive vector representation of the positive graph using the machine learning model; generate a negative vector representation of the negative graph using the machine learning model; determine a first similarity between the query vector representation and the positive vector representation; determine a second similarity between the query vector representation and the negative vector representation; generate a triplet loss based on the first similarity and the second similarity; and train the machine learning model based on the triplet loss.

[0111] In an embodiment, the device is also configured to use backpropagation and gradient descent to determine the weights of the machine learning model when training a machine learning model based on triplet loss.

[0112] In this embodiment, the triplet loss is calculated according to the following formula:

[0113] According to a ninth aspect, a computer-implemented method for location identification is provided. The computer-implemented method includes: obtaining information about an image identifying a first scene; identifying a plurality of object instances in the information identifying the image, the plurality of object instances including: a first object instance; and a second object instance; generating a feature vector set from the information identifying the image, the feature vector set including: a first feature vector associated with the first object instance; and a second feature vector associated with the second object instance; generating a graph of the scene, the graph including: a first node representing the first object instance, the first node being associated with the first feature vector; and a second node representing the second object instance, the second node being associated with the second feature vector; determining a distance between the first object instance and the second object instance, and adding a first edge to the graph between the first node and the second node in response to determining that the distance is less than a distance threshold; generating a vector representation of the graph; and calculating a similarity metric between the vector representation of the graph and a reference vector representation, wherein the reference vector representation is associated with the second scene; and determining that the first scene and the second scene are associated with the same location in response to determining that the similarity metric is less than a second threshold.

[0114] According to a tenth aspect, an apparatus for location identification is provided, the apparatus being configured to: obtain information of an image identifying a first scene; identify a plurality of object instances in the information identifying the image, the plurality of object instances including: a first object instance; and a second object instance; generate a feature vector set from the information identifying the image, the feature vector set including: a first feature vector associated with the first object instance; and a second feature vector associated with the second object instance; generate a graph of the scene, the graph including: a first node representing the first object instance, the first node being associated with the first feature vector; and a second node representing the second object instance, the second node being associated with the second feature vector; determine a distance between the first object instance and the second object instance, and in response to determining that the distance is less than a distance threshold, add a first edge to the graph between the first node and the second node; generate a vector representation of the graph; and calculate a similarity metric between the vector representation of the graph and a reference vector representation, wherein the reference vector representation is associated with the second scene; and in response to determining that the similarity metric is less than a second threshold, determine that the first scene and the second scene are associated with the same location.

[0115] This paper presents example use cases of the methods described herein. While the example use cases below involve autonomous navigation of robotic systems inside buildings, it is important to emphasize that the methods described herein can be used for various applications. For example, the visual location recognition method described herein can be used for vehicle localization / UAV position estimation and / or control.

[0116] Figure 1AA robot 100 according to an example is shown. The robot 100 includes a drive component 101, such as a full drive. The drive component 101 controls the movement of the robot 100, for example, when the robot 100 moves around in an indoor environment.

[0117] The robot 100 also includes a wireless input / output port 102 communicatively coupled to the antenna 103 for transmitting and receiving signals. The input / output port 102 is configured to transmit and receive radio signals, preferably short-range radio communication signals.

[0118] The robot 100 also includes a microprocessor 104 coupled to a non-volatile memory 105. The microprocessor 104 is configured to execute instructions stored in the non-volatile memory 105. Execution of these instructions causes the robot 100 to perform some of the method steps described herein, including but not limited to the visual location recognition methods discussed herein.

[0119] The example robot 100 also includes a camera 106 configured to acquire images of the robot's environment. Specifically, the camera is configured to capture images (i.e., visual representations) of the robot's local environment. In the example, the image comprises multiple pixels, each associated with a data value that can be used to generate a visual representation of the robot's local environment (e.g., red-green-blue values, YUV chromaticity and luminance values, etc.).

[0120] Figure 1B An example use case for visual location recognition is shown. Figure 1B The example method shown is in a robot (e.g.) Figure 1A This discussion is within the context of robot navigation within a building. In this example, successful navigation requires the robot to locate its position within the building so that it can determine a path to its destination. The method begins in step 150 with robot 100 receiving a request for location information. For example, this request could be generated by robot 100 itself in response to receiving a destination target to navigate to.

[0121] In step 151, the robot captures a visual representation of its environment. In this example, camera 106 of robot 100 is used to obtain the visual representation.

[0122] In step 152, robot 100 uses the visual representation (i.e., image data) obtained in step 151 to perform visual location recognition. Visual location recognition is performed according to the method described herein. The output of visual location recognition may include information identifying the location of the image captured in step 151. For example, robot 100 may maintain a reference dataset including images of an indoor environment, where each image is associated with information identifying a location / location (e.g., “Room F, 3rd Floor, Building X”). In this example, the visual location recognition method described herein is configured to identify the image in the reference dataset that is most similar to the image captured by camera 106 of robot 100 in step 151. Based on the matching image from the reference set, robot 100 determines its current location.

[0123] In step 153, robot 100 provides a response to the requester, which includes the location information obtained in step 152. In this example, the requester 100 is also the robot (i.e., a different process being performed by robot 100). In another example, robot 100 uses the location information received from step 153 to map a route to the target destination. Robot 100 then navigates (e.g., by moving using drive unit 101) to the target destination.

[0124] Figure 2 A method for visual location recognition according to an embodiment is illustrated. The method begins in step 201 by obtaining a rectangular image.

[0125] In the example, obtaining a rectangular image includes obtaining image data and generating a rectangular image (i.e., a rectangular projection of the image data). Figure 5A An example of a rectangular image is shown.

[0126] Acquiring image data can take many forms. In one example, acquiring image data includes receiving information representing an omnidirectional image. Image data includes the RGB values ​​of each pixel in the image. As is known in the art, an omnidirectional image is an image that covers approximately the entire sphere or at least the entire circle in a horizontal plane surrounding the image capturing device. Optionally, the information representing the omnidirectional image is generated by an omnidirectional capturing device such as an omnidirectional camera. Examples of omnidirectional cameras include, but are not limited to, those mentioned above. Max, Insta360 and Theta.

[0127] In this example, after obtaining information representing the omnidirectional image, the method further includes converting the omnidirectional image into an isorectangular image using an isorectangular projection. As is known in the art, an isorectangular image is a rectangular image (optionally having a 2:1 aspect ratio) that covers 360 degrees horizontally and 180 degrees vertically (i.e., includes image data of 360 degrees horizontally and 180 degrees vertically).

[0128] In another example, obtaining image data and generating an isorectangular image involves obtaining multiple images (e.g., from a perspective camera such as a pinhole camera) and generating an isorectangular image by stitching these images together.

[0129] In another example, the image data is obtained from a fisheye camera. As is known in the art, a fisheye camera includes an ultra-wide-angle lens.

[0130] While it is advantageous to generate a complete rectangular image (i.e., an image with image data (e.g., pixel values) associated with the full angular range of the image (e.g., 360 degrees horizontally and 180 degrees vertically), it should be emphasized that an incomplete rectangular image can be generated in step 201 (e.g., where the input image data is missing a portion of the rectangular image).

[0131] In another example, step 201 includes obtaining a rectangular image directly (e.g., from a storage device or as part of a received request).

[0132] The method proceeds to step 202. Step 202 includes performing instance segmentation on the equirectangular image generated in step 201.

[0133] We will now discuss this using a specific example of detecting object instances and grouping pixels within an image. Figure 2 The method is described. However, to avoid any ambiguity, it should be emphasized that the invention is not limited thereto. Specifically, in a more general example, pixels within an input image are grouped together to form pixel clusters (or, in other words, pixel clusters). In one example, a pixel cluster represents an object instance. In an alternative example, pixels within an image are grouped together using a different criterion than associating with an object, since that term is generally understood by humans (i.e., materials that can be seen and touched).

[0134] In the specific examples discussed below, performing instance segmentation involves identifying object instances within an image and identifying pixels of the isorectangular image associated with each identified object instance. As will be discussed in more detail below, the categories (i.e., types) of objects identified by the image segmentation component can include: “bookshelf,” “chair,” “door,” “table,” etc. Figure 5B A visual representation of instance segmentation generated according to an embodiment is shown.

[0135] In one example, an instance segmentation component is used to perform instance segmentation. The instance segmentation component is configured to receive information about the identified image (e.g., RGB values) and output information identifying each instance of an object detected in the input image, along with an object mask associated with the detected object instance (i.e., an indication of which pixels in the input image correspond to the identified object instance).

[0136] Optionally, the instance segmentation component is configured to use a machine learning model to generate information identifying each object instance and its associated object mask. In one example, the input image is an isorectangular image. In this case, the machine learning model is trained based on the isorectangular image to detect object instances and their associated object masks.

[0137] In another example, instance segmentation is performed according to "Mask R-CNN" as described in "K. He, G. Gkioxari, P. Dollár and R. Girshick, 'Mask R-CNN', IEEE International Conference on Computer Vision (ICCV) 2017, pp. 2980-2988, doi:10.1109 / ICCV.2017.322" which is incorporated herein by reference.

[0138] Although Mask R-CNN is discussed in detail in the following examples, it is important to emphasize, to avoid any ambiguity, that other instance-level segmentation methods can be used instead. For example, one could use "Associative Embedding," as described in "Alejandro Newell, Zhiao Huang, and Jia Deng (2017), Associative Embedding: End-to-End Learning for Joint Detection and Grouping, https: / / arxiv.org / abs / 1611.05424" (incorporated hereby), to cluster pixels into different instances. Another example method that can be used for instance-level segmentation includes "Discriminate Loss," as described in "Bert DeBrabandere, Davy Neven, and Luc Van Gool (2017), Semantic Instance Segmentation with a Discriminative Loss Function, https: / / arxiv.org / abs / 1708.02551" (incorporated hereby),

[0139] As is known in the art, images can be segmented using instance-level segmentation or semantic segmentation. In short, semantic segmentation associates each pixel with a category label (e.g., "table"). In contrast, instance segmentation treats multiple objects of the same category as distinct, separate objects (e.g., first "table", second "table", etc.). The method in this paper uses instance segmentation to correctly capture the geometric information in the scene (i.e., the relationships between objects, even if they may belong to the same category).

[0140] In short, "Mask R-CNN" is a technique that uses a region-based convolutional neural network (CNN) configured to take an image representation as input and to identify objects and generate for each candidate object: a class label, information identifying the object's bounding box, and an object mask.

[0141] As is known in the art, an object mask identifies the spatial layout of an object and includes information that identifies which pixels in an image (or which pixels in a portion thereof, such as which pixels within an object bounding box) are associated with the identified object.

[0142] Example category labels for objects identified by “Mask R-CNN” trained according to the examples disclosed herein include, but are not limited to: “beam”, “board”, “bookshelf”, “ceiling”, “chair”, “door”, “floor”, “table”, “wall”, “window” and “unknown”.

[0143] Figure 3 The implementation of instance segmentation using "Mask R-CNN" based on the example is shown. In step 301, candidate object bounding boxes (also called "regions of interest") and associated objectivity scores are generated.

[0144] “Mask R-CNN” comprises two stages. The first stage (step 301) is the Region Proposal Network (RPN), which is configured to propose candidate object bounding boxes (also known as regions of interest). The RPN is implemented using a fully convolutional network, taking the image as input and generating candidate object bounding boxes and associated object confidence scores (i.e., a measure of confidence that a candidate object bounding box belongs to a set of object categories rather than the background) as outputs. The implementation of the RPN is described by reference to “S. Ren, K. He, R. Girshick, and J. Sun, Faster R-CNN: Towards real-time object detection with region proposal networks, NIPS, 2015”, which is incorporated herein by reference.

[0145] In step 302, a category label is generated for each region of interest (RoI). In step 303, an object mask is generated for each region of interest (RoI).

[0146] Using the two-stage analogy above; the second stage includes assigning a category label and bounding box offset to each identified region of interest (RoI). The second stage also includes generating an object mask for each RoI in parallel.

[0147] Mask R-CNN uses the RoIAlign layer to extract features from each candidate object bounding box in a way that minimizes the misalignment between the region of interest (RoI) and the feature map. The implementation of the RoIAlign layer is provided in “Mask R-CNN, IEEE International Conference on Computer Vision (ICCV) 2017, pp. 2980-2988, doi:10.1109 / ICCV.2017.322”, which is incorporated herein by reference.

[0148] The second stage of “Mask R-CNN” involves using “RoIAlign” to extract features from the image data associated with each candidate object bounding box and performing classification and object bounding box regression on the extracted features. The implementation of classification and bounding box regression is provided in “Mask R-CNN, IEEE International Conference on Computer Vision (ICCV) 2017, pp. 2980-2988, doi:10.1109 / ICCV.2017.322” (which cites “R. Girshick, “Fast R-CNN”, IEEE International Conference on Computer Vision (ICCV) 2015, pp. 1440-1448, doi:10.1109 / ICCV.2015.169”).

[0149] In parallel with classification and object bounding box regression, the second stage of "Mask R-CNN" also predicts an object mask for each identified region of interest (RoI) based on the output of the "RoIAlign" layer (i.e., step 303). In this embodiment, "Mask R-CNN" uses a fully convolutional network to generate an object mask for each RoI. As is known in the art, a fully convolutional network comprises (artificial) neural networks that perform only convolutions and deconvolutions (in addition to pooling). In other words, a fully convolutional network does not contain any fully connected layers as in a convolutional neural network. The object mask generated for each RoI identifies the pixels of the image associated with an instance of the object.

[0150] In the example, a fully convolutional network is implemented as described in "Mask R-CNN" (which cites "J. Long, E. Shelhamer and T. Darrell, Fully convolutional networks for semanticsegmentation, CVPR, 2015"), which is also incorporated in this paper.

[0151] Various pre-trained models exist for “Mask R-CNN”. However, these models have been trained on datasets that include planar images (e.g., on the “COCO” dataset as described in “Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C. Lawrence Zitnick, Microsoft COCO: Common objects in context, ECCV, pp. 740-755, 2014”). It has been found that machine learning models trained on planar images produce unstable segmentation results for equirectangular images.

[0152] In the example, “Mask R-CNN” is trained for image segmentation using a training set that includes images of equal rectangles. The example training set used for this purpose includes at least a portion of the “Stanford-2D3DS” dataset as described in “Iro Armeni, Sasha Sax, Amir R. Zamir and Silvio Savarese, (2017), Joint 2D-3D-Semantic Data for Indoor Scene Understanding”, which is incorporated herein by reference.

[0153] The Stanford-2D3DS dataset comprises rectangular images and associated semantic segmentation images. The Stanford-2D3DS dataset uses 13 object categories: “ceiling,” “floor,” “wall,” “beam,” “column,” “window,” “door,” “table,” “chair,” “sofa,” “bookshelf,” “board,” and “miscellaneous” (also known as “unknown”). Therefore, a machine learning model trained using Stanford-2D3DS can detect objects in all 13 categories. However, to avoid any confusion, it is important to emphasize that machine learning models can be trained using different datasets with different categories. An example of training “Mask R-CNN” is described by reference in “Mask R-CNN, IEEE International Conference on Computer Vision (ICCV) 2017, 2017, pp. 2980-2988, doi:10.1109 / ICCV.2017.322,” which is incorporated herein by reference.

[0154] In step 304, a set of detections is generated. In the example, this set of detections includes a category label, an associated object mask, an associated objectness score, and an associated object bounding box for each identified object instance.

[0155] In summary, "Mask R-CNN" is configured to receive information from an identifiable image (e.g., RGB pixel values) as input and to output candidate object bounding boxes that identify regions of interest (RoIs), category labels that identify the category to which the detected object instances belong, object masks that identify pixels in the input image associated with the identified object instances, and object confidence scores associated with the candidate bounding boxes.

[0156] As discussed above, the omnidirectional image spans 360 degrees around the horizontal plane. However, the equirectangular projection of the omnidirectional image introduces discontinuities in the scene (e.g., along the Y-axis, or in other words, at the vertical edges of the equirectangular image). This discontinuity can reduce the accuracy of instance segmentation and cause some object instances to go undetected.

[0157] In light of this, a method is described below that reduces the impact of such discontinuities when segmenting equirectangular images. Although the following examples are presented using the output of "Mask R-CNN", it will be understood that the following method can be used with the output of any image segmentation technique, as long as the output includes candidate object bounding boxes and associated object confidence scores.

[0158] Figure 4The example illustrates a method for generating a set of detections. In the example, performing instance segmentation involves generating two sets of detections. One set of detections includes object bounding boxes identifying regions of interest (RoIs) and object confidence scores associated with those bounding boxes. The other set of detections also includes category labels identifying the class to which the detected object instances belong and object masks identifying pixels in the input image associated with the identified object instances.

[0159] Step 401 includes generating a first set of detections by performing instance segmentation on the “raw” isorectangular projection of the input image.

[0160] In step 402, the input image is rotated around the vertical axis. After the rotation, in step 403, a second set of detections is generated based on the rotated version of the "original" isorectangular projection. In the example, the "original" isorectangular projection is rotated 180 degrees around the vertical axis (i.e., in the horizontal plane) to generate the rotated version of the original isorectangular projection. Rotations at other angles are also possible, provided that, in general, discontinuities of objects in the "original" projection will not appear in the rotated version.

[0161] The first set of projections and the second set of projections are compared to generate a final set of detections. In step 404, the first set of detections and the second set of detections are filtered using non-maximum suppression (NMS) to generate the final set of detections.

[0162] As is known in the art, nonmaximum suppression (NMS) is a technique for filtering predictions in object detectors. Filtering detections using NMS involves generating a combined list of object bounding boxes from a first set of detections and a second set of detections. Next, an object bounding box associated with the highest object confidence score is selected from this combined list and added to a final list of object bounding boxes. The method also includes calculating an intersection-over-union (IoU) value for the selected object bounding boxes in the combined list and for each of the remaining object bounding boxes. In one example, the IoU value is calculated by determining the number of pixels common to each object bounding box being compared and dividing it by the number of pixels covered by the union of the object bounding boxes being compared. If the IoU is greater than a first threshold, the object bounding box (from the combined list) being used in the comparison is removed from the combined list. This method is repeated by selecting a new object bounding box associated with the highest object confidence score from the combined list, adding that candidate to the final list, and removing the remaining object bounding boxes from the combined list whose IoU value is greater than the first threshold. The first threshold is also called the intersection-over-union (IoU) threshold. In this embodiment, the IoU threshold is set to 0.5.

[0163] Optionally, the first and second sets of detections are filtered before forming the combined list of object bounding boxes, such that only object bounding boxes associated with object confidence scores greater than a second threshold are added to the combined list for subsequent filtering via non-maximum suppression (NMS). In the example, the second threshold (also known as the object confidence threshold) is an object confidence score greater than or equal to 0.7.

[0164] The bounding boxes of objects in the final list are each associated with an object mask and a category label. These are the same object masks and category labels associated with the bounding boxes of objects in the first / second set of detections.

[0165] Step 405 includes outputting the final set of detections.

[0166] Then, based on the final detection list, the instance segmentation output is generated (i.e., the output of step 202). As a reminder, the instance segmentation output includes output information identifying each object instance detected in the input image and an object mask associated with the detected object instance (i.e., an indication of which pixels in the input image correspond to the identified object instance).

[0167] In another example, the instance segmentation output in step 202 is generated based solely on the first set of detections (i.e., not based on, for example, the detections of the first set of detections). Figure 4 The rotated version of the described "original" image generates a second set of detections.

[0168] Return to Figure 2 In step 203, the method obtains depth information. As will become clearer from the following description, the extraction of camera-invariant relationships between objects is achieved in part by upscaling the input image into 3D geometry. If the depth of the pixels is known, a 3D scene can be generated based on a 2D isorectangular input image. Therefore, step 203 includes obtaining the depth information of the pixels in the 2D isorectangular input image. Figure 5C A visual representation of depth information generated according to an embodiment is shown.

[0169] Depth information can be obtained in several ways. In one example, LiDAR (Light Detection and Ranging) is used to obtain the depth information of the scene. In another example, depth information is obtained directly from the image data obtained in step 201. Specifically, if the image data includes depth information (e.g., the image data is provided in RGBD (Red, Green, Blue, Depth) format), the depth information in step 203 is obtained directly from the image data obtained in step 201. In this example, the information obtained in step 201 was captured by an RGBD camera.

[0170] In another example, depth information is obtained by generating a depth map (i.e., a representation of the input image where each pixel is associated with a depth value) from a rectangular projection of the input image (i.e., without explicit depth data being provided).

[0171] Optionally, a machine learning model can be used to generate the depth map. An example machine learning model is “RectNet”, as described in “Nikolaos Zioulis, Antonis Karakottas, Dimitrios Zarpalas and PetrosDaras, Omnidepth: Dense depth estimation for in-doors spherical panoramas, ECCV, pp. 448-465, 2018”, which is incorporated herein by reference.

[0172] In short, "RectNet" is a convolutional neural network (CNN) architecture that follows an autoencoder structure, where the encoder encodes the input by progressively reducing its spatial dimensions to a smaller representation, and the decoder regresses to the desired output by scaling up the representation. The individual layers of "RectNet" are designed to generate depth maps from 360-degree images.

[0173] In the example, the input to "RectNet" is the equirectangular image generated in step 201, and the output of "RectNet" is a depth map that associates each pixel in the equirectangular input image with a depth value.

[0174] In another example, the machine learning model, specifically "RectNet," is trained on at least a portion of the "Stanford-2D3DS" dataset, which includes input images and associated depth maps. Optionally, the isorectangular images are scaled to 512×256 pixels and then provided as input to "RectNet."

[0175] Although the discussion above about "RectNet" focused on obtaining depth information from image data, it will be understood that any technique used to generate depth maps from equirectangular projections can be used instead.

[0176] Return to Figure 2 In step 204, the method uses the depth map generated in step 203 to generate the 3D position of each pixel in the isorectangular image.

[0177] In one example, the pixel positions of the rectangular image are first converted to spherical coordinates. As is known in the art, a spherical coordinate system is a coordinate system in three-dimensional space, where the position of a point is determined by its radial distance (ρ), azimuth (θ), and polar angle. To specify. The first transformation converts the pixel positions of the rectangular image to azimuth (θ) and polar angle. In the example: the rectangular image comprises W × H pixels, each pixel has image coordinates (x, y), and the origin is the center of the image. Therefore: θ = 2πx / W and

[0178] The depth information obtained in step 203 is used to determine the radial distance (ρ) of each pixel. As a result, each pixel is associated with full spherical coordinates. Then, a second transformation (e.g., ...) is used. (z = ρcos 0) converts the spherical coordinates to Cartesian coordinates.

[0179] Although specific examples have been provided above, it will be understood that any method used to convert pixels of a rectangular image to 3D (Cartesian) coordinates can be used instead. In another example, 3D Cartesian coordinates are generated as described in “3DGeometry for Panorama, Jianxiong Xiao, MIT,” which is incorporated herein by reference.

[0180] In step 205, the method extracts features from the isorectangular projection of the input image. As is known in the art, feature extraction is a core component of the computer vision pipeline and involves extracting useful features from an image that clearly define objects within the image. In computer vision, features are measurable data that can be used to identify objects. For example, features can be different colors in an image, specific shapes such as lines or edges, or image fragments.

[0181] Step 205 includes optionally using a machine learning model to extract features for each pixel in the equirectangular image. Although feature extraction has been discussed in detail with “NetVLAD”, it is worth noting, for the sake of clarity, that other feature extraction methods can be used, provided that the method takes the equirectangular image as input and generates information about the identifying feature map for each pixel in the equirectangular image as output.

[0182] In the embodiments, feature extraction and encoding are performed based on "NetVLAD," as described in "Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pa-jdla and Josef Sivic, NetVLAD: CNN architecture for weakly supervised place recognition," which is incorporated herein by reference. Advantageously, "NetVLAD" has been trained for place recognition and has proven to perform well.

[0183] "NetVLAD" consists of a convolutional neural network (for feature extraction) and a trainable VLAD layer that can be trained via backpropagation to simulate VLAD within a convolutional neural network (CNN) framework.

[0184] VLAD stands for Local Feature Aggregation Descriptor, and is described by reference in “H. J'egou, M. Douze, C. Schmid, and P. P'erez, Aggregating local descriptors into a compact image representation, CVPR Proceedings, 2010”, which is incorporated herein by reference. In short, VLAD provides a descriptor pooling method where residuals (difference vectors between the descriptors of features and the cluster centers in the feature space) are generated. Formally, given N D-dimensional local image descriptors {x... i} as input (e.g., from previous feature extraction steps) and K cluster centers (“visual words”) {c k As VLAD parameters, the output VLAD image represents V as a K×D dimensional matrix. The (j, k) elements of V are calculated as follows:

[0185]

[0186] Where x i (j) and c k (j) represents the j-th dimension of the i-th descriptor and the k-th cluster center, respectively, and a k (x i ) represents the descriptor x i The membership degree of the k-th visual word, i.e., if cluster c k The closest descriptor x i If the clustering is 1, then the result is 0; otherwise, the result is 0.

[0187] "NetVLAD" modified the weighting function (a k (x i )), making the descriptor x i For cluster c k The weights of the cluster centers are proportional to their proximity, but relative to the proximity of other cluster centers. NetVLAD also introduces a parameter (α) that controls the decay of the response with distance. Furthermore, in NetVLAD, the anchor points for clustering (from which residuals are calculated) are trainable. In standard VLAD, anchor points are chosen as cluster centers to ensure a uniform distribution of residuals across the database. However, in NetVLAD, cluster centers can be trained such that the dot product between two descriptors known to belong to different images is small (i.e., making the residuals more distinct between different images).

[0188] The VLAD descriptor used in “NetVLAD” is:

[0189]

[0190] Among them, w k =2αc k And b k =-α||c k || 2 In "NetVLAD", {w k}、{b k} and {c k} are trainable parameters.

[0191] Therefore, given N D-dimensional local pixel descriptors as input and K automatically discovered cluster centers, the output of "NetVLAD" at each pixel is a K×D matrix based on the descriptor distances to each cluster center in the feature space. In "NetVLAD," all pixel matrices are simply aggregated into a single matrix. This matrix is ​​then flattened and used as an image embedding for image retrieval.

[0192] However, in the method disclosed in this paper, instead of summing all N pixel matrices in an unordered manner, the spatial structure is preserved to allow for the application of instance-level and graph-level aggregations. Therefore, the modified "NetVLAD" according to the example described herein generates an H×W×K×D feature map for an image of size H×W. Or in other words, the "NetVLAD" modified in this paper outputs information identifying the VLAD descriptor associated with each pixel.

[0193] As discussed above, the implementation of “NetVLAD” used in the example includes a convolutional neural network (CNN) for feature extraction. Alternatively, the CNN can be implemented using a pre-trained VGG-16 backbone as described in “K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition, ICLR Proceedings, 2015”, cited in this paper. In this embodiment, the weights of the trainable VLAD layers are the same as those used in “Titus Cieslewski, Siddharth Choudhary and DavideScara-muzza, Data-efficien decentralized visual SLAM, ICRA, pp. 2466-2473, IEEE, 2018”, cited in this paper.

[0194] In the example, the equirectangular image from step 201 is first upsampled to 2048×1024 pixels and then fed as input to the "NetVLAD" feature extractor. The output of the convolutional neural network (CNN) (i.e., VGG-16) is obtained when the size has been reduced to 128×64, where each pixel has a descriptor of 64×512.

[0195] In summary, after completing steps 202, 203, 204, and 205, the method has generated an object mask indicating the category label of each pixel in the equirectangular image from step 202, the 3D position of each pixel from steps 203 and 204, and information from step 205 identifying the feature vector associated with each pixel.

[0196] Optionally, steps 202 and 204 are each followed by a resizing step, where the size of the information is adjusted so that the output image size of each step (i.e., 202, 204, and 205) is the same. For example, when performing feature extraction using "NetVLAD" with an output size of 128×64, the output size of steps 202 and 204 is also resized to 128×64 pixels. Specifically, the instance segmentation information and depth information are resized to match the spatial dimensions of "NetVLAD" (i.e., 128×64 in this example). Optionally, downsampling is performed using nearest neighbor downsampling. In nearest neighbor downsampling, the coordinates of each pixel in the output image are first determined. Then, for each output pixel, the pixel value of the input pixel that is closest to that coordinate in the output image is selected as the output value.

[0197] exist Figure 2 Steps 202, 203, 204, and 205 are shown in parallel to illustrate that these operations do not necessarily depend on each other's outputs and can therefore be executed in parallel. Alternatively, Figure 2 Each branch shown can be executed sequentially.

[0198] In step 206, the method generates a scene graph. The scene graph G = (V, E) is represented as a set of nodes V and a set of edges E. Optionally, each node i ∈ V is associated with a feature vector x. i They are correlated, and each edge (i,j)∈E is associated with the feature vector x. ij Related.

[0199] In step 206, nodes of the graph are generated. Each node in the graph is associated with an object instance detected in step 202. Each pixel associated with the object (identified from the object mask generated in step 202) is determined, and the corresponding feature vectors (identified from step 205) associated with the determined pixels are aggregated by summation to determine a single feature vector for that object instance.

[0200] Each 3D coordinate (generated in step 204) associated with the object (identified from the object mask generated in step 202) is also determined. Median 3D coordinate values ​​are generated based on the 3D coordinates of each pixel associated with the object, thereby generating a single median 3D coordinate associated with each object instance.

[0201] Therefore: each object is associated with a node, the feature vector associated with the node is the sum of the feature vectors associated with each pixel of the object instance; and the node is also associated with 3D coordinates, where the 3D coordinates are the median of the 3D coordinates associated with the object instance.

[0202] Then, edges between the nodes are constructed. In one example, if the distance (optionally Euclidean distance) is less than a distance threshold, an edge is added between the nodes. There is no descriptor (i.e., feature vector x). ij It is associated with the identified edge. Advantageously, good performance can be achieved by using a distance threshold of 3 meters.

[0203] In another example, edges are added between nodes based on different criteria. In one example, each node is associated with an object instance, and each object instance is associated with a category label. In this case, if the object instances associated with nodes have the same category label, an edge is added between the first and second nodes (e.g., an edge is added between two nodes associated with instances of the "chair" category).

[0204] In another example, if a node is associated with a category label from a set of categories, an edge is added between the two nodes. For example, if the set of categories includes {"chair", "table"}, an edge is added between the first node associated with an object instance with the category label "chair" and the second node associated with an object instance with the category label "table".

[0205] More generally, in one example, if the associated object instances of two nodes have similar properties (such as label similarity, known related labels, 3D coordinates), then an edge is added between the two nodes. In another example, pairwise checks based on node properties generate an edge between the nodes.

[0206] Figure 6AA 3D representation of a scene with superimposed nodes and edges according to an embodiment is shown. Figure 6B A 2D representation of the scene graph based on the example is shown. However, it should be emphasized that the method described in this paper uses a 3D scene graph.

[0207] Return to Figure 2 With the scene graph already generated, the method proceeds to step 207.

[0208] In step 207, a graph embedding is generated by converting the generated scene graph into a vector that can be used for subsequent comparisons (e.g., for image recognition). Optionally, the graph embedding is generated by a graph embedding network.

[0209] Figure 7 A graph embedding network according to an embodiment is shown. Figure 7 A graph embedding network 701 is shown, which includes an encoder 702, a propagation layer 703, and a graph aggregator 704. Although Figure 7 The encoder 702, propagation layer 703, and graph aggregator 704 are shown as separate components, but it will be understood that their functions can be combined into a single component.

[0210] Encoder 702 is configured to map node and edge features from the scene graph generated in step 206 to initial node and edge vectors, respectively. and in It is the initial (i.e., time = 0) node vector associated with the i-th node, where i ∈ V, and V is the set of nodes forming the graph. Initial node vector It is set to be equal to the feature vector x associated with the i-th node. i (Right now ), while edge vectors (The edge between nodes i and j) is not used.

[0211] As discussed above, each node is associated with an object in the scene (i.e., the input image), and in this example, the feature vector associated with that node is an aggregation of the "NetVLAD" descriptors at each pixel associated with that object (i.e., the sum of the residuals of K clusters with D-dimensional features as a K×D matrix). Therefore, in this example, the initial node vector... It is a K×D matrix. Advantageously, "NetVLAD" is used to extract the feature map for each pixel, resulting in already rich encoding.

[0212] Encoder 702 initializes the node vector of each node. Transmitted to propagation layer 703. The node vector h (i.e., ...) of the i-th node is then transmitted. Let's discuss the function of the propagation layer, where i ∈ V, and V is the set of nodes that form the graph. However, it will be understood that the same function is applied to other nodes in the node set.

[0213] The propagation layer 703 is configured to (symbolically) pass messages along the edges connecting nodes between nodes in the graph to generate new representations of the nodes. As discussed above, the purpose of the graph embedding network 701 is to transform the graph representations into vectors that can be used for subsequent comparisons. As will be discussed in more detail below, the graph embedding network is trained such that two input graphs representing similar scenes produce output vectors that are more similar than output vectors generated from two input graphs representing different scenes. The propagation layer controls the process of transforming the feature vectors associated with each node into the final node representation, which is ultimately used to generate the final graph representation.

[0214] Figure 8 A method performed by the propagation layer 703 of a graph embedding network 701 according to an embodiment is shown. In step 801, the initial node vector of node i is obtained. As discussed above, the initial node vector Output by encoder 702. In the example, the initial node vector of node i is obtained. This includes the initial node vector Set to be equal to the feature vector x associated with the i-th node. i (Also known as the characteristic matrix).

[0215] In addition, step 801 also includes obtaining information E that identifies the edges connected to node i. i In the example, the information identifying the edges connected to node i is obtained from the scene graph.

[0216] In step 802, it is determined whether there is any edge connecting node i (i.e., whether there is any edge (j, i)). If there is at least one edge connecting node i to another node, the method proceeds to step 803.

[0217] In step 803, a message m is generated for each edge connected to node i according to the following formula. j→i :

[0218] and

[0219]

[0220] in:

[0221] σ(x) is the logical sigmoid function.

[0222] MLP gate(x) is a multilayer perceptron configured to generate gated weights based on the input x; and

[0223] ⊙ represents element-wise multiplication.

[0224] As will be clear from the equation above, f m (Also known as the message function) uses a gating function. As is known in the art, a gating function is configured to control the proportion of the input that is output. This is done by using gating weights (e.g., Its range is between 0 and 1) and node vector This is achieved through element-wise multiplication. The gating weights are generated by the machine learning model based on the input. In the example above, the gating weights are generated by the multilayer perceptron based on the input. Generated, among which It is the current node vector of the i-th node and It is the current node vector of the j-th node.

[0225] Let represent a multilayer perceptron that generates gating weights for gating the node vector from the i-th node during message generation (denoted as m). This multilayer perceptron has trainable weights to generate gating weights based on the input (at the output), where ... It is the current node vector of the i-th node and It is the current node vector of the j-th node. Similarly, It also has trainable weights.

[0226] Advantageously, by gating the representation of each node vector and learning how to apply that gating, the message function activates more reliable features.

[0227] In step 803, a message is generated for each edge connected to the i-th node. For example, if the i-th node is connected to two other nodes j1 and j2 via an edge, according to... and Calculate two messages. However, regardless of the node, f m The weights of the multilayer perceptrons in the model are all the same (i.e., The learnable weights do not change with j). Therefore, in this example, for and They all use the same Of course, I will understand, by The resulting gating weights (i.e., outputs) may differ because The inputs will be different.

[0228] After generating a message for each edge connected to the i-th node, the method proceeds to step 804. In step 804, the updated node vector is generated according to the following formula:

[0229]

[0230] in It also includes all edges (j, i) connected to the i-th node, where E is the set of edges in the scene graph, and ||E i || is E i The cardinality (i.e., the number of edges connected to node i). As will be clear from the above, the node function f n The sum of the messages generated in step 803 is used as a parameter.

[0231] In addition to using different multilayer perceptrons to generate gating weights (i.e., using...), and to replace and In addition to the node function f n Similar to message function f m Node function f n It is given by the following formula:

[0232]

[0233] and They all have learnable weights.

[0234] In generating updated node vectors (i.e., After that, the method proceeds to step 805, where it is determined whether time t+1 equals the time threshold T. If the time does equal the time threshold, the method proceeds to step 807, where the propagation layer 703 outputs the final node representation. If time t+1 is not equal to the time threshold (i.e., time is less than the time threshold T), the method proceeds to step 808, in which the time step is incremented (i.e., t = t+1), and then proceeds to step 802, in which the updated node vector is calculated.

[0235] Return to step 802. If it is determined in step 802 that there is no edge (j, i) connecting to the i-th node, the method proceeds to step 806, in which the updated node vector is... Set to equal to the current node vector The method then proceeds to step 805 as discussed above.

[0236] although Figure 8This is shown with respect to a single node of the graph (i.e., the i-th node), but it will be understood that the method is applicable to generating a final node representation for every node i in the graph. That is, by repeating for each node Figure 8 The method shown, or by adjusting Figure 8 The steps ensure that each operation is performed at each node i across the graph.

[0237] For example, regarding the message function f m The discussion focuses on how, when considering different edges, the same multilayer perceptron (i.e., with the same weights) is used for each node i in the graph when generating updates across nodes. In other words, and The weights are the same for all i, where i∈V.

[0238] In summary, the trainable parameters of graph embedding networks are The weights of the various multilayer perceptrons mentioned above are fixed over time (i.e., do not change with time t). Advantageously, using the same parameters over time reduces the amount of training required. In another embodiment, the weights of the multilayer perceptron change over time.

[0239] Return to Figure 7 The propagation layer 703 is configured to output the final node representation of each node i in the graph based on the output of the propagation layer 703. Specifically, the graph aggregator 704 is configured to be based on the final node representation of each node i in the graph. Generate a graph representation h G .

[0240] In this embodiment, the graph aggregator 704 calculates the graph-level representation according to the following formula:

[0241]

[0242] MLP gate and MLP node These are all row kernels that are convolved with the features of each of the K clusters (in the feature vector). MLP gate It is a multilayer perceptron, which is configured to input (i.e., the final representation of the i-th node) through the input. The MLP uses row kernel convolutions to generate gated weights as its output. node It is a multilayer perceptron, which is configured to input (i.e., the final representation of the i-th node) through the input. The output is generated by convolution with row kernels. The logical sigmoid function is applied to MLPs. gate The output is then multiplied element-wise by the MLP.node The output.

[0243] Node features (e.g.) ) and K×D eigenvectors associated with nodes (i.e., x i The K clusters have the same dimension because they represent K clusters with D-dimensional features. Advantageously, using row kernels (e.g., 1×D matrices) means performing convolutions within each cluster, which helps to preserve the structure in the data while reducing the node representation to a K×1 matrix.

[0244] This operation is performed on each node i in the node set V, and the output of each node is summed. A principal component analysis (PCA)-based transformation is then applied to the sum to obtain a final compact descriptor of the scene. As is known in the art, principal component analysis (PCA) is a dimensionality reduction method used to reduce the dimensionality of large datasets by transforming a large set of variables into a smaller set of variables that still contains most of the information from that large set.

[0245] In the examples discussed above, the PCA-based transformation uses a multilayer perceptron (i.e., MPL). G This is implemented using PCA-based transformations. An example of using PCA-based transformations is described above in "NetVlaD," which is incorporated herein by reference. In the embodiments, MPL... G The weights are predetermined (e.g., obtained as part of the weights of the pre-trained "NetVlaD"). In this case, the weights for the PCA-based transformation are generated during the training of the "NetVlaD" system while the other weights are being trained.

[0246] After dimensionality reduction, the graph aggregator 304 outputs a graph-level representation h. G .

[0247] In the example, the multilayer perceptron discussed above (i.e., and MLP node At least some (optionally all) of these are implemented using a single linear layer. In another example, each of the multilayer perceptrons includes multiple (e.g., 1, 2, and 3) propagation layers.

[0248] Return to Figure 2 After generating the graph embedding in step 207, the method proceeds to step 208.

[0249] Figure 2 The method involves reasoning. Therefore, the image data in step 201 can be obtained in response to a query performed on the image data to perform location identification and / or image retrieval.

[0250] In step 208, the method determines the graphical representation h of the input image generated in step 207. G Euclidean distances between multiple graph representations associated with images from a reference image set.

[0251] In the example, a reference image set is maintained. This reference image set includes multiple images. Optionally, each image is associated with location information. Once the system has been trained (e.g., once the graph embedding network has been trained), a graph embedding is obtained for each image in the reference set. This is done by providing the images as input in step 201 and from... Figure 2 Step 207 in the process is used to obtain the output. The graph embedding associated with each image is stored as part of the reference set.

[0252] During inference, Euclidean distances are determined between the graph embedding associated with the query image and the graph embeddings associated with each image in the reference set. The image associated with the smallest Euclidean distance between the graph embeddings is selected as the matching image and provided as the output of image retrieval. Optionally, the matching image may be provided to the user, for example, in response to a request received in step 201.

[0253] When the system is used for location identification, the location associated with the matching image is retrieved and provided to the user, thereby identifying the location associated with the input image received in step 201.

[0254] During inference, the trainable weights of the graph embedding network 301 (i.e., The weights are predetermined. In one embodiment, inference is performed after the system has been trained (discussed in more detail below). As a result, the weights are generated during the training phase. In another embodiment, the weights are pre-computed (e.g., from training phases at different times) and provided to the system before inference (e.g., by retrieving them from storage).

[0255] Now consider training the trainable weights of a graph embedding network. Training involves training the weights of the graph embedding network to produce vector representations, where similar scenes are associated with vectors that have greater similarity than vector representations associated with two dissimilar scenes.

[0256] In the example of implementing a graph embedding network as described above, training the graph embedding network includes training the learnable weights of the propagation layer 703 (i.e., (weight).

[0257] To avoid confusion, it is important to emphasize that training the graph embedding network does not include training the following: weights for feature extraction (e.g., the "NetVLAD" parameters), weights for depth estimation (where depth estimation is implemented using a machine learning model), and weights for instance segmentation. As discussed above, the weights for each of these tasks can be predetermined and / or generated by specifically training the relevant components.

[0258] Figure 9 A method for training a graph embedding network according to an embodiment is illustrated. Training the graph embedding network involves initializing trainable weights (i.e., The weights are then assigned, and the process begins in step 901. Optionally, trainable weights are initialized using random values.

[0259] In step 902, the method obtains the query graph G from the training set. q , positive graph G p and negative graph G n The training set comprises multiple triplets. The query graph is a graph representation of a scene. Positive graphs are graph representations of similar scenes (e.g., associated with the same location / location). Negative graphs are graph representations of dissimilar scenes (e.g., associated with different locations / locations). Graph representations of various scenes can be obtained by performing steps 201, 202, 203, 204, 205, and 206 on each image.

[0260] Optionally, batch training can be used to train the graph embedding network (i.e., training the graph embedding network on a subset of a larger training set). When using batch training, triplets are obtained from a batch of training samples. In the example, the number of training samples in the batch is 16.

[0261] The graph embedding network is trained using supervised learning and configured to learn a set of weights that minimizes the loss function. After obtaining the triads from the training set, the method proceeds to step 903.

[0262] In step 903, for querying graph G q , positive graph G p and negative graph G n Each generated graph embedding vector is then used. Using the currently set weights of the graph embedding network (which are the initial weights from step 901 for the first training epoch), according to... Figure 2 Step 207 generates graph embedding vectors, thereby providing a basis for querying graph G. q , positive graph G p and negative graph G n Each of the generated graph embedding vectors. To avoid confusion, it should be emphasized that the graph embedding network uses the same trainable weights to generate the vector representation of each graph.

[0263] In step 904, based on the query graph G q The generated graph embedding vector (i.e., ), is a positive graph G p The generated graph embedding vector (i.e. ) and the negative graph G n The generated graph embedding vector (i.e. The training method optimizes the margin-based triplet loss as described in "Florian Schroff, Dmitry Kalenichenko, and James Philbin, Facenet: A unified embedding for face recognition and clustering, CVPR, pp. 815-823, 2015" (incorporated by reference to this paper).

[0264] In the example, the method uses a boundary-based triplet loss calculated according to the following formula (i.e., a triplet loss that requires the similarity between the query graph and the positive graph to be greater than the similarity between the query graph and the negative graph by one boundary):

[0265]

[0266] in:

[0267] And it represents the graph embedding vector of the query graph. Graph embedding vectors of the positive graph The square of the Euclidean distance between them;

[0268] γ is the boundary limit, which can be optionally set to 1.0;

[0269] The loss based on the triad is the expected value of the loss. In the example, the set of training samples is uniformly sampled, and the sum of the calculated losses is normalized using the number of samples, thus producing the expected value of the loss (i.e., the average loss). The number of samples used to generate the expected value of the loss is greater than or equal to 1.

[0270] Therefore, when d(G) q G p ) compared to d(G q G n When the threshold γ is smaller than 0, the loss function generates a value of 0.

[0271] After the triplet loss is determined in step 904, the method proceeds to step 905.

[0272] In step 905, backpropagation based on triplet loss is used to determine changes to the trainable weights (for each trainable weight).

[0273] As is known in the art, backpropagation, an abbreviation for "propagation of error in the opposite direction," is an algorithm for supervised learning of artificial neural networks using gradient descent. Given an artificial neural network and an error function, this method computes the gradient of the error function with respect to the weights of the neural network and generates weight increments (i.e., update values) based on the learning rate and the computed gradients for specific weights.

[0274] Given trainable weights in a graph embedding network, and for a given training triad, the method is configured to generate three adjustments. The first adjustment is based on: the loss function (L) relative to the vector representation of the query graph. The differential, and the vector representation The second adjustment is based on the derivative of the loss function (L) with respect to the vector representation of the positive graph. The differential, and the vector representation The third adjustment is based on the derivative of the loss function (L) with respect to the vector representation of the negative graph. The differential, and the vector representation The derivative with respect to a given weight (w). The first, second, and third adjustments can be averaged to determine the final adjustment for the given weights based on the training triad.

[0275] The differential of the loss function (L) with respect to the vector representations of various graphs can be obtained by differentiating the loss function (L) described above. The chain rule and the structure of the graph embedding network can be used to determine the differential of the vector representation with respect to a given weight.

[0276] When using batch training to train a graph embedding network, it will be understood that the adjustments for each training example are averaged across the batch to generate the final weight adjustments for that training epoch.

[0277] In step 906, each trainable weight is adjusted based on the determined change of each weight determined in step 905.

[0278] In one embodiment, the learning rate used to train the graph embedding network is optionally set to 10. -4The Adam optimizer. As is known in the art, the Adam optimizer is an optimization algorithm for stochastic gradient descent used to train machine learning models. The Adam optimizer used in the examples is described by reference in "Diederik P. Kingma and Jimmy Ba, (2017), Adam: AMethod for Stochastic Optimization", which is incorporated herein by reference.

[0279] After adjusting the trainable weights of the graph embedding network, the number of training epochs performed by the training method is increased by one (i.e., the number of training epochs = the number of training epochs + 1), and the method proceeds to step 907.

[0280] In step 907, it is determined whether the number of training epochs already performed is equal to a threshold number of training epochs (maximum training epochs). As is known in the art, a training epoch refers to one pass through the training dataset (i.e., one repetition of training a machine learning model using samples in the dataset). In the case of batch learning, an epoch refers to the training iteration completed based on each training sample in the batch. In this embodiment, the graph matching network is trained for up to 1000 epochs (i.e., maximum training epochs = 1000).

[0281] If the maximum number of rounds has not yet been reached, the method proceeds to step 902. If the maximum number of rounds has been reached, the method proceeds to step 908, which completes the training.

[0282] Optionally, upon completion Figure 9 The weights learned through this method are then stored in a storage device for future use in inference.

[0283] The implementation (titled "GraphVLAD") was compared with other methods for image retrieval / location recognition. In "GraphVLAD", features are extracted using "NetVLAD" with a pre-trained VGG-16 backbone having the same weights used in "Titus Cieslewski, Siddharth Choudhary and Davide Scara-muzza, Data-efficiende centralized visual slam, ICRA, pp. 2466-2473, IEEE, 2018" as cited above and incorporated herein by reference. The outputs of instance segmentation, 3D coordinate generation, and feature extraction are resized to 128×64 pixels as needed. The graph embedding network 701 is implemented based on the specific example discussed above. In all evaluations, the graph matching network 701 was used with a learning rate set to 10. -4 The Adam optimizer is trained with batch sizes ranging from 16 to 1000 epochs.

[0284] To evaluate GraphVLAD, a new dataset called OmniClevr was developed. OmniClevr is inspired by the Clevr dataset described in Justin Johnson, Bharath Hariharan, Laurens VanDerMaaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick, Clevr: A diagnostic dataset for compositional language and elementary visual reasoning, CVPR 2017, pp. 2901–2910, which is incorporated herein by reference. OmniClevr comprises isorectangular representations of compositional environments with object components.

[0285] During the evaluation presented in this paper, ground-based instance segmentation and depth information were used (i.e., an embodiment of generating perfect predictions by creating instance segmentation and depth information components).

[0286] The table below shows a performance comparison between "NetVLAD," as described in "Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pa-jdla and Josef Sivic, NetVLAD: CNN architecture for weakly supervised place recognition," and "GraphVLAD," as described in this paper, for image retrieval.

[0287] method Distance (m) Recall rate @1 Recall rate @5 Recall rate @10 NetVLAD - 73.2 84.6 90.7 GraphVLAD 0 90.6 95.9 97.4 GraphVLAD 1 90.9 96.5 97.6 GraphVLAD 3 92.1 97.1 98.0 <![CDATA[NetVLAD t ]]> - 83.2 94.0 96.3 <![CDATA[GraphVLAD t ]]> 3 93.2 97.2 98.2

[0288] As discussed above, "GraphVLAD" generates a scene graph with "NetVLAD" features at the nodes, and generates edges between the nodes of the graph if the median 3D coordinate interval associated with a node is less than a threshold distance d. Results are generated for different distance thresholds (i.e., varying d). Note that a distance threshold of 3m for edge generation (i.e., d = 3m, requiring node intervals to be less than or equal to 3m to establish edges between nodes) performs best.

[0289] The table above reports the recall of the top N retrieved images in both the object split and camera split. As is known in the art, recall is the fraction (expressed as a percentage) of the number of times a relevant image is retrieved in the top N results. In this case, relevant images are images from the same location / location.

[0290] “GraphVLAD” improves upon “NetVLAD” with a recall of 92.1 to 73.2 @1. Note that the off-the-shelf “NetVLAD” is pre-trained on real images and is expected to perform poorly on the OmniClevr dataset. Therefore, “NetVLAD” is trained from scratch on “OmniClevr”. This instance of “NetVLAD” trained from scratch on “OmniClevr” is denoted as: “NetVLAD”. t Similarly, “GraphVLAD” t "Feature extraction was performed using these retrained NetVLAD features."

[0291] For "NetVLAD" t "The recall rate @1 increased to 83.2%, while "GraphVLAD" t "Score: 93.2. Note that overall performance was improved by using a graph (itself), and GraphVLAD..." t "Further benefit from the improved NetVLAD" t "feature.

[0292] Figure 10 The “GraphVLAD” according to an embodiment is shown. t "and NetVLAD" t Recall for each validation type on the "OmniClevr" dataset. Figure 10 It shows "NetVLAD" t "and GraphVLAD" t "Receiver operating characteristic (ROC) curves on the "OmniClevr" validation set for different groups."

[0293] Static scenes (represented as "none") are relatively easy to retrieve images from, while dynamic objects (such as object movement and changes) present a more challenging task. Here, "NetVLAD" t "It is particularly difficult to handle the addition or deletion of objects, resulting in recall rates of 71.6 and 63.2 respectively @1. In contrast, GraphVLAD..." t"Benefiting from graph embedding, which is more robust than appearance alone, it results in significantly better recalls of 78.4 and 80.4@1 for additions and deletions, respectively. Object motion is less challenging, but the method presented in this paper still outperforms NetVLAD." t ".

[0294] Figure 10 Another key factor in the evaluation shown is the view change caused by camera motion, especially between the reference set and the query set. It has been demonstrated that "NetVLAD..." t "The camera performs particularly poorly under scrolling and zoom conditions, with recall rates of only 72.4 and 82.4% @1, respectively. In stark contrast, thanks to our rotation- and translation-invariant graph embedding, GraphVLAD..." t "Recall rates of 96.0% and 95.2% were achieved respectively @1. Similarly, "GraphVLAD" t The translation performance of "" has also been improved.

[0295] Message passing plays a crucial role during graph similarity learning. Specifically, in the example described in this paper, we control context awareness through a larger receptive field via an edge distance threshold in our graph. In the table above, “GraphVLAD” (with pre-trained weights) is applied using a varying distance threshold d. Without message passing (d = 0m), recall @1 drops to 90.6, while d = 1m improves performance to 90.9. Overall, we achieve optimal performance at the expense of efficiency by utilizing a larger edge set. The best result in the above evaluation is achieved with a recall @1 of 92.1 using d = 3m. Therefore, accumulated neighborhood is considered beneficial.

[0296] With "NetVLAD" t In contrast, GraphVLAD, as described in the example in this article... t "It demonstrates its strength under challenging scene changes. Specifically, the method disclosed in this paper enables correct matching under camera translation and rotation. In addition, appearance distortion caused by zoom is handled because 3D position is estimated and used for graph construction. Finally, the method disclosed in this paper is robust to dynamics when new objects are added to the scene due to message passing between neighboring objects."

[0297] Representing a scene as a graph offers further advantages. For example, representing a scene as a graph allows for easier manipulation, enabling the addition or removal of nodes (i.e., objects) from the scene graph representation of the input image.

[0298] In the example, one approach involves modifying (e.g., by step 206) the scene graph to add additional nodes (associated with the additional objects) and edges connecting the nodes to other nodes in the graph. In this way, the scene graph can be modified / enhanced to include objects not present in the image obtained in step 201. Adding edges between nodes controls the position of the additional objects within the scene. Alternatively or additionally, the scene graph generated in step 206 may be modified to remove nodes from the graph representation before generating the graph embedding (e.g., in step 207). For example, removing nodes may be advantageous when the image data obtained in step 201 is known to include temporary objects that may not exist in the reference set. Thus, by removing the node, the associated object will not be considered when performing a similarity comparison with the graph embedding in the reference set. Optionally, a modified / enhanced scene graph is generated based on the output of step 206 (generating the scene graph) and provided as input to step 207 (generating the graph embedding).

[0299] Figure 11 A system according to an embodiment is shown. Figure 11 A location recognition system 1100 including an image formatter 1101 is shown. The image formatter 1101 is configured to implement the functions described with respect to step 201. The location recognition system 1100 also includes: an instance segmenter 1102 configured to implement the functions described with respect to step 202; a depth estimator 1103 configured to implement the functions described with respect to step 203; a 3D coordinate generator 1104 configured to implement the functions described with respect to step 204; a feature extractor 1105 configured to implement the functions described with respect to step 205; a scene graph generator 1106 configured to implement the functions described with respect to step 206; and a graph embedding network 1107 configured to implement the functions described with respect to step 207. Figure 11 A similarity detector 1108 is also shown, configured to perform the functions described with respect to step 208 (i.e., determining the distance (optionally Euclidean distance) between the generated graph embedding and the graph embedding from the reference set to determine the location / location). Optionally, the location recognition system may also include a training component 1109, configured to perform the function related to... Figure 9 The methods described are as follows. Each of the components described above is communicatively coupled to each other. Although the system is shown above as separate components, it will be understood that the functionality of these components can be combined in a single component. A system configured to implement the methods described above is also disclosed.

[0300] In the specific example above, the pixels of the image are grouped according to the detected object instances. However, as discussed above, the invention is not limited to this. More generally, the method described herein can be used when the input image is grouped into multiple pixel clusters. In this case, when the pixel cluster is an object instance, the 3D coordinates of each pixel cluster are generated as discussed above. Similarly, when the pixel cluster is an object instance, features are extracted for each pixel cluster (e.g., by summing the feature vectors associated with each pixel in the pixel cluster) as discussed above. Similarly, a scene graph is generated, where each node corresponds to a pixel cluster.

[0301] Figure 12 A schematic diagram of hardware that can be used to implement the method according to an embodiment is shown. The hardware includes a computing system 1200. In this particular example, the components of the system will be described together. However, it will be understood that these components are not necessarily located in the same location.

[0302] The components of the computing system 1200 include, but are not limited to, a processing unit 1213 (such as a central processing unit, CPU, or "processor"), system memory 1201, and a system bus 1211 that couples various system components, including system memory 1201, to the processing unit 1213. The system bus 1211 can be any of a variety of bus architectures, including a memory bus or memory controller, a local bus using any of a variety of bus architectures, and a peripheral bus. The computing system 1200 may also include a memory 1215 connected to the system bus 1211. The memory 1215 may be external memory.

[0303] System memory 1201 includes computer storage media in the form of volatile / or non-volatile memory, such as read-only memory (ROM). System memory 1201 stores computer-readable instructions that, when executed by processing unit 1213 (e.g., processor), cause processing unit 1213 (also referred to as "processor") to perform the functions described herein, particularly as per the description of... Figure 2 (reasoning) and Figure 9 (Training) methods of discussion.

[0304] System memory 1201 may also include program data 1209, application programs 1207, and operating system 1205 used by processing unit 1213. In an embodiment, system memory 1201 also includes a basic input / output system (BIOS) 1203 containing routines that help transfer information between components within the computer, such as information typically stored in system memory 1201 during startup.

[0305] The computing system also includes an interface 1225 communicatively connected to the system bus 1211. In this example, interface 1225 includes functionality for communication via wired and / or wireless connections, specifically for receiving and sending information to other devices. Additionally or alternatively, interface 1225 includes camera input for receiving input (e.g., an image of a scene) and output for providing a user with location identification / image retrieval results. In another example, interface 1225 may be a user interface of various forms that allow the user to interact with the system, such as generating commands and / or questions and receiving responses.

[0306] exist Figure 12 In the example, a video interface 1217 is provided. The video interface 1217 includes a graphics processing unit (GPU) 1219 communicatively connected to a graphics processing memory 1221.

[0307] The graphics processing unit (GPU) 1219 is particularly suitable for training machine learning models, such as neural network training, because it is well-suited for data-parallel operations. Therefore, in an embodiment, processing for training the machine learning model can be divided between the CPU 1213 and the GPU 1219.

[0308] It should be noted that in some embodiments, different hardware can be used to train the machine learning model and to perform state updates. For example, training the machine learning model can occur on one or more local desktop or workstation computers or on devices in a cloud computing system, which may include one or more discrete desktop or workstation GPUs, one or more discrete desktop or workstation CPUs, such as processors with PC-oriented architectures, and a large amount of volatile system memory, such as 16GB or more. However, for example, the execution (i.e., inference) of the machine learning model can use other hardware.

[0309] although Figure 12 A specific example of hardware that can be used to implement the methods described herein is shown, but it will be understood that this is only an example and other arrangements may also be used.

[0310] As discussed above, the examples presented herein can be used for image retrieval and location recognition. Example use cases include vehicle localization / drone position estimation and / or control for robots / cars. Optionally, images are acquired from smartphones, cameras, and / or robots, and the methods are performed by smartphones, cameras, and / or robots. The methods disclosed herein are robust to appearance changes caused by object dynamics and camera movement. Other use cases include cameras mounted on cars and drones subjected to different rotations. Furthermore, training with abstract data allows for easy expansion.

[0311] While certain arrangements have been described, they are presented as examples only and are not intended to limit the scope of protection. The inventive concepts described herein can be implemented in many other forms. Furthermore, various omissions, substitutions, and changes may be made to the specific embodiments described herein without departing from the scope of protection defined in the appended claims.

Claims

1. A computer-implemented method for location identification, comprising: Obtain information about the image that identifies the first scene; The information identifying the image identifies multiple pixel clusters, including: a first pixel cluster; and a second pixel cluster; A feature vector set is generated from the information identifying the image, the feature vector set including: a first feature vector associated with the first pixel cluster; and a second feature vector associated with the second pixel cluster; A graph of the scene is generated, the graph comprising: a first node representing the first pixel cluster, the first node being associated with the first feature vector; and a second node representing the second pixel cluster, the second node being associated with the second feature vector; In response to determining that a first attribute associated with the first pixel cluster is similar to a second attribute associated with the second pixel cluster, a first edge is added between the first node and the second node; Generate a vector representation of the graph; Calculate a measure of similarity between the vector representation of the graph and a reference vector representation, wherein the reference vector representation is associated with a second scene; and In response to the determination that the similarity metric is less than a threshold, it is determined that the first scene and the second scene are associated with the same location.

2. The computer-implemented method according to claim 1, wherein, The image includes multiple pixels, which include a first group of pixels associated with the first pixel cluster and a second group of pixels associated with the second pixel cluster, and generating the feature vector set includes: Generate a third feature vector associated with the first pixel in the first group of pixels; Generate a fourth feature vector associated with the second pixel in the first group of pixels; and The first feature vector is generated by aggregating the third feature vector and the fourth feature vector.

3. The computer-implemented method according to any of the preceding claims, wherein, The vector representation of the graph is generated as follows: Generate a first node vector, wherein the first node vector is associated with the first node; Generate a second node vector, wherein the second node vector is associated with the second node; and A query graph representation is generated based on the sum of the first node vector and the second node vector.

4. The computer-implemented method according to claim 3, wherein, The second node in the graph is connected to the first node in the graph through the first edge, and generating the first node vector includes: Set the first node vector to be equal to the first feature vector; Generate a message between the second node and the first node based on the first node vector and the second node vector; and The first node vector is updated based on its value and the message.

5. The computer-implemented method according to claim 4, wherein, Generating messages between the second node and the first node includes: Based on the first node vector and the second node vector, a first machine learning model is used to generate a first set of gating weights; The first node vector is gated by multiplying it with the first set of gate weights to form a first gate vector; Based on the first node vector and the second node vector, a second machine learning model is used to generate a second set of gating weights; The second node vector is gated by multiplying it with the second set of gate weights to form a second gated vector; and The message is generated based on the sum of the first gating vector and the second gating vector.

6. The computer-implemented method according to claim 5, wherein: The graph of the scene also includes a third node representing a third pixel cluster, which is associated with a third feature vector and is connected to the first node via a second edge; and The method further includes: Generate a third node vector, which is associated with the third node; and The second message between the third node and the first node is generated based on the first node vector and the third node vector through the following processing: Based on the first node vector and the third node vector, the first machine learning model is used to generate a third set of gating weights; The first node vector is gated by multiplying it with the third set of gate weights to form the third gate vector; Based on the first node vector and the third node vector, the second machine learning model is used to generate a fourth set of gating weights; The third node vector is gated by multiplying it with the fourth set of gate weights to form the fourth gate vector; The second message is generated based on the sum of the third gating vector and the fourth gating vector; and wherein: Updating the first node vector based on the value of the first node vector and the message includes: The first node vector is updated based on the value of the first node vector and the sum of the message and the second message.

7. The computer-implemented method according to claim 5 or 6, wherein, Updating the first node vector based on the value of the first node vector and the message includes: Based on the first node vector and the message, a third machine learning model is used to generate a fifth set of gating weights; The first node vector is gated by multiplying it with the fifth set of gate weights to form the fifth gate vector; Based on the first node vector and the message, a fourth machine learning model is used to generate a sixth set of gating weights; The message is gated by multiplying it with the sixth set of gate weights to form a sixth gate vector; and The first node vector is updated based on the sum of the fifth gating vector and the sixth gating vector.

8. The computer-implemented method according to any one of claims 4 to 6, wherein, Generating the query graph representation based on the sum of the first node vector and the second node vector includes: Based on the first node vector, the fifth machine learning model is used to generate the seventh set of gating weights; Based on the first node vector, a sixth machine learning model is used to generate the first node representation of the first node vector; The seventh set of gating weights is multiplied by the first node representation of the first node vector to obtain the final first node representation; Based on the second node vector, the fifth machine learning model is used to generate the eighth set of gating weights; Based on the second node vector, the sixth machine learning model is used to generate the second node representation of the second node vector; Multiply the eighth set of gating weights by the second node representation of the second node vector to obtain the final second node representation; and The query graph representation is generated by adding the final first node representation and the final second node representation.

9. The computer-implemented method according to claim 8, wherein, The fifth and sixth machine learning models implement row kernels such that the final first node representation and the final second node representation each include a matrix with a single column.

10. The computer-implemented method according to claim 8, wherein, Generating the query graph representation includes: Determine the sum of the final first node representation and the final second node representation; and Reduce the dimensions of the sum to form the query graph representation.

11. The computer-implemented method according to claim 4, wherein, Generating the first node vector also includes: The time value is incremented in response to updating the first node vector; Determine whether the time value is less than a time threshold; and In response to determining that the time value is less than the time threshold: Based on the first node vector and the second node vector, regenerate the message between the second node and the first node; and The first node vector is updated based on its value and the message.

12. The computer-implemented method according to any one of claims 1, 2, 4, 5, 6, 9, 10, and 11, wherein, The image is a rectangular image.

13. The computer-implemented method according to claim 12, wherein, Identifying the plurality of pixel clusters in the information identifying the image includes: A first set of detections is generated by performing instance segmentation on the image, the first set of detections including the first pixel cluster; Rotate the image to form a rotated image; A second set of detections is generated by performing instance segmentation on the rotated image; this second set of detections includes the second pixel cluster. The first group of detections and the second group of detections are combined to form the plurality of pixel clusters.

14. The computer-implemented method according to any one of claims 1, 2, 4, 5, 6, 9, 10, 11, and 13, wherein, The first feature vector is represented using a local feature aggregation descriptor.

15. The computer-implemented method according to any one of claims 1, 2, 4, 5, 6, 9, 10, 11, and 13, wherein, Determining that the first attribute associated with the first pixel cluster is similar to the second attribute associated with the second pixel cluster includes: Determine the first set of 3D coordinates associated with the first pixel cluster; Determine the second set of 3D coordinates associated with the second pixel cluster; Calculate the distance between the first set of 3D coordinates and the second set of 3D coordinates; and When the distance is less than the distance threshold, the first attribute is determined to be similar to the second attribute.

16. The computer-implemented method according to claim 15, wherein, Determining the first set of 3D coordinates associated with the first pixel cluster includes: Estimate the depth of the first pixel cluster; and The first set of 3D coordinates is generated based on the depth and the information identifying the image.

17. A non-transitory computer-readable medium comprising computer program instructions that, when executed by a processor, cause the processor to perform the method of any of the preceding claims.

18. An apparatus for location identification, the apparatus being configured to: Obtain information about the image that identifies the first scene; The information identifying the image identifies multiple pixel clusters, which include: First pixel cluster; Second pixel cluster; A feature vector set is generated from the information identifying the image, the feature vector set including: a first feature vector associated with the first pixel cluster; and a second feature vector associated with the second pixel cluster; A graph of the scene is generated, comprising: a first node representing the first pixel cluster, the first node being associated with the first feature vector; and a second node representing the second pixel cluster, the second node being associated with the second feature vector; In response to determining that a first attribute associated with the first pixel cluster is similar to a second attribute associated with the second pixel cluster, a first edge is added between the first node and the second node; Generate a vector representation of the graph; Calculate a measure of similarity between the vector representation of the graph and a reference vector representation, wherein the reference vector representation is associated with a second scene; and In response to the determination that the similarity metric is less than a threshold, it is determined that the first scene and the second scene are associated with the same location.

19. The apparatus according to claim 18, wherein, The image includes a plurality of pixels, the plurality of pixels including a first group of pixels associated with the first pixel cluster and a second group of pixels associated with the second pixel cluster, and the device is further configured to, when generating the feature vector set: Generate a third feature vector associated with the first pixel in the first group of pixels; Generate a fourth feature vector associated with the second pixel in the first group of pixels; and The first feature vector is generated by aggregating the third feature vector and the fourth feature vector.

20. The apparatus according to claim 18 or 19, wherein, The apparatus is also configured to, when generating a vector representation of the graph: Generate a first node vector, wherein the first node vector is associated with the first node; Generate a second node vector, which is associated with the second node; as well as A query graph representation is generated based on the sum of the first node vector and the second node vector.

21. The apparatus according to claim 20, wherein, The second node in the graph is connected to the first node in the graph via the first edge, and the device is further configured to, when generating the first node vector: Set the first node vector to be equal to the first feature vector; Generate a message between the second node and the first node based on the first node vector and the second node vector; as well as The first node vector is updated based on its value and the message.

22. The apparatus according to claim 21, wherein, The device is also configured to, when generating a message between the second node and the first node: Based on the first node vector and the second node vector, a first machine learning model is used to generate a first set of gating weights; The first node vector is gated by multiplying it with the first set of gate weights to form a first gate vector; Based on the first node vector and the second node vector, a second machine learning model is used to generate a second set of gating weights; The second node vector is gated by multiplying it with the second set of gate weights to form a second gated vector; and The message is generated based on the sum of the first gating vector and the second gating vector.

23. The apparatus according to claim 22, wherein: The scene graph also includes a third node representing a third pixel cluster, which is associated with a third feature vector and connected to the first node via a second edge; and The device is also configured to: Generate a third node vector, which is associated with the third node; and The second message between the third node and the first node is generated based on the first node vector and the third node vector through the following processing: Based on the first node vector and the third node vector, the first machine learning model is used to generate a third set of gating weights; The first node vector is gated by multiplying it with the third set of gate weights to form the third gate vector; Based on the first node vector and the third node vector, the second machine learning model is used to generate a fourth set of gating weights; The third node vector is gated by multiplying it with the fourth set of gate weights to form the fourth gate vector; The second message is generated based on the sum of the third gating vector and the fourth gating vector; and wherein: Updating the first node vector based on the value of the first node vector and the message includes updating the first node vector based on the value of the first node vector and the sum of the message and the second message.

24. A computer-implemented method for training a graph matching network for location recognition, wherein the graph matching network includes a machine learning model, wherein the machine learning model includes one of the following: a first machine learning model or a second machine learning model used in the computer-implemented method of claim 5 or 6, a third machine learning model or a fourth machine learning model used in the computer-implemented method of claim 7, and a fifth machine learning model or a sixth machine learning model used in the computer-implemented method of claim 8, the computer-implemented method comprising: Obtain a training tripartite, which includes a query graph, a positive graph, and a negative graph; The machine learning model is used to generate the query vector representation of the query graph; The machine learning model is used to generate the positive vector representation of the positive graph; The machine learning model is used to generate the negative vector representation of the negative graph; Determine the first similarity between the query vector representation and the positive vector representation; Determine the second similarity between the query vector representation and the negative vector representation; A triplet loss is generated based on the first similarity and the second similarity; and The machine learning model is trained based on the aforementioned triplet loss.

25. The computer-implemented method according to claim 24, wherein, Training the machine learning model based on the aforementioned triplet loss includes: Backpropagation and gradient descent are used to determine the weights of a machine learning model.

Images