Apparatus and method for clustering and matching feature descriptors

Abstract

Methods, systems, and apparatus are provided for clustering and matching image feature descriptors used in various systems. For example, a computing device receives a location from a remote device. The computing device applies a first clustering process to a plurality of descriptors associated with the location to determine a number of descriptor clusters. The computing device also applies a second clustering process to the number of descriptor clusters to determine a descriptor cluster center for each of the number of descriptor clusters. Further, the computing device generates descriptor cluster data characterizing similarities between the plurality of descriptors and the descriptor cluster center. The computing device then sends the descriptor cluster data to the remote device. The remote device may match the descriptors to the descriptor clustering center based on the descriptor clustering data.

Description

Technical Field

[0001] The present disclosure generally relates to processes for determining feature descriptors and, more particularly, to clustering and matching image feature descriptors for use in a driving system. Background Art

[0002] Various applications rely on capturing images and determining features based on the captured images. For example, extended reality applications such as augmented reality and mixed reality applications can capture two-dimensional images and can determine image descriptors that characterize the image features. In some applications, a vehicle (such as an autonomous vehicle) can operate with a vehicle monitoring system, and the vehicle monitoring system along with other systems can enhance the driver's experience and safety. For example, the vehicle monitoring system can capture a two-dimensional image of the vehicle's environment to determine the vehicle's location, or can capture a two-dimensional image of the driver to determine the driver's pose (e.g., the direction the driver is looking). In such examples, an image descriptor is determined from the captured two-dimensional image and compared to a feature database to determine the location or pose. However, in these and other examples, for various reasons such as changes in lighting conditions, seasonal changes, and object movement, there may be a lack of comparison and inaccurate results may be produced. As a result, various applications that employ such processes may be affected. For example, in the example of the vehicle monitoring system, the driver's experience and safety within the vehicle cab may be negatively impacted. Additionally, as the feature database grows, the amount of memory required to store additional features increases, as well as the processing power required to process the additional features, thereby increasing system costs and processing requirements. Summary of the Invention

[0003] According to one aspect, an apparatus includes a non-transitory machine-readable storage medium storing instructions, and at least one processor coupled to the non-transitory machine-readable storage medium. The at least one processor is configured to execute the instructions to apply a first clustering process to a plurality of descriptors associated with a geographical location to determine a number of descriptor clusters. Additionally, the at least one processor is configured to execute the instructions to apply a second clustering process to the number of descriptor clusters to determine a descriptor cluster center for each of the number of descriptor clusters. The at least one processor is further configured to execute the instructions to generate descriptor cluster data representative of the similarity between the plurality of descriptors and the descriptor cluster centers. The at least one processor is further configured to execute the instructions to store the descriptor cluster data in a data repository.

[0004] According to another aspect, a method executed by at least one processor includes applying a first clustering process to a plurality of descriptors associated with a geographical location to determine a number of descriptor clusters. Additionally, the method includes applying a second clustering process to the number of descriptor clusters to determine a descriptor cluster center for each of the number of descriptor clusters. The method further includes generating descriptor cluster data representative of the similarity between the plurality of descriptors and the descriptor cluster centers. The method also includes storing the descriptor cluster data in a data repository.

[0005] According to another aspect, a non-transitory machine-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations including applying a first clustering process to a plurality of descriptors associated with a geographical location to determine a number of descriptor clusters. Additionally, the operations include applying a second clustering process to the number of descriptor clusters to determine a descriptor cluster center for each of the number of descriptor clusters. The operations further include generating descriptor cluster data representative of the similarity between the plurality of descriptors and the descriptor cluster centers. The operations also include storing the descriptor cluster data in a data repository. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a block diagram of an exemplary vehicle monitoring system according to some specific implementations;

[0007] Figure 2 is a block diagram showing an exemplary portion of a vehicle monitoring system according to some specific implementations Figure 1 of;

[0008] Figure 3 shows an iterative descriptor clustering process according to some specific implementations;

[0009] Figure 4 shows a message passing graph according to some specific implementations;

[0010] Figure 5 is a flowchart of an exemplary process for clustering descriptors according to some specific implementations; and

[0011] Figure 6 is a flowchart of an exemplary process for performing operations based on matching image features with descriptors according to some specific implementations. DETAILED DESCRIPTION

[0012] Although the features, methods, devices, and systems described herein may be embodied in various forms, some exemplary and non-limiting embodiments are shown in the drawings and described below. Some of the components described in this disclosure are optional, and some specific implementations may include additional components, different components, or fewer components than those explicitly described in this disclosure.

[0013] The embodiments described herein relate to a computing environment that generates a database of cluster descriptors and corresponding cluster centers and uses an iterative process to update the cluster descriptors and corresponding cluster centers based on performance criteria.

[0014] For example, in some particular implementations, a vehicle monitoring system such as an advanced driver assistance (ADAS) system can include multiple vehicles such as autonomous vehicles and a server-side computing system such as a cloud computing system. The vehicle monitoring system can perform operations as described herein to detect objects and, for example, perform localization operations to determine the position of the vehicle in the real world (e.g., with respect to objects such as curbs, roads, other vehicles, etc.). Additionally, based on the localization, the vehicle monitoring system can perform operations to determine the position of the vehicle within a map such as a high-definition (HD) map. For example, a vehicle can use a camera such as a monochrome camera to capture an image of its environment, such as a two-dimensional (2D) image. In some cases, the vehicle can include a pose device such as a head-mounted display (HMD) device that captures an image of the field of view of the vehicle driver. Additionally, the vehicle computing system can detect features within the captured image (e.g., 2D observations) and can generate a 2D image descriptor based on the detected features. The image descriptor can correspond to a particular geographic location (e.g., a coordinate location, a 3D reference point).

[0015] The vehicle monitoring system can employ a process such as a simultaneous localization and mapping (SLAM) process to generate three-dimensional (3D) image descriptors based on the 2D image descriptors generated by the vehicle. Each 3D image descriptor can correspond to a geographic location (e.g., a coordinate location, a 3D reference point). For example, each 3D image descriptor can correspond to a geographic location and be associated with multiple 2D features and their corresponding 2D image descriptors. For example, a vehicle can include a camera that captures an image of the corresponding environment, such as a 2D image. Each 2D image can include multiple 2D points (e.g., feature points), which can be referred to as observations of corresponding 3D points. For example, each 3D point can have multiple corresponding 2D points (e.g., based on the images captured by the vehicle's camera). A 3D image descriptor corresponding to the 3D point can be generated based on the 2D points corresponding to the 3D point and their corresponding 2D image descriptors. The vehicle can send the 2D image descriptors to a server, such as a cloud-based server, and the server can generate the 3D image descriptors based on the received 2D image descriptors for the same 3D points.

[0016] In addition, a vehicle monitoring system (e.g., a cloud computing system) may perform operations as described herein to cluster 3D image descriptors to generate 3D image descriptor clusters and determine a cluster center for each cluster. In addition, the vehicle monitoring system may perform operations to generate descriptor cluster data representing the similarity (e.g., probability) between the 3D image descriptors and the cluster centers for each 3D image descriptor cluster.

[0017] As the vehicle moves in the environment, the vehicle may obtain descriptor cluster data corresponding to the geographical location of the vehicle. The vehicle may match 2D features detected in an image captured at the geographical location of the vehicle with the cluster centers identified in the descriptor cluster data. For example, the vehicle may determine the distance (e.g., Euclidean distance) between the 2D feature and each cluster center and match the 2D feature with the nearest cluster center (the calculated shortest distance). For example, the determined cluster center may correspond to a corner of a building or some other object or object feature. In some examples, the vehicle may determine that the 2D feature is one of a building or other object or feature based on the matched feature.

[0018] Thus, the embodiments allow the vehicle monitoring system to identify objects, such as objects along the road, road markings, or other objects, based on 2D-3D feature matching. In addition, based on the matching, the vehicle monitoring system may perform additional operations, such as operations to determine alignment, localization, or driver pose as part of a SLAM process.

[0019] In some cases, the vehicle monitoring system performs operations to update the descriptor cluster data based on the matched 3D image descriptors. For example, each vehicle may send matching information to the cloud computing system, the matching information including a number of matched reference points (e.g., matched 3D reference points), the ratio of the number of matched reference points to the total number of reference points at the geographical location (e.g., the location corresponding to the position where the image was captured), and a number of descriptor matches (e.g., the ratio of the number of matched 3D image descriptors to the total number of 3D image descriptors at the corresponding geographical location). The matching information may also include one or more of the matched 2D image descriptors and the matched 3D image descriptors at the corresponding geographical location (i.e., the 3D image descriptors generated based on the captured image and matched with the 3D image features of the descriptor cluster data).

[0020] A cloud computing system can perform operations to determine the matching performance of a geographical location based on membership information and can regenerate (e.g., update) descriptor clustering data for the geographical location based on the matching performance. For example, the cloud computing system can determine one or more of the total number of matching reference points, the ratio of the matching reference points to the total number of reference points, the total number of descriptor matches, the total number of 2D image descriptors that match, and the total number of 3D image descriptors that match based on matching information for the same geographical location received from multiple vehicles. Additionally, each 3D image descriptor, and in some cases, each cluster center associated with each 3D image descriptor, can be associated with a weighting value (e.g., a coefficient value, an importance coefficient). The cloud computing system can adjust the weighting value based on the frequency with which a 3D image descriptor is successfully matched (e.g., a vehicle successfully matches a 3D image descriptor corresponding to a geographical location with a 3D image descriptor of the received descriptor clustering data). For example, when a 3D image descriptor is successfully matched, the weighting value of the 3D image descriptor and the weighting value of the cluster center corresponding to the 3D image descriptor can be increased, while when the 3D image descriptor is not successfully matched, it can be decreased. The greater the weighting value of a 3D image descriptor, the more cluster centers can be retained, thereby increasing the matching opportunity for the corresponding 3D points. The greater the weighting value of a cluster center, the more likely the next iteration of the cluster center will be close to the cluster center, and the more likely the cluster center can be matched.

[0021] In some cases, the matching performance includes a statistical measure of a number of features that are successfully matched with at least one descriptor cluster center. For example, the matching performance can include the ratio of the number of vehicles (e.g., over a period of time) that successfully match 3D image descriptors to the total number of vehicles (e.g., the total number of vehicles that move through the same geographical location over the period of time) that attempt to match 3D image descriptors. In some cases, the matching performance of a cluster center includes the ratio of the number of times (e.g., over a period of time) that a 3D image descriptor corresponding to the cluster center is successfully matched to the total number of times (e.g., the total number of vehicles that move through the same geographical location over the period of time multiplied by the number of 3D image descriptors of the cluster center) that the 3D image descriptor corresponding to the cluster center can be matched.

[0022] Then, the cloud computing system can recluster the weighted 3D image descriptors and, in some cases, any newly received 3D image descriptors (e.g., 3D image descriptors received for the first time) to regenerate the 3D image descriptor clustering and re - determine the cluster centers of the regenerated 3D image descriptor clustering.

[0023] To re - determine the cluster centers, in some examples, the cloud computing system determines the matching performance for each geographical location, such as a statistical metric. For example, the cloud computing system can determine the ratio of the number of times a geographical location is successfully matched (e.g., over a period of time) to the total number of times the geographical location could have been matched (e.g., over that period of time). The cloud computing system can also determine the ratio of the number of times each cluster center associated with a geographical location is successfully matched (e.g., over a period of time) to the total number of times each cluster center could have been matched (e.g., over that period of time).

[0024] Based on the matching performance of each geographical location, the number of cluster centers can be increased or decreased. For example, if the matching performance of a geographical location meets a predetermined criterion, the number of cluster centers can be increased (e.g., since the current cluster centers are stable). Otherwise, if the matching performance of a geographical location does not meet the predetermined criterion, the number of cluster centers can be decreased (e.g., to allow for more stable cluster centers). In some cases, the predetermined criterion indicates that the ratio of the number of times a geographical location is successfully matched is higher than a first threshold. In some cases, additionally or alternatively, the predetermined criterion can identify that the ratio of the number of times each cluster center associated with a geographical location is successfully matched is higher than a second threshold, or that the average of all ratios of the clusters is higher than the second threshold.

[0025] Furthermore, once the cluster centers are established, the vehicle monitoring system can perform operations to regenerate descriptor clustering data that characterizes the similarity between the 3D image descriptors and the cluster centers of each 3D image descriptor cluster. Thus, embodiments can provide a process for iteratively improving the descriptor clustering data for geographical locations.

[0026] Among other advantages, embodiments reduce storage requirements by, at least, not requiring the storage of all descriptors associated with geographical locations, but rather relying on descriptor clustering data that characterizes the similarity between descriptors and cluster centers. Additionally, compared to conventional techniques, embodiments can require fewer processing resources (e.g., power and time) to match descriptors and can allow for more accurate and efficient object detection by employing an iterative process of updating the descriptor clustering data based on newly acquired descriptors (e.g., feedback data). Those of ordinary skill in the art with the present disclosure can also recognize these and other advantages of the embodiments.

[0027] Figure 1FIG. 0 is a block diagram of a vehicle monitoring system 100 that includes an advanced driver assistance (ADAS) system 102 for a vehicle 109 and a cloud computing system 180. Each of the ADAS system 102 and the cloud computing system 180 can be operably connected to one or more communication networks, such as communication network 150, and interconnected across the one or more communication networks. Examples of the communication network 150 include, but are not limited to, a wireless local area network (LAN) (e.g., a "Wi-Fi" network), a network utilizing a radio frequency (RF) communication protocol, a near field communication (NFC) network, a wireless metropolitan area network (MAN) connecting multiple wireless LANs, and a wide area network (WAN) (e.g., the Internet).

[0028] It should be understood that Figure 1 the specific configurations of the components shown in FIG. 0 and the communication interfaces between different components are merely exemplary, and other configurations of the components and / or other vehicle monitoring systems having the same or different components can also be configured to implement the operations and processes of the present disclosure.

[0029] The ADAS system 102 can include one or more processors 112, one or more sensors 117, a transceiver 119, a global positioning system (GPS) device 110, a display interface 126 communicatively coupled to a display 128, a memory controller 124, a system memory 130, and an instruction memory 132, which are configured to communicate with each other via a bus 129. The bus 129 can include any one of various bus structures, such as a third-generation bus (e.g., a HyperTransport bus or an InfiniBand bus), a second-generation bus (such as an Advanced Graphics Port bus, a Peripheral Component Interconnect (PCI) Express bus, or an Advanced eXtensible Interface (AXI) bus), or other types of buses or device interconnections.

[0030] At least some of the functions of the ADAS system 102 can be implemented in one or more processors, one or more field-programmable gate arrays (FPGAs), one or more application-specific integrated circuits (ASICs), one or more state machines, digital circuits, any other suitable circuits, or any suitable hardware.

[0031] The processor 112 can include any suitable processor, such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, or any other suitable processor. The processor 112 can be configured to execute instructions to perform one or more operations described herein. For example, the processor 112 can read instructions from the instruction memory 132 and execute the instructions to perform operations.

[0032] Sensor 117 may include, for example, one or more optical sensors, such as a camera, that is configured to capture images of the environment of vehicle 109. For example, sensor 117 may be a camera that is configured to capture images of the environment of vehicle 109. For example, the camera may capture images of tree 137, building 139, and / or road 135 on which vehicle 109 is traveling. In some examples, the camera may have a field of view in any direction relative to vehicle 109, such as a forward field of view, a rearward field of view, a lateral field of view, an inclined field of view, or any other suitable field of view. Sensor 117 may capture the images and may store the images in a memory device (e.g., an internal memory device, system memory 130, etc.). Processor 112 may obtain the captured images from the memory device or, in some examples, directly from sensor 117.

[0033] GPS device 110 may generate position data representative of the position of vehicle 109 based on GPS. For example, processor 112 may receive data representative of, for example, the latitude and longitude of the position of GPS device 110 from GPS device 110. Additionally, transceiver 119 is configured to receive data from and transmit data to communication network 150. Additionally, display interface 126 is configured to output a signal that causes graphical data to be displayed on display 128 (e.g., a dashboard display).

[0034] Memory controller 124 provides access to system memory 130 and instruction memory 132. System memory 130 may store program modules and / or instructions and / or data accessible to processor 112. For example, system memory 130 may store user applications (e.g., instructions for a camera application) and the resulting images from sensor 117. System memory 130 may also store rendered images, such as three-dimensional (3D) images, rendered by processor 112. System memory 130 may additionally store information used by and / or generated by other components of ADAS system 102. For example, system memory 130 may act as the device memory for processor 112. Examples of system memory 130 include one or more volatile or non-volatile memory or storage devices, such as RAM, SRAM, DRAM, EPROM, EEPROM, flash memory, magnetic data media, cloud-based storage media, or optical storage media.

[0035] The instruction memory 132 may store instructions that can be accessed (e.g., read) and executed by one or more processors 112. For example, the instruction memory 132 may store instructions that cause one or more processors 112 to perform one or more operations described herein when executed by the one or more processors 112. For example, the instruction memory 132 may include instructions that cause one or more processors 112 to perform the following operations when executed by the one or more processors 112: applying one or more feature detection processes (e.g., machine learning processes) to a captured image to detect features and associating the detected features with descriptor cluster centers identified within received descriptor cluster data.

[0036] For example, the feature detection model data 132A may include instructions that cause one or more processors 112 to perform the following operations when executed by the one or more processors 112: applying a feature detection process to an image (such as an image captured by the sensor 117) to determine 2D features. The descriptor generation model data 132B may include instructions that cause one or more processors 112 to perform the following operations when executed by the one or more processors 112: applying a descriptor generation process to the 2D features to generate 3D image descriptors. Additionally, the descriptor matching model data 132C may include instructions that cause one or more processors 112 to perform the following operations when executed by the one or more processors 112: matching the 3D image descriptors with descriptor cluster centers (e.g., descriptor cluster centers within received descriptor cluster data) and identifying an object or object features based on the matched 3D image descriptors.

[0037] In some examples, the ADAS system 102 includes a pose device, such as a head-mounted display (HMD) device, that may measure the viewing direction of the head of the driver of the vehicle 109. For example, the sensor 117 may include one or more of a gyroscope, an accelerometer, an inertial measurement unit, and any other type of sensor that may be configured to detect, measure, or generate sensor data associated with the position and / or orientation (e.g., head pose) of the driver's head. In some cases, the sensor 117 includes a camera that is positioned to capture an image of the field of view in the direction the driver's head is facing.

[0038] The cloud computing system 180 may include one or more servers 180A communicatively coupled to a communication network 150. The server 180A may be any suitable computing device. For example, the server 180A may include one or more processors 179, which may execute instructions stored in an instruction memory 178. In this example, the instruction memory 178 includes a clustering engine 182, a matrix generation engine 184, and a performance determination engine 186. In some cases, the server 180A executes a hypervisor that maintains virtual machines, where one or more of the clustering engine 182, the matrix generation engine 184, and the performance determination engine 186 are executed by one or more of the virtual machines.

[0039] The clustering engine 182 may include instructions that, when executed by one or more of the processors 179, cause one or more of the processors 179 to apply a first clustering process to a plurality of 3D image descriptors to determine a number of clusters. When the instructions are executed by one or more of the processors 179, the instructions may also cause one or more of the processors 179 to apply a second clustering process to the number of clusters to determine a cluster center for each of the number of clusters. Additionally, the matrix generation engine 184 may include instructions that, when executed by one or more of the processors 179, cause one or more of the processors 179 to generate descriptor clustering data that characterizes the similarity (e.g., probability) between the plurality of descriptors and the cluster centers. The performance determination engine 186 may include instructions that, when executed by one or more of the processors 179, cause one or more of the processors 179 to determine the matching performance of 3D descriptors for a geographical location (e.g., a 3D reference point) based on matching information received from, for example, a plurality of vehicles 109.

[0040] For example, the vehicle 109 may travel along a road 135. The sensors 117 of the vehicle 109 may capture images that include, for example, portions of the road 135, portions of a tree 137, and / or portions of a building 139. As described herein, the processor 112 may generate 2D image features based on the captured images and may generate 3D descriptors based on the 2D image features. Additionally, the processor 112 may obtain position data that characterizes the position of the vehicle 109 (e.g., GPS coordinates) from a GPS device 110. Further, the processor 112 may receive descriptor clustering data corresponding to the position of the vehicle 109 from the cloud computing system 180 via a transceiver 119 and through the communication network 150. As described herein, the cloud computing system 180 may maintain descriptor clustering data, such as a descriptor clustering matrix, for each geographical location (e.g., a 3D reference point), where the descriptor clustering data for each geographical location characterizes the similarity (e.g., proximity, probability) between the clustering descriptors and the corresponding cluster centers.

[0041] In addition, the processor 112 can associate 2D image features with the cluster centers based on the descriptor clustering data. For example, the vehicle can match the 3D image descriptors generated from the 2D image features with the cluster centers of the descriptor clustering data. Based on the match, the vehicle can associate the 2D features with the matched cluster centers of the descriptor clustering data. For example, the determined cluster centers can correspond to the corners of the building 139, the tree 137, the road markings of the road 135, or some other object or object feature. The processor 112 can perform operations to identify objects (such as the building 139, the tree 137, the road markings of the road 135) based on the match. In addition, based on the match, the vehicle 109 can perform one or more operations. For example, the vehicle 109 can perform positioning and / or alignment. For example, the vehicle 109 can be an autonomous vehicle, and based on object detection, can perform operations to stay within the identified road markings of the road 135, or can perform operations to avoid objects.

[0042] Although Figure 1 the components and operations are described with respect to the vehicle monitoring system 100, in other examples, other systems and / or devices can include the same or similar components and implement some or all of the operations described herein. For example, in some examples, an extended reality (XR) system (such as an augmented reality (AR) system, a virtual reality (VR) system, or a mixed reality (MR) system) can include a database of descriptor clustering data that characterizes the similarity between the representation descriptors and the descriptor cluster centers as described herein. An XR device (e.g., such as a head-mounted display (HMD) device) can capture an image (e.g., a real-world scene) and can generate a descriptor based on the captured image. In addition, the XR device can obtain the descriptor clustering data from the database. As described herein, the descriptor clustering data can include multiple values that characterize the similarity between the representation descriptors and the descriptor cluster centers. For example, each of the multiple values can characterize the similarity between one of the multiple descriptors and one of the descriptor cluster centers. The XR device can then determine the features within the captured image based on the generated descriptor and the descriptor clustering data. For example, the XR device can match the generated descriptor with one of the descriptor cluster centers and determine the features based on the match as described herein. The XR device can determine one or more objects based on the determined features and can project an image including the one or more objects on a display (e.g., in the case of providing an image for viewing in the extended reality space).

[0043] Figure 2 is shown including the ADAS system 102 and the cloud computing system 180 Figure 1A diagram of an exemplary portion of a vehicle monitoring system 100. In some cases, one or more operations performed by the ADAS system 102 and the cloud computing system 180 are performed as part of a SLAM process. In this example, the ADAS system 102 includes a feature detection engine 202, a descriptor generation engine 204, and a descriptor matching engine 206. In some examples, each of the feature detection engine 202, the descriptor generation engine 204, and the descriptor matching engine 206 may include instructions that, when executed by one or more processors 112, cause one or more of the processors 112 to perform corresponding operations. For example, the feature detection engine 202 may include feature detection model data 132A, the descriptor generation engine 204 may include descriptor generation model data 132B, and the descriptor matching engine 206 may include descriptor matching model data 132C.

[0044] In addition, the cloud computing system 180 includes a clustering engine 238, a matrix generation engine 240, and a performance determination engine 228. In some examples, one or more of the feature detection engine 202, the descriptor generation engine 204, the descriptor matching engine 206, the clustering engine 238, the matrix generation engine 240, and the performance determination engine 228 may be implemented in hardware, such as within one or more FPGAs, ASICs, digital circuits, or any other suitable hardware, or a combination of hardware and software.

[0045] The cloud computing system 180 also includes a memory 252, which may be a ROM, RAM, hard drive, disk drive, cloud-based storage device, or any other suitable memory. The memory 252 stores descriptor matching data 207, location-based descriptor data 230, and 3D reference point performance data 229, as well as other data. As described further below, descriptor matching data 207 may be received from the vehicle 109 as the vehicle 109 travels through various geographical locations.

[0046] The descriptor matching data 207 may include 3D reference points 207A that identify geographical locations (e.g., the location corresponding to the captured image), matching descriptors 207B that characterize descriptors that have successfully matched to features, matching cluster centers 207C that characterize the cluster centers of descriptor cluster data 230C to which descriptors that have matched to features belong, unmatched descriptors 207E that characterize descriptors that do not match, and / or descriptors (e.g., new image descriptors) generated by the vehicle 109 that cannot be matched to the cluster centers of the descriptor cluster data 230C, as well as matching data 207D that characterizes descriptor matching information, such as the ratio of the total number of successful matches between the 3D descriptors of the descriptor cluster data 230C and the 3D image descriptors of the descriptor cluster data 230C corresponding to the geographical location.

[0047] The location-based descriptor data 230 may include 3D reference points 230A that identify one or more geographical locations, compressed combined descriptor data 230B that represents one or more compressed combined descriptors for each corresponding 3D reference point 230A, descriptor clustering data 230C that represents one or more membership matrices for each corresponding 3D reference point 230A, and weighting values 230D associated with the compressed combined descriptor data 230B. As described herein, 3D point descriptors may be generated based on multiple 2D observations and the corresponding individual descriptors of their corresponding 3D points. Additionally, the number of cluster centers may be determined based on weighting values (e.g., importance coefficients), where the clustered 3D point descriptors may be referred to as compressed combined descriptors.

[0048] In this example, the sensor 117 captures an image, such as an image of the environment of the vehicle 109, and generates image data 201 that represents the captured image. Additionally, the feature detection engine 202 receives the image data 201 and applies a feature detection process to the image data 201 to detect features and generate image feature data 203 that represents the detected features. For example, the feature detection process may include a machine learning process that is trained to identify features within the image. In some examples, the feature detection process includes establishing a deep learning model or a convolutional network to detect features from the image data 201. For example, the feature data 203 may identify 2D image features within the image data 201.

[0049] The descriptor generation engine 204 receives the feature data 203 and applies a feature extraction process to the feature data 203 to generate descriptor data 205 that represents image descriptors. For example, the descriptor generation engine 204 may establish an oriented gradient histogram (HOG) feature extraction process, a speeded-up robust features (SURF) feature extraction process, or any other suitable feature extraction process, and apply the established feature extraction process to the feature data 203 to generate the descriptor data 205. For example, the descriptor data 205 may represent 2D image descriptors.

[0050] As described herein, the cloud computing system 180 can generate descriptor clustering data 230C that characterizes the similarity between descriptors (e.g., 3D image descriptors) and descriptor cluster centers. For example, referring to the cloud computing system 180, the clustering engine 182 can obtain a plurality of 3D descriptors from the memory 252, such as matching descriptors 207B and / or non-matching descriptors 207E. The clustering engine 182 can apply a first clustering process to the plurality of 3D descriptors to determine a number of descriptor clusters corresponding to a geographical location (such as the 3D reference point 207A). As described herein, the plurality of descriptors may have been generated by the SLAM process based on 2D image features generated by one or more vehicles 109 passing through the same geographical location (e.g., as identified by the corresponding 3D reference point 207A). In some cases, the first clustering process can include establishing an adaptive model that determines the number of descriptor clusters based on coefficient values (e.g., importance coefficients) corresponding to each geographical location. As described herein, the coefficient values can be determined based on 3D reference point performance data 229 that characterizes the matching performance, which is the matching performance of a geographical location (such as a 3D reference point). For example, a 3D point can be associated with a corresponding weighted value (e.g., importance coefficient). The higher the weighted value of the 3D point, the more cluster centers the first clustering process can generate and associate with the 3D point. In addition, the weighted value associated with each 3D point can be based on, for example, the number of observers of each 3D point, and the calculated distance from the camera that captured the corresponding image to the 3D point (e.g., the distance in 3D space from the camera that captured the image to the 3D point). The clustering engine 182 can calculate the distance based on, for example, the current position of the vehicle 109 and the 3D point.

[0051] In addition, the clustering engine 182 can apply a second clustering process to the number of descriptor clusters to determine an initial descriptor cluster center for each of the descriptor clusters. For example, the clustering engine 182 can apply a k-means clustering process, such as the k-means++ process, to the number of descriptor clusters to generate an initial descriptor cluster center. Based on the first clustering process and the second clustering process, the clustering engine 182 generates cluster center data 239 that characterizes the initial descriptor cluster center.

[0052] The matrix generation engine 184 receives the cluster center data 239 from the clustering engine 182 and further obtains 3D descriptors, such as the matching descriptor 207B and / or the non-matching descriptor 207E, from the memory 252. Based on the descriptor cluster centers identified in the cluster center data 239 and the multiple 3D descriptors obtained from the memory 252, the matrix generation engine 185 generates compressed combined descriptor data 230B, which characterizes the compressed combined descriptor (e.g., the subsequent descriptor cluster center) for a geographical location (e.g., for the corresponding 3D reference point 207A). The matrix generation engine 185 may store the compressed combined descriptor data 230B and its corresponding 3D reference point 230A (which may correspond to the 3D reference point 207A from which the cluster center data 239 is generated) within the memory 252. In some cases, once the compressed combined descriptor data 230B is generated, the matrix generation engine 185 deletes the corresponding multiple 3D descriptors, such as the matching descriptor 207B and / or the non-matching descriptor 207E, from the memory 252, thus saving memory space. In some examples, after each clustering process, the center and descriptor cluster matrices are retained, and the previous cluster centers and the 2D descriptors of the current iteration are deleted from the memory.

[0053] In addition, the matrix generation engine 185 may generate descriptor cluster data 230C that characterizes the similarity between the descriptor cluster center of the compressed combined descriptor data 230B and the multiple 3D descriptors. For example, the descriptor cluster data 230C may include descriptor cluster data that characterizes the descriptor cluster center and may also include probability values that characterize the probability that each of the multiple 3D descriptors belongs to each of the descriptor cluster centers. The probability values may be based on, for example, the fuzzy c-means (FCM) algorithm. In some examples, the descriptor cluster data 230C includes a matrix of probability values, where each element of the matrix includes a probability value that characterizes the probability that a 3D descriptor belongs to a descriptor cluster center. The matrix generation engine 185 may store the descriptor cluster data 230C within the memory 252.

[0054] In addition, as described herein, the cloud computing system 180 may send the descriptor cluster data 230C to the vehicle 109, such as the vehicle 109 traveling through the geographical location corresponding to the descriptor cluster data 230 (e.g., the 3D reference point 230A). For example, the vehicle 109 may determine its approximate location (e.g., via GPS) and may send its approximate location to the cloud computing system 180. In response, the cloud computing system 180 may determine one or more 3D reference points 230A corresponding to the approximate location and send the descriptor cluster data 230C corresponding to each of the one or more 3D reference points 230A to the vehicle 109.

[0055] As Figure 2As shown, the descriptor matching engine 206 receives descriptor data 205 from the descriptor generation engine 204, which may characterize 2D image descriptors captured for a specific geographic location. Additionally, the descriptor matching engine 206 may receive descriptor cluster data 230C corresponding to a specific geographic location (identified, e.g., by the 3D reference point 230A) from the cloud computing system 180. The descriptor matching engine 206 performs any of the operations described herein to match the 2D image descriptors identified within the descriptor data 205 with the descriptor cluster centers identified by the descriptor cluster data 230C based on the probability values of the descriptor cluster data 230C.

[0056] For example, the descriptor matching engine 206 may calculate the distance between each 2D image descriptor and each descriptor cluster center of the descriptor cluster data 230C, such as the Euclidean distance. Additionally, the descriptor matching engine 206 may determine the amount of similarity (e.g., membership degree) between the calculated distance of each 2D image descriptor and the descriptor cluster center based on the probability values of the descriptor cluster data 230C. For example, the descriptor matching engine 206 may multiply the calculated distance between the 2D image descriptor and the descriptor cluster center by the probability value corresponding to the same descriptor cluster center to determine the similarity between the calculated distance of each 2D image descriptor and the descriptor cluster center. The descriptor matching engine 206 may determine the descriptor cluster center most similar to each 2D image descriptor (e.g., the closest descriptor cluster center) based on the amount of similarity, thereby matching each 2D image descriptor with the closest descriptor cluster center. For example, a vehicle (such as vehicle 109) may match the cluster center of the 3D reference points with the individual descriptors of the 2D points (e.g., those from the camera frames of a moving vehicle trying to locate itself) based on the descriptor cluster data 230C to obtain a more reasonable matching result. When calculating the distance from an individual descriptor (seen in the camera frame) to the cluster of compressed combined descriptors of the 3D reference points, the descriptor matching engine 206 may calculate the distance to each cluster center separately and then may multiply by the membership degree. Once the distances to each cluster center are calculated, the descriptor matching engine 206 selects the cluster center with the minimum distance as the final result.

[0057] Based on the matching, vehicle 109 may identify objects within the captured image. For example, as part of the SLAM process, vehicle 109 (e.g., using the processor 112) may perform alignment operations (e.g., obstacle avoidance procedures), path planning operations, and / or positioning operations based on the identified objects. For example, vehicle 109 may perform a positioning operation to determine the vehicle's position in the real world (e.g., relative to an object) and determine the vehicle's position within a map (such as a high-definition (HD) map) based on that positioning (e.g., map operations).

[0058] In addition, the descriptor matching engine 206 can generate descriptor matching data 207 that characterizes the matching 2D image descriptors. For example, in some cases, the descriptor matching data 207 can identify one or more of the matching 2D image descriptors 207B, the matching cluster centers 207C, and the corresponding 3D reference points 207A. The descriptor matching data 207 can also include matching data 207D, which can characterize one or more of a number of the matching 3D reference points 207A. The descriptor matching data 207 can also include unmatched descriptors 207E, which can include unmatched 2D image descriptors (e.g., new image descriptors). The vehicle 109 can send the descriptor matching data 207 to the cloud computing system 180.

[0059] The cloud computing system 180 can receive descriptor matching data 207 for one or more locations from one or more vehicles, and can adjust (e.g., update) the corresponding location-based descriptor data 230 based on the received descriptor matching data 207. For example, the performance determination engine 186 can receive descriptor matching data 207, which, as described herein, can include one or more of the 3D reference points 207A, the matching descriptors 207B, the matching cluster centers 207C, and the matching data 207D, and can store the descriptor matching data 207 in the memory 252.

[0060] In addition, the performance determination engine 186 can determine the matching performance of each 3D reference point 230A based on the descriptor matching data 207, and can regenerate (e.g., update) the descriptor cluster data 230C for the 3D reference point 230A in the memory 252 based on the matching performance. For example, each compressed combined descriptor (e.g., as identified by the compressed combined descriptor data 230B) can be associated with a weighting value 230D (e.g., a coefficient value). The performance determination engine 186 can adjust the weighting value based on the frequency with which each compressed combined descriptor is successfully matched (e.g., the vehicle successfully matches the 2D image descriptor with the compressed combined descriptor of the received descriptor cluster data 230C). For example, as indicated by the descriptor matching data 207, the weighting value of the compressed combined descriptor can be increased when the compressed combined descriptor is successfully matched, and can be decreased when the compressed combined descriptor is not successfully matched.

[0061] In some cases, the performance determination engine 186 determines the matching performance of the compressed combination descriptor. For example, the performance determination engine 186 may determine the ratio of the number of vehicles that successfully match the compressed combination descriptor (e.g., over a period of time) to the total number of vehicles attempting to match the compressed combination descriptor (e.g., the total number of vehicles passing through the same geographical location during that period of time). In some cases, the performance determination engine 186 determines, for a compressed combination descriptor, the ratio of the number of times the 3D image descriptor corresponding to the compressed combination descriptor is successfully matched (e.g., over a period of time) to the total number of times the 3D image descriptor corresponding to the compressed combination descriptor could have been matched (e.g., the total number of vehicles passing through the same geographical location during that period of time multiplied by the number of 3D image descriptors of the descriptor cluster center).

[0062] In some examples, the performance determination engine 186 determines the matching performance of the 3D reference point 230A. For example, the performance determination engine 186 may determine the ratio of the number of vehicles that successfully match the 3D reference point 230A (e.g., over a period of time) to the total number of vehicles attempting to match the 3D reference point 230A (e.g., the total number of vehicles passing through the same geographical location during that period of time). In some examples, the performance determination engine 186 determines the matching performance of the 3D reference point 230A based on the number of observers (e.g., vehicle 109) for each 3D reference point 230A that is determined and the distance of each 3D reference point 230A from the camera, as described herein.

[0063] Based on the matching performance of each geographical location, the performance determination engine 186 may determine whether the matching performance of the geographical location meets a predetermined criterion. If the predetermined criterion is met, the number of cluster centers may be increased (e.g., since the current cluster centers are stable). Otherwise, if the matching performance of the geographical location does not meet the predetermined criterion, the number of cluster centers may be decreased (e.g., to allow for more stable cluster centers). In some cases, the predetermined criterion includes the ratio of the number of times the 3D reference point 230A is successfully matched, and whether that ratio is higher than a first threshold. In some cases, additionally or alternatively, the predetermined criterion may include the ratio of the number of times each descriptor cluster center associated with the 3D reference point 230A is successfully matched, and whether that ratio is higher than a second threshold. In some examples, additionally or alternatively, the predetermined criterion may indicate whether the average value of the ratios of all descriptor cluster centers corresponding to the 3D reference point 230A is higher than a third threshold.

[0064] In addition, the performance determination engine 186 may generate 3D reference point performance data 229 characterizing one or more of these matching performance determinations, and store the 3D reference point performance data 229 in the memory 252.

[0065] In some cases, the clustering engine 182 clusters (e.g., reclusters) the matching descriptors 207B received within the descriptor matching data 207 with the compressed combined descriptors identified within the compressed combined descriptor data 230B to generate updated cluster center data 239 as described herein. For example, the clustering engine 182 may apply a first clustering process to the matching descriptors 207B and the compressed combined descriptors to determine updated clusters of descriptors corresponding to a geographical location (e.g., 3D reference point 207A).

[0066] In some examples, the first clustering process includes determining the updated clusters of descriptors at least in part based on 3D reference point performance data 229. For example, based on the matching performance of each 3D reference point 207A identified by the corresponding 3D reference point performance data 229 (e.g., the proportion of vehicles that can match the 3D reference point 207A, the proportion of times the 3D reference point 207 is successfully matched, the proportion of times each cluster center associated with the 3D reference point 207A is successfully matched, the average value of all proportions for the clusters, etc.), the number of cluster centers of the descriptors can be increased or decreased. For example, if the matching performance of the 3D reference point 207A meets a predetermined criterion (e.g., is at or above a corresponding threshold), the number of cluster centers can be increased (e.g., because the current cluster centers are stable). Otherwise, if the matching performance of the 3D reference point 207A does not meet the predetermined criterion (e.g., is below the corresponding threshold), the number of cluster centers can be decreased (e.g., to allow for more stable cluster centers).

[0067] In addition, the clustering engine 182 may apply a second clustering process to the updated clusters of descriptors to determine updated cluster centers for each of the updated clusters of descriptors. Based on the first clustering process and the second clustering process, the clustering engine 182 generates updated cluster center data 239 characterizing the updated cluster centers of the descriptors.

[0068] In addition, the matrix generation engine 184 may receive the updated cluster center data 239 characterizing the updated cluster centers of the descriptors and may generate updated compressed combined descriptor data 230B and updated descriptor cluster data 230C based on the updated cluster center data 239. The matrix generation engine 184 may store the updated compressed combined descriptor data 230B and the updated descriptor cluster data 230C in the memory 252 (e.g., by overwriting the corresponding previous compressed combined descriptor data 230B and descriptor cluster data 230C).

[0069] Figure 3Illustrates an iterative descriptor clustering process 300 that generates descriptor clustering data, such as a descriptor clustering matrix 330, based on clustering 3D reference points 304 generated from images of an image sequence 302. As shown, 2D image descriptors 310 are determined based on the images of the image sequence 302. For example, as described herein, the 3D reference points 304 may be associated with various 2D reference points of the image sequence 302. The image sequence 302 may include, for example, images captured by a vehicle 109. Additionally, each 3D reference point 304 may be associated with a plurality of 3D image descriptors 314. At a cloud computing system, such as cloud computing system 180, a first clustering process 311, such as a k-means++ process, may be applied to the plurality of 3D image descriptors 314 and any current descriptor cluster centers 312 (e.g., determined by a previous iteration of the iterative descriptor clustering process 300) to determine a number of initial descriptor cluster centers 320 for the corresponding 3D reference points 304.

[0070] Additionally, at cloud computing system 180, a second clustering process 313, such as a fuzzy c-means clustering process, is applied to the initial descriptor cluster centers 320 and the plurality of 3D image descriptors 314 to generate a descriptor clustering matrix 330 that characterizes a compressed combined descriptor for each of a plurality of final cluster centers. For example, each column of the descriptor clustering matrix 330 represents a final cluster center (e.g., as represented by "a", "b", "c", and "d"), and each row of the descriptor clustering matrix 330 represents a 3D point descriptor 314 (e.g., as represented by "1" to "n"). Additionally, the descriptor clustering matrix 330 includes probability values for each pair of a final cluster center and a 3D point descriptor 314 that characterize the probability that each 3D point descriptor 314 belongs to each final cluster center. For example, the element "a1" includes a probability value that characterizes the probability that the 3D point descriptor "1" belongs to the final cluster center "a", and the element "dn" includes a probability value that characterizes the probability that the 3D point descriptor "n" belongs to the final cluster center "d".

[0071] The cloud computing system 180 may send the descriptor clustering matrix 330 to a vehicle, such as vehicle 109, traveling through a location corresponding to the 3D reference points 304.

[0072] Vehicle 109 may receive the descriptor clustering matrix 330 and may perform operations (e.g., a fuzzy 3D-2D matching operation) to match the 2D image descriptors of the images of the image sequence 302 with the descriptor cluster centers (e.g., "a", "b", "c", or "d") of the descriptor clustering matrix 330. For example, as described herein, vehicle 109 may apply a feature detection process to the images of the image sequence 302 to detect 2D image features. Additionally, vehicle 109 may apply a descriptor generation process to the detected 2D image features to generate 2D image descriptors 310.

[0073] Vehicle 109 may then perform any of the operations described herein to determine the distance between each 2D image descriptor 310 and each descriptor cluster center, such as the Euclidean distance, and may match the 2D image descriptor 310 with the nearest descriptor cluster center (shortest computed distance).

[0074] In some examples, vehicle 109 may generate a 2D descriptor matrix that characterizes the distances from one or more 2D image descriptors 310 to each descriptor cluster center. Additionally, vehicle 109 may multiply the 2D descriptor matrix by the probability values of the transpose of the descriptor clustering matrix 330 to determine distance values, and may also determine the minimum distance value among the distance values. Vehicle 109 may match each 2D image descriptor 310 with the descriptor cluster center corresponding to the minimum distance value.

[0075] As described herein, vehicle 109 may perform operations such as SLAM operations (e.g., alignment, localization, etc.) based on the descriptor cluster centers that match the 2D image descriptors 310. For example, vehicle 109 may perform SLAM operations in response to the match. In some examples, vehicle 109 may determine objects (e.g., corners of building 139, tree 137, road markings on road 135, etc.) based on the match, and may perform SLAM operations based on the determined objects.

[0076] Figure 4 A messaging diagram 400 between vehicle 109 and server 180A of the cloud computing system 180 is shown. Vehicle 109 may travel along a road such as road 135 and may determine an initial position. For example, vehicle 109 may determine its position based on GPS (e.g., using GPS device 110). Vehicle 109 may generate estimated position data 402 that characterizes the initial position, and may send the estimated position data 402 to server 180A of the cloud computing system 180.

[0077] In addition, based on the estimated location data 402, the server 180A can perform any of the processes described herein to generate descriptor cluster array data that characterizes a descriptor cluster matrix (such as the descriptor cluster matrix 330) for that location. As described herein, the descriptor cluster matrix can identify and characterize one or more probability values that characterize the probability that each of a plurality of 3D descriptors belongs to each of a plurality of descriptor cluster centers. The vehicle 109 can receive the descriptor cluster matrix data 404 and can perform 2D-3D point matching operations to match the 2D features with the plurality of descriptor cluster centers. For example, the vehicle 109 can capture one or more images of the location identified by the estimated location data 402 and can determine 2D image features based on the captured images, as described herein. In addition, as described herein, the vehicle 109 can perform any of the operations described herein to match each 2D image feature with one of the plurality of descriptor cluster centers of the descriptor cluster matrix 404 based on the probability values of the descriptor cluster matrix data 404.

[0078] In addition, based on the 2D-3D point matching 406, the vehicle 109 can perform one or more operations, such as SLAM operations. For example, the vehicle 109 can identify one or more objects based on the 2D-3D point matching 406. Based on the identified objects, the vehicle 109 can perform SLAM localization and / or SLAM alignment. In some examples, the vehicle 109 can send data to the server 180A, such as data identifying the object type, and any SLAM operations performed based on the identified objects, such as localization and / or alignment.

[0079] The vehicle 109 can also generate descriptor matching data 410 that characterizes the matching of a 2D image feature with one of the plurality of descriptor cluster centers of the descriptor cluster matrix data 404. For example, as described herein, the descriptor matching data 410 can identify the 3D reference point 207A, the matching descriptor 207B, the matching cluster center 207C, and the matching data 207D. The vehicle 109 can send the descriptor matching data 410 to the server 180A.

[0080] Server 180A may receive descriptor matching data 410 and may perform operations to cluster the descriptor cluster centers of the descriptor cluster matrix data with the matching image descriptors identified in the descriptor matching data 410. For example, Server 180A may apply a first clustering process, such as a k-means++ process, to the descriptor cluster centers and the image descriptors to determine a number of updated descriptor cluster centers for the corresponding 3D reference points. In some cases, the first clustering process includes weighting the descriptor cluster centers based on the matching data 207D, as described herein. Additionally, Server 180A may apply a second clustering process, such as a fuzzy c-means clustering process, to the number of updated descriptor cluster centers and the image descriptors to generate a compressed combined descriptor for each of the multiple final cluster centers. For example, Server 180A may replace any individual image descriptors of the 3D reference points with the compressed combined descriptors within the memory 252.

[0081] Additionally, based on the compressed combined descriptors, vehicle 109 may perform any of the operations herein to generate descriptor cluster matrix data 414 that characterizes a descriptor cluster matrix (such as descriptor cluster matrix 330). The descriptor cluster matrix may include probability values that characterize the probability that each image descriptor belongs to each of the multiple final cluster centers.

[0082] Figure 5 is a flowchart of an exemplary process 500 for clustering descriptors such as image descriptors. For example, one or more computing devices (such as Server 180A) may perform one or more operations of the exemplary process 500 as described below with reference to Figure 5 as described.

[0083] Refer to Figure 5 , at block 502, Server 180A may apply a first clustering process to a plurality of descriptors to determine a number of clusters. For example, as described herein, Server 180A may apply a k-means clustering process (such as a k-means++ process) to the plurality of descriptors to determine a number of clusters. For example, Server 180A may determine a number of cluster centers based on corresponding coefficient values (such as importance coefficients) and may apply a k-means++ model according to the determined number of cluster centers to generate initial cluster centers (i.e., execute the k-means++ model and apply the executed k-means++ model to the plurality of descriptors to generate a number of cluster centers).

[0084] In addition, at block 504, server 180A applies a second clustering process to a number of clusters and a plurality of descriptors to determine a compressed combined descriptor for each of the number of clusters. For example, as described herein, server 180A applies a fuzzy c-means clustering process to the number of clusters and the plurality of descriptors to determine compressed combined descriptor data 230B. At step 506, server 180A generates a descriptor-cluster matrix based on the compressed combined descriptors and the plurality of descriptors. The descriptor-cluster matrix characterizes the probability that each of the plurality of descriptors belongs to each of the compressed combined descriptors. For example, server 180A may generate a descriptor-cluster matrix 330 that characterizes the probability that any one of descriptors 1–n belongs to any of the descriptor clusters “a”, “b”, “c”, and “d”.

[0085] Moving to block 508, server 180A receives matching data that characterizes a match between a descriptor and at least one of the number of clusters. For example, server 180A may receive descriptor matching data 207 from at least one vehicle 109, which may include one or more of a 3D reference point 207A that identifies a geographic location (e.g., the location corresponding to the captured image), a matching descriptor 207B of a descriptor that characterizes the descriptor cluster data 230C of a successful match, a matching cluster center 207C that characterizes the descriptor cluster center of the match, and matching data 207D that characterizes the descriptor matching information. For example, vehicle 109 may generate descriptor matching data 207 based on a match of 2D image features to the descriptor cluster centers of descriptor-cluster matrix 330 and may send descriptor matching data 207 to server 180A based on the match.

[0086] At block 510, server 180A updates a coefficient value (e.g., a weighting value) based on the matching data. For example, when the matching data indicates that the descriptor cluster center has successfully matched a 3D reference point by more than a threshold percentage (e.g., over a period of time, such as a month), server 180A may increase the coefficient value of the descriptor cluster center, and when the matching data indicates that the descriptor center has not successfully matched a 3D reference point by at least the threshold percentage, may decrease the coefficient value of the descriptor cluster center.

[0087] Figure 6 is a flowchart of an exemplary process 600 for performing operations based on matching image features to descriptors. For example, one or more computing devices (such as one or more processors 112) may perform one or more operations of exemplary process 600, as described below with reference to Figure 6 described. In some examples, an XR device (such as an AR device, a VR device, or an MR device) may perform one or more operations of exemplary process 600.

[0088] Starting from block 602, processor 112 receives an image of the environment. For example, sensor 117 can be a camera, and the camera can capture an image of the environment of vehicle 109. Processor 112 can receive the captured image from the camera. At block 604, processor 112 applies a feature detection process to the image to determine at least one feature. For example, processor 112 can apply a trained machine learning process to the image to determine at least one feature. Additionally, at block 606, processor 112 applies a feature extraction process to the at least one feature to generate at least one descriptor. For example, and as described herein, processor 112 can apply a HOG feature extraction process to the at least one feature to generate at least one descriptor.

[0089] Proceeding to block 608, processor 112 determines an initial position. For example, processor 112 can obtain GPS data characterizing the position of vehicle 109 from GPS device 110. At block 610, processor 112 sends the initial position to a server, such as server 180A of cloud computing system 180.

[0090] Additionally, at block 612, processor 112 receives from the server a descriptor clustering matrix corresponding to the initial position. The descriptor clustering matrix characterizes the similarity between a plurality of descriptors and a number of cluster centers. For example, and as described herein, server 180A can apply one or more clustering processes to the plurality of descriptors associated with the initial position to generate a descriptor clustering matrix including probability values, such as descriptor clustering matrix 330, where each probability value is the probability that a descriptor belongs to a compressed combined descriptor.

[0091] Proceeding to block 614, processor 112 matches the at least one feature with at least one of the plurality of descriptors based on the descriptor clustering matrix. For example, processor 112 can calculate the distance between each descriptor of the plurality of descriptors and each compressed combined descriptor of the descriptor clustering matrix, such as the Euclidean distance. Additionally, processor 112 can determine the amount of similarity (e.g., membership degree) between each descriptor and each compressed combined descriptor based on the calculated distance and the corresponding probability value of the descriptor clustering matrix. Processor 112 can then match the at least one feature with the most similar compressed combined descriptor (e.g., as indicated by the highest amount of similarity among the calculated amounts of similarity).

[0092] At block 616, processor 112 may perform at least one operation based on the match. For example, the most similar compressed combination descriptor may correspond to a label of building 139, tree 137, or road 135. Based on this match, processor 112 may determine that the at least one feature is a feature of a label of building 139, tree 137, or road 135, and may perform alignment operations (e.g., obstacle avoidance procedures), path planning operations, and / or positioning operations, etc. based on the identified object.

[0093] The following numbered clauses further describe implementation examples:

[0094] 1. A device, comprising:

[0095] A non-transitory machine-readable storage medium storing instructions; and

[0096] At least one processor coupled to the non-transitory machine-readable storage medium, the at least one processor configured to execute the instructions to:

[0097] Apply a first clustering process to a plurality of descriptors associated with a geographical location to determine a number of descriptor clusters;

[0098] Apply a second clustering process to the number of descriptor clusters to determine a descriptor cluster center for each of the number of descriptor clusters;

[0099] Generate descriptor cluster data characterizing the similarity between the plurality of descriptors and the descriptor cluster centers; and

[0100] Store the descriptor cluster data in a data repository.

[0101] 2. The device according to clause 1, wherein the at least one processor is further configured to execute the instructions to generate the descriptor cluster data to include a plurality of values characterizing the similarity between the plurality of descriptors and the descriptor cluster centers, wherein each of the plurality of values characterizes the similarity between one of the plurality of descriptors and one of the descriptor cluster centers.

[0102] 3. The device according to clause 2, wherein each of the plurality of values identifies the probability that one of the plurality of descriptors belongs to one of the descriptor cluster centers.

[0103] 4. The device according to any one of clauses 1-3, wherein the at least one processor is further configured to execute the instructions to:

[0104] Receive location data characterizing the geographical location from a remote device; and

[0105] In response to receiving the location data, send the descriptor clustering data to the remote device.

[0106] 5. The apparatus according to clause 4, wherein the at least one processor is further configured to execute the instructions to:

[0107] Receive descriptor matching data from the remote device, the descriptor matching data characterizing a matching result of the descriptor clustering data; and

[0108] Adjust the descriptor clustering data in the data repository based on the descriptor matching data.

[0109] 6. The apparatus according to clause 5, wherein the descriptor matching data includes a number of descriptor matches for at least one of the descriptor clustering centers, and the at least one processor is configured to execute the instructions to:

[0110] Determine a matching performance of at least one of the descriptor clustering centers based on the number of descriptor matches; and

[0111] Adjust the descriptor clustering data based on the matching performance.

[0112] 7. The apparatus according to any one of clauses 5 - 6, wherein the at least one processor is further configured to execute the instructions to:

[0113] Weight the descriptor clustering centers based on the descriptor matching data;

[0114] Apply the first clustering process to the plurality of descriptors and the weighted descriptor clustering centers to determine a second number of descriptor clusters;

[0115] Apply the second clustering process to the second number of descriptor clusters to determine a second descriptor clustering center for each of the second number of descriptor clusters; and

[0116] Adjust the descriptor clustering data to characterize a similarity between the plurality of descriptors and the second number of descriptor clustering centers.

[0117] 8. The apparatus according to any one of clauses 1 - 7, wherein the at least one processor is configured to execute the instructions to:

[0118] Receive descriptor matching data for the geographical location and a statistical measure based on a number of features successfully matched with at least one of the descriptor clustering centers from a plurality of remote devices, the descriptor matching data characterizing a matching result of at least one descriptor clustering center and a number of features;

[0119] Determine a matching performance for the geographical location based on the matching results of the at least one descriptor clustering center and the several features and the statistical metric; and

[0120] Adjust the descriptor clustering data based on the matching performance.

[0121] 9. The apparatus according to clause 8, wherein the at least one processor is configured to execute the instructions to:

[0122] Adjust the weights of the at least one descriptor clustering center based on the matching performance;

[0123] Weight the at least one descriptor clustering center based on the weights;

[0124] Apply the first clustering process to the multiple descriptors and the descriptor clustering centers to determine a second several descriptor clusters, the descriptor clustering centers including the at least one weighted descriptor clustering center;

[0125] Apply the second clustering process to the second several descriptor clusters to determine a second descriptor clustering center for each of the second several descriptor clusters; and

[0126] Adjust the descriptor clustering data to characterize the similarity between the multiple descriptors and the second several descriptor clustering centers.

[0127] 10. The apparatus according to any one of clauses 1-9, wherein the at least one processor is configured to execute the instructions to establish the first clustering process as a k-means++ clustering process.

[0128] 11. The apparatus according to any one of clauses 1-10, wherein the at least one processor is configured to execute the instructions to establish the second clustering process as a fuzzy c-means clustering process.

[0129] 12. A method executed by at least one processor, the method comprising:

[0130] Apply a first clustering process to multiple descriptors associated with a geographical location to determine several descriptor clusters;

[0131] Apply a second clustering process to the several descriptor clusters to determine a descriptor clustering center for each of the several descriptor clusters;

[0132] Generate descriptor clustering data characterizing the similarity between the multiple descriptors and the descriptor clustering centers; and

[0133] Store the descriptor clustering data in a data repository.

[0134] 13. The method according to clause 12, including generating the descriptor clustering data to include a plurality of values characterizing the similarity between the plurality of descriptors and the descriptor clustering centers, wherein each of the plurality of values characterizes the similarity between one of the plurality of descriptors and one of the descriptor clustering centers.

[0135] 14. The method according to clause 13, wherein each of the plurality of values identifies the probability that one of the plurality of descriptors belongs to one of the descriptor clustering centers.

[0136] 15. The method according to any one of clauses 12-14, including:

[0137] Receiving location data characterizing the geographical location from a remote device; and

[0138] In response to receiving the location data, sending the descriptor clustering data to the remote device.

[0139] 16. The method according to clause 15, including:

[0140] Receiving descriptor matching data from the remote device, the descriptor matching data characterizing the matching result of the descriptor clustering data; and

[0141] Adjusting the descriptor clustering data in the data repository based on the descriptor matching data.

[0142] 17. The method according to clause 16, wherein the descriptor matching data includes a number of descriptor matches for at least one of the descriptor clustering centers, the method including:

[0143] Determining the matching performance of at least one of the descriptor clustering centers based on the number of descriptor matches; and

[0144] Adjusting the descriptor clustering data based on the matching performance.

[0145] 18. The method according to any one of clauses 16-17, including:

[0146] Weighting the descriptor clustering centers based on the descriptor matching data;

[0147] Applying the first clustering process to the plurality of descriptors and the weighted descriptor clustering centers to determine a second number of descriptor clusters;

[0148] Apply the second clustering process to the second plurality of descriptor clusters to determine a second descriptor cluster center for each of the second plurality of descriptor clusters; and

[0149] Adjust the descriptor cluster data to characterize the similarity between the plurality of descriptors and the second plurality of descriptor cluster centers.

[0150] 19. The method according to any one of clauses 12 - 18, comprising:

[0151] Receive, from a plurality of remote devices, descriptor matching data for the geographical location and statistical metrics of a plurality of features successfully matched with at least one descriptor cluster center, the descriptor matching data characterizing the matching result between at least one descriptor cluster center and a plurality of features;

[0152] Determine a matching performance for the geographical location based on the matching result between the at least one descriptor cluster center and the plurality of features and the statistical metrics; and

[0153] Adjust the descriptor cluster data based on the matching performance.

[0154] 20. The method according to clause 19, comprising:

[0155] Adjust the weight of the at least one descriptor cluster center based on the matching performance;

[0156] Weight the at least one descriptor cluster center based on the weight;

[0157] Apply the first clustering process to the plurality of descriptors and the descriptor cluster centers to determine a second plurality of descriptor clusters, the descriptor cluster centers including the weighted at least one descriptor cluster center;

[0158] Apply the second clustering process to the second plurality of descriptor clusters to determine a second descriptor cluster center for each of the second plurality of descriptor clusters; and

[0159] Adjust the descriptor cluster data to characterize the similarity between the plurality of descriptors and the second plurality of descriptor cluster centers.

[0160] 21. The method according to any one of clauses 12 - 20, comprising establishing the first clustering process as a k - means++ clustering process.

[0161] 22. The method according to any one of clauses 12 - 21, comprising establishing the second clustering process as a fuzzy c - means clustering process.

[0162] 23. A non - transitory machine - readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations including the following:

[0163] Apply a first clustering process to a plurality of descriptors associated with a geographical location to determine a number of descriptor clusters;

[0164] Apply a second clustering process to the number of descriptor clusters to determine a descriptor cluster center for each of the number of descriptor clusters;

[0165] Generate descriptor cluster data characterizing the similarity between the plurality of descriptors and the descriptor cluster centers; and

[0166] Store the descriptor cluster data in a data repository.

[0167] 24. The non - transitory machine - readable storage medium according to clause 23, wherein when the instructions are executed by the at least one processor, the at least one processor is caused to perform operations including the following: Generate the descriptor cluster data to include a plurality of values characterizing the similarity between the plurality of descriptors and the descriptor cluster centers, where each of the plurality of values characterizes the similarity between one of the plurality of descriptors and one of the descriptor cluster centers.

[0168] 25. The non - transitory machine - readable storage medium according to clause 24, wherein each of the plurality of values identifies the probability that one of the plurality of descriptors belongs to one of the descriptor cluster centers.

[0169] 26. The non - transitory machine - readable storage medium according to any one of clauses 23 - 25, wherein when the instructions are executed by the at least one processor, the at least one processor is caused to perform operations including the following:

[0170] Receive location data characterizing the geographical location from a remote device; and

[0171] In response to receiving the location data, send the descriptor cluster data to the remote device.

[0172] 27. The non - transitory machine - readable storage medium according to clause 26, wherein when the instructions are executed by the at least one processor, the at least one processor is caused to perform operations including the following:

[0173] Receive descriptor match data from the remote device, the descriptor match data characterizing the match result of the descriptor cluster data; and

[0174] Adjust the descriptor clustering data in the data repository based on the descriptor matching data.

[0175] 28. The non - transitory machine - readable storage medium according to clause 27, wherein the descriptor matching data includes a number of descriptor matches for at least one of the descriptor clustering centers, and wherein when the instructions are executed by the at least one processor, cause the at least one processor to perform operations including the following:

[0176] Determine the matching performance of at least one of the descriptor clustering centers based on the number of descriptor matches; and

[0177] Adjust the descriptor clustering data based on the matching performance.

[0178] 29. The non - transitory machine - readable storage medium according to any one of clauses 27 - 28, wherein when the instructions are executed by the at least one processor, cause the at least one processor to perform operations including the following:

[0179] Weight the descriptor clustering centers based on the descriptor matching data;

[0180] Apply the first clustering process to the plurality of descriptors and the weighted descriptor clustering centers to determine a second number of descriptor clusters;

[0181] Apply the second clustering process to the second number of descriptor clusters to determine a second descriptor clustering center for each of the second number of descriptor clusters; and

[0182] Adjust the descriptor clustering data to characterize the similarity between the plurality of descriptors and the second number of descriptor clustering centers.

[0183] 30. The non - transitory machine - readable storage medium according to any one of clauses 23 - 29, wherein when the instructions are executed by the at least one processor, cause the at least one processor to perform operations including the following:

[0184] Receive, from a plurality of remote devices, descriptor matching data for the geographical location and a statistical measure based on a number of features successfully matched with at least one of the descriptor clustering centers, the descriptor matching data characterizing the matching result of at least one descriptor clustering center with a number of features;

[0185] Determine the matching performance for the geographical location based on the matching result of the at least one descriptor clustering center with the number of features and the statistical measure; and

[0186] Adjust the descriptor clustering data based on the matching performance.

[0187] 31. The non-transitory machine-readable storage medium according to any one of clauses 30, wherein when the instructions are executed by the at least one processor, the at least one processor is caused to perform operations including the following operations:

[0188] Adjust the weights of the at least one descriptor cluster center based on the matching performance;

[0189] Weight the at least one descriptor cluster center based on the weights;

[0190] Apply the first clustering process to the plurality of descriptors and the descriptor cluster centers to determine a second plurality of descriptor clusters, the descriptor cluster centers including the at least one weighted descriptor cluster center;

[0191] Apply the second clustering process to the second plurality of descriptor clusters to determine a second descriptor cluster center for each of the second plurality of descriptor clusters; and

[0192] Adjust the descriptor cluster data to characterize the similarity between the plurality of descriptors and the second plurality of descriptor cluster centers.

[0193] 32. The non-transitory machine-readable storage medium according to any one of clauses 23-31, wherein when the instructions are executed by the at least one processor, the at least one processor is caused to perform an operation including establishing the first clustering process as a k-means++ clustering process.

[0194] 33. The non-transitory machine-readable storage medium according to any one of clauses 23-32, wherein when the instructions are executed by the at least one processor, the at least one processor is caused to perform an operation including establishing the second clustering process as a fuzzy c-means clustering process.

[0195] 34. An apparatus, comprising:

[0196] Components for applying a first clustering process to a plurality of descriptors associated with a geographical location to determine a plurality of descriptor clusters;

[0197] Components for applying a second clustering process to the plurality of descriptor clusters to determine a descriptor cluster center for each of the plurality of descriptor clusters;

[0198] Components for generating descriptor cluster data characterizing the similarity between the plurality of descriptors and the descriptor cluster centers; and

[0199] Components for storing the descriptor cluster data in a data repository.

[0200] 35. The apparatus according to clause 34, comprising means for generating the descriptor clustering data to include a plurality of values representative of the similarity between the plurality of descriptors and the descriptor clustering centers, wherein each of the plurality of values represents the similarity between one of the plurality of descriptors and one of the descriptor clustering centers.

[0201] 36. The apparatus according to clause 35, wherein each of the plurality of values identifies the probability that one of the plurality of descriptors belongs to one of the descriptor clustering centers.

[0202] 37. The apparatus according to any one of clauses 34 - 36, comprising:

[0203] means for receiving location data representative of the geographical location from a remote device; and

[0204] means for sending the descriptor clustering data to the remote device in response to receiving the location data.

[0205] 38. The apparatus according to clause 37, comprising:

[0206] means for receiving descriptor matching data from the remote device, the descriptor matching data representing the matching result of the descriptor clustering data; and

[0207] means for adjusting the descriptor clustering data in the data repository based on the descriptor matching data.

[0208] 39. The apparatus according to clause 38, wherein the descriptor matching data includes a plurality of descriptor matches for at least one of the descriptor clustering centers, the method comprising:

[0209] means for determining the matching performance of at least one of the descriptor clustering centers based on the plurality of descriptor matches; and

[0210] means for adjusting the descriptor clustering data based on the matching performance.

[0211] 40. The apparatus according to any one of clauses 38 - 39, comprising:

[0212] means for weighting the descriptor clustering centers based on the descriptor matching data;

[0213] means for applying the first clustering process to the plurality of descriptors and the weighted descriptor clustering centers to determine a second plurality of descriptor clusters;

[0214] A component for applying the second clustering process to the second plurality of descriptor clusters to determine a second descriptor cluster center for each of the second plurality of descriptor clusters; and

[0215] A component for adjusting the descriptor cluster data to characterize the similarity between the plurality of descriptors and the second plurality of descriptor cluster centers.

[0216] 41. The apparatus according to any one of clauses 34 - 40, comprising:

[0217] A component for receiving, from a plurality of remote devices, descriptor matching data for the geographical location and a statistical measure based on a number of features successfully matched with at least one descriptor cluster center, the descriptor matching data characterizing a matching result between at least one descriptor cluster center and a number of features;

[0218] A component for determining a matching performance for the geographical location based on the matching result between the at least one descriptor cluster center and the number of features and the statistical measure; and

[0219] A component for adjusting the descriptor cluster data based on the matching performance.

[0220] 42. The apparatus according to clause 41, comprising:

[0221] A component for adjusting the weight of the at least one descriptor cluster center based on the matching performance;

[0222] A component for weighting the at least one descriptor cluster center based on the weight;

[0223] A component for applying the first clustering process to the plurality of descriptors and the descriptor cluster centers to determine a second plurality of descriptor clusters, the descriptor cluster centers including the weighted at least one descriptor cluster center;

[0224] A component for applying the second clustering process to the second plurality of descriptor clusters to determine a second descriptor cluster center for each of the second plurality of descriptor clusters; and

[0225] A component for adjusting the descriptor cluster data to characterize the similarity between the plurality of descriptors and the second plurality of descriptor cluster centers.

[0226] 43. The apparatus according to any one of clauses 34 - 42, comprising a component for establishing the first clustering process as a k - means++ clustering process.

[0227] 44. The apparatus according to any one of clauses 34 - 43 includes components for establishing the second clustering process as a fuzzy c-means clustering process.

[0228] 45. An apparatus includes:

[0229] A non-transitory machine-readable storage medium storing instructions; and

[0230] At least one processor coupled to the non-transitory machine-readable storage medium, the at least one processor being configured to execute the instructions to:

[0231] Generate an image descriptor based on an image;

[0232] Receive descriptor clustering data, where the descriptor clustering data characterizes the similarity between a plurality of descriptors and a plurality of descriptor cluster centers;

[0233] Determine at least one of the plurality of descriptor cluster centers based on the image descriptor and the descriptor clustering data; and

[0234] Determine a location based on the at least one of the plurality of descriptor cluster centers.

[0235] 46. The apparatus according to clause 45, wherein the at least one processor is further configured to execute the instructions to:

[0236] Calculate the distance between the image descriptor and each of the plurality of descriptor cluster centers; and

[0237] Determine the at least one of the plurality of descriptor cluster centers based on the distance.

[0238] 47. The apparatus according to clause 46, wherein the descriptor clustering data includes a plurality of values, each of the plurality of values identifying the probability that one of the plurality of descriptors belongs to one of the descriptor cluster centers, and wherein the at least one processor is further configured to execute the instructions to determine the at least one of the plurality of descriptor cluster centers based on the plurality of values.

[0239] 48. The apparatus according to any one of clauses 45 - 47 includes at least one camera, wherein the at least one camera is configured to capture the image.

[0240] 49. The apparatus according to any one of clauses 45 - 48 includes a display, wherein the at least one processor is further configured to execute the instructions to provide an extended reality image including an object to the display.

[0241] 50. A method performed by at least one processor, the method comprising:

[0242] Generating an image descriptor based on an image;

[0243] Receiving descriptor clustering data, wherein the descriptor clustering data characterizes the similarity between a plurality of descriptors and a plurality of descriptor cluster centers;

[0244] Determining at least one of the plurality of descriptor cluster centers based on the image descriptor and the descriptor clustering data; and

[0245] Determining a position based on the at least one of the plurality of descriptor cluster centers.

[0246] 51. The method according to clause 50, comprising:

[0247] Calculating the distance between the image descriptor and each of the plurality of descriptor cluster centers; and

[0248] Determining the at least one of the plurality of descriptor cluster centers based on the distance.

[0249] 52. The method according to clause 51, wherein the descriptor clustering data includes a plurality of values, and each of the plurality of values identifies the probability that one of the plurality of descriptors belongs to one of the descriptor cluster centers, the method comprising determining the at least one of the plurality of descriptor cluster centers based on the plurality of values.

[0250] 53. The method according to any one of clauses 50 - 52, comprising causing a camera to capture the at least one image.

[0251] 54. The method according to any one of clauses 50 - 53, comprising providing an extended reality image including an object to a display.

[0252] 55. A non - transitory machine - readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations including the following:

[0253] Generating an image descriptor based on an image;

[0254] Receiving descriptor clustering data, wherein the descriptor clustering data characterizes the similarity between a plurality of descriptors and a plurality of descriptor cluster centers;

[0255] Determining at least one of the plurality of descriptor cluster centers based on the image descriptor and the descriptor clustering data; and

[0256] Determine a location based on at least one of the plurality of descriptor cluster centers.

[0257] 56. The non-transitory machine-readable storage medium according to clause 55, wherein when the instructions are executed by the at least one processor, the at least one processor is caused to perform operations including the following:

[0258] Calculate the distance between the image descriptor and each of the plurality of descriptor cluster centers; and

[0259] Determine at least one of the plurality of descriptor cluster centers based on the distance.

[0260] 57. The non-transitory machine-readable storage medium according to clause 56, wherein the descriptor cluster data includes a plurality of values, each of the plurality of values identifying the probability that one of the plurality of descriptors belongs to one of the descriptor cluster centers, and wherein when the instructions are executed by the at least one processor, the at least one processor is caused to perform an operation including determining at least one of the plurality of descriptor cluster centers based on the plurality of values.

[0261] 58. The non-transitory machine-readable storage medium according to any one of clauses 55-57, wherein when the instructions are executed by the at least one processor, the at least one processor is caused to perform an operation including causing a camera to capture the at least one image.

[0262] 59. The non-transitory machine-readable storage medium according to any one of clauses 55-58, wherein when the instructions are executed by the at least one processor, the at least one processor is caused to perform an operation including providing an extended reality image including an object to a display.

[0263] 60. An apparatus, comprising:

[0264] means for generating an image descriptor based on an image;

[0265] means for receiving descriptor cluster data, wherein the descriptor cluster data characterizes the similarity between a plurality of descriptors and a plurality of descriptor cluster centers;

[0266] means for determining at least one of the plurality of descriptor cluster centers based on the image descriptor and the descriptor cluster data; and

[0267] means for determining a location based on at least one of the plurality of descriptor cluster centers.

[0268] 61. The apparatus according to clause 60, comprising:

[0269] A component for calculating the distance between the image descriptor and each of the multiple descriptor cluster centers; and

[0270] A component for determining at least one of the multiple descriptor cluster centers based on the distance.

[0271] 62. The apparatus according to clause 61, wherein the descriptor cluster data includes a plurality of values, each of the plurality of values identifying the probability that one of the plurality of descriptors belongs to one of the descriptor cluster centers, the apparatus including a component for determining at least one of the multiple descriptor cluster centers based on the plurality of values.

[0272] 63. The apparatus according to any one of clauses 60 - 62, including a component for causing a camera to capture the at least one image.

[0273] 64. The apparatus according to any one of clauses 60 - 63, including a component for providing an extended reality image including an object to a display.

[0274] Although the methods described above refer to the illustrated flowcharts, many other ways of performing the actions associated with these methods can be used. For example, the order of some operations can be changed, and some embodiments can omit one or more of the described operations and / or include additional operations.

[0275] In addition, the methods and systems described herein can be embodied at least in part in the form of computer-implemented processes and apparatus for practicing these processes. The disclosed methods can also be embodied at least in part in the form of a tangible, non-transitory machine-readable storage medium encoded with computer program code. For example, these methods can be embodied in hardware, in executable instructions executed by a processor (e.g., software), or a combination of both. The medium can include, for example, RAM, ROM, CD-ROM, DVD-ROM, BD-ROM, hard disk drive, flash memory, or any other non-transitory machine-readable storage medium. When the computer program code is loaded into and executed by a computer, the computer becomes an apparatus for practicing the method. The method can also be embodied at least in part in the form of a computer into which the computer program code is loaded or executed, such that the computer becomes a special-purpose computer for practicing the method. When implemented on a general-purpose processor, the computer program code segments configure the processor to create specific logic circuits. Alternatively, the method can be embodied at least in part in a special integrated circuit for performing the method.

[0276] The subject matter has been described according to exemplary embodiments. Since they are merely examples, the claimed invention is not limited to these embodiments. Changes and modifications may be made without departing from the spirit of the claimed subject matter. The claims are intended to cover such changes and modifications.

Claims

1. A device, comprising: A non - transitory machine - readable storage medium storing instructions; And At least one processor coupled to the non - transitory machine - readable storage medium, the at least one processor being configured to execute the instructions to: Apply a first clustering process to a plurality of descriptors associated with a geographical location to determine a number of descriptor clusters; Apply a second clustering process to the number of descriptor clusters to determine a descriptor cluster center for each of the number of descriptor clusters; Generate descriptor cluster data characterizing the similarity between the plurality of descriptors and the descriptor cluster centers; And Store the descriptor cluster data in a data repository.

2. The device according to claim 1, wherein the at least one processor is further configured to execute the instructions to generate the descriptor cluster data to include a plurality of values characterizing the similarity between the plurality of descriptors and the descriptor cluster centers, wherein each of the plurality of values characterizes the similarity between one of the plurality of descriptors and one of the descriptor cluster centers.

3. The device according to claim 2, wherein each of the plurality of values identifies the probability that one of the plurality of descriptors belongs to one of the descriptor cluster centers.

4. The device according to claim 1, wherein the at least one processor is further configured to execute the instructions to: Receive location data characterizing the geographical location from a remote device; and In response to receiving the location data, send the descriptor cluster data to the remote device.

5. The device according to claim 4, wherein the at least one processor is further configured to execute the instructions to: Receive descriptor matching data from the remote device, the descriptor matching data characterizing a matching result of the descriptor cluster data; and Adjust the descriptor cluster data in the data repository based on the descriptor matching data.

6. The device according to claim 5, wherein the descriptor matching data includes a number of descriptor matches for at least one of the descriptor cluster centers, and the at least one processor is configured to execute the instructions to: Determine a matching performance of at least one of the descriptor cluster centers based on the number of descriptor matches; and Adjust the descriptor cluster data based on the matching performance.

7. The device according to claim 5, wherein the at least one processor is further configured to execute the instructions to: Weight the descriptor cluster centers based on the descriptor matching data; Apply the first clustering process to the plurality of descriptors and the weighted descriptor cluster centers to determine a second number of descriptor clusters; Apply the second clustering process to the second number of descriptor clusters to determine a second descriptor cluster center for each of the second number of descriptor clusters; And Adjust the descriptor cluster data to characterize the similarity between the plurality of descriptors and the second number of descriptor cluster centers.

8. The apparatus according to claim 1, wherein the at least one processor is configured to execute the instructions to: Receive, from a plurality of remote devices, descriptor matching data for the geographical location and statistical metrics based on a number of features successfully matched with at least one descriptor cluster center, the descriptor matching data characterizing a matching result between at least one descriptor cluster center and a number of features; Determine a matching performance for the geographical location based on the matching result between the at least one descriptor cluster center and the number of features and the statistical metrics; And Adjust the descriptor cluster data based on the matching performance.

9. The apparatus according to claim 8, wherein the at least one processor is configured to execute the instructions to: Adjust the weights of the at least one descriptor cluster center based on the matching performance; Weight the at least one descriptor cluster center based on the weights; Apply the first clustering process to the plurality of descriptors and the descriptor cluster centers to determine a second number of descriptor clusters, the descriptor cluster centers including the at least one weighted descriptor cluster center; Apply the second clustering process to the second number of descriptor clusters to determine a second descriptor cluster center for each of the second number of descriptor clusters; And Adjust the descriptor cluster data to characterize a similarity between the plurality of descriptors and the second number of descriptor cluster centers.

10. A method executed by at least one processor, the method comprising: Applying a first clustering process to a plurality of descriptors associated with a geographical location to determine a number of descriptor clusters; Applying a second clustering process to the number of descriptor clusters to determine a descriptor cluster center for each of the number of descriptor clusters; Generating descriptor cluster data characterizing a similarity between the plurality of descriptors and the descriptor cluster centers; And Storing the descriptor cluster data in a data repository.

11. The method according to claim 10, including generating the descriptor cluster data to include a plurality of values characterizing a similarity between the plurality of descriptors and the descriptor cluster centers, wherein each of the plurality of values characterizes a similarity between one of the plurality of descriptors and one of the descriptor cluster centers.

12. The method according to claim 10, including: Receiving location data representing the geographical location from a remote device; And In response to receiving the location data, sending the descriptor cluster data to the remote device.

13. The method according to claim 12, including: Receiving descriptor matching data from the remote device, the descriptor matching data characterizing a matching result of the descriptor cluster data; And Adjusting the descriptor cluster data in the data repository based on the descriptor matching data.

14. The method according to claim 13, including: Weighting the descriptor cluster centers based on the descriptor matching data; Apply the first clustering process to the plurality of descriptors and the weighted descriptor cluster centers to determine a second plurality of descriptor clusters; Apply the second clustering process to the second plurality of descriptor clusters to determine second descriptor cluster centers for each of the second plurality of descriptor clusters; and Adjust the descriptor cluster data to characterize the similarity between the plurality of descriptors and the second plurality of descriptor cluster centers.

15. The method according to claim 10, comprising: Receiving descriptor matching data for the geographical location and a statistical metric based on a plurality of features successfully matched with at least one of the descriptor cluster centers from a plurality of remote devices, the descriptor matching data characterizing a matching result between at least one of the descriptor cluster centers and the plurality of features; Determining a matching performance for the geographical location based on the matching result between at least one of the descriptor cluster centers and the plurality of features and the statistical metric; and Adjusting the descriptor cluster data based on the matching performance.

16. An apparatus, comprising: A non-transitory machine-readable storage medium storing instructions; and At least one processor coupled to the non-transitory machine-readable storage medium, the at least one processor configured to execute the instructions to: Generate image descriptors based on an image; Receive descriptor cluster data, wherein the descriptor cluster data characterizes the similarity between a plurality of descriptors and a plurality of descriptor cluster centers; Determine at least one of the plurality of descriptor cluster centers based on the image descriptors and the descriptor cluster data; and Determine a location based on at least one of the plurality of descriptor cluster centers.

17. The apparatus according to claim 16, wherein the at least one processor is further configured to execute the instructions to: Calculate a distance between each of the image descriptors and each of the plurality of descriptor cluster centers; and Determine at least one of the plurality of descriptor cluster centers based on the distance.

18. The apparatus according to claim 17, wherein the descriptor cluster data includes a plurality of values, each of the plurality of values identifying a probability that one of the plurality of descriptors belongs to one of the descriptor cluster centers of the plurality of descriptor cluster centers, and wherein the at least one processor is further configured to execute the instructions to determine at least one of the plurality of descriptor cluster centers based on the plurality of values.

19. The apparatus according to claim 16, comprising at least one camera, wherein the at least one camera is configured to capture the image.

20. The apparatus according to claim 16, comprising a display, wherein the at least one processor is further configured to execute the instructions to provide an extended reality image including an object to the display.

Images