System and method for processing image data

Abstract

Disclosed is a system for processing image data, the system comprising a processor, the processor configured to: obtain a multi-modal dataset, the multi-modal dataset comprising two- dimensional (2D) ground-view image data of a scene and a corresponding point-cloud data representing the scene; extract, using a first neural network backbone module, a first set of feature representation from the 2D ground-view image data; extract, using a second neural network backbone module, a second set of feature representation from the corresponding point- cloud data; fuse, using a fusion embedding construction module, the first set of feature representation and the second set of feature representation to generate a fusion embedding; generate, using a modal-wise fusion module, a 2D modality-specific descriptor and a 3D modality-specific descriptor based on the fusion embedding; and combine the 2D modality- specific descriptor and the 3D modality-specific descriptor to form a fused ground scene descriptor.

Description

SYSTEM AND METHOD FOR PROCESSING IMAGE DATACROSS-REFERENCE TO RELATED APPLICATION

[0001] The application claims the benefit of priority of Singapore patent application No. 10202401872S, filed 25 June 2025, the content of it being hereby incorporated by reference in its entirety for all purposes.TECHNICAL FIELD

[0002] Various aspects of this disclosure relate to methods, server apparatuses, and systems for processing image data, and in particular image data that may be used for location or place recognition (PR)BACKGROUND

[0003] The following discussion of the background art is intended to facilitate an understanding of the present disclosure only. It should be appreciated that the discussion is not an acknowledgement or admission that any of the material referred to was published, known or is part of the common general knowledge of the person skilled in the art in any jurisdiction as of the priority date of the disclosure.

[0004] Place recognition (PR) addresses localization by treating the localization as a retrieval task. This may involve maintaining a database of geographically tagged sensor data — such as images or point clouds — captured at previously visited locations. During operation, PR calculates similarity metrics between incoming query data and stored database entries to identify the closest matches. The technology may be useful for applications like autonomous navigation, augmented reality (AR), and robotic mapping systems.

[0005] PR models typically transform raw sensor inputs into compact, high-level scene descriptors Conventional approaches utilize manually designed features, such as Bag of Words (BoW) or Vector of Locally Aggregated Descriptors (VLAD), to summarize data. While functional, these methods depend on human-defined parameters and may struggle in complex environments. Recent advancements have shifted toward data-driven solutions, typically single-modal systems where queries and database entries share the same sensor modality, such as images or point clouds.

[0006] Emerging multi-modal PR systems integrate data from multiple sensor types, enabling richer environmental representations and improved performance compared to single-modal approaches. A separate challenge involves cross-view scenarios, where database data is captured from aerial perspectives (e.g., drones or satellites) and queries originate from groundlevel sensors (e g., robots, vehicles, or handheld devices). Cross-view PR leverages aerial maps for broad field-of-view coverage, which is particularly advantageous for ground robots operating in challenging terrains like forests or mountains. Aerial data collection is often more efficient than ground-based methods, and aerial maps provide geometric references for precise localization.

[0007] Current cross-view PR systems focus on matching ground images to aerial images, neglecting the potential benefits of fusing ground-level images with point cloud data.

[0008] Aerial-ground PR is a PR system that has emerged as a technology for enabling localization in systems where ground-based queries are matched against aerial -reference databases. This cross-view PR framework seek to address inherent perspective discrepancies between ground-level sensor data (e.g., images or point clouds) and aerial data (e.g., satellite imagery or drone-captured maps).

[0009] Prior methodologies attempt to mitigate view differences through geometric alignment techniques. For example, some approaches apply polar transformations to align panoramic ground images with aerial views, while others employ generative models to refine such transformations. These methods rely on geometric correspondences between aerial and ground perspectives. Alternative strategies focus on descriptor-level alignment, such as using shared neural network architectures to extract compatible features from both views, implementing contrastive loss strategies to distinguish cross-view relationships, or incorporating angular similarity metrics to optimize feature matching.

[0010] Despite progress, existing aerial-ground PR systems remain mostly limited to single- modal inputs, typically matching ground images to aerial images. Although multi-modal PR systems have been developed and utilized for PR systems, such PR systems are generally limited to ground-ground same-view situation, where the construction of a ground-view database requires tedious collection of ground data.

[0011] There exists a need to provide an improved technical solution for place recognition.SUMMARY

[0012] The present disclosure was conceptualized to provide a technical solution for integrating multi-modal inputs in cross-view systems to enhance robustness and accuracy for place recognition (PR) applications The disclosure may relate to PR for localization tasks,specifically to methods and systems for cross-view, multi-modal place recognition using ground-level multi-sensor data (images and LiDAR point clouds) and aerial-view data (satellite images, maps) The PR method and system of the present disclosure aims at retrieving the query place from a database, which may play a crucial role in various applications, including, but not limited to, navigation, autonomous driving, and augmented reality. The present disclosure, which may be referred to as the “AGPlace” or “AGPlace model”, seeks to effectively integrate information from ground- view sensors (e g., cameras and LiDARs), to achieve accurate aerial-ground PR. By incorporating differential geometry concepts into the manifold point motion state evolution process (parameterized by neural ordinary differential equations (ODEs)), AGPlace achieves effective multi-modal feature fusion from camera images and LiDAR point clouds. From the experiments described, AGPlace surpasses existing multi-modal baselines on benchmark datasets, demonstrating robustness and efficacy with Recall@l exceeding 50% and Recall@10 surpassing 80%. AGPlace represents a significant advancement in multi-modal aerial-ground PR, with promising implications for real-world applications. The AGPlace leverages neural ODEs to achieve effective multi-sensor feature fusion inspired by point movement on a manifold, establish a benchmark to evaluate AGPlace, which consists of different multi-sensor ground data and various aerial database maps.

[0013] According to an aspect of the present disclosure there is provided a system for processing image data, the system comprising a processor, the processor configured to: obtain a multi-modal dataset, the multi-modal dataset comprising two-dimensional (2D) ground-view image data of a scene and a corresponding point-cloud data representing the scene; extract, using a first neural network backbone module, a first set of feature representation from the 2D ground-view image data; extract, using a second neural network backbone module, a second set of feature representation from the corresponding point-cloud data; fuse, using a fusion embedding construction module, the first set of feature representation and the second set of feature representation to generate a fusion embedding; generate, using a modal-wise fusion module, a 2D modality-specific descriptor and a 3D modality-specific descriptor based on the fusion embedding; and combine the 2D modality-specific descriptor and the 3D modalityspecific descriptor to form a fused ground scene descriptor.

[0014] In some embodiments, the system may be deployed as a, or as part of, an aerial ground place recognition system. The aerial ground place recognition system may be, or form part of a navigation system for autonomous driving, and / or an augmented reality system.

[0015] In some embodiments, the fusion embedding construction module comprises a plurality of neural ordinary differential equation (ODE) blocks.

[0016] In some embodiments, each of the plurality of neural ODE blocks is configured to progressively refine the fusion embedding by integrating one or more feature trajectories over a time period, such that an output of each neural ODE block is used as an input to a next neural ODE block.

[0017] In some embodiments, the input to the next neural ODE block comprises a set of features, wherein the set of features comprises a summation of feature output(s) from one or more previous neural ODE blocks.

[0018] In some embodiments, the one or more feature trajectories of the fusion embedding within each neural ODE block is parameterized.

[0019] In some embodiments, the modal -wise fusion module is trained using a triplet marginal loss function using an anchor parameter, a positive sample, and a negative sample.

[0020] In some embodiments, the modal-wise fusion module is configured to process the fusion embedding using a first chart function to generate a 2D mapped fusion embedding, and a second chart function to generate a 3D mapped fusion embedding.

[0021] In some embodiments, the first chart function and the second chart function are learnable manifold chart functions.

[0022] In some embodiments, the modal-wise fusion module is further configured to project the 2D mapped fusion embedding using a 2D feature, and project the 3D mapped fusion embedding using a 3D feature.

[0023] In some embodiments, the first neural network backbone module and the second neural network backbone module are each configured to generate a sequence of a number of feature extraction blocks and output a 2D feature map and a 3D feature map respectively.

[0024] In some embodiments, the system further comprising a database, the database configured to store a plurality of aerial images, and wherein each of the plurality of aerial images is associated with an aerial-view descriptor.

[0025] In some embodiments, the processor further comprises a comparator, the comparator configured to compare the fused ground scene descriptor with each of the aerial -view descriptor to determine a match, wherein in a positive determination of a match, a matched aerial image associated with the matched aerial-view descriptor is sent to a user device.

[0026] In some embodiments, the system further comprises an image capturing device configured to obtain the ground-view image data of the scene, and a light detection and ranging (lidar) sensor configured to obtain corresponding point-cloud data representing the scene.

[0027] According to another aspect of the present disclosure there is provided a vehicle, the vehicle comprising an image capturing device configured to obtain a two-dimensional (2D) ground-view image data of a scene, a light detection and ranging (lidar) sensor configured to obtain corresponding point-cloud data representing the scene; a processor, the processor comprising: an input module arranged in signal or data communication with the image capturing device and the lidar sensor, the input module configured to obtain the 2D groundview image data and the corresponding point-cloud data representing the scene; an neural network module, the neural network module configured to: extract, using a first neural network backbone module, a first set of feature representation from the 2D ground-view image data; extract, using a second neural network backbone module, a second set of feature representation from the corresponding point-cloud data; fuse, using a fusion embedding construction module, the first set of feature representation and the second set of feature representation to generate a fusion embedding; generate, using a modal-wise fusion module, a 2D modality-specific descriptor and a 3D modality-specific descriptor based on the fusion embedding; and combine the 2D modality-specific descriptor and the 3D modality-specific descriptor to form a fused ground scene descriptor.

[0028] In some embodiments, the vehicle further comprises a data communication device arranged in data communication with an aerial-view map database, the aerial-view map database comprising a plurality of aerial images, wherein each of the plurality of aerial images is associated with an aerial -view descriptor.

[0029] In some embodiments, the processor comprises a comparator, the comparator configured to compare the fused ground scene descriptor with each of the aerial -view descriptor to determine a match, wherein in a positive determination of a match, a matched aerial image associated with the matched aerial-view descriptor is sent to a user device.

[0030] According to another aspect of the present disclosure there is provided a method for processing image data comprising: obtaining a multi-modal dataset, the multi-modal dataset comprising two-dimensional (2D) ground-view image data of a scene and a corresponding point-cloud data representing the scene; extracting, using a first neural network backbone module, a first set of feature representation from the 2D ground-view image data; extracting, using a second neural network backbone module, a second set of feature representation fromthe corresponding point-cloud data; fusing, using a fusion embedding construction module, the first set of feature representation and the second set of feature representation to generate a fusion embedding; generating, using a modal-wise fusion module, a 2D modality-specific descriptor and a 3D modality-specific descriptor based on the fusion embedding; and combining the 2D modality-specific descriptor and the 3D modality-specific descriptor to form a fused ground scene descriptor.

[0031] In some embodiments, the fusion embedding construction module comprises a plurality of neural ordinary differential equation (ODE) blocks.

[0032] In some embodiments, each of the plurality of neural ODE blocks is configured to progressively refine the fusion embedding by integrating one or more feature trajectories over a time period, such that an output of each neural ODE block is used as an input to a next neural ODE block.

[0033] According to another aspect of the present disclosure, there is provided a non- transitory computer-readable medium storing computer executable code comprising instructions for processing image data according to any one of the aforementioned methods.BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The disclosure will be better understood with reference to the detailed description when considered in conjunction with the non-limiting examples and the accompanying drawings, in which:- FIG. 1 A is a schematic block diagram of a server apparatus for processing image data.- FIG. IB is a schematic block diagram comprising the server apparatus of FIG 1 A, with a user device, implemented as a system for processing image data.- FIG. 2 is a schematic diagram illustrating an example pipeline of multi-modal aerialground PR according to some embodiments.- FIG. 3 shows an embodiment of a pipeline overview of an embodiment of the server apparatus and / or system for image processing, comprising a fusion embedding construction stage and a modal-wise fusion stage.- FIG. 4A and FIG. 4B show embodiments of fusion embedding evolution processes starting from the last block and ending at the first block according to some embodiments.- Fig. 5 illustrates possible input multi-modal dataset into the server apparatus or system, comprising a KITTI-AG and a Ithaca365-AG datasets.- FIG. 6 is a table illustrating the performance comparison on KITTI-AG and Ithaca365- AG datasets across different multi-modal PR system (the data entry marked as AGPlace is the multi-modal PR system of the present disclosure).- FIG. 7 is a table illustrating the performance comparison using different aerial data sources, including satellite RGB images, building maps, and terrain maps. The test is built based on the KITTI-AG dataset.- FIG. 8 illustrates different aerial data sources, including satellite RGB images, building maps, and terrain maps.- FIG. 9 is a table illustrating a main ablation study on the proposed modules, with incremental inclusion of stage 1 and stage 2, based on the KITTI-AG dataset- FIG. 10 is a table illustrating fusion direction comparison, based on the KITTI-AG dataset.- FIG. 11 is a table illustrating state updating comparison, based on the KITTI-AG dataset.- FIG. 12 is a table illustrating ground modality comparison, based on the KITTI-AG dataset.- FIG. 13 is a table illustrating point cloud representation comparison, based on the KITTI-AG dataset.- FIG. 14 is a table illustrating runtime performance comparison on a Tesla A100, based on the KITTI-AG dataset.- FIG. 15 shows an aerial-ground scene descriptor distance visualization heatmap images, with fusion manifold labelled as 1501, without fusion manifold labelled as 1502, and the aerial -ground geographical distances.- FIG. 16 illustrates data visualization from the KITTI360-AG dataset, which is used for another experiment.- FIG. 17 is a table illustrating the aerial-ground PR results on the KITTI360-AG dataset of FIG. 16 using satellite or road map aerial sources. denotes the model is frozen and purely relies on pre-trained weights. "C" denotes the model is designed for cross- view / cross-modal PR. Hard negative mining is applied to all models.- FIG. 18 is a table illustrating aerial -ground PR results on another dataset, the nuScenes- AG dataset, using satellite image database, "fail" denotes dropping the modality input during testing. All models are trained with both modalities.- FIG. 19 is a table illustrating ground-ground PR results on another dataset, the Oxford benchmark datasets. denotes the result is not provided.- FIG. 20 is a table illustrating module ablation, based on the KITTI360-AG dataset of FIG. 16.- FIG. 21 illustrates t-SNE plots of ground scene descriptors (from a consecutive frame sequence), indicating that ODEs can help build more consistent descriptors that align with the consecutive geometry.- FIG. 22 is a table comparing state updating method and fusion direction.- FIG. 23 is a table illustrating ground and aerial modality comparison.- FIG. 24 is a salience map visualization for identifying which part of ground and aerial inputs are focused in the present place recognition system.- FIG. 25 is a table illustrating runtime performance comparison on a Tesla A100, with respective modal spaces to achieve further modal-wise feature extraction.- FIG. 26 is a flowchart showing a generalized method for processing image data, according to some embodiments of the present disclosure.DETAILED DESCRIPTION

[0035] The following detailed description refers to the accompanying drawings that show, by way of illustration, specific details, and embodiments in which the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure. Other embodiments may be utilized, and structural and logical changes may be made without departing from the scope of the disclosure. The various embodiments are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

[0036] Features that are described in the context of an embodiment may correspondingly be applicable to the same or similar features in the other embodiments. Features that are described in the context of an embodiment may correspondingly be applicable to the other embodiments, even if not explicitly described in these other embodiments. Furthermore, additions and / orcombinations and / or alternatives as described for a feature in the context of an embodiment may correspondingly be applicable to the same or similar feature in the other embodiments.

[0037] In the context of various embodiments, the articles “a”, “an” and “the” as used with regard to a feature or element include a reference to one or more of the features or elements. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0038] While such terms as "first," "second," etc., may be used to describe various elements, such elements must not be limited to the above terms. The above terms are used only to distinguish one element from another, and do not define corresponding elements, for example, an order and / or significance of the elements. Without departing from the scope of rights of the specification, a first element may be referred to as a second element, and similarly, the second element may be referred to as the first element.

[0039] As used herein, the term “data” may be understood to include information in any suitable analog or digital form, for example, provided as a file, a portion of a file, a set of files, a signal or stream, a portion of a signal or stream, a set of signals or streams, and the like. Image data may comprise still images, video frames, image sequences, or any other visual information captured by digital cameras, sensors, or other imaging devices. The term data, however, is not limited to the aforementioned examples and may take various forms and represent any information as understood in the art.

[0040] As used herein, the term “processor” refers to a circuit, including analog circuits, digital circuits, or hybrid circuits, or their constituent components. Any other kind of implementation of the respective functions which will be described in more detail below may also be understood as a “circuit” in accordance with an alternative embodiment. A digital circuit may be understood as any kind of a logic implementing entity, which may be special purpose circuitry or a processor executing software stored in a memory, or a firmware.

[0041] As used herein, the term “module” refers to, forms part of, or includes an Application Specific Integrated Circuit (ASIC); an electronic circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor (shared, dedicated, or group) that executes code; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip. The term module may include memory (shared, dedicated, or group) that stores code executed by the processor. A single module or a combination of modules may be regarded as a device. A processor may include one or more modules For example, multiple modules described in this disclosure may form a processor.

[0042] As used herein, the term “associate”, “associated”, and “associating” indicate a defined relationship (or cross-reference) between two items.

[0043] As used herein, “memory” may be understood as a non-transitory computer-readable medium in which data or information can be stored for retrieval. References to “memory” included herein may thus be understood as referring to volatile or non-volatile memory, including random access memory (“RAM”), read-only memory (“ROM”), flash memory, solid-state storage, magnetic tape, hard disk drive, optical drive, etc., or any combination thereof. Furthermore, it is appreciated that registers, shift registers, processor registers, data buffers, etc., are also embraced herein by the term memory. It is appreciated that a single component referred to as “memory” or “a memory” may be composed of more than one different type of memory, and thus may refer to a collective component including one or more types of memory. It is readily understood that any single memory component may be separated into multiple collectively equivalent memory components, and vice versa. Furthermore, while memory may be depicted as separate from one or more other components (such as in the drawings), it is understood that memory may be integrated within another component, such as on a common integrated chip.

[0044] As used herein, the term “device” may be understood to refer to any apparatus, equipment, or component, whether standalone or integrated, that performs a specific function or set of functions. This includes, but is not limited to, mechanical, electrical, electronic, optical, or electromechanical systems, subsystems, and assemblies. A device may comprise one or more components, modules, or units that are designed to interact with each other to achieve a particular purpose.

[0045] As used herein, the term “place recognition” may be understood to refer to a process by which a system, apparatus, or method is used to identify or determine one or more locations of an agent, device, or image associated with an environment or set of reference locations, based on the analysis of visual or sensor data. In particular, in the context of computer vision and robotics, place recognition enables the recognition of a specific place or location by comparing query observations — such as images, sensor readings, or other data representations — with reference observations (which may include images) previously stored or mapped. The process may be robust to variations in viewpoint, lighting, weather, or other environmental conditions, and may involve the use of feature extraction, machine learning, or deep learning techniques to achieve invariance and accuracy in recognition. In some embodiments, place recognition may be further characterized by its ability to identify a locationeven when a query observation is obtained from a different perspective, time, or under different environmental conditions compared to the reference observation, provided there exists sufficient visual or sensor overlap between the observations. Place recognition may be used for various applications, non-limiting examples include autonomous navigation, localization in robotics, augmented reality, and intelligent surveillance systems.

[0046] As used herein, the term “neural network backbone module” refers to a foundational set of initial layers or a neural network architecture that is primarily responsible for extracting features from raw input data, such as from images or sensor readings. The backbone module may be configured to process the input data and transforms the input data into a compact, informative representation — often referred to as feature maps — that captures essential patterns, textures, and structures present in the input. The backbone module may comprise a sequence of layers, which may include convolutional layers, pooling layers, and activation functions, arranged to progressively learn hierarchical features from the input data. Early layers may detect basic elements such as edges and comers, while deeper layers combine such features to recognize more complex structures or objects. The output of the backbone module serves as the basis for subsequent processing by additional network components (such as "neck" or "head" modules), which may further refine, aggregate, or utilize the extracted features for specific tasks such as object detection, segmentation, or classification. The backbone module may be distinct from the entire neural network and is often implemented using architectures such as ResNet, VGG, MobileNet, or AlexNet, which may be pre-trained on large datasets to enhance performance and enable transfer learning for downstream applications.

[0047] As used herein, the term “fusion embedding” refers to a computational process that integrates heterogeneous data representations (e.g., visual, textual, or sensor modalities) into a unified feature space through mathematical operations or learned transformations This embedding enables subsequent machine learning models to process combined information while preserving critical relationships between modalities.

[0048] As used herein, the term “modality-specific descriptor” refers to one or more computational constructs or feature vectors that encode distinctive characteristics of input data unique to a particular data modality, such as, but not limited to, visual, auditory, textual, or tactile information. In artificial intelligence and computer vision, a modality-specific descriptor may be generated by analyzing raw data from a single modality (e g., from images, point clouds, audio waveforms, or text sequences) and transforming it into a numerical representation that captures the patterns or features inherent to that modality. In some embodiments, themodality-specific descriptor may be designed to be robust to variations within its own modality, such as changes in lighting or perspective for images, or accent and background noise for audio. Modality-specific descriptors may enable efficient comparison, retrieval, and recognition tasks by providing a standardized way to represent and process data from each input type, while preserving the unique attributes that distinguish one modality from another In some embodiments, the process of generating modality-specific descriptors may involve specialized algorithms or neural network architectures tailored to the particular characteristics of the input data. It is contemplated that in multi-modal systems, modality-specific descriptors may be combined with descriptors from other modalities to achieve a comprehensive understanding of the input, but each descriptor may retain the distinctive information relevant to its original data source.

[0049] As used herein, the term “sensor” includes any device, apparatus, system, and / or software component that detects, measures, monitors, or records physical, environmental, or operational conditions, phenomena, or properties, and generates output indicative of those conditions. The output may be in the form of electrical, mechanical, optical, or other signals, and may be processed by hardware or software systems for further analysis or control purposes. A sensor may include both hardware components (e.g., transducers, detectors, circuits) and software components (e.g., algorithms, data processing modules) that together enable the detection, measurement, and interpretation of the desired parameters. Some non-limiting examples of sensors include image capturing sensors, for example, cameras, and light detection and ranging sensor (lidar).

[0050] As used herein, the term “configured to” broadly refers to the design, arrangement, or adaptation of a system, device, component, or module to perform a specific function or achieve a particular outcome. The term includes both hardware and software implementations wherein in a hardware implementation, the physical components are arranged, programmed, or structured to carry out the intended function(s), and in the context of programming and software, a device is operable under executable instructions (e.g., software, firmware) to perform the specified function(s) when executed by one or more processors. The resultant configuration allows the system or component to perform the stated function, either inherently or after suitable programming or activation, without requiring substantial modifications to its structure or operational logic.

[0051] According to various embodiments, a circuit may include analog circuits or components, digital circuits or components, or hybrid circuits or components. Any other kindof implementation of the respective functions which will be described in more detail below may also be understood as a "circuit" in accordance with an alternative embodiment. A digital circuit may be understood as any kind of a logic implementing entity, which may be special purpose circuitry or a processor executing software stored in a memory, firmware, or any combination thereof Thus, in various embodiments, a "circuit" may be a digital circuit, e.g., a hard-wired logic circuit or a programmable logic circuit such as a programmable processor, e.g., a microprocessor (e.g., a Complex Instruction Set Computer (CISC) processor or a Reduced Instruction Set Computer (RISC) processor). A "circuit" may also include a processor executing software, e.g., any kind of computer program, e g., a computer program using a virtual machine code such as e.g., Java.

[0052] According to an aspect of the disclosure and with reference to FIG. 1A, there is provided a server apparatus for processing image data. The image data may include two- dimensional image data of a scene, for example raw RGB image data, and / or three dimensional image data, such as point-cloud data representing the scene.

[0053] The server apparatus may comprise a processor and a memory, the processor is capable of being configured to execute instructions stored in the memory to receive a request for resources. In the embodiment illustrated in FIG 1A, the server apparatus may be a communications server apparatus. The communications server apparatus may be in the form of a server computer 10, the server computer 10 may be a single server as illustrated schematically in FIG. 1A, or have the functionality performed distributed across multiple server components.

[0054] In some embodiments, the server computer 10 includes a communication interface 12 (e.g. configured to receive data, i.e. requests, service requests). The communication interface 12 may include a transmitter module and / or a receiver module allowing the server computer 10 to communicate over a communications network. The communication interface 12 may include one or more user-interfaces configured to provide users for user control and may include, for example, one or more computing peripheral devices such as display monitors, computer keyboards and the like.

[0055] The server computer 10 may further include a processor in the form of processing unit 14 and a memory 16. The memory 16 may be used by the processing unit 14 to store, for example, data to be processed, including the image data and any associated data, metadata, etc.

[0056] As shown in FIG. IB, there is a system 100 for processing image data utilizing the server computer 10. In operation, the processing unit 14 of the server computer 10 may beconfigured to receive, for example from one or more computer devices 15, a request for a matched image 1001 corresponding to a ground scene captured by an image sensor. Upon receipt of the request for a matched image 1001, the processing unit 14 may be configured to: obtain a multi-modal dataset, the multi-modal dataset comprising two-dimensional (2D) ground-view image data 1002 of a scene and a corresponding point-cloud data 1003 representing the scene; extract, using a first neural network backbone module 112, a first set of feature representation 1004 from the 2D ground-view image data 1002; extract, using a second neural network backbone module 114, a second set of feature representation 1005 from the corresponding point-cloud data 1003; fuse, using a fusion embedding construction module 116, the first set of feature representation 1004 and the second set of feature representation 1005 to generate a fusion embedding 1006; generate, using a modal-wise fusion module 118, a 2D modality-specific descriptor 1007 and a 3D modality-specific descriptor 1008 based on the fusion embedding 1006; and combine the 2D modality-specific descriptor 1007 and the 3D modality-specific descriptor 1008 to form a fused ground scene descriptor 1009. The fused ground scene descriptor 1009 may subsequently be used to retrieve the matched image 1110.

[0057] The image data 1002 of the scene may be obtained from a first sensor 110, and the point cloud data 1003 may be obtained from a second sensor 120. In some embodiments, the second sensor 120 may include one or more lidar sensors.

[0058] As illustrated in FIG. IB, the various modules and data flow may be facilitated by a network 180. The network 180 may be an internal network, such as an intranet, or may be the internet. In some embodiments, the network 180 may form part of a cloud network. It is contemplated that although the different modules 112, 114, 116 and 118 are illustrated as separate units from the processing unit 14, one or more of the modules may be integrated with the processing unit 14 to form further embodiments

[0059] In some embodiments, the system of FIG. IB may be utilized or adopted in a multimodal aerial-ground PR system, as illustrated in the system 200, which is represented as a pipeline of multi-modal aerial-ground PR.

[0060] As shown in FIG. 2, the system 200 comprises a database arranged in data and / or signal communication with the server apparatus 10. The database may be, or may comprise, an aerial -view map database 202 configured to store a plurality of aerial images 212 of different scenes, and wherein each of the plurality of aerial images 212 is associated with an aerial-view descriptor 210. A multi-modal dataset 204, which may include ground-view image data, may be configured as input into the server computer 10. The multi-modal dataset 204 may beprocessed by the server computer 10 to obtain a ground-view query descriptor 206, which may include the fused ground scene descriptor 1009.

[0061] The ground-view query descriptor 206 may then be compared, using a comparator 208, with each of the aerial-view descriptor 210 to determine a match, wherein in a positive determination of a match, a matched aerial image associated with the matched aerial-view descriptor will be selected. The selected matched aerial-view descriptor may then be sent to a computer device, such as a user device, having a user interface for displaying the matched aerial image.

[0062] In some embodiments, the ground data collected with multiple modalities, may comprise both images and point clouds. In some embodiments, the aerial data may encompass various formats, including real captured aerial RGB images as well as human-built semantic maps, such as, but not limited to, terrain maps and building maps. In some embodiments, the aerial images 212 may be generated or stored using an aerial PR model. In some embodiments, the aerial PR model may be constructed using off-the-shelf pre-trained 2D vision models 214 without any fine-tuning (e g., ResNet [1]). This may be done to facilitate the generalization of database representations, making it consistent with off-the-shelf pre-trained vision models.

[0063] In some embodiments, the system 200 may comprise an image capturing device (not shown), the image capturing device configured to obtain the ground-view image data of the scene, and a light detection and ranging (lidar) sensor configured to obtain corresponding pointcloud data representing the scene.

[0064] FIG. 3 shows an architecture of an embodiment of the system 100, in the form of a pipeline 300, of a scene descriptor model for multi-modal aerial-ground PR. The pipeline utilizes a manifold to fuse multi-modal information by learning a plurality of manifold chart functions 306 to relate points in 2D or 3D spaces to the manifold. A fusion embedding may then be constructed using, for example, neural ordinary differential equations (ODEs) 308 based functions, which describe the movement of points on the manifold. Subsequently, the updated fusion embedding acts as guidance for the extraction of respective modal features to formulate the final scene descriptor.

[0065] The architecture comprises a first stage 302 A and a second stage 302B. In the first stage 302A, a ground image 304A and Bird’s Eye View (BEV) point cloud 304B are processed with separate backbone branches, the output features of which are used to build the fusion embedding 1006 (denoted as efuse). In the second stage 302B, the constructed fusion embedding 1006 is mapped into the respective modal spaces to achieve further modal-wise featureextraction. The respective model spaces may comprise the first chart function 312 to generate the 2D mapped fusion embedding 314, and a second chart function 316 to generate a 3D mapped fusion embedding 318 Further details of the workings may be made with reference to Equations (2) to (16).

[0066] The 2D and 3D mapped fusion embedding 314, 318 may be input into a 2D decoder block 332 and a 3D decoder block 334, and respective pooling functions 336 and 338, which in turn is used to generate the 2D modality-specific descriptor 1007 and the 3D modalityspecific descriptor 1008.

[0067] In the embodiment of FIG. 3, the aerial-ground multi-modal place recognition query model may be formulated as an optimization problem for cross-modal retrieval between ground-level and aerial imagery data, mathematically expressed in Equation (1), as follows.where fGdenotes the ground multi-modal query model, fAdenotes the aerial database model, IGand PGare ground query images and point clouds respectively, IAis the aerial database maps, Rlis the positive neighborhood set of query z, T is the indicator function, and £ is a feature distance function.

[0068] It may be appreciable that the Equation (1) defines an optimization objective that maximizes a parameter fa, fAand sums over all query instances i and database instances j . An indicator function T is used to count successful retrievals. The objective function counts how often the retrieved aerial item (based on minimal feature distance) is in the correct positive neighborhood for each ground query.

[0069] For each ground query i, the system retrieves the aerial item j with the most similar features, and checks if it is in the set R1(i.e., the set of relevant items for query z).

[0070] In some embodiments, T may be an indicator function that returns 1 if the condition is met, 0 otherwise.

[0071] Referring to FIG. 3, and in relation to modal -specific feature extraction, i.e. the extraction of the first set of feature representation from the 2D ground-view image data using the first neural network backbone module 112, and the extraction of a second set of feature representation from the corresponding point-cloud data using a second neural network backbone module 114, a cross-view visual place recognition solution that uses two differentneural networks (i.e., not sharing weights) to handle aerial and ground inputs respectively. For the aerial part, denoted as.4(-), the visual place recognition (VPR) for the 2D modality may be assumed to be well-established, and therefore an off-the-shelf 2D VPR methods (e g., ResNet- GeM[2]) to extract aerial descriptors may be contemplated.

[0072] Tn relation to the ground part, denoted as / G ( , ) that receives the multi-modal query inputs comprising the 2D image and 3D point cloud query inputs, separate 2D and 3D backbones may be used to extract the respective basic features in each modality first. Using hierarchical information may be used to extract multiple feature maps. The feature maps at the l-th backbone block may be denoted as F m and F D, and all the gathered feature maps are denoted respectively in Equation (2) and Equation (3), mathematically expressed as:where Nlis the number of feature vectors at block / , Clis the number of corresponding feature channels, and L is the total number of backbone blocks. For images, a standard 2D format may be utilized. For point clouds, a Bird’s Eye View (BEV) projection technique may be leveraged to convert the point clouds into one or more 2D representations. Such approach may be referenced in point cloud place recognition (PR) as well as one or more other related tasks. By projecting point clouds, the 2D representations may be used to extract point cloud features. Such an approach may eliminate the need for additional CUDA-dependent frameworks for complex 3D processing, making it more suitable for practical deployment and increasing its compatibility.

[0073] In some embodiments, the BEV projection performed on the point cloud {p, =mapping each point p, to (x,, y;) coordinates, generating the BEV point cloud IBEVG IF?Vxlv. where each pixel value corresponds to a height value. The BEV projection may be mathematically represented in Equation (4) as follows:where xmaxand ymaxrepresent the maximum LiDAR range in the x and y directions (roll and pitch axes), respectively.

[0074] As an alternative embodiment to the BEV projection, a spherical projection may be used. For the spherical projection, the Cartesian coordinates may be first converted into polar coordinates, mathematically expressed in Equation (5) as follows:<y »> . = arc 0 zn; = arc 4t.an l I ^ —l\ I , r£= x^ 2 + i y^ 2 + i z^ 2. (5)

[0075] Projecting all the N points, an image Isph6 IRWxlvmay be obtained, in which each pixel contains the radius value denoted as n, mathematically represented in Equation (6) as follows:

[0076] In focusing on aerial-ground PR, the BEV method may inherently align the data representation of the ground point cloud and aerial maps more effectively.

[0077] As illustrated in FIG. 3, a two-stage fusion strategy between different modalities in the multi-modal PR system may be adopted. In some embodiments, in the first stage, in addition to the vanilla 2D and 3D feature branches, a third branch may be introduced as a surrogate space for fusing feature information from the 2D and 3D branches and constructing the fusion embedding. In the second stage, the fusion embedding in the surrogate space guides the respective 2D and 3D branches for feature updating.

[0078] At the fusion embedding construction stage, the extracted feature representations, in the form of feature maps F2D and F3D, may be leveraged and fused from each modal backbone to build the fusion embedding. Inspired by the concept of space point motion on manifolds in differential geometry, the fusion embedding construction may be modeled as a multi-block (multiple neural ODE blocks 402) state evolution process, as illustrated in the fusion embedding evolution processes shown in FIG. 4A and FIG. 4B. The state evolution process may start from a last block 402C and progress towards a first block (i.e., L 1), allowing the embedding state to incorporate more global -level information at later blocks that is beneficial for the PR task. As illustrated in FIG. 4B, there comprises three blocks 402C, 402B, 402A, with the first block being 402A, and the L block as 402C.

[0079] In other words, each of the plurality of neural ODE blocks may be configured to progressively refine the fusion embedding by integrating one or more feature trajectories over a time period, such that an output of each neural ODE block is used as an input to a next neural ODE block. The input to the next neural ODE block comprises a set of features, wherein theset of features comprises a summation of feature output(s) from one or more previous neural ODE blocks.

[0080] In some embodiments, the evolution may comprise two sub-processes: (a) state initialization and (b) state updating at each block.

[0081] Tn the state initialization stage, at each backbone block / , the initial fusion state, denoted as n the C-dimensional manifold Mc, may be mathematicallydefined in Equation (7) as follows:t — f ? t / wherein T is the state updating end time, andra fctrats=i is the fusion state momentum. Specifically, mzmay be constructed using feature maps from the respective / -th backbone block, mathematically expressed in Equation (8) as follows: (8)

[0082] Here, f^D, f^Dearethe 2D and 3D embeddings from the respective modal space to the fusion manifold Mc, mathematically expressed in Equation (9) as follows: (9)wherein Pooling is the global average pooling function, and <P2D> <P3Darelearnable manifold chart functions representing homeomorphisms between the 2D / 3D space and the fusion manifold.

[0083] As an alternative to Equation (8), mzmay be mathematically expressed in Equation (8 A), as follows: (8 A)arelearnable manifold chart functions representing connections between the 2D / 3D space and the fusion manifold A chart for a manifold M provides a local coordinate system, capturing local structures of M .fgDinEquation (8A) are the pooled 2D and 3D features from the respective modal space, mathematically represented in Equation (9 A), as follows:where Pool(.) is the global average pooling function. The pooling function may highly summarize the global information of the whole scene, which is beneficial for constructing a fusion embedding with rich global multi-modal domain representations and thus benefits thePR task. This differs from dense prediction tasks (e.g. depth estimation, segmentation) that more dig into local patterns.

[0084] In some embodiments, a chart for a manifold M provides a local coordinate system by establishing a homeomorphism <p from an open subset ! / to an open subset of a Euclidean space. The chart can also be denoted as a pair (! / , <p). An atlas for a manifold M encompasses an indexed family of charts, each capturing local structures of M.

[0085] In some embodiments, the state may be updated, based on, or inspired by, Riemannian geometry, the movement of a point on a manifold can be described by a smooth parametric curve. The process of updating the fusion embedding (state) may be modeled as the movement of a point in the manifold

[0086] In some embodiments, ordinary differential equations (ODEs) may serve as a tool for describing the dynamics of dynamical systems. ODEs may be used to express how a system state evolve over time, capturing relationships between the system variables and the rates of change. Based on this, the smooth movement dynamics on a manifold can be described by an ODE, mathematically expressed in Equation (10) as follows:wherein y(t) denotes the state trajectory.

[0087] In the proposed pipeline, the fusion embedding state may be updated from its initial state with parameterized neural ODEs at each block / , which can be regarded as point movements on the manifold, mathematically defined in Equation (11) as follows:where yl(t) denotes the fusion embedding state trajectory at block I, f denotes a non-linear neural block with learnable parameters 6lthat describes the trajectory dynamics. By solving (11), the updated state yz(T) at the end time T can be obtained. At a next block I - 1, the start state yz-1(0) may be initialized based on the last end state yz(T) and momentum mz-1, as shown in FIG. 4 and Equation (7). After progressive updating, the end state at the first block y1(T) as the final constructed fusion embedding efuse= y1(T) may be obtained.

[0088] In some embodiments, the fusion embedding efusecaptures rich multi-modal fusion information, which in turn serves as effective guidance for individual modal feature learning. In some embodiments, the fusion embedding may be adapted into the corresponding modal space to obtain 2D and 3D embeddings, mathematically expressed in Equation (12) as follows:(12) wherein i / huse^D, Vhuse.sDarechart functions that may be used to establish a homeomorphism from the fusion manifold to the respective 2D and 3D spaces. As shown in FIG. 18, with the help of included fusion embedding in this stage 2 (compare w / o fusion embedding with the present disclosure), the final scene descriptor could be more robust against sensor failing. These 2D and 3D embeddings capture informative multi-modal features, which are further combined with the original modal features F2Dand F(Dfrom the last block L to construct more effective representations through decoder blocks g2v, 9SD , and 5fUse, mathematically expressed in Equations (13) to (15), as follows:wherein © denotes broadcast addition, and i^2D,fuse, V'sDjusearechart functions. The decoded embeddings e2D, e3D, and e[useis used to construct the final scene descriptor.

[0089] In some embodiments, the fused ground scene descriptor may comprise three parts, the three parts comprising the fusion embedding, as well as two modal-specific embeddings. The fused ground scene descriptor, denoted as efina], may be mathematically expressed in Equation (16) and Equation (16A), as follows:wherein X is a learnable weight.

[0090] In some embodiments, the network may be trained using a triplet marginal loss function, mathematically expressed in Equation (17), as follows: (17)wherein ea, ep, enare the descriptors of an anchor, a positive sample, and a negative sample, respectively, and m is the margin hyperparameter and II II2is the G-norm.

[0091] In some embodiments of the multi-modal cross-view PR task, the features may be interacting not only in different modalities (2D and 3D) but also in different views (aerial and ground). Thus, sufficient feature alignment across various domains is critical for this PR task. To achieve this goal, a multi-view multi-modal (MVMM) loss function may be proposed. In some embodiments, the MVMM loss function may treat PR as a binary classification problem,where positive pairs may have smaller feature distances and are assigned ground-truth labels “0”, while negative pairs have ground-truth labels “1”.

[0092] In some embodiments, given two sets of embeddings Di={eii}ie[Ni], D2 ={e2j}je[N2] from two domains, the basic domain distance loss may be defined based on Equation (18), mathematically expressed as follows:+ (1 - yd log(i - dhu - e2ihl) ) - / 1 O,(18 ) where yij GE {0, 1 } indicates positive and negative pair labels, and o( ) denotes a sigmoid function. The MVMM loss function may then be defined based on Equation (19) as follows:where DA = {eAijieM is a set of aerial descriptors. DG = {ecjJjeN, DG2D = {f2Dj}j£N, DGJD = { fsDj !i ' N are sets of corresponding ground final, 2D, and 3D embeddings. It may be appreciable that the MVMM loss aligns features not only in different domains but also within the same domains. That is, the MVMM loss also optimizes the ground-ground and aerial-aerial feature relationships, which could better help cross-view PR. As shown in FIG. 24, the “roof’ shown in the image can only be perceived by aerial views is emphasized, which indicates the aerial-aerial same-view discrimination is implicitly hidden in the whole cross-view discrimination process. In addition, the MVMM loss also directly aligns ground 2D and 3D features to the aerial features, which may implicitly contribute to more effective aerial -ground alignment for cross-view PR.

[0093] The above MVMM loss function may be regarded as the direct guidance that forces pair distances close to some target values.

[0094] In some embodiments, the negative pairs may be configured to have greater distances than positive pairs, which can be regarded as the relative guidance. Specifically, the traditional triplet loss may be used to achieve the relative guidance based on Equation (20), mathematically expressed as follows:where m is the margin. e , e” are positive and negative aerial descriptors sampled from DA. In some embodiments, hard negative mining may be adopted during training. The final loss function is a combination of the directly guided MVMM loss and the relatively-guided triplet loss with the weight a, mathematically expressed in Equation (21) as:£ = a^MVMM + Cfri (21)Experiments

[0095] To evaluate the performance of multi-modal aerial-ground PR models, benchmark datasets that consist of comprehensive ground sensor data and corresponding aerial database maps were created. In the examples shown in FIG. 5, two datasets, namely KITTI-AG and Ithaca365-AG, were specifically constructed. These two datasets are designed to provide a diverse range of scenarios for evaluating the models’ robustness and effectiveness. FIG 5 further shows a Ground query data visualization from the KITTI-AG and Ithaca365-AG datasets. Each block consists of a ground image, a ground point cloud, and the corresponding most positive aerial image.

[0096] KITTI-AG dataset — For the ground dataset, a vanilla KITTI [3] dataset may be used as the base to provide ground-view images, point clouds, and global navigation satellite system (GNSS) coordinates. The KITTI dataset is widely recognized in computer vision and autonomous driving research, offering a diverse range of real-world data for algorithm development and evaluation. KITTI incorporates various sensor modalities, including cameras, LiDAR sensors, and GNSS or inertial navigation systems, providing a comprehensive view of the environment.

[0097] The aerial data may be generated based on the GNSS coordinates of the ground data frames. GNSS coordinates within the area covered by the KITTI dataset may be randomly sampled, and Google Maps Static API may be used to download the corresponding aerial images, including, but not limited to, satellite images, terrain maps, and building maps. To introduce variability in the aerial data, noise may be added to the original GNSS coordinates. The Google Maps Static API parameters may be set as follows: scale=l, zoom=19, size=640x640. Each aerial image covers an approximately 100x100 m2area.

[0098] Ithaca365-AG dataset — The evaluation on the KITTI-AG dataset primarily focuses on assessing the model’s performance in normal driving conditions without significant perturbations. However, it may be important to ensure the robust operation of these systems under diverse weather conditions, including challenging scenarios under adverse conditionsassociated with snow and rain, to meet stringent safety requirements. To evaluate the model’s robustness in challenging environments, the Ithaca365-AG dataset may be introduced. The ground data is based on the vanilla Ithaca365 [4] dataset, which offers a comprehensive representation of real-world driving conditions. The dataset may include various scenes, weather conditions (snow, rain, and sunny weather), times of the day, and traffic scenarios (urban, highway, rural, and campus scenes). For the aerial data, the same processing procedure may be followed as mentioned above.

[0099] In the implementation, state-of-the-art (SOTA) multi-modal PR models as ground network baselines for query descriptor construction, including advanced neural network models or algorithms for PR such as MinkLoc++ , AdaFusion , and LCPR may be used. The VSGP-PR algorithm is also implemented as another baseline. For the aerial database, ResNet- 18 with GeM as the database model may be used. The aerial -view network is frozen while only updating the ground-view network during training. All compared models are trained for 50 epochs. The recall@K (K = 1, 5, 10) (R@K) is used as the metric. The positive retrieval threshold is set as 25 m. All experiments are conducted on a Tesla A100 GPU.

[0100] Main Results — Satellite RGB images. An extensive evaluation of the present PR system and system 100, 200, 300 is performed based on the KITTI-AG and Ithaca365-AG datasets. The results shown in the table of FIG. 6 demonstrate the robustness and efficacy of the present PR system and system 100, 200, 300 in tackling the multi-modal aerial-ground PR task. In the KITTI-AG dataset which contains normal situations, all models achieve commendable performance metrics, with R@1 scores surpassing 30% and R@10 scores exceeding 50%. The best results are highlighted in. The present PR system and system 100, 200, 300 is demonstrated to outperform all other baselines, with significant improvements in R@l, R@5, and R@10 scores.

[0101] In the more challenging dataset Ithaca365-AG, which consists of snowy, rainy, and cloudy weather conditions, all models experience a significant drop in performance. However, present PR system and system 100, 200, 300 continues to achieve the highest recall scores, demonstrating its superior robustness against these challenging perturbations.

[0102] Semantic Maps — Other types of aerial data sources, such as semantic maps, may also be used for evaluation. In addition to satellite RGB images, the performance of terrain maps and building maps, which contain only semantic information, are evaluated. The visualization of these three aerial data sources (including satellite RGB images, building maps, and terrain maps) are depicted in FIG. 8. The performance of the present PR system and system100, 200, 300 with other baseline models across different aerial scenarios, as presented in the table of FIG. 7.

[0103] It may be appreciable that terrain maps, which contain less detailed information, exhibit the lowest performance compared to satellite RGB images and building maps. On the other hand, building maps, which include additional information about building structures, achieve similar performance to satellite RGB images. This suggests that semantic maps, which are easily accessible and created without the need for satellites, UAVs, or airplanes, can already provide sufficient information for practical PR tasks. This finding may open up applications in one or more fields.

[0104] In some embodiments, ablation studies may also be carried out. In module ablation, the effectiveness of each module may be evaluated by ablating each module in the network. As shown in FIG. 9, both the two fusion stages introduced can contribute to a better final performance, which verifies the effectiveness.

[0105] Fusion Direction and ODE Updating — The performance of different fusion directions, as outlined in the table in FIG. 10, may be evaluated. The direction “block L block 1”, which transfers information from high-level summaries to detailed low-level representations, emerges as the preferred choice for fusion embedding construction. During the state evolution process, the fusion embedding state trajectories may be parameterize as learnable neural ODEs. Notably, upon removing the ODE updating mechanism, performance deteriorates across all metrics, underscoring the necessity of the ODE-based state updating process (see table depicted in FIG. 11)

[0106] Modality and Point Cloud Representations — The performance of various modality inputs, including pure 2D, pure 3D, and combined 2D+3D inputs, as illustrated in FIG. 12, may be evaluated. The findings demonstrate that leveraging multiple sensors leads to significant performance improvements compared to single-sensor inputs, highlighting the effectiveness of multi-modal inputs.

[0107] Moreover, the representation of the point cloud may be another consideration. Three different point cloud formats (vanilla, BEV projection, spherical projection) are compared in FIG. 13. Among them, the BEV would be the optimal representation format, as it could inherently better align the aerial image data.

[0108] Runtime Speed — The runtime speed and memory consumption of the model may also be assessed or evaluated. FIG. 14 shows a table of a runtime performance comparison on a Tesla A100 using different PR models. The present PR system and system 100, 200, 300demonstrates medium-level real-time performance, satisfying basic deployment and application requirements. All compared models exhibit comparable memory usage, with no significant discrepancies observed in this metric.

[0109] The aforementioned experiments demonstrate that AGPlace outperforms previous baselines and performs well in both normal and challenging situations, facilitating the suitability for deployment in real-world environments.

[0110] While the experiments primarily focuses on ground query data collected in urban environments, it is appreciable that the dataset and model may be expanded to include more diverse and challenging scenarios, such as forests, deserts, and coastlines, with an aim to establish a more comprehensive and generalized multi-modal cross-view PR pipeline.

[0111] FIG. 15 shows an aerial-ground scene descriptor distance visualization heatmap images, with fusion manifold labelled as 1501, and without fusion manifold labelled as 1502, and the aerial-ground geographical distances labelled as 1503.

[0112] The visualization displays heat maps showing cross-view geometry relationships between aerial and ground perspectives, with distance measurements represented ranging from close to far. The scene descriptor distance is the Euclidean (L2) distance between ground scene descriptors and aerial scene descriptors

[0113] In some other experiments, the mentioned two datasets, namely KITTI360-AG and nuScenes-AG, shown in FIG 16, may be tested on the Oxford RobotCar benchmark datasets used in previous multi-modal ground-ground PR evaluations, the data visualization of which is shown in FIG. 16.

[0114] For the ground dataset, the vanilla KITTI360 [5] dataset as the base to provide ground-view images, point clouds, and GNSS coordinates. A total of 7 sequences was used, in which the first 85% frames are for training and the last 15% frames are for testing, such that the test frames consist of seen and unseen areas for both fitting and generalization evaluation. The aerial data is generated based on the GNSS coordinates of the ground data frames. The Google Maps Static APB was used to download the corresponding aerial images, including satellite images, and road maps.

[0115] The Google Maps Static API parameters may be set as follows: scale=l, zoom=20, size=640><640. Each aerial image covers an approximately 75x75 m2area.[001 16] nuScenes-AG. The nuScenes [6] dataset provides a rich collection of sensor data collected from real urban driving scenarios, which also contains multi -view cameras. The aerial data based on GNSS coordinates with similar configurations mentioned in KITTI360-AG maybe downloaded. The official train / test split is used in our setting, such that test scenes contain seen and unseen areas for both fitting and generalization evaluation.

[0117] In some embodiments, the Oxford RobotCar Benchmark may be used The Oxford RobotCar benchmark dataset is a public benchmark used for multi-modal ground-ground PR. There are two different splits used by MinkLoc++ (Oxford-Mink+) and AdaFusion (Oxford- Ada) respectively. The PR model may be compared under both settings.

[0118] Implementation. The selected baselines mainly comprise four types, including ground- view 2D / 3D / multi-modal baselines and aerial-ground 2D baselines. The ResNet-18- Avg may be used as the aerial network for all PR works except for AnyLoc for a fair comparison (with a head to align descriptor dimensions). SOTA multi-modal models may be used as ground network baselines for query descriptor construction, mainly including advanced neural network algorithms / models such as MinkLoc++, AdaFusion, LCPR, MSSPlace, and UMF, VSGP-PR does not provide open-source codes and EINet provides data splits without model codes, which may hinder comparison. The neural backbones may be constructed based on MinkLoc++. Recall@K (K = 1, 5, 10) (R@.K), average recall@l%(AR@l%) may be used, and average recall @1(AR@1) as the metrics. The positive retrieval threshold is set as 25 metres (m). All experiments are conducted on a Tesla Al 00 GPU.

[0119] Based on the use of the KITTI360-AG (Satellite and Roadmap Databases) dataset, an evaluation of the present PR system and system 100, 200, 300 on the KITTI360-AG dataset demonstrated in FIG. 17 (See the best is in bold and second best is underlined). The present PR system and system 100, 200, 300 is shown to outperform all other baselines in both satellite and road map settings. On the other hand, when using the road map aerial database, most models can already show considerable recall performance. This suggests that semantic maps, which are easily accessible and created without the need for satellites, drones, or airplanes, can already provide sufficient information for practical PR tasks. The finding may open up possibilities in the navigation field. The 2D VPR methods were also tested / configured with the strong foundation backbone DINOV2[7], some of which can even surpass 3D counterparts.

[0120] nuScenes-AG (Ground Sensor Failing) dataset. The nuScenes-AG dataset that comprises a LiDAR and multiple cameras is also tested / evaluated, where the present PR system and system 100, 200, 300 can also achieve significant performance. With the fusion embedding injected into the respective 2D / 3D spaces in the second stage (stage 2), the final scene descriptor demonstrates stronger robustness both camera and LiDAR failing. Thisdemonstrates that the present PR system and system 100, 200, 300 can be more applicable in real deployment conditions.

[0121] Oxford Benchmark Dataset. To investigate the inherent power of the present PR system and system 100, 200, 300, a comparison was performed with previous SOTA baselines using the ground-ground PR task. As shown in FIG. 19, which shows a table comparing ground-ground PR Results on the Oxford benchmark datasets ("-" denotes the result is not provided), the present PR system and system 100, 200, 300 (without using extra data, reranking, or multiple cameras) achieves better performance than previous SOTA counterparts, which further verifies the effectiveness of the proposed system 100, 200, 300.

[0122] In some embodiments, ablation studies may also be carried out. In module ablation, the effectiveness of each module may be evaluated by ablating each module in the network. As shown in the table of FIG. 20, the two fusion stages illustrated in FIG. 3, and the MVMM loss can contribute to the better final performance, confirming the efficacy of the present approach (see system 100, 200, 300).

[0123] ODE Updating and Fusion Direction. During the state evolution process, the fusion embedding state trajectories may be parameterize as learnable neural ODEs. Notably, in the table depicted in FIG. 22, ODEs is found to perform better than multi-layer perceptron (MLP) and attention, both of which cannot guarantee outputs’ differences.

[0124] In addition, the t-SNE visualization in FIG. 21 shows ODEs can build more consecutive scene descriptors, underscoring the necessity of the ODE-based fusion state updating process for PR.

[0125] Next, the performance of different fusion directions, as outlined in the table shown in FIG. 22, are compared. The direction "block L ->• block 1”, which transfers information from high-level summaries to detailed low-level representations, emerges as the preferred choice for fusion embedding construction.

[0126] Aerial Modality and Salience Visualization — The fusing of satellite images and road maps for more representative aerial descriptors was explored. As shown in the table illustrated in FIG 23, the fusion on both ground and aerial sides provides an even better performance. To investigate which part of ground and aerial inputs are focused in the PR network, the salience maps are visualized as shown in FIG. 24. The geometric landmarks are emphasized (e.g. buildings and roads), which verifies the geo-feature extraction effectiveness of the proposed system 100, 200, 300 and the alignment effectiveness of the loss function.

[0127] Runtime Performance — The runtime speed and memory consumption of the proposed system 100, 200, 300 were also assessed, see the table illustrated in FIG. 25. The proposed system 100, 200, 300 is shown to demonstrate acceptable real-time performance, satisfying basic deployment and application requirements. All compared models exhibit comparable memory usage, with no significant discrepancies observed in this metric.

[0128] In summary, the proposed multi-modal model for addressing the aerial -ground PR problem, leverages neural ODEs rooted in differential geometry to achieve effective multimodal feature interaction. The experiments demonstrate that the proposed system 100, 200, 300 outperform previous baselines and performs well in large-scale scenes, making it suitable for deployment in real-world environments. In some embodiments, the dataset and model may be expanded to establish a more comprehensive and generalized multi-modal cross-view PR pipeline.

[0129] According to an aspect of the present disclosure, there is provided a vehicle, the vehicle comprising an image capturing device configured to obtain a two-dimensional (2D) ground-view image data of a scene; a light detection and ranging (lidar) sensor configured to obtain corresponding point-cloud data representing the scene; a processor, the processor comprising: an input module arranged in signal or data communication with the image capturing device and the lidar sensor, the input module configured to obtain the 2D groundview image data and the corresponding point-cloud data representing the scene; an neural network module, the neural network module configured to: extract, using a first neural network backbone module, a first set of feature representation from the 2D ground-view image data; extract, using a second neural network backbone module, a second set of feature representation from the corresponding point-cloud data; fuse, using a fusion embedding construction module, the first set of feature representation and the second set of feature representation to generate a fusion embedding; generate, using a modal-wise fusion module, a 2D modality-specific descriptor and a 3D modality-specific descriptor based on the fusion embedding; and combine the 2D modality-specific descriptor and the 3D modality-specific descriptor to form a fused ground scene descriptor.

[0130] In some embodiments, the vehicle further comprises a data communication device arranged in data communication with an aerial-view map database, the aerial-view map database comprising a plurality of aerial images, wherein each of the plurality of aerial images is associated with an aerial -view descriptor.

[0131] In some embodiments, the processor comprises a comparator, the comparator configured to compare the fused ground scene descriptor with each of the aerial -view descriptor to determine a match, wherein in a positive determination of a match, a matched aerial image associated with the matched aerial-view descriptor is sent to a user device.

[0132] According to another aspect and with reference to the flowchart of FIG 26, there is provided a generalized method 600 for processing image data comprising the following:

[0133] Step S601 : obtaining a multi-modal dataset, the multi-modal dataset comprising two-dimensional (2D) ground-view image data of a scene and a corresponding point-cloud data representing the scene;

[0134] Step S602: extracting, using a first neural network backbone module, a first set of feature representation from the 2D ground-view image data;

[0135] Step S603: extracting, using a second neural network backbone module, a second set of feature representation from the corresponding point-cloud data,

[0136] Step S604: fusing, using a fusion embedding construction module, the first set of feature representation and the second set of feature representation to generate a fusion embedding;

[0137] Step S605: generating, using a modal-wise fusion module, a 2D modality-specific descriptor and a 3D modality-specific descriptor based on the fusion embedding; and

[0138] Step S606: combining the 2D modality-specific descriptor and the 3D modalityspecific descriptor to form a fused ground scene descriptor.

[0139] In the step S604, the fusion embedding construction module may comprise a plurality of neural ordinary differential equation (ODE) blocks. In some embodiments, each of the plurality of neural ODE blocks is configured to progressively refine the fusion embedding by integrating one or more feature trajectories over a time period, such that an output of each neural ODE block is used as an input to a next neural ODE block.

[0140] It may be appreciable that the method may be implemented in a non-transitory computer-readable medium storing computer executable code comprising instructions for processing image data.

[0141] It is contemplated that the outcome of the present disclosure, i.e., the fused ground scene descriptor, may be used to obtain a matched image

[0142] While the disclosure has been particularly shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of thedisclosure as defined by the appended claims. The scope of the disclosure is thus indicated by the appended claims and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced.REFERENCES[1]. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778 (2016)[2], Filip Radenovi'c, Giorgos Tolias, and Ond’rcj Chum. Finetuning CNN image retrieval with no human annotation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(7): 1655-1668, 2018.[3], Geiger, A., Lenz, P., Stiller, C., Urtasun, R : Vision meets robotics: The kitti dataset. The International Journal of Robotics Research 32(11), 1231-1237 (2013)[4], Diaz-Ruiz, C.A., Xia, Y., You, Y., Nino, J., Chen, J., Monica, J., Chen, X., Luo, K., Wang, Y., Emond, M , et al.: Ithaca365: Dataset and driving perception under repeated and challenging weather conditions. In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 21383-21392 (2022)[5], Yiyi Liao, Jun Xie, and Andreas Geiger. KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2D and 3D. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3292-3310, 2022.[6], Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE / CVF conference on Computer Vision and Pattern Recognition, pages 11621-11631, 2020.[7], Maxime Oquab, Timothee Darcet, Theo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin ELNouby, et al. DINOv2: Learning robust visual features without supervision. ArXiv preprint arXiv:2304.07193, 2023.

Claims

CLAIMS1. A system for processing image data, the system comprising a processor, the processor configured to: obtain a multi-modal dataset, the multi-modal dataset comprising two-dimensional (2D) ground-view image data of a scene and a corresponding point-cloud data representing the scene; extract, using a first neural network backbone module, a first set of feature representation from the 2D ground-view image data; extract, using a second neural network backbone module, a second set of feature representation from the corresponding point-cloud data, fuse, using a fusion embedding construction module, the first set of feature representation and the second set of feature representation to generate a fusion embedding; generate, using a modal-wise fusion module, a 2D modality-specific descriptor and a 3D modality-specific descriptor based on the fusion embedding; and combine the 2D modality-specific descriptor and the 3D modality-specific descriptor to form a fused ground scene descriptor.

2. The system of claim 1, wherein the fusion embedding construction module comprises a plurality of neural ordinary differential equation (ODE) blocks.

3. The system of claim 2, wherein each of the plurality of neural ODE blocks is configured to progressively refine the fusion embedding by integrating one or more feature trajectories over a time period, such that an output of each neural ODE block is used as an input to a next neural ODE block.

4. The system of claim 3, wherein the input to the next neural ODE block comprises a set of features, wherein the set of features comprises a summation of feature output(s) from one or more previous neural ODE blocks5. The system of claim 4, wherein the one or more feature trajectories of the fusion embedding within each neural ODE block is parameterized.

6. The system of any one of the preceding claims, wherein the modal-wise fusion module is trained using a triplet marginal loss function using an anchor parameter, a positive sample, and a negative sample7. The system of any one of the preceding claims, wherein the modal-wise fusion module is configured to process the fusion embedding using a first chart function to generate a 2D mapped fusion embedding, and a second chart function to generate a 3D mapped fusion embedding.

8. The system of claim 7, wherein the first chart function and the second chart function are learnable manifold chart functions.

9. The system of claim 7 or 8, wherein the modal-wise fusion module is further configured to project the 2D mapped fusion embedding using a 2D feature, and project the 3D mapped fusion embedding using a 3D feature10. The system of any one of the preceding claims, wherein the first neural network backbone module and the second neural network backbone module are each configured to generate a sequence of a number of feature extraction blocks and output a 2D feature map and a 3D feature map respectively.

11. The system of any one of the preceding claims, further comprising a database, the database configured to store a plurality of aerial images, and wherein each of the plurality of aerial images is associated with an aerial-view descriptor.

12. The system of claim 11, wherein the processor further comprises a comparator, the comparator configured to compare the fused ground scene descriptor with each of the aerialview descriptor to determine a match, wherein in a positive determination of a match, a matched aerial image associated with the matched aerial-view descriptor is sent to a user device.

13. The system of any one of the preceding claims, further comprising an image capturing device configured to obtain the ground-view image data of the scene, and a light detection and ranging (lidar) sensor configured to obtain corresponding point-cloud data representing the scene.

14. A vehicle, the vehicle comprising an image capturing device configured to obtain a two-dimensional (2D) ground-view image data of a scene; a light detection and ranging (lidar) sensor configured to obtain corresponding pointcloud data representing the scene; a processor, the processor comprising: an input module arranged in signal or data communication with the image capturing device and the lidar sensor, the input module configured to obtain the 2D ground-view image data and the corresponding point-cloud data representing the scene; an neural network module, the neural network module configured to: extract, using a first neural network backbone module, a first set of feature representation from the 2D ground-view image data; extract, using a second neural network backbone module, a second set of feature representation from the corresponding point-cloud data, fuse, using a fusion embedding construction module, the first set of feature representation and the second set of feature representation to generate a fusion embedding; generate, using a modal-wise fusion module, a 2D modality-specific descriptor and a 3D modality-specific descriptor based on the fusion embedding; and combine the 2D modality-specific descriptor and the 3D modality-specific descriptor to form a fused ground scene descriptor.

15. The vehicle of claim 14, further comprising a data communication device arranged in data communication with an aerial-view map database, the aerial-view map database comprising a plurality of aerial images, wherein each of the plurality of aerial images is associated with an aerial-view descriptor.

16. The vehicle of claim 15, wherein the processor comprises a comparator, the comparator configured to compare the fused ground scene descriptor with each of the aeri l -view descriptor to determine a match, wherein in a positive determination of a match, a matched aerial image associated with the matched aerial-view descriptor is sent to a user device.

17. A method for processing image data comprising: obtaining a multi-modal dataset, the multi-modal dataset comprising two-dimensional (2D) ground-view image data of a scene and a corresponding point-cloud data representing the scene;extracting, using a first neural network backbone module, a first set of feature representation from the 2D ground-view image data; extracting, using a second neural network backbone module, a second set of feature representation from the corresponding point-cloud data, fusing, using a fusion embedding construction module, the first set of feature representation and the second set of feature representation to generate a fusion embedding; generating, using a modal -wise fusion module, a 2D modality-specific descriptor and a 3D modality-specific descriptor based on the fusion embedding; and combining the 2D modality-specific descriptor and the 3D modality-specific descriptor to form a fused ground scene descriptor.

18. The method of claim 17, wherein the fusion embedding construction module comprises a plurality of neural ordinary differential equation (ODE) blocks.

19. The method of claim 18, wherein each of the plurality of neural ODE blocks is configured to progressively refine the fusion embedding by integrating one or more feature trajectories over a time period, such that an output of each neural ODE block is used as an input to a next neural ODE block.

20. A non-transitory computer-readable medium storing computer executable code comprising instructions for processing image data according to any one of the methods of claim 17 to 19.

Images