Performing image-based feature matching on images in a scene

Abstract

In some examples, a method of image-based feature matching of images in a scene includes obtaining a set of images including a plurality of images in the scene, wherein respective images of the plurality of images are captured under different lighting conditions; for each image of the plurality of images in the set of images, using unsupervised domain adaptation (UDA) to provide respective semantic labels for features representing segmented objects and / or regions in the image; using the semantic labels to compare the plurality of images in the set of images, thereby creating a plurality of image pairs according to semantic similarity, wherein each image pair of the plurality of image pairs includes a first image captured in the scene under a first lighting condition and a second image captured in the scene under a second lighting condition.

Description

Technical Field

[0001] This invention generally relates to image-based feature matching of images in a scene. Various aspects of this invention relate to device-friendly and semantically guided methods for day / night feature matching. Background Technology

[0002] Image-based feature matching is fundamental to many downstream computer vision tasks, such as localization, 3D reconstruction, image retrieval, and augmented reality. While deep learning-based feature matching of images taken under good lighting conditions (i.e., during the day) has been extensively studied, day-night scenes remain an ongoing challenge.

[0003] Existing day-night feature matching methods are primarily trained on daytime data (or daytime style transfer data) and tested using day-night benchmark datasets (e.g., the Aachen day-night benchmark dataset). However, the purpose of benchmark datasets is mainly to test the robustness of daytime models, rather than to build an efficient pipeline specifically for handling day-night scenarios.

[0004] If a feature matching algorithm is to be built specifically for handling day-night scenes, it would be best to incorporate nighttime data into the training. However, when considering nighttime data for training, current methods struggle to generalize well to nighttime data due to the significant visual domain gap. Therefore, to train on day-night data pairs, ground truth correspondences between the pairs must be manually prepared, which imposes a heavy annotation burden. Furthermore, to capture paired day-night data for training, researchers must be equipped with accurate GPS devices and assume that GPS signals are always stable at the data acquisition location, thus posing stringent requirements for conducting such research. Summary of the Invention

[0005] The purpose of this invention is to provide a mechanism that allows nighttime data to be easily incorporated into training and to build a pipeline for processing day and night scenes in an unsupervised manner.

[0006] The above and other objectives are achieved through the features of the independent claims.

[0007] Other implementations are obvious from the dependent claims, the specification and the drawings.

[0008] A first aspect of the present invention provides a method for image-based feature matching of images in a scene. The method includes: acquiring a set of images comprising multiple images in the scene, wherein corresponding images in the multiple images are captured under different lighting conditions; and for each image in the set of images, using unsupervised domain adaptation... Adaptation (UDA) provides corresponding semantic labels for features representing segmented objects and / or regions in the image; the semantic labels are used to compare the plurality of images in the set of images, thereby creating a plurality of image pairs based on semantic similarity, wherein each of the plurality of image pairs includes a first image captured under a first lighting condition and a second image captured under a second lighting condition in the scene; for each of the plurality of image pairs, a renormalized first image indicative of the first image under the second lighting condition is generated based on the image features of the second image, and a renormalized second image indicative of the second image under the first lighting condition is generated based on the image features of the first image; for each of the plurality of image pairs, pseudo-correspondences are generated for the first image and the second image based on the comparison results between the first image and the renormalized second image and between the second image and the renormalized first image, respectively, using the semantic labels provided by the UDA; the generated pseudo-correspondences are used to train a neural network.

[0009] Therefore, UDA-assisted data processing can easily incorporate nighttime data into training, thereby constructing a pipeline specifically for handling day-night scenarios in an unsupervised manner. Furthermore, the proposed method is device-friendly because it does not rely on GPS to create day-night data pairs, making it usable even in privacy-preserving areas (or other areas without GPS signals). By employing semantically aware renormalization, the appearance differences between daytime and nighttime images can be reduced for training purposes. Moreover, since renormalized data and UDA semantic labels aid in training machine learning models, reliable pixel-wise correspondences can be selected to create pseudo-labels.

[0010] Providing corresponding semantic labels for features representing segmented objects and / or regions in the image may include: training a segmentation model on a labeled dataset using unsupervised domain adaptation (UDA), and applying the segmentation model to the set of images, thereby providing the corresponding semantic labels.

[0011] The segmented object may include static objects, and comparing the plurality of images in the set of images may include: dividing the static objects into a first group and a second group, wherein the first group includes static objects of a first category and the second group includes static objects of a second category; calculating a label overlap rate based on a weighted average label overlap rate between the first group and the second group; and comparing the plurality of images in the set of images based on the calculated label overlap rate.

[0012] The first category of static objects may include static objects with a first predefined response, and the second category of static objects may include static objects with a second predefined response.

[0013] The generation of the renormalized first image and the renormalized second image may include: performing channel-wise normalization on at least one image feature of the first image relative to the second image and the second image relative to the first image, based on the corresponding semantic tags associated with the first image and the second image, wherein the at least one image feature includes mean and / or variance.

[0014] The channel-wise normalization may be performed using the semantic labels provided by the UDA, such that the class-wise features of the first image are used to normalize the corresponding features of the second image, and the class-wise features of the second image are used to normalize the corresponding features of the first image.

[0015] The method may further include: generating feature descriptors for the renormalized first image and the renormalized second image; comparing the similarity of the feature descriptors generated for the first image and the renormalized second image, and determining a first group of pixel positions of the feature descriptors based on the similarity being greater than a threshold; comparing the similarity of the feature descriptors generated for the second image and the renormalized first image, and determining a second group of pixel positions of the feature descriptors based on the similarity being greater than the threshold; comparing the first group of pixel positions and the second group of pixel positions to determine a set of pixel positions that appear simultaneously in the first group of pixel positions and the second group of pixel positions; generating a set of pseudo-labels for the set of pixel positions that appear simultaneously; and removing pseudo-labels from the set of pseudo-labels whose semantic information represented by the semantic labels does not match between the first group of pixel positions and the second group of pixel positions, based on the semantic labels provided by the UDA.

[0016] A second aspect of the present invention provides a computer-readable storage medium. The computer-readable storage medium includes computer program code and is accessible by a device including a processor to provide instructions and / or data to the device, the computer program code being configured to cause the device to perform the following operations via the processor: acquiring a set of images comprising a plurality of images in a scene, wherein corresponding images among the plurality of images are captured under different lighting conditions; and for each of the plurality of images in the set of images, using unsupervised domain adaptation... Domain Adaptation (UDA) provides corresponding semantic labels for features representing segmented objects and / or regions in the image; the semantic labels are used to compare the plurality of images in the set of images, thereby creating a plurality of image pairs based on semantic similarity, wherein each of the plurality of image pairs includes a first image captured under a first lighting condition and a second image captured under a second lighting condition in the scene; for each of the plurality of image pairs, a renormalized first image indicative of the first image under the second lighting condition is generated based on the image features of the second image, and a renormalized second image indicative of the second image under the first lighting condition is generated based on the image features of the first image; for each of the plurality of image pairs, pseudo-correspondences are generated for the first image and the second image based on the comparison results between the first image and the renormalized second image and between the second image and the renormalized first image, respectively, using the semantic labels provided by the UDA; the generated pseudo-correspondences are used to train a neural network.

[0017] The computer program code used by the processor to cause the device to provide the corresponding semantic labels for the features representing the segmented objects and / or regions in the image may include program code used by the processor to cause the device to perform the following operations: train a segmentation model on a labeled dataset using the unsupervised domain adaptation (UDA); apply the segmentation model to the set of images, thereby providing the corresponding semantic labels.

[0018] The segmented objects may include static objects, and the computer program code for causing the device to compare the plurality of images in the set of images via the processor may include program code for causing the device to perform the following operations via the processor: dividing the static objects into a first group and a second group, wherein the first group includes static objects of a first category and the second group includes static objects of a second category; calculating a label overlap rate based on a weighted average label overlap rate between the first group and the second group; and comparing the plurality of images in the set of images based on the calculated label overlap rate.

[0019] The computer program code for causing the device to generate the renormalized first image and the renormalized second image via the processor may include program code for causing the device to perform the following operations via the processor: performing channel-wise normalization on at least one image feature of the first image relative to the second image and the second image relative to the first image, based on the corresponding semantic tags associated with the first image and the second image, wherein the at least one image feature includes mean and / or variance.

[0020] The computer-readable storage medium may further include program code for causing the device to perform the following operations via the processor: generating feature descriptors for the renormalized first image and the renormalized second image; comparing the similarity of the feature descriptors generated for the first image and the renormalized second image, and determining a first set of pixel positions of the feature descriptors based on the similarity being greater than a threshold; comparing the similarity of the feature descriptors generated for the second image and the renormalized first image, and determining a second set of pixel positions of the feature descriptors based on the similarity being greater than the threshold; comparing the first set of pixel positions and the second set of pixel positions to determine a set of simultaneously occurring pixel positions existing in the first set of pixel positions and the second set of pixel positions; generating a set of pseudo-labels for the set of simultaneously occurring pixel positions; and removing pseudo-labels from the set of pseudo-labels whose semantic information represented by the semantic labels does not match between the first set of pixel positions and the second set of pixel positions, based on the semantic labels provided using the UDA.

[0021] The computer-readable storage medium may further include program code for causing the device to perform the following operation via the processor: calculating descriptor loss based on the set of pseudo-labels.

[0022] A third aspect of the invention provides an apparatus for image-based feature matching of images in a scene. The apparatus includes a processor and a memory coupled to the processor, the memory storing program code executable by the processor, the program code including one or more instructions, thereby causing the apparatus to perform the following operations: acquiring a set of images comprising a plurality of images in the scene, wherein corresponding images of the plurality of images are captured under different lighting conditions; and for each of the plurality of images in the set of images, using unsupervised domain adaptation. Adaptation (UDA) provides corresponding semantic labels for features representing segmented objects and / or regions in the image; the semantic labels are used to compare the plurality of images in the set of images, thereby creating a plurality of image pairs based on semantic similarity, wherein each of the plurality of image pairs includes a first image captured under a first lighting condition and a second image captured under a second lighting condition in the scene; for each of the plurality of image pairs, a renormalized first image indicative of the first image under the second lighting condition is generated based on the image features of the second image, and a renormalized second image indicative of the second image under the first lighting condition is generated based on the image features of the first image; for each of the plurality of image pairs, pseudo-correspondences are generated for the first image and the second image based on the comparison results between the first image and the renormalized second image and between the second image and the renormalized first image, respectively, using the semantic labels provided by the UDA; the generated pseudo-correspondences are used to train a neural network.

[0023] These and other aspects of the invention will be apparent from one or more embodiments described below. Attached Figure Description

[0024] To facilitate understanding of the present invention, various embodiments of the present invention are described below by way of example with reference to the accompanying drawings, in which: Figure 1 This is a flowchart illustrating a method for performing image-based feature matching on images in a scene; Figure 2 This is a schematic diagram illustrating the extraction of day and night data pairs based on semantic similarity with an example; Figure 3 This is a schematic diagram illustrating semantic renormalization as an example. Figure 4 This is a schematic illustration of the training and loss calculation for cross-illumination feature matching provided as an example; Figure 5This is a schematic diagram of an example device. Detailed Implementation

[0025] The exemplary embodiments are described below in sufficient detail to enable those skilled in the art to implement and carry out the systems and processes described herein. It is important to understand that embodiments may be provided in many alternative forms and should not be construed as limited to the examples set forth herein.

[0026] Therefore, while embodiments may be modified in various ways and take various alternative forms, specific embodiments are shown in the accompanying drawings and described in detail below as examples. It is not intended to limit one to the particular forms disclosed. Rather, all modifications, equivalents, and alternatives falling within the scope of the appended claims should be included. Throughout the drawings and detailed descriptions, elements of the exemplary embodiments are indicated in appropriate places by the same reference numerals.

[0027] The terminology used to describe embodiments herein is not intended to be limiting. "An" and "the" are used in the singular because they refer to only one object; however, the use of the singular form in this document should not preclude the possibility of more than one object. In other words, unless the context clearly indicates otherwise, an element mentioned in the singular form may be one or more elements. It should also be understood that the term "comprising" as used herein indicates the presence of the stated feature, item, step, operation, element, and / or component, but does not preclude the presence or addition of one or more other features, items, steps, operations, elements, components, and / or combinations thereof.

[0028] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall be interpreted in accordance with the conventions of the art. It should also be understood that, unless explicitly defined herein, terms in common usage should be interpreted as their customary usage in the relevant field, and not as idealized or overly formal meanings.

[0029] Due to the reduced visibility of nighttime data, existing technologies can only be trained on daytime datasets and evaluated using day-night benchmark datasets without manual annotation of pixel-by-pixel correspondences. In other words, they cannot be trained on paired day-night data to handle such challenging cross-lighting scenarios.

[0030] For example, the general semantic segmentation network used in SFD2 (Semantic Guided Feature Detection and Description) is not specifically trained for custom datasets, so it cannot produce high-quality semantic predictions even on unseen daytime images, let alone the more challenging nighttime images. Therefore, joint day-night training is not possible using SFD2.

[0031] Robot cars provide datasets for cross-domain training of feature matching, where pixel-by-pixel correspondences require manual annotation. However, creating pixel-by-pixel labeled training sets for day-night feature matching is both expensive and labor-intensive. Furthermore, to collect paired day-night data for training, researchers must be equipped with accurate GPS devices and assume that GPS signals are always stable at the locations chosen for data acquisition, which may not be the case in reality.

[0032] Based on one example, a mechanism is provided that allows for easy incorporation of nighttime data into training and the construction of a pipeline for processing day-night scenes in an unsupervised manner. Advantageously, the provided mechanism is device-friendly because it does not rely on using GPS to create day-night data pairs, thus enabling its application even in privacy-preserving areas (or other areas without GPS signal). By employing semantically aware renormalization, the appearance differences between daytime and nighttime images can be reduced for training purposes. Furthermore, since renormalized data and UDA semantic labels aid in training machine learning models, reliable pixel-wise correspondences can be selected to create pseudo-labels.

[0033] The various examples in this invention can be provided as methods, systems, or machine-readable instructions, such as any combination of software, hardware, firmware, etc. Such machine-readable instructions can be included in computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-readable program code on or in such media.

[0034] This invention is described in conjunction with flowcharts and / or block diagrams of methods, apparatus, and systems provided in various embodiments of the invention. While the flowcharts described above illustrate a specific order of execution, the order of execution may differ from that described. A box associated with one flowchart may be combined with a box from another flowchart. In some examples, some boxes in the flowcharts may be unnecessary and / or additional boxes may be added. It should be understood that each flow and / or box in a flowchart and / or block diagram, and combinations of flows and / or diagrams in flowcharts and / or block diagrams, can be implemented using machine-readable instructions.

[0035] For example, machine-readable instructions can be executed by a processor of a machine such as a general-purpose computer, a user device such as a smart device (e.g., a smartphone), a special-purpose computer, an embedded processor, or other programmable data processing device to implement the functions described in the specification and figures. Specifically, a processor or processing device can execute machine-readable instructions. Therefore, modules in the device (e.g., modules implementing comparator units or firewall structures) can be implemented by a processor that executes machine-readable instructions stored in memory or a processor that operates according to instructions embedded in logic circuitry. The term "processor" should be interpreted broadly to include CPUs, processing units, ASICs, logic units, or programmable gate groups, etc. All methods and modules can be executed by a single processor or distributed among several processors.

[0036] Such machine-readable instructions can also be stored in computer-readable storage that can instruct a computer or other programmable data processing device to operate in a specific mode. For example, instructions can be provided in a non-transitory computer-readable storage medium that is encoded with instructions and can be executed by a processor.

[0037] Figure 1 This is a flowchart illustrating an example of a method for performing image-based feature matching on images in a scene. The method includes, in box 101, acquiring a set of images comprising multiple images in the scene, wherein corresponding images among the multiple images were captured under different lighting conditions. Here, the term "different lighting conditions" can refer to images captured under a first lighting condition and images captured under a second lighting condition, and so on. For example, the set of images may include at least one pair of images in the same scene, one image captured during the day and one image captured at night (or any other time when light intensity is reduced, such as late afternoon and / or evening).

[0038] The method described above includes, in box 102, for each of multiple images in a set of images, using unsupervised domain adaptation (UDA) to provide corresponding semantic labels for features representing segmented objects and / or regions in the image. Here, the term "segmented object" can refer to different objects present in the image.

[0039] Providing corresponding semantic labels may include training a segmentation model on a labeled dataset using UDA and applying the segmentation model to a set of images, thereby providing corresponding semantic labels.

[0040] In other words, to train a model applicable to day-night scenarios, the first step is to create paired day-night datasets from the captured raw data (see box 103 described below). However, existing approaches to creating such datasets largely rely on comparing the GPS locations of captured images to create day-night data pairs based on images captured at the same locations. This approach requires precise GPS devices and stable GPS signals at all locations, which may not be feasible.

[0041] In contrast, this technical solution utilizes a domain adaptation technique to train a segmentation model on an open-source labeled dataset and transfers knowledge to the collected unlabeled data. Therefore, this invention can generate high-quality semantic labels for each image captured during the day or night. Segmented objects can include objects present in the captured images, such as traffic lights, buildings, etc. Segmented objects can be categorized into static objects (i.e., objects whose position does not change between images) and / or dynamic objects (i.e., objects whose position may change between images, such as cars). Semantic labels can be generated using open-source datasets such as the DarkZurich dataset.

[0042] The method described above includes: in box 103, using semantic labels to compare multiple images in a set of images, thereby creating multiple image pairs based on semantic similarity, wherein each of the multiple image pairs includes a first image captured in the scene under a first lighting condition and a second image captured in the scene under a second lighting condition.

[0043] As described above, the first lighting condition can include lighting conditions corresponding to daytime, and the second lighting condition can include lighting conditions different from daytime (i.e., nighttime, late night, etc.). The method described above can include creating these day-night data pairs based on the semantic similarity of the day-night data pairs using a UDA segmentation mask. Specifically, only the semantic similarity of static categories can be considered. After iterative processing of the entire dataset, the label overlap ratio (LOR) can be considered according to the following equation.

[0044]

[0045] If the LOR (Leadership in Relationship) is greater than a certain threshold (tau), a day-night data pair can be found. To calculate the semantic overlap rate, static categories can be divided into two groups: shift-sensitive classes (SSCs) (classes with a predefined response to light, such as traffic lights, traffic signs, or utility poles) and other static categories (classes that are not sensitive to light, such as roads and buildings). Unlike existing solutions, to ensure that all static categories contribute to semantic similarity comparison, the LOR can be calculated based on a weighted average between SSCs and non-SSCs, where the criteria are prioritized according to SSCs. If the overlap is greater than the threshold, a day-night data pair can be created.

[0046] To help understand the creation of image pairs, refer to Figure 2 . Figure 2 This is a schematic illustration of day-night data pair extraction based on semantic similarity with an example. The first image 201 may include a daytime image, and the second image 202 may include a nighttime image. As mentioned above, when considering semantic similarity, only the static category 203 may be considered. Specifically, weights may be assigned to the shift-sensitive category 204.

[0047] Return to reference Figure 1 In block 104, the above method includes: for each of a plurality of image pairs, generating a renormalized first image indicating a first image under a second illumination condition based on image features of a second image, and generating a renormalized second image indicating a second image under a first illumination condition based on image features of the first image.

[0048] In other words, to train a strong feature matching algorithm for day and night scenes using the image pair created in box 103 and the semantic information generated in box 102, semantic-aware renormalization can be applied. Specifically, a pair of day and night images can be considered, and channel-wise normalization (e.g., with respect to mean and variance) can be performed relative to each other to reduce the appearance difference between the first and second images. Figure 3 This is a schematic illustration of semantic renormalization provided as an example. A renormalized first image 301-2 can be generated based on a first image 301-1, and a renormalized second image 302-2 can be generated based on a second image 302-1. As described above, a renormalized first image 301-2 can indicate the first image 301-1 under a second lighting condition (i.e., the lighting condition associated with the second image 302-1), and a renormalized second image 302-2 can indicate the second image 302-1 under a first lighting condition (i.e., the lighting condition associated with the first image 301-1).

[0049] However, instead of blindly normalizing the entire image using statistical information associated with paired images, semantic labels can be considered. For example, night sky statistics can be used to normalize the sky region in a daytime image, and vice versa. By using this method, the resulting renormalized images 301-2 and 302-2 are closer to the paired images. Semantic labels can be obtained unsupervised, using domain adaptation transferred from labeled open-source datasets.

[0050] The method described above includes: in box 105, for each of a plurality of image pairs, generating pseudo-correspondences for the first image and the second image based on comparisons between the first image and the renormalized second image, and between the second image and the renormalized first image, using semantic labels provided by UDA. In box 106, the method described above includes: training a neural network using the generated pseudo-correspondences.

[0051] Figure 4 This is a schematic illustration of the training and loss calculation for cross-illumination feature matching, built upon the method steps in boxes 105 and 106, provided as an example. To train the cross-illumination feature matching network, in each training iteration, all four images (i.e., first image 401-1 under the first illumination condition, second image 402-1 under the second illumination condition in the same scene, renormalized first image 401-2, and renormalized second image 402-2) are fed into the network, and the output of these images is obtained. Simultaneously, a descriptor loss is calculated with semantic guidance. The output feature descriptor can be obtained from first image 401-1, second image 402-1, renormalized first image 401-2, and renormalized second image 402-2. Here, the term "feature descriptor" can refer to the pixel-wise / position-wise raw output of the neural network.

[0052] Since directly matching features between the first image 401-1 and the second image 402-1 can be quite difficult, an intermediate step can be introduced between the two. Specifically, since the renormalized first image 401-2 and the second image 402-1 have a more similar appearance (i.e., the lighting conditions between the two images are the same or very similar), it may be easier to find the closest feature descriptors between them based on a threshold and obtain the first set of pixel positions of similar feature descriptors. Similarly, a second set of pixel positions with similar feature descriptors can also be found for the renormalized second image 402-2 and the first image 401-1. With two sets of pixel positions of similar feature descriptors, a consistency check can be performed between them to determine a set of pixel positions that appear simultaneously, while only leaving the simultaneously appearing pixel positions as pseudo-labels 403. That is, pixel positions of similar feature descriptors found only after a single comparison between the image and its renormalized version can be removed.

[0053] Then, the semantic consistency of the pseudo-labels 403 can be checked. Specifically, pseudo-labels 403 whose semantic information does not match between the first and second set of pixel locations can be removed from a set of pseudo-labels 403. For example, if a road pixel has already matched a tree pixel, a pseudo-label 403 can be removed. Due to different camera poses, correctly matched descriptors typically do not appear at the same pixel location in the image. Incorrectly matched descriptors may also appear due to different lighting. Therefore, pseudo-labels can be interpreted as the locations of reliably matching descriptors. Once these locations are known, the neural network can be encouraged to minimize the distance between feature descriptors between two images.

[0054] The remaining pseudo-labels 403 can be used to calculate the descriptor loss between the first image 401-1 and the second image 402-1. This ensures that the descriptor loss directly applied to the matching image pairs is reliable. Furthermore, since this process requires no manual annotation and is completely unsupervised, it is less costly and less labor-intensive compared to currently available methods.

[0055] Figure 5 This is a schematic illustration of an example apparatus. Apparatus 500 may include a processor 503 and a memory 505. Memory 505 is coupled to processor 503 and is used to store instructions or program code 507 executable by processor 503. Apparatus 500 may include program code 507 for causing the apparatus to perform the image-based feature matching method for images in a scene as described herein.

[0056] According to one example, machine-readable instructions can be loaded onto a computer or other programmable data processing apparatus to cause the computer or other programmable data processing apparatus to perform a series of operations to perform computer-implemented processing. Thus, the instructions that execute on the computer or other programmable apparatus provide operations for implementing the functions specified by one or more flows in a flowchart and / or one or more boxes in a block diagram.

[0057] Furthermore, the teachings herein can be implemented in the form of a computer or software product, such as a non-transitory machine-readable storage medium, in which computer software or products are stored and include a plurality of instructions, such as machine-readable instructions, for causing a computer device to implement the methods described in the various examples of the present invention.

[0058] In some examples, certain methods can be executed in a cloud-based or web-based environment. Cloud environments provide a variety of services and applications over the internet. These cloud-based services (e.g., Software as a Service, Platform as a Service, Infrastructure as a Service, etc.) can be accessed through a user's device's web browser or other remote interfaces. The various functionalities described herein can be provided through a remote desktop environment or any other cloud-based computing environment.

[0059] While various embodiments have been described and / or illustrated herein in the context of a full-featured computing system, one or more of these exemplary embodiments may be distributed as a program product in various forms, regardless of the specific type of computer-readable storage medium used for the actual execution of the distribution. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include scripts, batch files, or other executable files that may be stored on a computer-readable storage medium or in a computing system. In some embodiments, these software modules may configure a computing system to perform one or more exemplary embodiments disclosed herein. Furthermore, one or more modules described herein may convert data, physical devices, and / or representations of physical devices from one form to another.

[0060] The above description is provided to enable those skilled in the art to best utilize the various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be exhaustive or limited to any precise forms disclosed. Many modifications and variations can be made without departing from the spirit and scope of the invention. The embodiments disclosed herein should be considered illustrative rather than restrictive in all respects. Reference should be made to the appended claims and their equivalents in determining the scope of the invention.

Claims

1. A method for image-based feature matching of images in a scene, characterized in that, The method includes: Acquire a set of images comprising multiple images in the scene, wherein corresponding images in the multiple images are captured under different lighting conditions (101). For each of the plurality of images in the set of images, unsupervised domain adaptation (UDA) is used to provide corresponding semantic labels for the features representing segmented objects and / or regions in the image (102). The semantic tags are used to compare the plurality of images in the set of images, thereby creating a plurality of image pairs based on semantic similarity, wherein each of the plurality of image pairs includes a first image captured in the scene under a first lighting condition and a second image captured in the scene under a second lighting condition (103). For each of the plurality of image pairs, a renormalized first image indicating the first image under the second illumination condition is generated based on the image features of the second image, and a renormalized second image indicating the second image under the first illumination condition is generated based on the image features of the first image (104). For each of the plurality of image pairs, based on the semantic labels provided by the UDA, pseudo-correspondences are generated for the first image and the second image respectively (105) according to the comparison results between the first image and the renormalized second image and between the second image and the renormalized first image. The generated pseudo-correspondence is used to train the neural network (106).

2. The method according to claim 1, characterized in that, The provision of corresponding semantic labels (102) for features representing segmented objects and / or regions in the image includes: The segmentation model is trained on the labeled dataset using the unsupervised domain adaptation (UDA). The segmentation model is applied to the set of images, thereby providing the corresponding semantic labels.

3. The method according to claim 1 or 2, characterized in that, The segmented object includes static objects, and comparing the plurality of images in the set of images includes: The static objects are divided into a first group and a second group, wherein the first group includes static objects of a first category and the second group includes static objects of a second category. The label overlap rate is calculated based on the weighted average label overlap rate between the first group and the second group; The multiple images in the set of images are compared based on the calculated label overlap rate.

4. The method according to claim 3, characterized in that, The first category of static objects includes static objects with a first predefined response, and the second category of static objects includes static objects with a second predefined response.

5. The method according to any one of claims 1 to 4, characterized in that, The generation of the renormalized first image and the renormalized second image (104) includes: Based on the corresponding semantic tags associated with the first image and the second image, channel-wise normalization is performed on at least one image feature of the first image relative to the second image and the second image relative to the first image. The at least one image feature includes mean and / or variance.

6. The method according to claim 5, characterized in that, The channel-wise normalization is performed using the semantic labels provided by the UDA, such that the class-wise features of the first image are used to normalize the corresponding features of the second image, and the class-wise features of the second image are used to normalize the corresponding features of the first image.

7. The method according to any one of the preceding claims, characterized in that, Also includes: Generate feature descriptors for the renormalized first image and the renormalized second image; The similarity between the feature descriptors generated for the first image and the renormalized second image is compared, and the first group of pixel positions of the feature descriptors is determined based on the similarity being greater than a threshold. The similarity between the feature descriptors generated for the second image and the renormalized first image is compared, and the second set of pixel positions of the feature descriptors is determined based on the similarity being greater than the threshold. Compare the first group of pixel positions and the second group of pixel positions to determine a set of pixel positions that appear simultaneously in either the first group of pixel positions or the second group of pixel positions. Generate a set of pseudo-labels for the set of simultaneously occurring pixel positions; Based on the semantic tags provided by the UDA, remove pseudo-tags from the set of pseudo-tags where the semantic information represented by the semantic tags does not match between the first set of pixel positions and the second set of pixel positions.

8. The method according to claim 7, characterized in that, Also includes: Calculate the descriptor loss based on the set of pseudo-labels.

9. A computer-readable storage medium, characterized in that, Includes computer program code and is accessible by a device including a processor to provide instructions and / or data to the device, wherein the computer program code is used to cause the device to perform the following operations via the processor: Acquire a set of images comprising multiple images in a scene, wherein corresponding images among the multiple images are captured under different lighting conditions; For each of the plurality of images in the set of images, unsupervised domain adaptation (UDA) is used to provide corresponding semantic labels for the features representing segmented objects and / or regions in the image; The semantic tags are used to compare the plurality of images in the set of images, thereby creating a plurality of image pairs based on semantic similarity, wherein each of the plurality of image pairs includes a first image captured in the scene under a first lighting condition and a second image captured in the scene under a second lighting condition; For each of the plurality of image pairs, a renormalized first image indicating the first image under the second illumination condition is generated based on the image features of the second image, and a renormalized second image indicating the second image under the first illumination condition is generated based on the image features of the first image. For each of the plurality of image pairs, based on the semantic tags provided by the UDA, pseudo-correspondences are generated for the first image and the second image respectively, according to the comparison results between the first image and the renormalized second image and between the second image and the renormalized first image. The generated pseudo-correspondence is used to train the neural network.

10. The computer-readable storage medium according to claim 9, characterized in that, The computer program code is used by the processor to cause the device to provide the corresponding semantic labels for the features representing the segmented objects and / or regions in the image, including program code for causing the device to perform the following operations by the processor: The segmentation model is trained on the labeled dataset using the unsupervised domain adaptation (UDA). The segmentation model is applied to the set of images, thereby providing the corresponding semantic labels.

11. The computer-readable storage medium according to claim 9 or 10, characterized in that, The segmented object includes a static object, and the computer program code for causing the device to compare the plurality of images in the set of images via the processor includes program code for causing the device to perform the following operations via the processor: The static objects are divided into a first group and a second group, wherein the first group includes static objects of a first category and the second group includes static objects of a second category. The label overlap rate is calculated based on the weighted average label overlap rate between the first group and the second group; The multiple images in the set of images are compared based on the calculated label overlap rate.

12. The computer-readable storage medium according to any one of claims 9 to 11, characterized in that, The computer program code for causing the device to generate the renormalized first image and the renormalized second image via the processor includes program code for causing the device to perform the following operations via the processor: Based on the corresponding semantic tags associated with the first image and the second image, channel-wise normalization is performed on at least one image feature of the first image relative to the second image and the second image relative to the first image. The at least one image feature includes mean and / or variance.

13. The computer-readable storage medium according to any one of claims 9 to 12, characterized in that, It also includes program code for causing the device to perform the following operations via the processor: Generate feature descriptors for the renormalized first image and the renormalized second image; The similarity between the feature descriptors generated for the first image and the renormalized second image is compared, and the first group of pixel positions of the feature descriptors is determined based on the similarity being greater than a threshold. The similarity between the feature descriptors generated for the second image and the renormalized first image is compared, and the second set of pixel positions of the feature descriptors is determined based on the similarity being greater than the threshold. Compare the first group of pixel positions and the second group of pixel positions to determine a set of pixel positions that appear simultaneously in either the first group of pixel positions or the second group of pixel positions. Generate a set of pseudo-labels for the set of simultaneously occurring pixel positions; Based on the semantic tags provided by the UDA, remove pseudo-tags from the set of pseudo-tags where the semantic information represented by the semantic tags does not match between the first set of pixel positions and the second set of pixel positions.

14. The computer-readable storage medium according to any one of claims 9 to 13, characterized in that, It also includes program code for causing the device to perform the following operations via the processor: Calculate the descriptor loss based on the set of pseudo-labels.

15. An apparatus (500) for image-based feature matching of images in a scene, characterized in that, The device includes: Processor (503); A memory (505) coupled to the processor (503) is used to store program code (507) executable by the processor, the program code (507) comprising one or more instructions that cause the device (500) to perform the following operations: Acquire a set of images comprising multiple images in the scene, wherein corresponding images in the multiple images are captured under different lighting conditions; For each of the plurality of images in the set of images, unsupervised domain adaptation (UDA) is used to provide corresponding semantic labels for the features representing segmented objects and / or regions in the image; The semantic tags are used to compare the plurality of images in the set of images, thereby creating a plurality of image pairs based on semantic similarity, wherein each of the plurality of image pairs includes a first image captured in the scene under a first lighting condition and a second image captured in the scene under a second lighting condition; For each of the plurality of image pairs, a renormalized first image indicating the first image under the second illumination condition is generated based on the image features of the second image, and a renormalized second image indicating the second image under the first illumination condition is generated based on the image features of the first image. For each of the plurality of image pairs, based on the semantic tags provided by the UDA, pseudo-correspondences are generated for the first image and the second image respectively, according to the comparison results between the first image and the renormalized second image and between the second image and the renormalized first image. The generated pseudo-correspondence is used to train the neural network.

Images