Device and method for automatic key point and description extraction

Abstract

An apparatus (102, 202, 302) for performing automatic keypoint and description extraction from image data (204A and 204B) is provided. The apparatus (102, 202, 302) comprises a data processing device (106) coupled to a data storage device (108), a communication network device (110), and a feature description device (112). The communication network device (i) removes regions in a plurality of images whose information content is below a given threshold; (ii) selects regions in each of the plurality of images that represent mutually common features; and (iii) generates one or more output feature vectors h representing features present in the image data. The feature description device receives the one or more output vectors h for the plurality of images and generates one or more output vectors z representing one or more keypoints (230A-N, 232A-N) present in each of the plurality of images of the image data.

Description

Technical Field

[0001] The present invention generally relates to image processing techniques. Furthermore, the present invention relates to an apparatus and method for performing automatic keypoint and description extraction from image data representing a plurality of images. Furthermore, the present invention relates to an autonomous vehicle including such an apparatus and a method for operating an autonomous vehicle including such an apparatus to enable the autonomous vehicle to navigate a spatial region. Background Art

[0002] In image processing, keypoint detection and description extraction are essential components of several geometric computer tasks, such as robust pose estimation and simultaneous localization and mapping (SLAM). Keypoint detection is used to find corresponding points between two or more images of the same scene or object. Keypoint detection has achieved great success in many computer vision and pattern recognition fields, including object recognition, motion tracking, wide-baseline stereo, texture recognition, image retrieval, robot navigation, video data mining, architectural panoramas, stereo matching, camera motion recovery, and 3D reconstruction. Keypoint detection identifies stable keypoints that match the position and scale of two or more images of the same scene or object. Stable keypoints are detected using the position and corresponding scale, while using appropriate neighborhoods when computing descriptors. Keypoint description involves constructing a unique descriptor for each keypoint by describing its adjacent region. Ideally, this involves creating a description of the keypoint. The description must be unique and invariant under various transformations, such as viewpoint changes, rotation, scaling, and illumination variations.

[0003] In known methods, keypoint detection and extraction relies on manually engineered features, where scale and rotation are estimated to compute a description based on local features of neighboring pixels. In another known method, keypoint detection and extraction relies on learning local features from labeled data or pseudo ground truth (i.e., labels are generated based on the method's predictions). This known method relies on a strong prior basis, where a given keypoint is constrained to be similar to a synthetic keypoint and generalization is restricted to angular features learned from the generated pseudo ground truth. In another known method, negative samples are generated from different images and positive samples are generated by applying data augmentation methods to the same image. The positive samples extracted from the image do not explicitly take into account the spatial information of the image. This known method is unable to identify keypoints with the same visual content in two similar images.

[0004] In the known methods mentioned above, local features are ambiguous, making key points and descriptions difficult for non-experts to interpret. Manually designed descriptors fail to capture the diverse input modalities of natural images. Existing learnable descriptors do not generalize well to distributions different from those they were trained on. Known methods require large labeled datasets and perform poorly under adverse conditions such as varying lighting and different viewpoints.

[0005] Therefore, it is necessary to address the above technical deficiencies of existing systems or technologies in automatic key point and description extraction. Summary of the Invention

[0006] The present invention aims to provide an apparatus for performing automatic keypoint and description extraction that is adaptable to different data distributions and robust to adverse conditions such as lighting variations and different viewpoints. Furthermore, the present invention aims to provide an improved method (i.e., a method) for using the apparatus for performing automatic keypoint and description extraction from image data representing a plurality of images. Furthermore, the present invention relates to an autonomous vehicle comprising the apparatus and to a method for operating an autonomous vehicle comprising the apparatus so as to enable the autonomous vehicle to navigate an area.

[0007] This object is achieved by the features of the independent claim. Further implementations are apparent from the dependent claims, the description and the drawings.

[0008] The present invention provides an apparatus and method for performing automatic keypoint and description extraction from image data representing a plurality of images. Furthermore, the present invention relates to an autonomous vehicle comprising the apparatus and a method for operating an autonomous vehicle comprising the apparatus so as to enable the autonomous vehicle to navigate a spatial region.

[0009] According to a first aspect, there is provided an apparatus for performing automatic keypoint and description extraction from image data representing a plurality of images. The image data is input to the apparatus. The apparatus comprises a data processing device coupled to a data storage device. The data processing device is configured to execute one or more neural network algorithms. The apparatus comprises a communication network device and a feature description device. The communication network device is implemented as an algorithm executed by the data processing device. The communication network device is configured to: process the image data to remove regions in the plurality of images whose information content is below a given threshold; and select at least one region in each of the plurality of images that represents mutually common features to generate one or more output feature vectors h representing features present in the image data. The feature description device is implemented as an algorithm executed by the data processing device. The feature description device is configured to: receive the one or more output vectors h for the plurality of images; and generate one or more output vectors z, the one or more output vectors z representing one or more keypoints present in each of the plurality of images of the image data.

[0010] The apparatus supports fully automated keypoint and description extraction. The apparatus performs the automatic keypoint and description extraction without any labeled real-time image data. The apparatus is configured to learn any type of basic visual features and basic semantic information from the image data. The apparatus may include visual embeddings for solving computer vision problems such as location recognition and classification tasks. The apparatus is continuously trained to learn to (i) adapt to different data distributions associated with different environments and weather conditions and to become (ii) robust to adverse conditions. The apparatus remains invariant to changes in the input space.

[0011] The one or more algorithms may utilize a keypoint detection algorithm that utilizes guided backpropagation through output neurons in a neural network to detect the one or more keypoints in the image data by generating an output gradient tensor g, and filters out less relevant keypoints from the output gradient tensor g using a hysteresis thresholding method applied to the detected one or more keypoints. Optionally, the keypoint detection algorithm is configured to detect output values of the most activated neurons stimulated by the keypoint detection algorithm, wherein the keypoint detection algorithm describes the most informative features present in the image data.

[0012] The keypoint detection algorithm uses a single input image to detect the one or more keypoints by guided backpropagation of the values of the most activated output neurons that describe the most informative visual features of the input image. The apparatus utilizes post-processing methods such as hysteresis thresholding and non-maximum suppression to detect the one or more keypoints to improve the robustness, accuracy, reliability, and repeatability of the one or more keypoints.

[0013] Optionally, the apparatus is trained using contrastive learning based on a sampling algorithm tailored for keypoint and description extraction to automatically detect visual similarities and differences between images. The sampling algorithm is configured to process a combination of positive and negative versions of at least one image of the input data that preserves the spatial relationship between the images. Positive samples are described as partially overlapping regions of the images, and negative samples are described as non-overlapping regions of one or the other image.

[0014] Optionally, the communication network device and the feature description device are implemented as an encoder network and a projection head network, respectively; wherein the encoder network is used to receive the image data and generate the one or more output feature vectors h; the projection head network is used to receive the one or more output feature vectors h and generate the one or more output vectors z therefrom, and the one or more output vectors z represent the one or more key points present in the image data.

[0015] Optionally, the apparatus is configured to generate a set of one or more output vectors z for each image. The apparatus is configured to calculate an independent variable parameter in a multiplication operation of the set of one or more output vectors z. The independent variable parameter indicates whether the same given feature is present in the plurality of images.

[0016] Optionally, the apparatus is configured to compute the keypoint k by extracting a local patch centered at a given keypoint k from the image data, the local patch being rescaled such that cosine similarity of the one or more output vectors z of the local patches of the plurality of images is used to determine whether the local patches represent the same feature for communication detection purposes.

[0017] Optionally, the apparatus is configured to compute a keypoint k by extracting a local patch centered at a given keypoint k from the image data. The apparatus is configured to use an additional neural network algorithm trained by contrastive learning based on a combination of positive and negative versions of the at least one image of the input data, wherein the at least one image of the input data preserves spatial relationships between the images. The additional neural network algorithm is configured to process the local patches extracted around the local patch centered at the keypoint k to learn visual similarities and differences from the input data; and the cosine similarity of the one or more output vectors z of the local patches of the multiple images with one or more neural networks is used to determine whether the local patches represent the same features for communication detection purposes.

[0018] According to a second aspect, a method for performing automatic keypoint and description extraction from image data representing a plurality of images using an apparatus is provided. The image data is input to the apparatus. The apparatus includes a data processing device coupled to a data storage device. The data processing device is configured to execute one or more neural network algorithms. The method includes configuring the apparatus to utilize a communication network device, the communication network device being implemented as an algorithm executed by the data processing device. The communication network device is configured to: process the image data to remove regions of the plurality of images whose information content is below a given threshold; and select at least one region in each of the plurality of images that represents mutually common features to generate one or more output feature vectors h representing features present in the image data. The method includes configuring the apparatus to utilize a feature description device, the feature description device being implemented as an algorithm executed by the data processing device. The feature description device is configured to: receive the one or more output vectors h for the plurality of images; and generate one or more output vectors z representing one or more keypoints present in each of the plurality of images of the image data.

[0019] The method may include a Siamese convolutional network that is trained using contrastive learning to automatically identify visual similarities and differences between the multiple images. The method supports fully automated key point and description extraction. The method performs the automatic key point and description extraction in the absence of any labeled real-time image data. The method enables the device to be adapted to learn any type of basic visual features and basic semantic information from the image data. The method may adopt visual embeddings that are used to solve computer vision problems such as location recognition and classification tasks. The method enables the device to be continuously trained to learn (i) to adapt to different data distributions associated with different environments and weather conditions, and to become (ii) robust to adverse conditions. The method remains unchanged against changes in the input space.

[0020] Optionally, the method includes configuring the one or more algorithms to use a key point detection algorithm, which utilizes guided backpropagation of output neurons in a neural network to detect the one or more key points in the image data by generating an output gradient tensor g, and uses a hysteresis threshold method applied to the detected one or more key points to filter out key points with low relevance from the output gradient tensor g.

[0021] The method may include post-processing methods such as hysteresis thresholding and non-maximum suppression to detect the key points to improve robustness, accuracy, reliability and repeatability metrics.

[0022] Optionally, the method comprises configuring the keypoint detection algorithm by detecting output values of the most highly activated neurons stimulated by the keypoint detection algorithm, the keypoint detection algorithm describing the most informative features present in the image data.

[0023] Optionally, the method includes training the apparatus to automatically detect visual similarities and differences between images using contrastive learning based on a sampling algorithm tailored for keypoint and description extraction. The sampling algorithm is configured to process a combination of positive and negative versions of at least one image of the input data that preserves the spatial relationship between the images. Positive samples are described as partially overlapping regions of the images, and negative samples are described as non-overlapping regions of one or other of the images.

[0024] Optionally, the method includes implementing the communication network device and the feature description device as an encoder network and a projection head network, respectively; wherein the encoder network is used to receive the image data and generate the one or more output feature vectors h; the projection head network is used to receive the one or more output feature vectors h and generate the one or more output vectors z therefrom, and the one or more output vectors z represent the one or more key points present in the image data.

[0025] Optionally, the method includes configuring the apparatus to generate a set of one or more output vectors z for each image. The apparatus is configured to calculate an independent variable parameter in a multiplication operation of the set of one or more output vectors z. The independent variable parameter indicates whether the same given feature is present in the plurality of images.

[0026] Optionally, the method includes configuring the apparatus to compute the keypoint k by extracting a local patch centered at a given keypoint k from the image data, the local patch being rescaled such that cosine similarity of the one or more output vectors z of the local patches of the plurality of images is used to determine whether the local patches represent the same feature for communication detection purposes.

[0027] Optionally, the method includes configuring the device to calculate the key point k by extracting a local patch centered at a given key point k from the image data; wherein the method includes configuring the device to use an additional neural network algorithm, which is trained by contrastive learning based on a combination of positive and negative versions of the at least one image of the input data, and the at least one image of the input data preserves the spatial relationship between the images; wherein the additional neural network algorithm is used to process the local patches extracted around the local patch centered at the key point k to learn visual similarities and differences from the input data; wherein the cosine similarity of the one or more output vectors z of the local patches of the multiple images and one or more neural networks is used to determine whether the local patches represent the same features for communication detection purposes.

[0028] According to a third aspect, there is provided an autonomous vehicle comprising means for performing automatic keypoint and description extraction from image data representing at least one image of a field of view captured from a spatial region surrounding the autonomous vehicle, to enable the autonomous vehicle to navigate in the spatial region.

[0029] According to a fourth aspect, there is provided a method of operating an autonomous vehicle, the autonomous vehicle comprising means for performing automatic keypoint and description extraction from image data representing at least one image of a field of view captured from a spatial region surrounding the autonomous vehicle, to enable the autonomous vehicle to navigate in the spatial region.

[0030] The autonomous vehicle supports fully automated keypoint and description extraction using the apparatus. The apparatus enables the autonomous vehicle to be continuously trained to learn to (i) adapt to different data distributions associated with different environments and weather conditions, and to become (ii) robust to adverse conditions.

[0031] According to a fifth aspect, there is provided a computer program product comprising a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, the computer-readable instructions being executable by a computerized device comprising processing hardware for performing the method.

[0032] It solves the technical problems in the existing technology, namely the extraction of key points and descriptions under adverse conditions such as lighting changes and different viewpoints.

[0033] Therefore, in contrast to the prior art, the present invention provides an apparatus and method for performing automatic keypoint and description extraction, wherein the apparatus provides clear and simple local features for non-expert interpretation of keypoints and descriptions. The apparatus captures a variety of input patterns from natural images and generalizes them to a distribution that is different from the training distribution. The apparatus performs the automatic keypoint and description extraction in the absence of any labeled real-time image data. Optionally, the apparatus is suitable for learning any type of basic visual features and basic semantic information from the image data. The apparatus includes a visual embedding that is used to solve computer vision problems such as location recognition and classification tasks. The apparatus is continuously trained to learn (i) to adapt to different data distributions associated with different environments and weather conditions, and to become (ii) robust to adverse conditions such as lighting changes and different viewpoints. The apparatus remains invariant to changes in the input space.

[0034] These and other aspects of the invention will become apparent from and elucidated with reference to the detailed description given hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Implementations of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0036] Figure 1 A block diagram of an apparatus for performing automatic key point and description extraction provided by an implementation of the present invention is shown;

[0037] Figure 2A An exemplary block diagram of an apparatus trained to perform automatic keypoint and description extraction provided by an implementation of the present invention is shown;

[0038] Figure 2B An exemplary block diagram of an apparatus for performing key point detection using image data provided by an implementation of the present invention is shown;

[0039] Figure 2C An exemplary block diagram of an apparatus for performing description extraction and communication detection using image data provided by an implementation of the present invention is shown;

[0040] Figure 3 An exemplary autonomous driving vehicle including an apparatus for performing automatic keypoint and description extraction provided by an implementation of the present invention is shown;

[0041] Figure 4 A flow chart illustrating a method for using an apparatus for performing automatic keypoint and description extraction from image data representing a plurality of images, provided by an implementation of the present invention;

[0042] Figure 5 A diagram of an apparatus for implementing an implementation of the present invention is shown. DETAILED DESCRIPTION

[0043] Implementations of the present invention provide an apparatus for performing automatic keypoint and description extraction from image data representing a plurality of images, wherein the automatic keypoints and descriptions are adapted to different data distributions and made robust to adverse conditions such as lighting variations and different viewpoints. Furthermore, implementations of the present invention provide a method (i.e., a method) for using the apparatus for performing automatic keypoint and description extraction from image data representing the plurality of images. Furthermore, the present invention relates to an autonomous vehicle comprising the apparatus, and a method for operating an autonomous vehicle comprising the apparatus so as to enable the autonomous vehicle to navigate a spatial region.

[0044] In order to make it easier for those skilled in the art to understand the solutions of the present invention, the following implementation of the present invention is described in conjunction with the accompanying drawings.

[0045] The terms "first," "second," "third," and "fourth" (if any) in the abstract of the specification, the claims, and the above-mentioned figures of the present invention are used to distinguish similar objects and are not necessarily used to describe a particular sequence or order. It should be understood that the terms used are interchangeable under appropriate circumstances, so that, for example, the implementation of the invention described herein can be implemented in a sequence different from the sequence shown or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not necessarily limited to the steps or units explicitly listed, but may include other steps or units that are not explicitly listed, or steps or units inherent to such process, method, product, or device.

[0046] Figure 1 A block diagram of an apparatus 102 for performing automatic keypoint and description extraction provided by an implementation of the present invention is shown. The apparatus 102 includes a data processing device 106 coupled to a data storage device 108, a communication network device 110, and a feature description device 112. The apparatus 102 performs the automatic keypoint and description extraction from image data representing a plurality of images. The image data is input (104) into the apparatus 102. The data processing device 106 is configured to execute one or more neural network algorithms. The communication network device 110 is implemented as an algorithm executed by the data processing device 106. The communication network device 110 is configured to: process the image data to remove regions in the plurality of images whose information content is below a given threshold; and select at least one region in each of the plurality of images that represents mutually common features to generate one or more output feature vectors h representing features present in the image data. The one or more output feature vectors h represent features present in the image data. The feature description device 112 is implemented as an algorithm executed by the data processing device 106. The feature description device 112 is configured to receive the one or more output vectors h of the plurality of images. The feature description device 112 is used to generate one or more output vectors z, and the one or more output vectors z represent one or more key points present in each of the multiple images of the image data.

[0047] The one or more neural algorithms executed by the data processing device 106 may use a key point detection algorithm, which utilizes guided backpropagation of output neurons in a neural network to detect the one or more key points in the image data by generating an output gradient tensor g, and uses a hysteresis threshold method applied to the detected one or more key points to filter out key points with low relevance from the output gradient tensor g.

[0048] The device 102 may be selected from, but not limited to, a mobile phone, a personal digital assistant (PDA), a tablet computer, a desktop computer, a server, or a laptop computer. The device 102 utilizes the guided backpropagation to detect the one or more key points in the image data. The image data may be associated with a single image.

[0049] The keypoint detection algorithm can be configured in the following manner: detecting an output value of a most activated neuron stimulated by the keypoint detection algorithm, the keypoint detection algorithm describing the most information-rich features present in the image data. The device 102 can use the keypoint detection algorithm to detect the one or more keypoints in the image data by directing backpropagation of the value of the most activated neuron describing the most information-rich features present in the image data.

[0050] The device 102 detects the one or more key points using post-processing methods such as hysteresis threshold method and non-maximum suppression method to improve the robustness, accuracy, reliability and repeatability of the one or more key points.

[0051] The apparatus 102 is trained using contrastive learning based on a sampling algorithm tailored for keypoint and description extraction to automatically detect visual similarities and differences between images. The sampling algorithm processes a combination of positive and negative versions of at least one image of the input data that preserves the spatial relationship between the images. Positive samples are described as partially overlapping regions of the images, and negative samples are described as non-overlapping regions of one or the other image.

[0052] The apparatus 102 may include a Siamese convolutional network trained using contrastive learning to automatically determine the visual similarities and differences between the plurality of images. The apparatus 102 may include a convolutional neural network trained using contrastive learning to sample the image data.

[0053] The sampling algorithm may include one or more data augmentation methods applied to the image data, including perspective transformation, to simulate the multiple images captured from different angles. The sampling algorithm enables the device 102 to continuously learn fine-grained visual features.

[0054] The apparatus 102 supports fully automated keypoint and description extraction. The apparatus 102 performs the automatic keypoint and description extraction without any labeled real-time image data. The apparatus 102 is adapted to learn any type of basic visual features and basic semantic information from the image data. The apparatus 102 includes visual embeddings that are used to solve computer vision problems such as location recognition and classification tasks. The apparatus 102 is continuously trained to learn to (i) adapt to different data distributions associated with different environments and weather conditions and to become (ii) robust to adverse conditions. The apparatus 102 remains invariant to changes in the input space.

[0055] Figure 2A An exemplary block diagram of an apparatus 202 provided by an implementation of the present invention and trained to perform automatic key point and description extraction is shown. The apparatus 202 includes a communication network apparatus and a feature description apparatus. The apparatus 202 implements the communication network apparatus as an encoder network (206A, 206B). The encoder network (206A, 206B) is used to receive image data (204A and 204B) and generate one or more output feature vectors h. The apparatus 202 implements the feature description apparatus as a projection head network (210A, 210B). The projection head network (210A, 210B) is used to receive the one or more output feature vectors h and generate the one or more output feature vectors z therefrom, the one or more output feature vectors z representing the one or more key points present in the image data (204A and 204B). The image data (204A and 204B) may be a plurality of images of one or more objects captured from different angles (e.g., a table with one or more objects captured from different angles, such as Figure 2A The one or more output feature vectors h may represent a visual embedding (208A and 208B) of the pair of image data (204A and 204B). The one or more output vectors z may represent an extracted description (212A and 212B) of the image data (204A and 204B). The image data (204A and 204B) is used to calculate the one or more output vectors z.

[0056] The device 202 is configured to generate the set of one or more output vectors z for each image. The device 202 is configured to calculate an independent variable parameter in a multiplication operation of the set of one or more output vectors z. The independent variable parameter indicates whether the same given feature exists in the multiple images.

[0057] The apparatus 202 may include calculating a cosine similarity between the image data (204A and 204B) using an output vector z between the image data (204A and 204B). If the image data (204A and 204B) are similar, selecting an output neuron by calculating the maximum argument in a multiplication operation of the one or more output vectors z of the image data (204A and 204B). The similarity score may be represented as sim(z, z') 214. Optionally, the image data (204A and 204B) are processed by applying guided backpropagation to obtain the one or more output vectors z representing the one or more keypoints.

[0058] refer to Figure 2A , Figure 2B An exemplary block diagram of an apparatus 202 for performing keypoint detection using image data (204A and 204B) provided by an implementation of the present invention is shown. The apparatus 202 uses an encoder network (206A, 206B) and a projection head network (210A, 210B) to calculate (i) one or more output vectors z and (ii) the maximum argument in a multiplication operation of the one or more output vectors z. The one or more output vectors z represent one or more keypoints (230A-N, 232A-N) in the plurality of images. The one or more output feature vectors h can represent a visual embedding (208A and 208B) of the image data (204A and 204B). The one or more output feature vectors z represent an extracted description (212A and 212B).

[0059] High activation values for the one or more output vectors z indicate the presence of the same visual feature in the multiple images. Guided backpropagation of the selected output vectors to the input image can generate an output gradient tensor g. The higher the output gradient tensor g, the greater the influence of the input image on the selected output vector. To increase the likelihood of the keypoints (230A-N or 232A-N) in the image data (204A and 204B), post-processing methods such as hysteresis thresholding and non-maximum suppression can be used.

[0060] refer to Figure 2A and Figure 2B , Figure 2CAn exemplary block diagram of an apparatus 202 for performing description extraction and communication detection using image data (204A and 204B) provided by an implementation of the present invention is shown. The apparatus 202 is configured to compute a keypoint k by extracting a local image patch (242A, 242B) centered at a given keypoint k from the image data (204A and 204B). The local image patch (242A, 242B) is rescaled such that the cosine similarity of the one or more output vectors z of the local image patches (242A, 242B) of the plurality of images is used to determine whether the local image patches (242A, 242B) represent the same features for communication detection purposes. The local image patches (242A, 242B) may include similar keypoints (230A, 232A).

[0061] Figure 3 An exemplary autonomous vehicle 304 including an apparatus 302 for performing automatic keypoint and description extraction provided by an implementation of the present invention is shown. Optionally, the autonomous vehicle 304 includes the apparatus 302 for performing automatic keypoint and description extraction from image data representing at least one image of a field of view captured from a spatial region surrounding the autonomous vehicle 304, so as to enable the autonomous vehicle 304 to navigate in the spatial region.

[0062] The autonomous vehicle 304 may be a two-wheeled vehicle, a four-wheeled vehicle, etc. The autonomous vehicle 304 supports fully automated key point and description extraction using the apparatus 302. The apparatus 302 enables the autonomous vehicle 304 to be continuously trained to learn (i) to adapt to different data distributions associated with different environments and weather conditions, and to become (ii) robust to adverse conditions.

[0063] Figure 4A flowchart of a method for using an apparatus for performing automatic keypoint and description extraction from image data representing multiple images, provided by an implementation of the present invention, is shown. The image data is input into the apparatus. The apparatus includes a data processing device coupled to a data storage device. The data processing device is configured to execute one or more neural network algorithms. In step 402, the apparatus is configured to utilize a communication network device, implemented as an algorithm executed by the data processing device. The communication network device is configured to: process the image data to remove regions in the multiple images whose information content is below a given threshold; and select at least one region in each of the multiple images that represents mutually common features to generate one or more output feature vectors h representing features present in the image data. In step 404, the apparatus is configured to utilize a feature description device, implemented as an algorithm executed by the data processing device. The feature description device is configured to: receive the one or more output vectors h for the multiple images; and generate one or more output vectors z representing one or more keypoints present in each of the multiple images of the image data.

[0064] The method may include a twin convolutional network that is trained using contrastive learning to automatically identify visual similarities and differences between the multiple images. The method supports fully automated key point and description extraction. The method performs the automatic key point and description extraction in the absence of any labeled real-time image data. The method enables the device to be adapted to learn any type of basic visual features and basic semantic information from the image data. The method may adopt visual embeddings that are used to solve computer vision problems such as location recognition and classification tasks. The method enables the device to be continuously trained to learn (i) different data distributions associated with different environments and weather conditions, and to become (ii) robust to adverse conditions such as lighting changes and different viewpoints. The method remains invariant to changes in the input space.

[0065] In a first implementation, the method includes configuring the one or more algorithms to use a key point detection algorithm, which utilizes guided backpropagation of output neurons in a neural network to detect the one or more key points in the image data by generating an output gradient tensor g, and uses a hysteresis threshold method applied to the detected one or more key points to filter out key points with low relevance from the output gradient tensor g.

[0066] In a second implementation, the method includes configuring the keypoint detection algorithm by detecting output values of the most highly activated neurons stimulated by the keypoint detection algorithm, the keypoint detection algorithm describing the most informative features present in the image data.

[0067] In a third implementation, the method includes training the apparatus to automatically detect visual similarities and differences between images using contrastive learning based on a sampling algorithm tailored for keypoint and description extraction. The sampling algorithm processes a combination of positive and negative versions of at least one image of the input data that preserves the spatial relationship between the images. Positive samples are described as partially overlapping regions of the images, and negative samples are described as non-overlapping regions of one or the other image.

[0068] In a fourth implementation, the method includes implementing the communication network device and the feature description device as an encoder network and a projection head network, respectively; wherein the encoder network is used to receive the image data and generate the one or more output feature vectors h; the projection head network is used to receive the one or more output feature vectors h and generate the one or more output vectors z therefrom, and the one or more output vectors z represent the one or more key points present in the image data.

[0069] In a fifth implementation, the method includes configuring the apparatus to generate, for each image, a set of one or more output vectors z. The apparatus is configured to calculate an independent variable parameter in a multiplication operation of the set of one or more output vectors z. The independent variable parameter indicates whether a given feature is present in the plurality of images.

[0070] In a sixth implementation, the method includes configuring the apparatus to compute a given keypoint k by extracting a local patch centered at the keypoint k from the image data. The local patch is rescaled such that cosine similarity of the one or more output vectors z of the local patches of the plurality of images is used to determine whether the local patches represent the same feature for communication detection purposes.

[0071] Optionally, the method includes post-processing methods such as hysteresis threshold method and non-maximum suppression method to detect the key points to improve robustness, accuracy, reliability and repeatability measurement.

[0072] The contrast loss function is used to train the device. The contrast loss function is defined as:

[0073]

[0074] Where, at the top, the division operation sim(z i ,z j ) evaluates the cosine similarity between positive samples; at the bottom, the cosine similarity sim(z) of the negative samples in the pair of image data (204A and 204B) is calculated and summed up i ,z j The positive samples are transformed versions of overlapping cropped regions of the same image. The negative samples are transformed cropped regions of different images and non-overlapping cropped regions of the same image.

[0075] In one implementation, a method of operating an autonomous vehicle is provided, the autonomous vehicle comprising the apparatus for performing automatic keypoint and description extraction from image data representing at least one image of a field of view captured from a spatial region surrounding the autonomous vehicle to enable the autonomous vehicle to navigate in the spatial region.

[0076] Figure 5 A diagram of a computerized device 500 is shown in which various architectures and functions of the various implementations described above can be implemented. As shown, the computerized device 500 includes at least one processor 504 connected to a bus 502. The computerized device 500 can be implemented using any suitable protocol, such as Peripheral Component Interconnect (PCI), PCI Express, Accelerated Graphics Port (AGP), HyperTransport, or any other bus or one or more point-to-point communication protocols. The computerized device 500 also includes a memory 506.

[0077] Control logic (software) and data are stored in the memory 506, which may be in the form of random-access memory (RAM). In the present invention, a single semiconductor platform may refer to a single semiconductor-based integrated circuit or chip. It should be noted that the term "single semiconductor platform" may also refer to a multi-chip module with higher connectivity, wherein the multi-chip module simulates an on-chip module with higher connectivity, wherein the on-chip module simulates on-chip operation, and wherein the multi-chip module achieves substantial improvements in utilizing traditional central processing units (CPUs) and bus implementations. Of course, various modules may also be provided separately or in combination with various semiconductor platforms, depending on user needs.

[0078] The computerized device 500 may also include an auxiliary memory 510. For example, the auxiliary memory 510 includes a hard disk drive and a removable storage drive, such as a floppy disk drive, a tape drive, an optical disk drive, a digital versatile disk (DVD) drive, a recording device, or a universal serial bus (USB) flash memory. The removable storage drive performs at least one of the following operations: reading from and writing to a removable storage unit in a well-known manner.

[0079] A computer program or computer control logic algorithm may be stored in at least one of the memory 506 and the secondary storage 510. Such a computer program, when executed, enables the computerized device 500 to perform the various functions described above. The memory 506, the secondary storage 510, and any other storage are possible examples of computer-readable media.

[0080] In one implementation, the architecture and functionality depicted in the above figures may be implemented in the context of the processor 504, a graphics processor coupled to the communication interface 512, an integrated circuit (not shown) capable of simultaneously having at least a portion of the functionality of the processor 504 and the graphics processor, a chipset (i.e., a group of integrated circuits designed to work and sold as a unit that performs related functions, etc.).

[0081] Furthermore, the architecture and functionality depicted in the above figures can be implemented in the context of a general-purpose computer system, a circuit board system, a gaming system dedicated to entertainment purposes, or a dedicated system. For example, the computerized device 500 can take the form of a desktop computer, a laptop computer, a server, a workstation, a gaming console, or an embedded system.

[0082] Furthermore, the computerized device 500 may take the form of various other devices, including but not limited to a personal digital assistant (PDA) device, a mobile phone device, a smart phone, a television, etc. In addition, although not shown, the computerized device 500 may be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a wired network, etc.) via an I / O interface 508 for communication.

[0083] It should be understood that the arrangements of components shown in the described figures are exemplary and that other arrangements are possible. It should also be understood that the various system components (and devices) defined by the claims, described below, and shown in the various block diagrams represent components in some systems configured according to the subject matter disclosed herein. For example, one or more of these system components (and devices) may be implemented in whole or in part by at least some of the components shown in the arrangements shown in the described figures.

[0084] Additionally, while at least one of these components may be at least partially implemented as electronic hardware components and thus constitute a machine, other components may be implemented in software which, when included in an execution environment, constitutes a machine, hardware, or a combination of software and hardware.

[0085] Although the present invention and its advantages have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the disclosure as defined by the appended claims.

Claims

1. An apparatus (102, 202, 302) for performing automatic keypoint and description extraction from image data (204A and 204B) representing a plurality of images, characterized in that The image data (204A and 204B) are input (104) into the device (102, 202, 302), wherein the device (102, 202, 302) includes a data processing device (106), the data processing device (106) is coupled to a data storage device (108) and is configured to execute one or more neural network algorithms, and the device (102, 202, 302) includes: (i) a communication network device (110) implemented as an algorithm executed by the data processing device (106), wherein the communication network device (110) is configured to: process the image data (204A and 204B) to remove regions of the plurality of images having an information content below a given threshold; select at least one region in each of the plurality of images representing mutually common features to generate one or more output feature vectors h representing features present in the image data (204A and 204B); (ii) a feature description device (112) implemented as an algorithm executed by the data processing device (106), wherein the feature description device (112) is configured to: receive the one or more output feature vectors h of the plurality of images; generate one or more output vectors z, the one or more output vectors z representing one or more key points (230A-N, 232A-N) present in each of the plurality of images of the image data (204A and 204B); The one or more neural algorithms use a keypoint detection algorithm that utilizes guided backpropagation of output neurons in a neural network to detect the one or more keypoints (230A-N, 232A-N) in the image data (204A and 204B) by generating an output gradient tensor g, and filters out less relevant keypoints from the output gradient tensor g using a hysteresis thresholding method applied to the detected one or more keypoints (230A-N and 232A-N).

2. The device (102, 202, 302) according to claim 1, characterized in that The keypoint detection algorithm is configured by detecting the output value of the most highly activated neurons stimulated by the keypoint detection algorithm, which describes the most informative features present in the image data (204A and 204B).

3. The device (102, 202, 302) according to claim 1 or 2, characterized in that The apparatus (102, 202, 302) is trained using contrastive learning based on a sampling algorithm customized for keypoint and description extraction to automatically detect visual similarities and differences between images; wherein the sampling algorithm is used to process a combination of positive and negative versions of at least one image of input data, wherein the at least one image of the input data preserves the spatial relationship between the images; wherein positive samples are described as partially overlapping regions of the images and negative samples are described as non-overlapping regions of one or the other image.

4. The device (102, 202, 302) according to claim 1 or 2, characterized in that The communication network device (110) and the feature description device (112) are respectively implemented as an encoder network (206A, 206B) and a projection head network (210A, 210B); wherein the encoder network (206A, 206B) is used to receive the image data (204A and 204B) and generate the one or more output feature vectors h; the projection head network (210A, 210B) is used to receive the one or more output feature vectors h and generate the one or more output vectors z therefrom, wherein the one or more output vectors z represent the one or more key points (230A-N, 232A-N) present in the image data (204A and 204B).

5. The device (102, 202, 302) according to claim 1 or 2, characterized in that The device (102, 202, 302) is used to generate the set of one or more output vectors z for each image, wherein the device (102, 202, 302) is used to calculate an independent variable parameter in a multiplication operation of the set of one or more output vectors z, wherein the independent variable parameter indicates whether the same given feature is present in the multiple images.

6. The device (102, 202, 302) according to claim 5, characterized in that The apparatus (102, 202, 302) is configured to compute a given keypoint k by extracting a local patch centered at the keypoint k from the image data (204A and 204B), wherein the local patch (242A, 242B) is rescaled such that the cosine similarity of the one or more output vectors z of the local patches (242A, 242B) of the plurality of images is used to determine whether the local patches (242A, 242B) represent the same feature for communication detection purposes.

7. The device (102, 202, 302) according to claim 5, characterized in that The apparatus (102, 202, 302) is configured to calculate a keypoint k by extracting a local patch centered at a given keypoint k from the image data (204A and 204B); wherein the apparatus (102, 202, 302) is configured to use an additional neural network algorithm that is trained by contrastive learning based on a combination of positive and negative versions of at least one image of input data, wherein the at least one image of the input data preserves spatial relationships between the images; wherein the additional neural network algorithm is configured to process local patches extracted around the local patch centered at the keypoint k to learn visual similarities and differences from the input data; wherein the cosine similarity of the one or more output vectors z of the local patches (242A, 242B) of the multiple images with one or more neural networks is configured to determine whether the local patches (242A, 242B) represent the same features for communication detection purposes.

8. A method for using a device (102, 202, 302), characterized in that The apparatus (102, 202, 302) is for performing automatic keypoint and description extraction from image data (204A and 204B) representing a plurality of images, wherein the image data (204A and 204B) is input (104) into the apparatus (102, 202, 302), wherein the apparatus (102, 202, 302) comprises a data processing device (106) coupled to a data storage device (108) and configured to execute one or more neural network algorithms; wherein the method comprises: (i) configuring the apparatus (102, 202, 302) to use a communication network apparatus (110), the communication network apparatus (110) being implemented as an algorithm executed by the data processing apparatus (106), wherein the communication network apparatus (110) is configured to: process the image data (204A and 204B) to remove regions of the plurality of images having an information content below a given threshold; select at least one region in each of the plurality of images representing mutually common features to generate one or more output feature vectors h representing features present in the image data (204A and 204B); (ii) configuring the apparatus (102, 202, 302) to use a feature description device (112), the feature description device (112) being implemented as an algorithm executed by the data processing device (106), wherein the feature description device (112) is configured to: receive the one or more output feature vectors h of the plurality of images; generate one or more output vectors z, the one or more output vectors z representing one or more key points (230A-N, 232A-N) present in each of the plurality of images of the image data (204A and 204B); The method includes configuring the one or more algorithms to use a keypoint detection algorithm that utilizes guided backpropagation through output neurons in a neural network to detect the one or more keypoints (230A-N, 232A-N) in the image data (204A and 204B) by generating an output gradient tensor g, and filtering out less relevant keypoints from the output gradient tensor g using a hysteresis thresholding method applied to the detected one or more keypoints (230A-N and 232A-N).

9. The method according to claim 8, characterized in that The method includes configuring the keypoint detection algorithm by detecting output values of the most highly activated neurons stimulated by the keypoint detection algorithm, the keypoint detection algorithm describing the most informative features present in the image data (204A and 204B).

10. The method according to claim 8 or 9, characterized in that The method comprises training the apparatus (102, 202, 302) using contrastive learning based on a sampling algorithm tailored for keypoint and description extraction to automatically detect visual similarities and differences between images; wherein the sampling algorithm is used to process a combination of positive and negative versions of at least one image of input data, wherein the at least one image of the input data preserves spatial relationships between the images; wherein positive samples are described as partially overlapping regions of the images and negative samples are described as non-overlapping regions of one or the other image.

11. The method according to claim 8 or 9, characterized in that The method includes implementing the communication network device (110) and the feature description device (112) as an encoder network (206A, 206B) and a projection head network (210A, 210B), respectively; wherein the encoder network (206A, 206B) is used to receive the image data (204A and 204B) and generate the one or more output feature vectors h; the projection head network (210A, 210B) is used to receive the one or more output feature vectors h and generate the one or more output vectors z therefrom, wherein the one or more output vectors z represent the one or more key points (230A-N, 232A-N) present in the image data (204A and 204B).

12. The method according to claim 8 or 9, characterized in that The method comprises configuring the apparatus (102, 202, 302) to generate the set of one or more output vectors z for each image, wherein the apparatus (102, 202, 302) is configured to calculate an independent variable parameter in a multiplication operation of the set of one or more output vectors z, wherein the independent variable parameter indicates whether the same given feature is present in the plurality of images.

13. The method according to claim 12, characterized in that The method comprises configuring the apparatus (102, 202, 302) to compute a given keypoint k by extracting a local patch centered at the keypoint k from the image data (204A and 204B), wherein the local patch (242A, 242B) is rescaled such that cosine similarity of the one or more output vectors z of the local patches (242A, 242B) of the plurality of images is used to determine whether the local patches (242A, 242B) represent the same feature for communication detection purposes.

14. The method according to claim 13, wherein: The method comprises configuring the apparatus (102, 202, 302) to compute a keypoint k by extracting a local patch centered around a given keypoint k from the image data (204A and 204B); wherein the method comprises configuring the apparatus (102, 202, 302) to use an additional neural network algorithm trained by contrastive learning based on a combination of positive and negative versions of at least one image of input data, wherein the at least one image of the input data preserves spatial relationships between the images; wherein the additional neural network algorithm is used to process the local patches extracted around the local patch centered around the keypoint k to learn visual similarities and differences from the input data; wherein the cosine similarity of the one or more output vectors z of the local patches (242A, 242B) of the multiple images with one or more neural networks is used to determine whether the local patches (242A, 242B) represent the same features for communication detection purposes.

15. An autonomous driving vehicle (304), characterized in that Comprising an apparatus (102, 202, 302) according to any one of claims 1 to 7, the apparatus (102, 202, 302) being for performing automatic keypoint and description extraction from image data (204A and 204B) representing at least one image of a field of view captured from a spatial region surrounding the autonomous vehicle (304), to enable the autonomous vehicle (304) to navigate in the spatial region.

16. A method of operating an autonomous vehicle (304), characterized in that The autonomous vehicle (304) comprises an apparatus (102, 202, 302) according to any one of claims 1 to 7, the apparatus (102, 202, 302) being configured to perform automatic keypoint and description extraction from image data (204A and 204B) representing at least one image of a field of view captured from a spatial region surrounding the autonomous vehicle (304) to enable the autonomous vehicle (304) to navigate in the spatial region.

17. A computer program product, characterized in that Comprising computer readable instructions executable by a computerized device (500) comprising processing hardware for performing the method according to any one of claims 8 to 14 or claim 16.

Images