Feature detector and descriptor

Abstract

An image processor comprising a plurality of modules including a first module and a second module. The image processor is configured to receive an input image and output a plurality of mathematical descriptors of a feature region of the input image; the first module is configured to implement a first trained artificial intelligence model to detect a set of feature regions in the input image; the second module is configured to implement a second trained artificial intelligence model to determine a mathematical descriptor of each feature region in the set of feature regions, wherein the first trained artificial intelligence model and the second trained artificial intelligence model are trained end-to-end.

Description

Technical Field

[0001] This invention relates to image processing, and more particularly to extracting features from an input image. Background Technology

[0002] Feature extraction is a crucial step in many computer vision applications, helping 2D and 3D vision systems build a reliable backbone that ensures the robustness and effectiveness of the application, thereby providing a satisfying user experience. The core problem of feature extraction is identifying meaningful locations within a scene that can be easily located with high precision from different viewpoints, while also being visually distinguishable from other candidate objects.

[0003] Modern 3D vision pipelines for simultaneous localization and mapping (SLAM), augmented reality, and photogrammetry all rely on robust feature extraction stages to function seamlessly. Furthermore, 2D vision applications requiring geometric understanding of scenes or image alignment (such as panoramic photo stitching) demand reliable feature extractors. Once feature regions and their mathematical descriptors are extracted, they can be matched not only to infer 2D relationships between images but also to leverage these relationships to generate 3D reconstructions and depth maps from images, and many other 3D applications.

[0004] A standard feature extraction pipeline typically includes: a detector, used to detect feature regions in an image called keypoints or patches; and a continuous descriptor, used to mathematically describe the regions surrounding the keypoints. Keypoints and descriptors are generated to describe these locations, even when the image is under different geometric or photometric transformations.

[0005] Therefore, this task is twofold. First, a reasonable number of reliable points need to be found within a specific region of interest; this could be a part of the image or the entire image. Second, using information from the vicinity of the keypoints, a function needs to be found to robustly extract meaningful representations of the keypoints.

[0006] Figure 1An example of a detector-descriptor pipeline is shown, and the process is illustrated with two exemplary images from different viewpoints, as shown in 101 and 104. The photographic images are formed by an RBG camera. In the first stage, the detector detects specific keypoints. The locations of these keypoints are represented by circles in the images at 102 and 105. These locations are discriminative points within the input images, calculated by the detector. They are further processed by a descriptor that assigns multidimensional values ​​to each individual keypoint and its neighborhood in the descriptor space. Keypoints can be matched by comparing differences in the descriptor spaces. In this example, given the metrics in the descriptor space, corresponding regions within the two images are matched if the descriptors are close to each other. Points below a specified threshold that are closest to the descriptor are matched. The matching is illustrated by the connecting line between the images at 103 and 106.

[0007] Since the detector's output is the descriptor's input, these two tasks are interdependent. Most current state-of-the-art methods focus on developing solutions for either of these tasks, or describing the entire image pixel by pixel.

[0008] Traditionally, detectors and descriptors are based on extracting features from human designs used in high-efficiency pipelines.

[0009] To address the problem under discussion, several hand-designed pipelines are introduced. Most of these methods utilize differential operators applied as filters to the image to generate gradient images and Hessian. While edge detectors and corner detectors can be efficiently designed as filters, blob detectors typically analyze a smoothed version of the image through simple differential operations (e.g., Gaussian and Laplacian sources, Gaussian differences, Hessian determinants). To ensure the algorithm is invariant to photometric transformations and geometrical variations (such as scale, rotation, and differences in more complex mappings), these filters are used in carefully designed pipelines to detect keypoints and characterize their neighborhoods.

[0010] An example of this type of algorithm is the scale-invariant feature transform (SIFT), as described by David Lowe in "Distinctive image features from scale-invariant keypoints" published in the *International Journal of Computer Vision* (IJCV) in 2004 and in US 6711293 B1. The SIFT pipeline is a manually designed method that infers orientation, location, and description from scale-space volume. This method uses a 128-dimensional vector to describe the found SIFT points. For localization, Gaussian differences are used, while scale invariance is achieved through a scale-space pyramid, and rotation invariance is handled through an orientation histogram.

[0011] While known methods have proven effective and accurate in many applications, their design philosophy is to remain invariant to a specific set of transformations and is primarily driven by observing how the human eye distinguishes certain key locations. Furthermore, methods like SIFT can only handle local geometric distortions to a limited extent. Overall, due to limited modeling capabilities, they cannot specifically analyze the large number of differences that may exist in an image.

[0012] Data-driven methods inherently analyze the variance of the fed image during parameter optimization. Doing this within the framework of convolutional neural networks allows for the optimization of an approximator based on an ideal kernel best suited to the available data. Most methods focus on learning a detector or a descriptor.

[0013] There is a need to develop an image processing system that can overcome the above problems. Summary of the Invention

[0014] According to one aspect, an image processor is provided, the image processor comprising a plurality of modules, the plurality of modules including a first module and a second module, wherein the image processor is configured to receive an input image and output a plurality of mathematical descriptors of feature regions of the input image; the first module is configured to implement a first trained artificial intelligence model to detect a set of feature regions in the input image; the second module is configured to implement a second trained artificial intelligence model to determine a mathematical descriptor of each feature region in the set of feature regions, wherein the first trained artificial intelligence model and the second trained artificial intelligence model are uniformly trained end-to-end.

[0015] The input to the second module may include the output of the first module. Therefore, the image processor can be implemented in an image processing pipeline.

[0016] The first module can be used to detect feature regions of the input image by combining manually designed features and learned features. The second module can be used to determine the mathematical descriptor of the feature regions by combining manually designed features and learned features. This can improve robustness to changes in luminance and geometry. The hybrid formula utilizes the manually designed features as prior knowledge of robustness while also possessing generality during the learning phase. In this way, meaningful representations can be computed without needing to extract robustness separately from the data, thereby achieving efficient computation.

[0017] At least one of the first and second modules can be used to aggregate data from regions of different sizes in the input image. This ensures that features can be extracted from images with different contexts based on the resolution of the input image.

[0018] The input image can be an RGB image. This ensures that the image processor can be used by devices equipped with cameras, such as smartphones.

[0019] The feature region can be the edges and / or corners of the input image. This ensures that the image processor can be used for computer vision applications.

[0020] According to a second aspect, a method is provided for implementation at an image processor, the image processor comprising a plurality of modules, including a first module and a second module, wherein the image processor is configured to receive an input image and output a plurality of mathematical descriptors for feature regions of the input image, the method comprising: implementing a first trained artificial intelligence model at the first module to detect a set of feature regions in the input image; implementing a second trained artificial intelligence model at the second module to determine a mathematical descriptor for each feature region in the set of feature regions; wherein the first trained artificial intelligence model and the second trained artificial intelligence model are uniformly trained end-to-end.

[0021] According to a third aspect, a method is provided for training a machine learning system to be implemented at an image processor, the system comprising a first module and a second module, the first module being configured to implement a first trained artificial intelligence model to detect a set of feature regions in an input image, and the second module being configured to implement a second trained artificial intelligence model to determine a mathematical descriptor for each feature region in the set of feature regions; the method comprising: uniformly training the first artificial intelligence model and the second artificial intelligence model end-to-end.

[0022] The method may include mutually optimizing the functions of the first trained artificial intelligence model and the functions of the second trained artificial intelligence model. This can improve performance.

[0023] The method may include: training a second artificial intelligence model based on the output of the training phase of the first artificial intelligence model. The method may also include: subsequently training the first artificial intelligence model based on the output of the training phase of the second artificial intelligence model. This ensures that the descriptor-based detector and the descriptor-based detector can refine each other.

[0024] The method may include: alternately learning a first trained artificial intelligence model and a second trained artificial intelligence model. The method may also include: iteratively updating the parameters of the first trained artificial intelligence model and the second trained artificial intelligence model. This ensures that the models can be continuously improved during training.

[0025] The method may include: updating the parameters of the first trained artificial intelligence model to improve the repeatability of the first trained artificial intelligence model. The method may also include: updating the parameters of the second trained artificial intelligence model to improve the discriminative score of the second trained artificial intelligence model. Therefore, the joint training of the models can ensure the detection of features that are both repeatable and distinguishable.

[0026] According to the fourth aspect, a machine learning system is provided for implementation at an image processor, the machine learning system being used to perform any of the methods described above. Attached Figure Description

[0027] The invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0028] Figure 1 This illustrates a traditional detector-descriptor pipeline;

[0029] Figure 2 The hybrid detector-descriptor pipeline is shown;

[0030] Figure 3 The bidirectional training of the detector and descriptor is shown;

[0031] Figure 4 Another example of bidirectional training of the detector and descriptor is shown;

[0032] Figure 5 The manual design of feature extraction for descriptors is illustrated;

[0033] Figure 6The feature extraction from descriptor learning is illustrated;

[0034] Figure 7 Hierarchical context aggregation is shown;

[0035] Figure 8 An example of hybrid hierarchical context aggregation is shown;

[0036] Figure 9 A method for implementation at an image processing system is shown;

[0037] Figure 10 A camera used to implement the image processor described herein is shown;

[0038] Figure 11 This demonstrates the impact of bidirectional training on keypoint scores;

[0039] Figures 12(a) to 12(c) A qualitative comparison of different detector-descriptor pipelines in feature matching is shown. Detailed Implementation

[0040] The image processor described in this paper formulates the problem of extracting features from images as a combination of two functions, h (detector) and d (descriptor). These functions can either fuse manually designed features or learn representations to combine the results of both approaches. These two functions are approximated using a general function approximator. Conveniently, this could be a convolutional neural network that jointly optimizes the mapping with specific input data while aggregating context from different levels.

[0041] Figure 2 An overview of an example of a hybrid image processing pipeline is shown. Input image 201 is fed to detector stage 202, which combines hand-designed, learnable features (described in more detail below) to detect distinguishable keypoints shown as circles in detection map 203. A sequential description stage 204 further combines the hand-designed features with the learned representations to output a descriptor map 205 describing the keypoint locations. The description can be sparse (keypoint locations) or dense (all pixels).

[0042] Traditionally, detectors and descriptors are two separate entities that are processed and trained. As described below, the detector and descriptor architecture described in this paper can be trained by training the two component modules alternately.

[0043] Detector and descriptor methods typically have two different objectives. Detectors tend to detect repeatable regions that can be detected consistently (or, preferably always), such as images taken from different viewpoints, while descriptors tend to present discriminative regions that ultimately lead to unique representations of these parts in image matching tasks.

[0044] For example, given input images of the same object taken from different viewpoints, the model's repeatability reflects the proportion of instances where the same keypoint can be detected from different viewpoints. The model's discriminative score reflects the degree to which it can distinguish different keypoints. A high discriminative score indicates that the model can distinguish different keypoints.

[0045] These two goals are often contradictory. For example, highly textured surfaces with repeatable patterns will be advantageous for detection, while descriptors may struggle to distinguish the detected parts within each instance of the repeating pattern. Therefore, joint training of the various components within the feature extraction pipeline (i.e., joint training of the detector and the descriptor) is beneficial.

[0046] Figure 3 An overview of the bidirectional mutual training process of detector 301 and descriptor 302 is shown. After performing optimization steps on the detector to improve the repeatability of keypoints, the descriptor stage is updated, which emphasizes the discriminativeness of features. This joint training ensures the ability to detect features that are both repeatable and discriminative. During the training of detector 301, updates improve the repeatability of features. Following these updates, descriptor stage 302 is updated, which improves the discriminative score of the features (i.e., the ability of features to distinguish each other). Figure 3 The arrows shown indicate that training is performed alternately between the detector and the descriptor. Therefore, the image processor is used to train the descriptor based on the output of the detector's training phase. The image processor can also train the detector based on the output of the descriptor's training phase. The image processor can then train the detector based on the output of the descriptor's training phase, and / or vice versa. Thus, the image processor can perform learning on the detector and descriptor alternately (i.e., one after another and preferably repeatedly), and can iteratively update the parameters of the model these modules are meant to implement. Therefore, there is a mutual refinement between the descriptor-based detector and the detector-based descriptor.

[0047] In terms of function approximation, the feature extraction pipeline can be viewed as a combination of two functions: h (the detector) and d (the descriptor). Training the neural network pipeline containing the detector and the descriptor can be considered an optimization problem in the function space, such that a composite function of the descriptor is applied after the detector is retrieved. Traditionally, these two problems are treated independently. The outcome of the entire pipeline depends on both parts. Mutual updates of these functions can improve the combination of these functions, which will be described in more detail below.

[0048] The detector function h assigns a score s from the score range S = [0,1] to each pixel location within image I. h can be viewed as a function that includes an identity and a scoring function, which retrieves the probability that a given pixel p is a keypoint, e.g., h(p) = (p,s), where s >> 0. Meanwhile, points that are not reliably detected are assigned a low score.

[0049] A descriptor can be viewed as a function d that assigns multidimensional values ​​to a given pixel within an image, which is considered to both accurately describe the image content of that portion and be robust to image transformations. If pixel regions correspond to the same world point, the multidimensional values ​​assigned to a particular pixel can be found in the hypercube C. n The values ​​are close in the middle, where C = [–1, 1], but if this is not the case, the description may be different.

[0050] To approximate the two functions h and d, the method described in this paper learns to assign a pseudo-metric to the embedding of the descriptor, enabling the system to distinguish different points well while ensuring consistency in the points generated by different projections of the same world point. In this implementation, metric learning can be used, leveraging the Mahalanobis distance between points obtained through triplet loss (see "Learning a distance metric from relative comparisons" by Schultz and Joachims, presented at Advances in Neural Information Processing Systems (NeurIPS) in 2004). The descriptor c of the candidate patch... j Relative to anchor patch c i In Euclidean embedding space C n For different points, the values ​​are pushed further apart, while if the anchor point and the candidate object are similar, their values ​​are pulled closer together. In practice, n=128 is used.

[0051] Training can be performed jointly in the sense that an optimization step for the descriptors is performed after the detector optimization step, similar to the expectation-maximization optimization scheme. These two functions can be optimized separately using stochastic gradient descent, using a common loss function d(h(c)). i To optimize the combination of these two functions, the different losses L for the detector and descriptor are used. det and L desc They can be used for mutual updates as described below, or L desc It can be used with alternating gradient step sizes along the direction given by the weights of d or h.

[0052] Each loss can be written as:

[0053] L det =‖d(h(c i ))–d(h(c j ))‖ p (1)

[0054] For plaque c i and c j According to the description, the distance update detector is as follows:

[0055]

[0056] For the descriptor, use the anchor patch c i Positive sample c j negative sample c k And the margin α between positive and negative pairs, learning a meaningful pseudo-metric in the embedding space. In practice, p=2 or a robust Huber-loss can be used. The network described above can be trained using these losses until convergence.

[0057] Therefore, for both detectors and descriptors, the loss formula is based on repeatability and discriminability, for example, through metric learning.

[0058] Figure 4 The bidirectional training of detector 401 and descriptor 402 is illustrated, each having functions h and d respectively. As described above, the descriptor and detector training are performed jointly. During the optimization of detector h with stochastic gradient d, a step is performed using a fixed detector approximation of d. In this example, relative to the common composite loss function L... det (Equation (1)) calculates the gradient. Regarding the triplet loss L... desc The metric learning of the Mahalanobis pseudometric in (Equation (2)) performs descriptor-driven steps such that similar points are assigned to similar descriptors, while points corresponding to different regions are pushed apart in the descriptor space (bottom). This process is repeated iteratively through mutual updates of the detector and descriptors until convergence.

[0059] Therefore, the architecture's function can be approximated as a combination of joint training and / or bidirectional training of the detector and descriptor.

[0060] Therefore, detectors and descriptors are correlated not only in reasoning but also during training.

[0061] As mentioned above, the detector stage can combine manually designed features and learnable features. Manually designed features refer to features derived using information present in the input image itself. For example, a basic edge detector algorithm can detect feature regions such as edges by detecting areas of sudden changes in image intensity. Learned features refer to features that can be learned from the input image data, i.e., features automatically obtained through machine learning algorithms. For example, convolutional networks can learn feature regions from the input image data because they are generated from the data.

[0062] For hybrid extraction by the detector, the architecture and implementation described in Key.Net (Axel Barroso et al., “Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters”, International Conference on Computer Vision (ICCV), 2019) can be applied. This architecture and implementation combine handcrafted features with learned features to extract keypoints from the input image.

[0063] For descriptors, an example of a manually designed feature extraction stage is as follows: Figure 5 As shown. In this example, the rotation-equivariant directional derivative kernel is optimally extracted using the method described by Farid and Simoncelli in "Optimally rotation-equivariant directional derivative kernels" published at the International Conference on Computer Analysis of Images and Patterns (ICIP) in 1997.

[0064] As shown in 502, image 501 is processed by a set of discrete differential operators, which are applied as filters to the image, as proposed by Farid and Simoncelli in their 1997 presentation, "Optimally rotation-equivariant directional derivative kernels," at the International Conference on Computer Analysis of Images and Patterns (ICIP). These filters rotate in 22° steps to produce outputs robust to rotations of 2D images. A set of discrete differential operators (i.e., Rotate in 22° increments and apply to the image individually. This increment improves robustness to rotations of image content.

[0065] Then, as shown in 503, 16 rotation filters are applied to the image, and in 504, the maximum activation of every three rotation filters is selected.

[0066] To ensure robustness to scale changes, a Gaussian blur pyramid can be applied, where dilated convolutions look at different contexts for each rotated filter to find the maximum activation across scale dimensions.

[0067] As shown in 505, a Gaussian blurred pyramid is created for the image, which is processed by dilated convolutions with increased stride, as shown in 506, to gather context from an ever-growing neighborhood. This information is compressed through max-pooling layers in the spatial dimension (see Worrall and Welling, “Deep Scale-spaces: Equivariance Over Scale,” arXiv, 2019), as shown in 507.

[0068] During training, a whitening transformation can be applied to the features across the entire batch to transform the feature statistics into identity covariance. This represents the features as uncorrelated signals with a variance of 1 within the hypersphere.

[0069] Figure 6Feature extraction of the learned descriptor is illustrated. In this example, the learned features used in the architecture proposed by Tian et al. in "L2-net: Deep learning of discriminative patch descriptor ineuclidean space" published in the IEEE Transactions on Computer Vision and Pattern Recognition in 2017 are extracted. L2-Net is a fully convolutional neural network whose filtering volume increases hierarchically with depth. The network applies individual convolutional blocks of convolutional layers, followed by batch normalization and ReLU activation.

[0070] Image 601 is processed by the learned feature extraction stage L2-Net. Seven convolutional blocks with hierarchically increasing filter banks are appended to the architecture, typically as shown in 602, to extract feature map 603. Each block includes a convolutional layer on which batch normalization and ReLU-form activation are applied. The first filter is a 3x3 convolution with 32 learned filters, and the last layer includes a 9x9 convolution with 128 learned filters.

[0071] Therefore, the architecture combines manually designed features with learnable features. A similar approach is used to combine manually designed features with learned features in the detector. This improves robustness to changes in photometric intensity and geometry by design.

[0072] In addition, the image processor can also perform hierarchical multi-context aggregation.

[0073] The first layer filters of a convolutional neural network (CNN) applied to an image have only limited spatial awareness of the context surrounding the point to which the filter is applied. While deeper information fusion can combine data from different spatial locations, the convolution operation itself is translation invariant. This means that applying a convolution kernel to an image does not include information about the location of the kernel within that image. While this is an advantage during CNN training, it also presents a problem when spatial context should be aggregated early in the CNN process, as only a finite set of filters can have trained weights. This can be overcome by using a hierarchical context aggregation scheme.

[0074] To achieve aggregation from different spatial locations, this architecture can advantageously include filters that operate on both local and global perception, allowing for the simultaneous inclusion of information from both spatially nearby and more distant locations. For this purpose, spatial sampling pyramids can be utilized by applying filters individually to extract features at different image resolutions. Performing this multi-context feature extraction in both the detector and descriptor makes the results more robust to perturbations related to scale variations in the image, while simultaneously aggregating information from different neighborhood sizes. Figure 7 As shown in Figure 701, a spatial pyramid of downsampled images is generated at reduced resolution. The feature extraction stage 702 operates on each image separately to retrieve information including different neighborhood sizes, where the receptive field of the early-stage filters includes different contexts within the image. The extracted features have different contexts depending on the resolution of the input image, as shown in Figure 703. The information aggregation stage combines the information from these context maps to arrive at a final decision, as shown in Figure 704.

[0075] Similar to the combination described above Figure 7 The described scale-space method can be used to fuse features of both artificially designed pipelines (HF) and learning phases (L). For example... Figure 8 As shown. In this hybrid hierarchical context aggregation, the spatial image pyramid extracts individually designed and learned features from three different layers to augment the spatial context of the applied filters into other neighborhoods at lower scales. The feature layers computed at a resized resolution are first upsampled to full resolution, then fed into a neural network that learns filter weights across scale dimensions, ultimately outputting a combined feature map with aggregated information.

[0076] The spatial image pyramid consists of three levels, reducing the resolution from 1 / 1 to 1 / 2 and 1 / 4, typically as shown in 801. Extracted artificially designed (HC) features and learned (L) features follow a process of increasing neighborhood size to reduce resolution and are stored at different resolutions with a feature dimension of 128, typically as shown in 802. The low-resolution feature map is upsampled by bilinear upsampling across the spatial domain, creating three feature maps at the same resolution, as shown in 803. A set of filters 804 is learned to combine the feature maps across the scale dimension. The final feature map 805 contains information aggregated based on the different contexts generated by the previous steps.

[0077] Therefore, the image processor can perform feature extraction and aggregation on multiple contexts for the detector and descriptor. Thus, the detector and descriptor can advantageously aggregate data from input image regions of different sizes surrounding the feature region. These different regions can be increasingly larger neighborhoods, which can have hierarchically increasing neighborhoods around the feature region.

[0078] Figure 9 A method for implementation at an image processor is summarized. As described above, the image processor includes multiple modules, including a first module and a second module, wherein the image processor is configured to receive an input image and output multiple mathematical descriptors of feature regions of the input image. In step 901, the method includes: implementing a first trained artificial intelligence model in the first module to detect feature regions in the input image. In step 902, the method includes: implementing a second trained artificial intelligence model in the second module to determine the mathematical descriptors of the feature regions. As described above, the first trained artificial intelligence model and the second trained artificial intelligence model are uniformly trained end-to-end.

[0079] Figure 10 An example of a camera is shown for implementing an image processor to process images captured by the image sensor 1002 in camera 1001. Such a camera 1001 typically possesses some in-vehicle processing capabilities. This can be provided by processor 1004. Processor 1004 can also be used for basic device functions. Cameras typically also include memory 1003.

[0080] Transceiver 1005 is capable of communicating with other entities 1010 and 1011 via a network. These entities may be physically located away from camera 1001. The network may be a publicly accessible network, such as the Internet. Entities 1010 and 1011 may be cloud-based. In one example, entity 1010 is a computing entity, and entity 1011 is a command and control entity. These entities are logical entities. In practice, each of these entities may be provided by one or more physical devices such as servers and data storage, and the functionality of two or more of these entities may be provided by a single physical device. Each physical device implementing an entity includes a processor and memory. The device may also include a transceiver for sending and receiving data to and from transceiver 1005 of camera 1001. The memory stores in a non-transient manner code that can be executed by the processor to implement the respective entity in the manner described herein.

[0081] As described above, the command and control entity 1011 can train the artificial intelligence model used in the system. This is typically a computationally intensive task, even if the resulting model can be efficiently described, and therefore can be highly efficient for developing algorithms to be executed in the cloud, where a large amount of energy and computing resources can be expected to be available. Predictably, this is more efficient than forming such a model in a typical camera.

[0082] In one implementation, once a deep learning algorithm is developed in the cloud, the command and control entity can automatically generate a corresponding model and transmit that model to the relevant camera device. In this example, the system is implemented at camera 1001 by processor 1004.

[0083] In another possible implementation, images can be captured by camera sensor 1002, and the image data can be sent to the cloud by transceiver 1005 for processing in the system. The resulting target image can then be sent back to camera 1001, such as... Figure 10 As shown in 1012.

[0084] Therefore, this method can be deployed in various ways, such as in the cloud, on a device, or alternatively in dedicated hardware. As shown above, cloud facilities can be used for training to develop new algorithms or improve existing ones. Training can be performed close to the source data or in the cloud, for example using an inference engine, depending on the computing power available near the data corpus. The system can also be implemented at a camera, in dedicated hardware, or in the cloud.

[0085] In some implementations, the joint training described above can have a significant impact on the results of the descriptors and detectors, such as Figure 11 As shown.

[0086] Figure 11 The impact of bidirectional training on keypoint scores is illustrated. The source image 1101 is fed into a detector-descriptor-frame that has undergone two separate and joint training sessions. Detector scores are visualized below for independent training (as shown in 1102) and joint training (as shown in 1103), where darker values ​​correspond to higher scores. For portions of the images shown in 1104 and 1105 in 1102 and 1103 respectively, 1106 shows these two regions in detail. For joint training, even with repetitive structures, the response along the edges in the zoom area is high. This response decreases to corner points, making the keypoints more localized and thus more reliable when performing sequential tasks such as matching.

[0087] In typical (independent) training, high scores are assigned to feature points regardless of their repetition in the image; joint training, on the other hand, produces easily distinguishable regions. Figure 11 In the image shown, this can be observed by looking at high gradient regions that have high responsiveness in independent training, while using joint detector descriptor training to reduce pixels along the image edges to feature structures at corner points.

[0088] Figures 12(a) to 12(c)A comparison of results from a feature matching task is shown, where the pipeline described in this paper and existing techniques are used to connect the closest descriptors for keypoints in an image from two different perspectives.

[0089] Keypoints and individual descriptors for a set of images are computed using SIFT in Figure 12(a), D2-Net (D2-Net: A Trainable CNN for Joint Description and Detection of Local Features, IEEE Transactions on Computer Vision and Pattern Recognition (CVPR), 2019) in Figure 12(b), and the pipeline described in Figure 12(c). Images are displayed as matches based on the nearest descriptor distance of keypoints detected by the three different methods. SIFT performs well under small viewpoint changes but poorly under more general transformations. The neural network D2-Net is more robust to larger viewpoint changes but suffers from repetitive structural issues. As described in this paper, the pipeline jointly trained for the detector and descriptors demonstrates the best results for matching under significant viewpoint changes, even with repetitive patterns in the scene.

[0090] The hybrid detector-descriptor architecture described in this paper tightly intertwines keypoint extraction and feature description tasks using efficient, hand-designed pipelines and data-driven approaches. Compared to existing methods, the method described in this paper combines hand-designed and learned features from both the detector and descriptor, bidirectional intertwining of the detector and descriptor during training, and methods for hierarchical context aggregation.

[0091] Human-designed features and learned features are extracted and combined from images to create a final representation. While the human-designed features are designed to be robust to specific geometric and photometric transformations, they also benefit from the generalization capabilities of learning-based methods that inherently leverage data statistics. The hybrid formula utilizes these human-designed features as prior knowledge of robustness while also possessing generality during the learning phase. This allows for the computation of meaningful representations without the need to extract robustness separately from the data, resulting in efficient computation. This enables the implementation and execution of pipelines on embedded devices such as mobile phones.

[0092] The applicant hereby separately discloses each individual feature described herein, as well as any combination of two or more such features. It is possible, to the ordinary knowledge of those skilled in the art, to implement such features or combinations as a whole based on this specification, regardless of whether such features or combinations of features solve any problem disclosed herein, and without prejudice to the scope of the claims. This application indicates that aspects of this disclosure can be constituted by any such individual feature or combination of features. Given that various modifications are possible to the foregoing description within the scope of this disclosure, it will be apparent to those skilled in the art.

Claims

1. An image processor, characterized in that, The image processor includes multiple modules, including a first module and a second module, wherein the image processor is used to receive an input image and output multiple mathematical descriptors of feature regions of the input image; The first module is used to implement a first trained artificial intelligence model to detect a set of feature regions in the input image; The second module is used to implement a second trained artificial intelligence model to determine the mathematical descriptor of each feature region in the set of feature regions; Wherein, at least one of the first module and the second module is used to aggregate data from regions of different sizes from the input image; the first trained artificial intelligence model and the second trained artificial intelligence model are uniformly trained end-to-end; the input of the second module includes the output of the first module; the parameters of the first trained artificial intelligence model and the second trained artificial intelligence model are iteratively updated; the second trained artificial intelligence model is trained based on the output of the training phase of the first trained artificial intelligence model, and the first trained artificial intelligence model is trained based on the output of the training phase of the second trained artificial intelligence model.

2. The image processor according to claim 1, characterized in that, The first module is used to detect feature regions of the input image by combining manually designed features and learned features.

3. The image processor according to claim 1 or 2, characterized in that, The second module is used to determine the mathematical descriptor of the feature region by combining manually designed features and learned features.

4. The image processor according to claim 1 or 2, characterized in that, The input image is an RGB image.

5. The image processor according to claim 1 or 2, characterized in that, The feature region is the edge and / or corner of the input image.

6. A method for implementation at an image processor, characterized in that, The image processor includes multiple modules, including a first module and a second module. The image processor receives an input image and outputs multiple mathematical descriptors for feature regions of the input image. The method includes: A first trained artificial intelligence model is implemented in the first module to detect a set of feature regions in the input image; A second trained artificial intelligence model is implemented in the second module to determine the mathematical descriptor of each feature region in the set of feature regions; Wherein, at least one of the first module and the second module is used to aggregate data from regions of different sizes from the input image; the first trained artificial intelligence model and the second trained artificial intelligence model are uniformly trained end-to-end; the input of the second module includes the output of the first module; the parameters of the first trained artificial intelligence model and the second trained artificial intelligence model are iteratively updated; the second trained artificial intelligence model is trained based on the output of the training phase of the first trained artificial intelligence model, and the first trained artificial intelligence model is trained based on the output of the training phase of the second trained artificial intelligence model.

7. A method for training a machine learning system for implementation at an image processor, characterized in that, The system includes a first module and a second module. The first module is used to implement a first trained artificial intelligence model to detect a set of feature regions in an input image, and the second module is used to implement a second trained artificial intelligence model to determine a mathematical descriptor for each feature region in the set of feature regions. At least one of the first module and the second module is used to aggregate data from regions of different sizes of the input image; the method includes: uniformly training a first artificial intelligence model and a second artificial intelligence model end-to-end; The second artificial intelligence model is trained based on the output of the training phase of the first artificial intelligence model; the first artificial intelligence model is trained based on the output of the training phase of the second artificial intelligence model; and the parameters of the first trained artificial intelligence model and the second trained artificial intelligence model are iteratively updated.

8. The method according to claim 7, characterized in that, The method includes: mutually optimizing the functions of the first trained artificial intelligence model and the functions of the second trained artificial intelligence model.

9. The method according to claim 7 or 8, characterized in that, The method includes: alternately learning a first trained artificial intelligence model and a second trained artificial intelligence model.

10. The method according to claim 7 or 8, characterized in that, The method includes updating the parameters of the first trained artificial intelligence model to improve the repeatability of the first trained artificial intelligence model.

11. The method according to claim 7 or 8, characterized in that, The method includes updating the parameters of the second trained artificial intelligence model to improve the discrimination score of the second trained artificial intelligence model.

12. A machine learning system implemented at an image processor, characterized in that, The machine learning system is used to perform the method according to any one of claims 7 to 11.

Images