Image processing method, computer system, and computer storage medium

Abstract

Image processing methods, computer systems, and computer storage media are disclosed. In one example, key points are selected in an image and different motion blurs are applied to the image to generate a plurality of blurred images. Based on neighborhoods of the key points in each of the plurality of blurred images, an artificial neural network is trained at each of a plurality of different scales to generate key point descriptors.

Description

Technical Field

[0001] This invention generally relates to methods and systems related to image processing. More specifically, embodiments of the invention provide methods and systems for performing feature matching in augmented reality applications. Background Technology

[0002] Augmented reality (AR) overlays virtual content onto the user's view of the real-world environment. With the development of AR software development kits (SDKs), the mobile industry has brought smartphone AR into the mainstream. AR SDKs typically provide six degrees-of-freedom (6DoF) tracking capabilities. Users can scan their environment using their smartphone's camera, and the smartphone performs visual inertial odometry (VIO) in real time. After continuously tracking the camera's pose, virtual objects can be placed into the AR scene to create the illusion that real and virtual objects are blended together.

[0003] Keypoints (or “points of interest”) are points in an image that are distinct from other points in the image, have a well-defined spatial location or are otherwise positioned within the image, and are stable under local and global changes (e.g., scale changes, illumination changes, etc.). A keypoint descriptor can be defined as a multi-element vector that describes the neighborhood (typically in scale space) of the keypoint in the image. Examples of keypoint descriptor frameworks include Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), and Binary Robust Invariant Scalable Keypoints (BRISK). Image features can be defined as keypoints and corresponding keypoint descriptors.

[0004] Matching features between different images (e.g., different frames of a video sequence) is a crucial part of many image processing applications. Images acquired while the image sensor is moving may exhibit significant motion blur. Since most widely used image feature representations are highly sensitive to motion blur, feature matching is difficult to achieve successfully when extracting features from images with varying degrees of motion blur. Therefore, there is a need in the art for improved methods to perform feature matching. Summary of the Invention

[0005] The embodiments of the present invention can be used in various applications of augmented reality and computer-based display systems.

[0006] A method for generating robust keypoint descriptors to motion blur, based on a typical configuration, includes: selecting multiple keypoints in an image; applying each of the multiple distinct motion blurs to the image to generate multiple blurred images; and training an artificial neural network (ANN) at each of multiple different scales based on the neighborhood of the keypoints in each of the multiple blurred images to generate keypoint descriptors. In this method, the training criterion is to minimize the distance measurement between instances of the generated keypoint descriptors, where the instances correspond to the same keypoint in different images of the multiple blurred images.

[0007] According to another general configuration, the computer system includes one or more processors and one or more memories for storing computer-readable instructions that, when executed by the one or more processors, configure the computer system to: select a plurality of keypoints in an image; apply each of the plurality of motion blurs to the image for each of the plurality of distinct motion blurs to generate a plurality of blurred images; and train an artificial neural network (ANN) at each of a plurality of different scales based on the neighborhood of the keypoints in each of the plurality of blurred images to generate keypoint descriptors. In this system, the training criterion is to minimize the distance measurement between instances of the generated keypoint descriptors, where the instances correspond to the same keypoint in different images of the plurality of blurred images.

[0008] According to another general configuration, one or more non-transitory computer storage media store instructions that, when executed by a computer system, cause the computer system to perform the following operations: select multiple keypoints in an image; for each of a plurality of distinct motion blurs, apply each of the plurality of motion blurs to the image to generate multiple blurred images; and train an artificial neural network (ANN) at each of a plurality of different scales based on the neighborhood of the keypoints in each of the plurality of blurred images to generate keypoint descriptors. The training criterion is to minimize the distance measurement between instances of the generated keypoint descriptors, where the instances correspond to the same keypoint in different images of the plurality of blurred images.

[0009] This invention achieves many advantages over conventional techniques. For example, the methods and systems disclosed herein use deep learning networks to generate keypoint descriptors robust to motion blur, thereby addressing the challenge of image feature matching when different motion blurs exist in camera frames. For example, embodiments of this invention can be used to improve the performance of localization and mapping in AR / VR applications. Furthermore, embodiments of this invention can increase the accuracy of feature matching when extracting features from acquired camera frames with different image motions. Therefore, SLAM calculations can be more accurate and stable when the device is moving rapidly because one or more previously wasted blurred camera frames can be used efficiently for localization and mapping estimation. Thus, embodiments of this invention avoid the disadvantages of existing methods, such as lower matching accuracy when motion is very slight or the image is blurred by similar motion. These and other embodiments of the invention, along with their many advantages and features, will be described in more detail below in conjunction with the accompanying drawings. Attached Figure Description

[0010] Figure 1 A simplified flowchart of an image processing method according to an embodiment of the present invention is shown.

[0011] Figure 2A An example of an image according to an embodiment of the present invention is shown.

[0012] Figure 2B An embodiment of the present invention is shown. Figure 2A Examples of key points in the image shown.

[0013] Figure 3 An example diagram of six degrees of freedom (6DOF) is shown.

[0014] Figure 4 A simplified flowchart of a method for generating training data according to an embodiment of the present invention is shown, the training data being used to train a network.

[0015] Figure 5 A simplified flowchart illustrating an example of a task to train an artificial neural network (ANN) to generate keypoint descriptors according to an embodiment of the present invention is shown.

[0016] Figure 6 A simplified flowchart illustrating the operation of the ANN-based keypoint descriptor comparison module according to an embodiment of the present invention is shown.

[0017] Figure 7 A simplified flowchart of a method for training an ANN with a keypoint descriptor comparison module according to an embodiment of the present invention is shown.

[0018] Figure 8A and Figure 8B An example of a training standard according to an embodiment of the present invention is shown.

[0019] Figure 9 A simplified block diagram of an apparatus according to an embodiment of the present invention is shown.

[0020] Figure 10 A simplified schematic diagram of a keypoint descriptor converter according to an embodiment of the present invention is shown.

[0021] Figure 11 A simplified flowchart of an image processing method according to an embodiment of the present invention is shown.

[0022] Figure 12 A simplified flowchart of a method for performing image processing according to an embodiment of the present invention is shown.

[0023] Figure 13 A simplified flowchart of a method for training an ANN according to an embodiment of the present invention is shown.

[0024] Figure 14 A simplified block diagram of an apparatus according to an embodiment of the present invention is shown.

[0025] Figure 15 A block diagram of a computer system according to an embodiment of the present invention is shown. Detailed Implementation

[0026] In the following description, various embodiments will be described. Specific configurations and details are set forth for illustrative purposes in order to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that the embodiments may also be practiced without these specific details. Furthermore, well-known features may be omitted or simplified so as not to obscure the described embodiments.

[0027] Many applications heavily rely on the performance of image feature matching. These applications can include image alignment (e.g., image stitching, image registration, panoramic mosaic), three-dimensional (3D) reconstruction (e.g., stereo vision), indexing and content retrieval, motion tracking, object recognition, and more.

[0028] A fundamental requirement for many augmented reality / virtual reality (AR / VR) applications is determining the device's position and orientation in 3D space. These applications can use simultaneous localization and mapping (SLAM) algorithms to determine the device's real-time position and orientation, and deduce the structure of the environment (or scene) in which the device operates. In one example of a SLAM application, a sequence of video frames from the device's camera is input into a module that executes the SLAM algorithm. Features are extracted from the frames, and the SLAM algorithm searches for matching features corresponding to the same point in the scene being filmed by matching features across different frames. By tracking the position of features across different frames, the SLAM module can determine the motion of the image sensors within the scene and deduce the main structure of the scene.

[0029] In traditional feature matching, keypoint detection is performed on each of a plurality of images (e.g., each frame of a video sequence), and a corresponding keypoint descriptor is computed from its neighborhood (typically in scale space) for each detected keypoint. Typically, the number of keypoints detected for each image is at least several dozen, possibly up to five hundred or more, and the neighborhood used to compute the keypoint descriptor usually has a radius of approximately fifteen pixels around the keypoint. Keypoint descriptors f0 from a first image I0 (e.g., a frame of a video sequence) and f1 from a second image I1 (e.g., another frame of a video sequence, such as consecutive frames in the sequence) are used to compute a matching score. The score metric is typically the distance d(f0,f1) between the keypoint descriptors f0 and f1 in descriptor space, such as the distance according to any of the following example distance metrics. The above score computation is repeated for different keypoint descriptor pairs in the two images, and the resulting scores can be thresholded to identify matching features, for example, to determine whether a particular pair of keypoint descriptors (and therefore, corresponding features in the two images) matches. In typical SLAM applications, a pair of matched features corresponds to a single point in the physical environment, and this correspondence generates mathematical constraints. Later in the SLAM computation, a model of camera motion and the environment can be derived as the optimal solution to satisfy multiple constraints, including those generated by the matched feature pairs.

[0030] Examples of distance metrics that can be used for matching score calculations include Euclidean distance, city-block distance, chi-squared distance, cosine distance, and Minkowski distance. Assume f0 and f1 are n-dimensional vectors, f0 = x 0,1 ,x 0,2 ,x 0,3 ,...,x0,n f1 = x 1,1 ,x 1,2 ,x 1,3 ,...,x 1,n Based on these distance metrics, the distance d(f0,f1) between f0 and f1 can be described as follows:

[0031] Euclidean distance:

[0032] Distance between city blocks:

[0033] Cosine distance: in,

[0034] Chi-square distance (assuming all elements of f0 and f1 are greater than 0):

[0035] Minkowski distance (also known as generalized Euclidean distance):

[0036] The obtained scores can be thresholded according to a process such as the following to determine whether the corresponding features match:

[0037]

[0038]

[0039] Where T represents the threshold.

[0040] In typical feature matching applications, the above feature point descriptor matching process is repeated for multiple different keypoint descriptor pairs (f0, f1) across consecutive frame pairs in a video sequence (e.g., within each consecutive frame pair). In one example, for each keypoint among multiple keypoints detected in a first image (each keypoint having a corresponding position in the first image), the keypoint descriptor matching process is repeated for each keypoint descriptor pair, wherein each keypoint descriptor pair includes the descriptor of that keypoint in the first image and the descriptor of a keypoint within a threshold distance (e.g., 20 pixels) at the same position in the second image.

[0041] Keypoint descriptor matching is not limited to source images of any specific size or format (i.e., first image I0 and second image I1). Examples from typical feature matching applications are provided here. Most current AR / VR devices use VGA format video (i.e., frame size 640×480 pixels), with each pixel having red, green, and blue components. The largest common frame format in such devices is 1280×720 pixels, so in typical applications, the maximum size of each image in the first and second images is approximately 1000×2000 pixels. In typical applications, because smaller image sizes may not be sufficient to support algorithms such as SLAM, the minimum size of the first and second images is approximately one-quarter of a VGA (i.e., 320×240 pixels).

[0042] Keypoint descriptor computation is typically implemented using existing keypoint descriptor frameworks such as Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), and Binary Robust Invariant Scalable Keypoints (BRISK). Such tasks may include computing the orientation of keypoints, which can involve determining how or in what direction the pixel neighborhood (also known as an "image patch") around the keypoint is oriented. Computing the orientation of keypoints may include detecting the most dominant orientation of gradient angles in the patch, which is typically performed on the patch at different scales in scale space. For example, the SIFT framework assigns a 128-dimensional feature vector to each keypoint based on the gradient orientations of pixels in its 16 local neighborhoods. Some keypoint descriptor frameworks (e.g., SIFT and SURF) combine keypoint detection and keypoint descriptor computation. Other keypoint descriptor frameworks (e.g., Binary Robust Independent Elementary Features (BRIEF)) include keypoint descriptor computation but not keypoint detection.

[0043] Examples of AR / VR devices include mobile phones and head-mounted devices (e.g., AR or "smart" glasses). Given the nature of AR / VR devices, many video frames are captured as the image sensor (e.g., a video camera) moves. Therefore, the captured frames may have significant motion blur. If the image sensor's motion during the acquisition of image I0 is the same as its motion during the acquisition of image I1, and each of the descriptors f0 and f1 corresponds to the same keypoint in both images, then the values ​​of descriptors f0 and f1 tend to be similar, and the computational distance between f0 and f1 is small. However, in practical AR / VR applications, the image sensor typically experiences different motion when acquiring each image, so descriptors f0 and f1 may be distorted due to varying degrees of motion blur. Almost all widely used image features (e.g., SIFT, SURF, BRISK) are highly sensitive to motion blur, so any significant motion blur (e.g., blur of five pixels or more) can cause distortion of the keypoint descriptors. As a result, even if descriptors f0 and f1 correspond to the same keypoint, their values ​​can differ significantly. Feature matching may fail when these features are extracted from images with different motion blurs and the scoring metric described above is used.

[0044] For many applications where the image sensor may be in motion (e.g., AR / VR applications), image motion during each acquisition interval can be quantified. For example, the direction and magnitude of motion can be estimated based on input from one or more motion sensors of the device, and / or the direction and magnitude of motion can be calculated based on two temporally adjacent frames in a video sequence. Motion sensors (which may include one or more gyroscopes, accelerometers, and / or magnetometers) can indicate changes in the device's displacement and / or orientation and can be implemented within an inertial measurement unit (IMU).

[0045] Examples of techniques for handling motion blur can include:

[0046] 1) Mismatch: Since image feature matching becomes very unreliable in blurred images, one possible solution is to abandon image feature matching altogether, at least not between image pairs with significant and distinct motion blur.

[0047] 2) First, deblur the image: Before using the image for feature extraction, perform a deblurring operation to remove motion blur from the image.

[0048] 3) Compensating for motion blur when computing keypoint descriptors: When computing keypoint descriptors, the effect of motion blur is compensated using an estimate of image motion. This method differs from "deblurring the image first" in that motion blur removal or compensation is performed in the neighborhood of the keypoints rather than over the entire image.

[0049] 4) Extracting Blur-Invariant Features: When calculating keypoint descriptors from the neighborhood of keypoints, only blur-invariant components are used, while those sensitive to motion blur are ignored. Therefore, even if the image has motion blur, the keypoint descriptors will remain largely unchanged.

[0050] 5) Motion-Blur Robust Image Feature Matching: In many feature matching applications (e.g., in most SLAM applications), motion blur can be quantized. Therefore, a deep learning-based motion blur transformer can be designed to simulate descriptor distortion caused by motion blur. Before feature matching, at least one of the descriptors to be matched is transformed, ensuring that the input descriptors contain the same motion blur effect. In another example, a deep learning-based motion-blur-aware descriptor comparison module can be designed to determine whether input features match given known image motion.

[0051] The disadvantages of the above solutions may include:

[0052] 1) Mismatch: To prevent mismatches of features in blurred images from degrading the quality of estimations in SLAM, feature matching can be omitted entirely when significant image motion is detected. For example, SLAM can be performed using only the motion sensor output at these moments. However, this approach results in the complete waste of the image sensor output at these moments, potentially leading to less accurate and unstable SLAM computations.

[0053] 2) First, deblur the image: Image deblurring typically involves a significant amount of computation. Since SLAM computations are usually performed on mobile platforms, the additional computation required for deblurring may not always be available or affordable. Furthermore, deblurring often introduces new artifacts into the original image, which in turn can negatively impact the accuracy of image feature matching.

[0054] 3) Compensating for motion blur when computing keypoint descriptors: Because it only involves the neighborhood of the keypoint, compensating for motion blur when computing keypoint descriptors usually requires less additional computation than deblurring the entire image. However, the drawback of introducing new artifacts into the image still exists.

[0055] 4) Extracting blur-invariant features: Because this method ignores components sensitive to motion blur, less information is available for feature matching. In other words, this method improves matching accuracy and stability when the camera is moving significantly, but at the cost of potentially reducing matching performance in other situations (e.g., when motion is not obvious).

[0056] 5) Motion-Blur-Robust Image Feature Matching: These proposed solutions improve upon the keypoint descriptor matching stage. However, traditional keypoint descriptors are easily distorted by motion blur because they are not designed to account for the camera's ongoing motion. Therefore, matching performance remains limited. This approach considers image motion in the matching scoring stage to address the matching failures discussed above. However, to fully realize the potential of the improved scoring scheme, it may be necessary to design a new descriptor that is less sensitive to motion blur than traditional descriptors.

[0057] It may be necessary to add robustness of keypoint descriptions to motion blur. Therefore, as disclosed herein, the embodiments implemented using appropriate systems, methods, apparatuses, devices, etc., can support improved accuracy of feature matching operations in applications prone to motion blur. The embodiments described herein can be implemented in any of a variety of applications using feature matching, including image alignment (e.g., image stitching, image registration, panoramic mosaic), 3D reconstruction (e.g., stereo vision), indexing and content retrieval, endoscopic imaging, motion tracking, object tracking, object recognition, autonomous navigation, SLAM, etc.

[0058] According to embodiments of the present invention, a deep learning network is trained and used to generate keypoint descriptors robust to motion blur to address the challenge of image feature matching posed by varying motion blur in camera frames. For example, embodiments of the present invention can be used to improve the performance of localization and mapping in AR / VR applications. Furthermore, embodiments of the present invention can increase the accuracy of feature matching when extracting features from camera frames with different image motions. Therefore, SLAM calculations can be more accurate and stable when the device is moving rapidly because one or more previously wasted blurred camera frames can be effectively used for localization and mapping estimation. Thus, embodiments of the present invention avoid the disadvantages of existing methods, such as lower matching accuracy when motion is very slight or the image is blurred by similar motion.

[0059] Figure 1 A simplified flowchart of a method for generating keypoint descriptors robust to motion blur according to an embodiment of the present invention is shown. Figure 1The method 100 for generating keypoint descriptors robust to motion blur, as shown, includes tasks 110, 120, and 130. Task 110 selects a plurality of corresponding keypoints in an image. For each of a plurality of distinct motion blurs, task 120 applies each of the plurality of motion blurs to the image to generate a plurality of blurred images. Task 130 trains an artificial neural network (ANN) at each of a plurality of different scales based on the neighborhood of the keypoints in each of the plurality of blurred images to generate keypoint descriptors, wherein the training criterion is to minimize the distance measurement between instances of the generated keypoint descriptors that correspond to the same feature in different images of the plurality of blurred images.

[0060] It should be understood that Figure 1 The specific steps shown provide a particular method for generating keypoint descriptors robust to motion blur according to embodiments of the present invention. As described above, other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the above steps in a different order. Furthermore, Figure 1 The steps shown may include multiple sub-steps, which can be performed in various orders depending on the individual steps. Furthermore, additional steps may be added or removed depending on the specific application. Those skilled in the art will recognize many variations, modifications, and substitutions.

[0061] Keypoints are points in an image that are distinct from other points in the image, have a well-defined spatial location or are otherwise located within the image, and are stable under local and global changes (e.g., scale changes, brightness changes, etc.).

[0062] Figure 2A An example of an image (e.g., a frame of a video sequence) according to an embodiment of the present invention is shown. Figure 2B An embodiment of the present invention is shown. Figure 2A Examples of key points in the image shown. (Reference) Figure 2B , Figure 2B The circles shown indicate the locations of several key points in the image, examples 210-220. In typical feature matching applications, the number of key points detected in each image is at least twelve, twenty-five, or fifty, and may be as high as one hundred, two hundred, five hundred, or more.

[0063] refer to Figure 1Task 110 selects multiple corresponding keypoints in an image. Examples of keypoint detectors that can be used to implement Task 110 include corner detectors (e.g., Harris corner detector, features from Accelerated Segment Test (FAST)) and blob detectors (e.g., Laplacian of Gaussian (LoG), Difference of Gaussians (DoG), Determinant of Hessian (DoH)). Such keypoint detectors can be used to blur (e.g., Gaussian blur) and resample the image with different blur widths and resolutions to create a scale space and detect corners and / or blobs at different scales. For example, Task 100 may include downsampling the original image to create versions at different resolutions. For example, for an image with an original resolution of 640×480 pixels, downsampling may be necessary to obtain the same image at resolutions such as 320×240 pixels and 160×120 pixels. For neighboring windows of the same size, a lower resolution allows for coverage of a larger scene area.

[0064] Refer again Figure 1 For each of a plurality of distinct motion blurs, Task 120 applies each of the plurality of motion blurs to an image to generate a plurality of blurred images. It may be desirable that the training data is sufficient to allow the trained ANN to encapsulate complex logic and computation. Image motion blur can be simulated using image processing operations (such as directional filtering), and one or more of these operations can be used to generate large amounts of synthetic training data.

[0065] Each of the multiple motion blurs can be implemented as a motion descriptor, which describes a trajectory or path in a coordinate space of one, two, or three spatial dimensions. In one example, such a trajectory can be described as a series of locations in a two-dimensional image plane, and each location can be represented as a motion vector relative to a previously sampled location (e.g., using the location of a keypoint as the starting position). In another example, such a trajectory can be described as a sequence of one or more locations of an image sensor acquiring the image, which are sampled at uniform intervals within a corresponding acquisition period, and each sampled location can be represented as a motion vector relative to a previously sampled location (e.g., using the position of the image sensor at the beginning of the acquisition period as the origin of the coordinate space).

[0066] It may be desirable to compute motion blur based on data collected during actual image acquisition, such as data collected during a real-world instance of a feature matching application where the descriptor generated by method 100 can be applied. For example, motion blur can be computed from the output of a motion sensor (e.g., an IMU) and / or from adjacent frames in a video sequence (e.g., frames acquired immediately before and after the frame being computed). The acquisition period for each frame in a video sequence is typically the reciprocal of the frame rate, but the acquisition period can be shorter. A typical video sequence (e.g., captured by an Android phone) has a frame rate of 30 frames per second (fps). iPhones or head-mounted devices can have frame rates up to 120 fps.

[0067] Each motion descriptor can be further implemented to describe six degrees of freedom (6DOF) motion. In addition to the three spatial dimensions, 6DOF motion can include rotations about one or more axes of these dimensions. For example... Figure 3 As shown, it illustrates a 6DOF diagram, where these rotations can be labeled as tilt, pitch, and yaw. For example, for each sampling position of the image sensor, the motion descriptor can include the direction of the image sensor's reference direction (e.g., gaze direction) relative to the direction of the previous sampling position (e.g., let the reference direction be the direction at the start of the acquisition cycle).

[0068] Refer again Figure 1 Task 130 trains an artificial neural network (ANN) to generate keypoint descriptors. The training performed in Task 130 is based on the neighborhood of keypoints in each of multiple blurred images and is performed at each of multiple different scales. For the case of Task 110, which involves blurring and resampling images with different blur widths and resolutions to create a scale space, the neighborhood of keypoints can be obtained from the same scale space created. Alternatively (or where different scale spaces are required), Task 130 may include blurring (e.g., Gaussian blurring) and resampling images with different blur widths and resolutions to create a scale space.

[0069] In one example, the input to the ANN is the pixel values ​​of the neighborhood windows of keypoints at different resolutions. The choice of the actual window size at each resolution can be considered in terms of matching and computational performance. Larger windows typically cover a larger neighborhood region, but also make the descriptor generation network larger. Therefore, more computation may be required to generate the corresponding descriptors.

[0070] The output of the ANN is a multi-element vector, which is the generated descriptor. The actual length of the descriptor can be determined based on considerations of matching and computational performance, as well as memory usage and memory access overhead when storing and retrieving a large number of descriptors. While longer descriptors typically encapsulate more neighborhood information, they also make the comparison network in the keypoint descriptor comparison module 600 (described below) larger. Therefore, more computation may be consumed when matching descriptors. In addition, more memory and / or more data bandwidth may be required to store and retrieve longer descriptors. When the number of descriptors is large, memory and / or bandwidth overhead can increase significantly.

[0071] Figure 4 A simplified flowchart of a method for generating training data for task 130 according to an embodiment of the present invention is shown. For each of a plurality of training images 420, a plurality of keypoints in the training images 422 are detected (e.g., as described herein with reference to task 110). A plurality of motion blurs M1 to Mn are also applied 424-1 to 424-n to the training images 420 to generate a plurality of corresponding blurred images B1 to Bn. From each of the plurality of blurred images B1 to Bn, for each of the detected keypoints, a neighborhood is extracted at each of a plurality of different scales to obtain a corresponding set of neighborhoods S1 to Sn. The number of neighborhoods may vary from set to set: for example, if the corresponding motion blur causes the neighborhood of a keypoint to include a region that extends beyond the image boundary, that neighborhood may be omitted from the set.

[0072] The training criterion performed in Task 130 is to minimize the distance metric between instances of the generated keypoint descriptors, where instances of the generated keypoint descriptors correspond to the same keypoints in different images of multiple blurred images. Examples of distance metrics that can be used in Task 130 include Euclidean distance, chi-square distance, etc.

[0073] It should be understood that Figure 4 The specific steps shown provide a particular method for generating training data according to embodiments of the present invention. As described above, other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the above steps in a different order. Furthermore, Figure 4 The steps shown may include multiple sub-steps, which can be performed in various orders depending on the individual steps. Furthermore, additional steps may be added or removed depending on the specific application. Those skilled in the art will recognize many variations, modifications, and substitutions.

[0074] Figure 5A simplified flowchart illustrating an example of a task 130 of training an artificial neural network (ANN) 500 to generate keypoint descriptors according to an embodiment of the present invention is shown. First, the trained ANN 500 receives the neighborhood of keypoint i from a blurred image Bj (i.e., from set Sj) as input from the training data and generates a descriptor Dij as output. Next, the trained ANN 500 receives the neighborhood of keypoint i from another blurred image Bk (i.e., from set Sk) as input from the training data and generates a descriptor Dik as output.

[0075] Loss function selector 510 compares motion blurs Mj and Mk corresponding to the two generated descriptors Dij and Dik. If selector 510 determines that the motion blurs are similar (e.g., the distance between the motion blurs does not exceed a threshold), then a first loss function 512 is selected. In one example, the first loss function 512 is used to minimize the distance between the generated descriptors (e.g., Euclidean distance, chi-square distance, etc.).

[0076] If selector 510 determines that the motion blur is different (e.g., the distance between the motion blurs exceeds a threshold), then a second loss function 514 is selected. In one example, the second loss function 514 is used to maximize the matching probability, as indicated by the ANN-based keypoint descriptor comparison module 600. While the first loss function 512 could be omitted and only the second loss function 514 used, using the distance-based loss function 512 can be expected to reduce computational requirements when keypoint descriptors f0 and f1 come from images with similar motion blur (e.g., during slow camera movement). Inference via a deep learning network typically involves a much larger amount of computation compared to calculating the distance between two keypoint descriptors.

[0077] It should be understood that Figure 5 The specific steps shown provide a particular method for training an ANN to generate keypoint descriptors according to embodiments of the present invention. As described above, other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the above steps in a different order. Furthermore, Figure 5 The steps shown may include multiple sub-steps, which can be performed in various orders depending on the individual steps. Furthermore, additional steps may be added or removed depending on the specific application. Those skilled in the art will recognize many variations, modifications, and substitutions.

[0078] Figure 6 A simplified flowchart illustrating the operation of an ANN-based keypoint descriptor comparison module 600 according to an embodiment of the present invention is shown. Figure 6As shown, the keypoint descriptor comparison module 600 is used to determine whether two keypoint descriptors match, even if these features belong to two images with different image motions. Figure 6 As shown, the keypoint descriptor comparison module 600 receives four inputs, including two keypoint descriptors to be compared (denoted as f0 and f1) and their corresponding motion blurs (denoted as M0 and M1). Before inputting into the ANN, the values ​​of the motion blurs may need to be normalized to have the same range as the values ​​of the generated keypoint descriptors. The output is a binary decision 610 (i.e., whether keypoint descriptors f0 and f1 match) and a value P (e.g., a probability value) 620 representing the confidence level of the binary decision.

[0079] In a manner similar to that described above, a deep learning network is trained to produce matching indications and confidence values ​​for the keypoint descriptor comparison module 600. In this case, the network can be implemented as a classifier network known in the field of deep learning. Given sufficient training data, the classifier will typically produce good outputs. It can be used as described above (e.g., referencing...). Figure 4 The training data obtained can be used to train such a network (e.g., a CNN). It may be necessary to augment the synthetic training data with descriptors computed from a relatively small number of images with realistic motion blur and manually labeled by humans.

[0080] It should be understood that Figure 6 The specific steps shown provide specific operations for the ANN-based keypoint descriptor comparison module according to embodiments of the present invention. As described above, other sequences of steps can also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the above steps in a different order. Furthermore, Figure 6 The steps shown may include multiple sub-steps, which can be performed in various orders depending on the individual steps. Furthermore, additional steps may be added or removed depending on the specific application. Those skilled in the art will recognize many variations, modifications, and substitutions.

[0081] Figure 7A simplified flowchart illustrating a method for training an ANN 700 of a keypoint descriptor comparison module 600 according to an embodiment of the present invention is shown. The ANN 700 can be implemented as a binary classifier such that the output of the ANN indicates the probability that the keypoint descriptor input represents the same feature in the corresponding image. For example, the output layer of the ANN can be configured to apply a sigmoid or softmax function, and the loss function can be implemented as a binary cross-entropy function, indicating whether the training input represents the same feature that provides the ground truth for the loss function during ANN training. In one example, the ANN is implemented such that both the input and output layers are 32×32 arrays. Since both ANN 500 and ANN 700 are based on deep learning, the parameters in the two networks can be cross-optimized. For example, the training of the two networks can be combined. In one such example, the training process begins using one or more conventional keypoint descriptor frameworks (e.g., SIFT, SURF, BRISK).

[0082] In addition to the aforementioned training criteria based on matching accuracy (used to generate matching descriptors corresponding to the same keypoints but with different motion blurs (i.e., reference loss functions 512 and 514)), it may be necessary to train ANN 700 to generate different descriptors for different keypoints. For example, for cases where the generated descriptors correspond to different keypoints, it might be desirable to implement task 130 to include corresponding loss functions to approximate the reference distance between descriptors calculated based on keypoints in the corresponding (unblurred) training images. For example, the reference distance can be calculated using existing keypoint descriptor frameworks such as SIFT, SURF, or BRISK.

[0083] It should be understood that Figure 7 The specific steps shown provide a particular method for training an ANN in a keypoint descriptor comparison module according to an embodiment of the present invention. As described above, other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the above steps in a different order. Furthermore, Figure 7 The steps shown may include multiple sub-steps, which can be performed in various orders depending on the individual steps. Furthermore, additional steps may be added or removed depending on the specific application. Those skilled in the art will recognize many variations, modifications, and substitutions.

[0084] Figure 8A Examples of three different training criteria are shown, which are applied to matching neighborhoods. These neighborhoods are blurred by similar or different motion blurs, and they are neighborhoods of the same or different keypoints. Figure 8BAs shown, task 130 may need further modification to train ANN 700 only on training data from images with significant motion blur. The generated descriptors obtained from this training criterion can be used for feature matching in combination with computed descriptor frameworks (e.g., SIFT, SURF, or BRISK), for example, such that the generated descriptors are used when significant motion blur is present, and the computed descriptors are used otherwise.

[0085] The trained ANN 500 can be used to generate keypoint descriptors (e.g., replacing traditional computational descriptor frameworks such as SIFT, SURF, BRISK, etc.) for feature matching as described herein. In a typical production environment, such as during manufacturing and / or pre-configuration, copies of the trained ANN 500 are stored in each of a series of devices with the same model of video camera (and possibly the same model of IMU).

[0086] Figure 9 A simplified block diagram of an apparatus according to an embodiment of the present invention is shown. As an example, Figure 9 The apparatus 900 shown can be used to generate key point descriptors on mobile devices (e.g., cellular phones such as smartphones or head-mounted devices) or other computing devices or systems according to a general configuration, which includes a key point selector 910, a motion blur applicator 920, and an ANN trainer 930.

[0087] A keypoint selector 910 is used to select multiple corresponding keypoints in the image (e.g., as described in Task 110 of this document). A motion blur applicator 920 is used to apply each of multiple distinct motion blurs to the image to generate multiple blurred images (e.g., as described in Task 120 of this document). An ANN trainer 930 is used to train an ANN at each of multiple different scales based on the neighborhood of the keypoints in each of the multiple blurred images (e.g., as described in Task 130 of this document), wherein the training criterion is to minimize the distance measurement between instances of the generated keypoint descriptors, which correspond to the same features in the different blurred images of the multiple blurred images.

[0088] In one example, device 900 is implemented within a device such as a mobile phone, which typically has a video camera for generating a frame sequence including a first image and a second image. The device may also include one or more motion sensors, which can be used to determine the device's 6DOF motion in space. In another example, device 900 is implemented in a head-mounted device, such as AR glasses, which may also have motion sensors and one or more cameras. Additionally or alternatively, such a device can be used to obtain training images (and possibly motion-blurred data) and / or apply the generated descriptors to feature matching.

[0089] According to other embodiments of the invention, deep learning networks can be used as keypoint descriptor motion blur converters or keypoint descriptor comparison modules to address the challenges of image feature matching when different motion blurs exist in camera frames. For example, descriptors generated according to the methods described herein can be implemented in conjunction with the descriptor transformation and / or comparison methods described in U.S. Provisional Patent Application No. 62 / 978,462 (Attorney General's No. 105184-1166208-002300US) to further improve feature matching accuracy when features are derived from acquired camera frames with different image motions. When camera motion is very slight or nonexistent, the proposed solution allows for high matching accuracy using a significantly simplified matching scoring process, thereby greatly reducing computational cost, compared to cases with significant camera motion.

[0090] Figure 10 A simplified schematic diagram of a keypoint descriptor converter 1000 according to an embodiment of the present invention is shown. Figure 10 As shown, a transformation is performed on the keypoint descriptor f0 generated by the trained ANN 500 from the neighborhood of keypoints in image I0. The descriptor converter receives three inputs: the keypoint descriptor f0, the motion descriptor M0 of the image motion when image I0 was acquired, and the motion descriptor M1 of the image motion when image I1 was acquired. The output of the converter is the transformed keypoint descriptor f1'. The design goal is that if descriptors f0 and f1 correspond to the same keypoint, the transformed keypoint descriptor f1' is similar to the keypoint descriptor f1 generated by the trained ANN 500 from the neighborhood of keypoints in image I1; however, if descriptors f0 and f1 correspond to different keypoints, the transformed descriptor f1' will be significantly different from descriptor f1. As described above, the first motion descriptor and the second motion descriptor describe the motion of the image sensor during the acquisition of the first and second images, respectively, and can be computed from the output of the motion sensor (e.g., IMU) and / or from adjacent frames in the video sequence (e.g., frames acquired immediately before and after the frame being computed).

[0091] Figure 11 A simplified flowchart of an image processing method according to an embodiment of the present invention is shown. Figure 11The image processing method 1100 shown includes tasks 1110, 1120, 1130, and 1140. Task 1110 uses a trained ANN 500 to generate descriptors for keypoints in a first image acquired by an image sensor during a first time period, wherein a first motion descriptor describes the motion of the image sensor during the first time period. Task 1120 uses the trained ANN 500 to generate descriptors for keypoints in a second image acquired by the image sensor during a second time period different from the first time period, wherein a second motion descriptor describes the motion of the image sensor during the second time period. Task 1130, based on the first and second motion descriptors, uses the trained ANN to convert the generated descriptors for keypoints in the first image into transformed descriptors. Task 1140 compares the transformed descriptors with the generated descriptors for keypoints in the second image. For example, Task 1140 may include calculating the distance (e.g., Euclidean distance, chi-square distance, etc.) between the transformed and generated descriptors in the descriptor space and comparing the calculated distance with a threshold.

[0092] It should be understood that Figure 11 The specific steps shown provide a particular method for performing image processing according to embodiments of the present invention. As described above, other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the above steps in a different order. Furthermore, Figure 11 The steps shown may include multiple sub-steps, which can be performed in various orders depending on the individual steps. Furthermore, additional steps may be added or removed depending on the specific application. Those skilled in the art will recognize many variations, modifications, and substitutions.

[0093] Figure 12 A simplified flowchart of a method for performing image processing according to another embodiment of the present invention is shown. Figure 12 As shown, Figure 12 Method 1200 shown uses the elements used in method 1100, as well as the additional elements described below. Therefore, where appropriate, with Figure 11 The relevant description also applies to Figure 12 Those skilled in the art will recognize many changes, modifications, and alternatives.

[0094] like Figure 12As shown, task 1210 selects keypoints in a first image, and task 1220 selects keypoints in a second image. Examples of keypoint detectors that can be used to implement tasks 1210 and 1220 include corner detectors (e.g., Harris corner detector, FAST) and blob detectors (e.g., LoG, DoG, DoH). Such keypoint detectors are typically configured to blur and resample images with different blur widths and sampling rates to create a scale space and detect corners and / or blobs at different scales.

[0095] Refer again Figure 12 Task 1130 uses a trained ANN to transform the descriptors of keypoints in a generated first image into transformed descriptors, based on a first motion descriptor and a second motion descriptor. Examples of ANNs that can be trained to perform such complex transformations between multi-element vectors include convolutional neural networks (CNNs) and autoencoders. It may be desirable to implement the ANN relatively small and fast, for example, including fewer than 10,000 parameters, or fewer than 5,000 parameters, and / or the trained ANN occupying less than 5 megabytes of storage. In one example, the ANN is implemented such that both the input and output layers are 32×32 arrays. In a typical production environment, such as during manufacturing and / or pre-configuration, copies of the trained ANN 500 are stored in each of a series of devices with the same model of video camera (and possibly the same model of IMU). Before inputting the trained ANN, it may be necessary to normalize the values ​​of the motion descriptors to have the same range as the values ​​of the generated keypoint descriptors.

[0096] It should be understood that Figure 12 The specific steps shown provide a particular method for performing image processing according to embodiments of the present invention. As described above, other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the above steps in a different order. Furthermore, Figure 12 The steps shown may include multiple sub-steps, which can be performed in various orders depending on the individual steps. Furthermore, additional steps may be added or removed depending on the specific application. Those skilled in the art will recognize many variations, modifications, and substitutions.

[0097] Figure 13 A simplified flowchart of a method for training an ANN for keypoint transformation according to an embodiment of the present invention is shown. (See reference...) Figure 4 As described, the keypoint descriptors used to train an ANN can be generated from the training data (images, motion blur, and keypoint neighborhoods) by the trained ANN 500. For example... Figure 13As shown, the corresponding generated keypoint descriptors provide the ground truth values ​​for the loss function during ANN training. It may be necessary to augment the synthetic training data with descriptors computed from a relatively small number of images with realistic motion blur and manually annotated.

[0098] Compared to calculating the distance between two keypoint descriptors, inference via a deep learning network typically involves more computation. Therefore, to save computation, it might be desirable to implement method 1100 or 1200 to compare the value of a first motion descriptor with the value of a second motion descriptor and avoid using a trained network (e.g., alternatively using a conventional scoring metric) when the first and second motion descriptors have similar values. For example, method 1100 or 1200 could be implemented by including a task of calculating the distance between the first and second motion descriptors and comparing that distance with a threshold. This implementation of method 1100 or 1200 could be configured to, in response to an indication from a motion descriptor comparison task that the motion blur of the first image is similar to that of the second image, use a scoring metric (e.g., distance as described above) instead of a trained network to determine whether a keypoint descriptor from the first image matches a keypoint descriptor from the second image.

[0099] Figure 14 A simplified block diagram of an apparatus according to an embodiment of the present invention is shown. As an example, Figure 14 The apparatus 1400 shown can be used to perform image processing on a mobile device (e.g., a cellular phone such as a smartphone or a head-mounted device) or other computing device or system according to a general configuration, which includes a keypoint descriptor generator 1410, a keypoint descriptor converter 1420, and a keypoint descriptor comparator 1430.

[0100] Keypoint descriptor generator 1410 includes an instance of a trained ANN 500 and is used to generate descriptors for keypoints in a first image and a second image, the first image being acquired by an image sensor during a first time period, and the second image being acquired by the image sensor during a second time period different from the first time period (e.g., as described herein with reference to tasks 1110 and 1120, respectively). Keypoint descriptor converter 1420 is used to convert the generated descriptors of keypoints in the first image into transformed descriptors based on the first and second motion descriptors, using the trained ANN, wherein the first motion descriptor describes the motion of the image sensor during the first time period, and the second motion descriptor describes the motion of the image sensor during the second time period (e.g., as described herein with reference to task 1130). Keypoint descriptor comparator 1430 is used to compare the transformed descriptors with the generated descriptors of keypoints in the second image (e.g., as described herein with reference to task 1140).

[0101] In one example, device 1400 is implemented within a device such as a mobile phone, which typically has a video camera for generating a frame sequence including a first image and a second image. The device may also include one or more motion sensors, which can be used to determine the device's 6DOF motion in space. In another example, device 1400 is implemented in a head-mounted device, such as AR glasses, which may also have motion sensors and one or more cameras.

[0102] In another embodiment of the invention, a trained keypoint descriptor comparison module 600 is used instead of the keypoint descriptor converter 1420 to perform feature matching. As described above, the keypoint descriptor comparison module 600 is used to determine whether two keypoint descriptors match, even if the features belong to two images with different image motions. Using the keypoint descriptor comparison module 600 is a more comprehensive way to address feature matching failures caused by motion blur. Figure 6 As shown, the keypoint descriptor comparison module 600 receives four inputs, including two keypoint descriptors f0 and f1, and motion descriptors M0 and M1 corresponding to the motion blur of the source images of f0 and f1. The output is a binary decision 610 (i.e., whether f0 and f1 match) and a value P 620 representing the confidence level of the binary decision of the output.

[0103] Compared to the keypoint descriptor converter 1420, the keypoint descriptor comparison module 600 encapsulates a score metric and tends to have higher matching accuracy. On the other hand, the comparison module 600 takes more inputs than the converter 1420 and typically includes a larger network. Therefore, this approach tends to consume more memory and computational resources. As mentioned above, it may be desirable to use a conventional score metric when keypoint descriptors f0 and f1 come from images with similar image motions to save computation.

[0104] The embodiments discussed herein can be implemented in various fields, including feature matching, such as image alignment (e.g., panoramic mosaic), 3D reconstruction (e.g., stereo vision), indexing, and content retrieval. Training images are not limited to images generated by visible light cameras (e.g., in RGB or other color spaces), but can also be images generated by cameras sensitive to non-visible light (e.g., infrared (IR), ultraviolet (UV)), images generated by structured light cameras, and / or images generated by image sensors other than cameras (e.g., imaging using radar, lidar, sonar, etc.). Furthermore, the embodiments described herein can be extended beyond motion blur to cover other factors that may distort keypoint descriptors, such as brightness variations.

[0105] Figure 15Examples of components of a computer system 1500 implemented according to methods described herein (e.g., method 100, method 1100, and / or method 1200) are shown. Although these components are shown as belonging to the same computer system 1500, the computer system 1500 may also be implemented such that the components are distributed (e.g., distributed across different servers, distributed across smartphones, and one or more network entities, etc.).

[0106] Computer system 1500 includes at least a processor 1502, memory 1504, storage device 1506, input / output (I / O) peripherals 1508, communication peripherals 1510, and interface bus 1512. Interface bus 1512 is used for communication, sending, and transmitting data, control, and commands between various components of computer system 1500. Memory 1504 and storage device 1506 include computer-readable storage media such as RAM, ROM, electrically erasable programmable read-only memory (EEPROM), hard disk drives, CD-ROMs, optical storage devices, magnetic storage devices, electronically non-volatile computer storage devices (e.g., flash memory), and other tangible storage media. Any such computer-readable storage medium can be used to store instructions or program code implementing aspects of this disclosure. Memory 1504 and storage device 1506 also include computer-readable signal media. Computer-readable signal media include propagated data signals containing computer-readable program code. Such propagated signals take any of a variety of forms, including but not limited to electromagnetic, optical, or any combination thereof. Computer-readable signal media includes any computer-readable medium that is not a computer-readable storage medium but can communicate, propagate or transmit programs for use in conjunction with computer system 1500.

[0107] In addition, memory 1504 includes an operating system, programs, and application programs. Processor 1502 is used to execute stored instructions and includes, for example, a logic processing unit, a microprocessor, a digital signal processor, and other processors. Memory 1504 and / or processor 1502 can be virtualized and can be hosted in another computer system, such as a cloud network or data center. I / O peripherals 1508 include user interfaces such as a keyboard, a screen (e.g., a touchscreen), a microphone, speakers, other input / output devices, and computing components such as a graphics processing unit, serial ports, parallel ports, a universal serial bus, and other input / output peripherals. I / O peripherals 1508 are connected to processor 1502 via any port coupled to interface bus 1512. Communication peripherals 1510 are used to facilitate communication between computer system 1500 and other computer systems via a communication network and include, for example, a network interface controller, a modem, wireless and wired interface cards, antennas, and other communication peripherals.

[0108] Although this subject matter has been described in detail with reference to specific embodiments thereof, it should be understood that changes, variations, and equivalents of these embodiments can be readily made by those skilled in the art upon acquiring an understanding of the foregoing. Therefore, it should be understood that this disclosure is presented for illustrative purposes rather than limiting, and does not exclude modifications, variations, and / or additions to the subject matter that will be obvious to those skilled in the art. In fact, the methods and systems described herein may be embodied in a variety of other forms. Furthermore, various omissions, substitutions, and changes may be made to the form of the methods and systems described herein without departing from the spirit of this disclosure. The appended claims and their equivalents are intended to cover such forms or modifications that fall within the scope and spirit of this disclosure.

[0109] Unless otherwise expressly stated, it should be understood that terms such as “processing,” “computing,” “determining,” and “identifying” used in the discussion of this specification refer to the actions or processes of a computer system (such as one or more computers or similar electronic computer systems or devices) that manipulate or convert data represented as physical electronic or magnetic quantities in the memory, registers, or other information storage, transmission, or display devices of a computing platform.

[0110] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computer system may include any suitable arrangement of components that provides a result conditioned on one or more inputs. Suitable computer systems include microprocessor-based multipurpose computer systems that access stored software that programs or configures the computer system from a general-purpose computing device to a dedicated computing device for implementing one or more embodiments of this subject matter. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software used for programming or configuring computing devices.

[0111] Embodiments of the methods disclosed herein can be executed in the operation of such a computer system. The order of the blocks presented in the above examples can be changed; for example, the boxes can be reordered, combined, and / or decomposed into sub-boxes. Some boxes or processes can be executed in parallel.

[0112] The terms “including,” “having,” etc., are synonyms and are used inclusively in an open-ended manner, not excluding other elements, features, actions, operations, etc. Furthermore, the term “or” is used in an inclusive (not exclusive) manner; for example, when used to connect lists of elements, the term “or” indicates one, some, or all of the elements in the list. The use of “for,” or “configured as,” herein is an open and inclusive language, not excluding devices used or configured to perform additional tasks or steps. The headings, lists, and numbering included herein are for illustrative purposes only and are not intended to be limiting.

[0113] The various features and processes described above can be used independently of each other or in combination in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Furthermore, certain method or process blocks may be omitted in some implementations. The methods and processes described herein are not limited to any particular order, and the blocks or states associated with them may be executed in other suitable orders. For example, the described blocks or states may be executed in an order different from that specifically disclosed, or multiple blocks or states may be combined in a single block or state. Example blocks or states may be executed sequentially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed examples. Similarly, the example systems and components described herein may be configured differently from those described. For example, elements may be added, removed, or rearranged compared to the disclosed examples.

[0114] Various elements of the embodiments of the devices or systems disclosed herein (e.g., device 900 or 1400, system 1500) can be implemented in any combination of hardware and software and / or firmware deemed suitable for the intended application. For example, these elements can be fabricated as electronic and / or optical devices, such as residing on the same chip or between two or more chips in a chipset. An example of such a device is a fixed or programmable array of logic elements, such as transistors or logic gates, and any of these elements can be implemented as one or more such arrays. Any two or more of these elements, or even all of them, can be implemented in the same one or more arrays. Such one or more arrays can be implemented within one or more chips (e.g., within a chipset comprising two or more chips). Such a device can also be implemented as including a memory for storing training images and / or neighborhood sets.

[0115] The processor or other device for processing disclosed herein can be manufactured as one or more electronic and / or optical devices, such as residing on the same chip or between two or more chips in a chipset. An example of such a device is a fixed or programmable array of logic elements, such as transistors or logic gates, and any of these elements can be implemented as one or more such arrays. Such one or more arrays can be implemented within one or more chips (e.g., within a chipset comprising two or more chips). Examples of such arrays include fixed or programmable arrays of logic elements, such as microprocessors, embedded processors, IP cores, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), application-specific standard products (ASSPs), and application-specific integrated circuits (ASICs). The processor or other device for processing disclosed herein can also be implemented as one or more computers (e.g., a machine comprising one or more arrays programmed to execute one or more sets of instructions or sequences of instructions) or other processors. The processor described herein can be used to perform tasks or other sets of instructions not directly related to the implementation of methods 100, 1100, or 1200 (or another method disclosed with reference to the operation of the apparatus or system described herein), such as tasks related to another operation of a device or system with an embedded processor (e.g., a voice communication device, such as a smartphone or smart speaker). A portion of the methods disclosed herein can also be executed under the control of one or more other processors.

[0116] Each task of the methods disclosed herein (e.g., methods 100, 1100, 1200) can be implemented directly in hardware, in a software module executed by a processor, or in a combination thereof. In typical applications of embodiments of the methods disclosed herein, an array of logic elements (e.g., logic gates) is used to perform one, more, or even all of the various tasks of the method. One or more (potentially all) tasks can also be implemented as code (e.g., a set or more sets of instructions) in a computer program product (e.g., one or more data storage media, such as disks, flash memory or other non-volatile memory cards, semiconductor memory chips, etc.), which can be read and / or executed by a machine (e.g., a computer) including an array of logic elements (e.g., a processor, microprocessor, microcontroller, or other finite state machine). Tasks of embodiments of the methods disclosed herein can also be performed by more than one such array or machine. In these or other embodiments, tasks can be performed within a device for wireless communication, such as a cellular phone or other device with such communication capabilities. Such a device can be used to communicate with circuit-switched and / or packet-switched networks (e.g., using one or more protocols such as VoIP). For example, such a device may include RF circuitry for receiving and / or transmitting encoded frames.

[0117] In one or more exemplary embodiments, the operations described herein may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, such operations may be stored as one or more instructions or code on or transmitted through a computer-readable medium. The term “computer-readable medium” includes computer-readable storage media and communication (e.g., transmission) media. By way of example and not limitation, a computer-readable storage medium may include an array of storage elements, such as semiconductor memory (which may include, but is not limited to, dynamic or static RAM, ROM, EEPROM, and / or flash RAM) or ferroelectric, magnetoresistive, elliptic, polymeric, or phase-change memory; CD-ROM or other optical disc storage; and / or magnetic disk storage or other magnetic storage devices. Such a storage medium may store information in the form of instructions or data structures that are accessible to a computer. A communication medium may include any medium that can be used to carry desired program code in the form of instructions or data structures and that is accessible to a computer, including any medium that facilitates the transfer of a computer program from one place to another. Furthermore, any connection is properly referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology such as infrared, radio, and / or microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology such as infrared, radio, and / or microwave are all included in the definition of media. As used herein, disks (Disk / disc) include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs. TM (Blu-Ray Disc Association, Universal City, Calif.), where disks typically reproduce data magnetically, while other disks use lasers to reproduce data optically. Combinations of these should also be included within the scope of computer-readable media.

[0118] In one example, a non-transitory computer-readable storage medium includes code that, when executed by at least one processor, causes the at least one processor to perform a method for generating keypoint descriptors robust to motion blur, such as method 100, 1100, or 1200 described herein. Other examples of such storage media include media that further include code that, when executed by at least one processor, causes the at least one processor to perform a method as described herein for generating keypoint descriptors robust to motion blur.

[0119] Unless expressly limited by its context, the term "signal" is used herein to indicate any of its general meanings, including the state of a storage location (or set of storage locations) expressed on a line, bus, or other transmission medium. Unless expressly limited by its context, the term "generate" is used herein to indicate any of its general meanings, such as calculation or otherwise generating. Unless expressly limited by its context, the term "calculate" is used herein to indicate any of its general meanings, such as calculating, evaluating, estimating, and / or selecting from multiple values. Unless expressly limited by its context, the term "acquire" is used to indicate any of its general meanings, such as calculating, deriving, receiving (e.g., from an external device), and / or retrieving (e.g., from a storage array element). Unless expressly limited by its context, the term "select" is used to indicate any of its general meanings, such as identifying, indicating, applying, and / or using at least one of two or more sets, but not all of them. Unless expressly limited by its context, the term "determine" is used to indicate any of its general meanings, such as deciding, establishing, summarizing, calculating, selecting, and / or evaluating. When the term “comprising” is used in this specification and claims, other elements or operations are not excluded. The term “based on” (e.g., “A is based on B”) is used to indicate any of its general meanings, including (i) “derived from” (e.g., “B is a precursor of A”), (ii) “based on at least” (e.g., “A is at least based on B”), and, if appropriate in a particular context, (iii) “equal to” (e.g., “A equals B”). Similarly, the word “responds” is used to indicate any of its general meanings, including “responding to at least”. Unless otherwise stated, the terms “at least one of A, B, and C”, “one or more of A, B, and C”, “at least one of A, B, and C”, and “between one or more A, B, and C” mean “A and / or B and / or C”. Unless otherwise stated, the terms “each of A, B, and C” and “each of A, B, and C” mean “A and B and C”.

[0120] Unless otherwise stated, any disclosure of operation of a device with specific characteristics is also expressly intended to disclose a method with similar characteristics (and vice versa), and any disclosure of operation of a device according to a specific configuration is also expressly intended to disclose a method according to a similar configuration (and vice versa). The term “configuration” may be used with reference to a method, apparatus, and / or system indicated by its specific context. Unless otherwise stated by the specific context, the terms “method,” “process,” “procedure,” and “technology” are generally used and interchangeably. A “task” having multiple subtasks is also a method. Unless otherwise stated by the specific context, the terms “apparatus” and “device” are also generally used and interchangeably. The terms “element” and “module” are generally used to refer to a part of a larger configuration. Unless expressly limited by its context, the term “system” is used herein to mean any of its general meanings, including “a set of elements that interact to serve a common purpose.”

[0121] Unless initially introduced by a definite article, ordinal terms used to modify claim elements (e.g., “first,” “second,” “third,” etc.) do not themselves indicate any priority or order of claim elements relative to another element, but merely distinguish claim elements from another claim element with the same name (but using ordinal terms). Unless explicitly limited by their context, each of the terms “a plurality” and “a set” is used herein to indicate an integer quantity greater than one.

[0122] The foregoing description is provided to enable those skilled in the art to make or use the disclosed embodiments. Various modifications to these embodiments will be apparent to those skilled in the art, and the principles defined herein can be applied to other embodiments without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the embodiments shown herein, but is to be given the widest possible scope consistent with the principles and novel features defined as in the appended claims.

Claims

1. An image processing method, characterized in that, The method includes: Multiple key points are selected from the image, wherein the multiple key points are detected by a key point detector; Apply each of the multiple motion blurs to the image to generate multiple blurred images; and Based on the neighborhood of key points in each of the plurality of blurred images, a key point descriptor for the image is generated; The step of generating a keypoint descriptor based on the neighborhood of keypoints in each of the plurality of blurred images includes: Based on the neighborhood of key points in each of the plurality of blurred images, an artificial neural network (ANN) is trained at each of the plurality of different scales to generate the key point descriptor; The criterion for training the ANN is to minimize the distance measurement between instances of the generated keypoint descriptors, wherein the instances correspond to the same keypoint in different blurred images among the plurality of blurred images.

2. The method according to claim 1, characterized in that, Each of the plurality of motion blurs has a corresponding trajectory in at least two spatial dimensions.

3. The method according to claim 1, characterized in that, Each of the plurality of motion blurs has six degrees of freedom of motion.

4. The method according to claim 1, characterized in that, The method further includes extracting the neighborhood of the key point from each of the plurality of blurred images at each of a plurality of different scales.

5. The method according to claim 4, characterized in that, The extraction includes applying at least one Gaussian blur to each of the plurality of blurred images.

6. The method according to claim 4, characterized in that, The method further includes downsampling each of the plurality of blurred images.

7. The method according to claim 1, characterized in that, The method further includes determining that the distance between two motion blurs among the plurality of motion blurs is not less than a threshold, wherein, based on the above determination, the training loss function is calculated using a trained second ANN.

8. The method according to any one of claims 1-7, characterized in that, The image includes a first image and a second image, and the generated keypoint descriptor includes a first keypoint descriptor of the first image and a second keypoint descriptor of the second image. The method further includes: Convert the first keypoint descriptor into a transformed descriptor; and The transformed descriptor is compared with the second keypoint descriptor for feature matching.

9. The method according to claim 8, characterized in that, The first image is acquired by an image sensor during a first time period, and the second image is acquired by the image sensor during a second time period different from the first time period. The step of converting the first keypoint descriptor into a transformed descriptor includes: Based on the first motion descriptor and the second motion descriptor, the descriptors generated for key points in the first image are converted into transformed descriptors using a trained third ANN. The first motion descriptor describes the motion of the image sensor during the first time period, and the second motion descriptor describes the motion of the image sensor during the second time period.

10. The method according to claim 9, characterized in that, The third ANN includes a convolutional neural network (CNN) and / or an autoencoder.

11. The method according to claim 9 or 10, characterized in that, The method further includes: The first motion descriptor and the second motion descriptor are normalized to have the same range as the first keypoint descriptor and the second keypoint descriptor.

12. The method according to any one of claims 8-11, characterized in that, The step of comparing the transformed descriptor with the second keypoint descriptor includes: Calculate the distance between the transformed descriptor and the second keypoint descriptor in the descriptor space, and compare the calculated distance with a threshold.

13. The method according to any one of claims 1-12, characterized in that, The selected image contains several key points, including: The multiple key points in the image are detected using a corner detector and / or a blob detector, wherein the corner detector and / or blob detector are configured to blur and resample the image with different blur widths and sampling rates to create a scale space and detect corners and / or blobs at different scales.

14. A computer system, characterized in that, include: One or more processors; as well as One or more memories are provided for storing computer-readable instructions that, when executed by the one or more processors, cause the computer system to perform the method according to any one of claims 1-13.

15. One or more non-transitory computer storage media, characterized in that, The system stores instructions that, when executed on a computer system, cause the computer system to perform the method according to any one of claims 1-13.

Images