Descriptor Generation System

Abstract

The present disclosure relates to sampling for feature detection. A computer-implemented method is provided for generating feature descriptors for locations in an image, thereby performing descriptor matching when analyzing the image, the method comprising: determining a sample set representing the location in the image by sampling scale-space data representing the image, the scale-space data comprising data representations of the image at multiple length scales; and generating the feature descriptors based on the determined sample set.

Description

Technical Field

[0001] The present disclosure relates to sampling data to generate feature descriptors for locations in an image for use in performing descriptor matching when analyzing the image. Background Art

[0002] In computer vision systems and image processing systems, it is useful to be able to detect and describe features such as local features in an image. A "feature" is a portion of the content of an image that can be used to track content in multiple images. A feature such as a local feature is a feature of a region of an image rather than a feature of the entire image, i.e., it relates to a portion of an image. It is useful that a feature is discriminable so that the feature can be matched between images with a certain degree of confidence. A feature may correspond to an object or a portion of an object in an image. The detection of features may be performed using algorithms known in computer vision systems. Feature detection algorithms typically compute an abstraction of image information for determining whether a particular type of feature (e.g., an edge or a corner) is present in a tested image region. A feature detection algorithm may output the location (e.g., pixel coordinates) of a portion of an image representing the tested feature. For example, a corner detection algorithm may output the pixel coordinates of an area in an image that represents a corner or that may represent a corner. Feature detection algorithms typically do not provide information about the nature of the detected feature. Detection of image features allows for the comparison of features between images, which allows for an understanding of how objects in a sequence of images may move, and / or how the camera viewpoint may change between sequences of images.

[0003] For example, one may consider a location or point in an image that corresponds to a feature and a location or point in another image that may correspond to the same feature. A small area of ​​the image around each location may be analyzed to see if the location corresponds to the same feature. This may be done by forming a descriptor that represents the image location under test, and therefore the feature at that location. The descriptor may be in the form of a descriptor vector that characterizes a particular feature. The descriptor for each location may be formed by extracting and processing samples from a small area around each location according to the descriptor pattern. The descriptors for different features in these images may then be compared to assess the likelihood that the location corresponds to the same feature.

[0004] Examples of algorithms that determine descriptors are the Scale Invariant Feature Transform (SIFT) algorithm and the Speeded Up Robust Features (SURF) algorithm. The Directed FAST [Fastened Segment Test Features] and Rotated BRIEF [Binary Robust Independent Elementary Features] (ORB) algorithms are alternatives to SIFT. Other algorithms that determine descriptors are the Binary Robust Invariant Scalable Keypoints (BRISK) algorithm and the Fast Retinal Keypoints (FREAK) algorithm. These algorithms differ, inter alia, with respect to the descriptor patterns used. Summary of the invention

[0005] This Summary is provided to introduce a selection of concepts that are further described below in the Detailed Description. This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0006] According to one aspect, there is provided a computer-implemented method for generating feature descriptors for locations in an image for use in performing descriptor matching when analyzing the image, the method comprising:

[0007] Determining a set of samples representing locations in the image by sampling scale-space data representing the image, the scale-space data comprising data representing the image at a plurality of length scales; and

[0008] A feature descriptor is generated according to the determined sample set.

[0009] The scale-space data representing the image may include pre-filtered representations of the image at multiple length scales. The pre-filtered representation of the image may have been filtered using a low-pass filter. The pre-filtered representation of the image may have been filtered using one or more of a Gaussian filter and a box filter.

[0010] The scale-space data representing the image may include an image pyramid.

[0011] The method may include sampling scale-space data representing the image using a descriptor pattern. The descriptor pattern may be one of or based on a BRISK, ORB or FREAK descriptor pattern.

[0012] The location in the image may be a point in the image. The location in the image may be a pixel location in the image. The location in the image may be a key point in the image.

[0013] Sampling scale-space data can include:

[0014] Analyzing portions of the scale-space data representing locations at different length scales to determine a likelihood metric for each analyzed portion, the likelihood metric representing the likelihood of the feature at the corresponding length scale

[0015] determining said portion of said spatial data that results in a turning point in the measure of likelihood;

[0016] Based on the determined portion, a length scale is determined at which the scale-space data is sampled to determine the sample set.

[0017] The relatively smaller scale space representation may be placed within the relatively larger scale space representation according to the identified length scale. The relatively smaller scale space representation may include a descriptor pyramid, and the relatively larger scale space representation may include an image pyramid. The method may include sampling the scale space data using the relatively smaller scale space representation at the identified length scale.

[0018] The method may include identifying a location in an image based on one or more location identifiers or detection algorithms. The method may include identifying at least one of a location and a length scale in scale-space data associated with the identified location in the image. Sampling the scale-space data may include sampling the scale-space data based on the identified length scale. Sampling the scale-space data may include sampling data at a level in the scale-space data associated with the identified length scale. Sampling the scale-space data may include interpolating between data at levels in the scale-space data associated with length scales above and below the identified length scale.

[0019] Scale-space data may include data filtered at different length scales.

[0020] The data in the scale-space data filtered at different length scales may correspond to filtered samples to be extracted with respect to different radial distances in the descriptor pattern from the center of the descriptor pattern.

[0021] The descriptor pattern may include at least one ring around a location in the image. The at least one ring may be one of a circle, a wavy circle, and a polygon. The descriptor pattern may include a plurality of rings. The rings in the plurality of rings may be concentric.

[0022] The determined set of samples may be stored in an array, and generating feature descriptors based on the determined set of samples may include forming a modified array.

[0023] The method may include determining a rotation metric for a location in an image, the rotation metric describing an angle between an orientation of the image and a characteristic direction of the image at the location; and generating a feature descriptor based on the determined rotation metric. Forming the modified array may include shifting elements of at least a portion of the array by a number of positions in a corresponding portion of the array, the number of positions depending on the determined rotation metric. Forming the modified array may include interpolating between two or more samples in the determined sample set. The two or more samples in the determined sample set may be obtained from a single ring of the descriptor pattern. Forming the modified array may include interpolating between two or more samples in the determined sample set along a portion of the shape of the ring to which the two or more samples correspond.

[0024] The two or more samples in the determined sample set may be obtained from a plurality of rings of the descriptor pattern. Two rings of the plurality of rings may be adjacent in the descriptor pattern. The two or more samples in the determined sample set may include N1 samples from a first ring and N2 samples from a second ring, where N1≤N2. The first ring may be radially inside the second ring.

[0025] The method may comprise discarding the modified array once the feature descriptor is generated.

[0026] According to another aspect, there is provided a descriptor generation system for generating feature descriptors for locations in an image for use in performing descriptor matching when analyzing the image, the descriptor generation system comprising:

[0027] A feature descriptor generator, the feature descriptor generator being configured to:

[0028] Determining a set of samples representing locations in the image by sampling scale-space data representing the image using a sampling unit, the scale-space data including data representing the image at a plurality of length scales; and

[0029] A feature descriptor is generated according to the determined sample set.

[0030] The scale-space data representing the image may include pre-filtered representations of the image at multiple length scales. The pre-filtered representation of the image may have been filtered using a low-pass filter. The pre-filtered representation of the image may have been filtered using one or more of a Gaussian filter and a box filter.

[0031] The scale-space data representing the image may include an image pyramid.

[0032] The feature descriptor generator may be configured to sample scale-space data representing the image using a descriptor pattern. The descriptor pattern may be one of or based on a BRISK, ORB or FREAK descriptor pattern.

[0033] The location in the image may be a point in the image. The location in the image may be a pixel location in the image. The location in the image may be a key point in the image.

[0034] The feature descriptor generator can be configured to sample the scale-space data in the following way:

[0035] analyzing portions of the scale-space data representing locations at different length scales to determine a likelihood metric for each analyzed portion, the likelihood metric representing the likelihood of the feature at the corresponding length scale;

[0036] determining a portion of the spatial data that results in a turning point in the measure of likelihood;

[0037] Based on the determined portion, a length scale is determined at which the scale-space data is sampled to determine the sample set.

[0038] The descriptor generation system may be configured to place the relatively smaller scale space representation within the relatively larger scale space representation according to the identified length scale. The relatively smaller scale space representation may include a descriptor pyramid, and the relatively larger scale space representation may include an image pyramid. The descriptor generation system may be configured to sample the scale space data using the relatively smaller scale space representation at the identified length scale.

[0039] The descriptor generation system may be configured to identify a location in an image based on one or more location identifiers or detection algorithms. The descriptor generation system may be configured to identify at least one of a location and a length scale in scale-space data associated with the location identified in the image. The feature descriptor generator may be configured to sample the scale-space data based on the identified length scale. The feature descriptor generator may be configured to sample data at a level in the scale-space data associated with the identified length scale. The feature descriptor generator may be configured to interpolate between data at levels in the scale-space data associated with length scales above and below the identified length scale.

[0040] The scale-space data may include data filtered at different length scales. The data in the scale-space data filtered at different length scales may correspond to filtered samples to be extracted with respect to different radial distances in the descriptor pattern from a center of the descriptor pattern.

[0041] The descriptor pattern may include at least one ring around a location in the image. The at least one ring may be one of a circle, a wavy circle, and a polygon. The descriptor pattern may include a plurality of rings. The rings in the plurality of rings may be concentric.

[0042] The descriptor generation system may be configured to store the determined sample set in an array, and the feature descriptor generator may be configured to generate the feature descriptor from the determined sample set by forming a modified array.

[0043] The descriptor generation system may be configured to determine a rotation metric of a location in an image, the rotation metric describing an angle between an orientation of the image and a characteristic direction of the image at the location; and to generate a feature descriptor based on the determined rotation metric. The descriptor generation system may be configured to form a modified array by shifting elements of at least a portion of an array by the number of positions in a corresponding portion of the array, the number of positions being determined based on the determined rotation metric. The descriptor generation system may be configured to form the modified array by interpolating between two or more samples in the determined sample set. The two or more samples in the determined sample set may be obtained from a single ring of the descriptor pattern.

[0044] The descriptor generation system may be configured to form the modified array by interpolating between two or more samples in the determined sample set along a portion of the shape of the ring to which the two or more samples correspond.

[0045] The two or more samples in the determined sample set may be obtained from a plurality of rings of the descriptor pattern. Two rings of the plurality of rings may be adjacent in the descriptor pattern. The two or more samples in the determined sample set may include N1 samples from a first ring and N2 samples from a second ring, where N1≤N2. The first ring is radially inside the second ring.

[0046] The descriptor generation system may be configured to discard the modified array once the feature descriptor is generated.

[0047] According to another aspect, there is provided a descriptor generation system configured to perform the method as described herein.

[0048] According to another aspect, there is provided a descriptor generation system as described herein, wherein the descriptor generation system is embodied in hardware on an integrated circuit.

[0049] According to another aspect, a method of manufacturing a descriptor generation system as described herein using an integrated circuit manufacturing system is provided.

[0050] According to another aspect, there is provided a method of manufacturing a descriptor generation system as described herein using an integrated circuit manufacturing system, the method comprising:

[0051] processing the computer-readable description of the graphics processing system using a layout processing system to generate a circuit layout description of an integrated circuit embodying the descriptor generation system; and

[0052] Using the integrated circuit generation system, a descriptor generation system based on the circuit layout description is manufactured.

[0053] According to another aspect, computer program code for performing a method as described herein is provided.

[0054] According to another aspect, a non-transitory computer-readable storage medium having computer-readable instructions stored thereon is provided, which, when executed at a computer system, causes the computer system to perform a method as described herein.

[0055] According to another aspect, an integrated circuit definition data set is provided which, when processed in an integrated circuit manufacturing system, configures the integrated circuit manufacturing system to manufacture a descriptor generation system as described herein.

[0056] According to another aspect, a non-transitory computer-readable storage medium having stored thereon a computer-readable description of a descriptor generation system as described herein is provided, which when processed in an integrated circuit manufacturing system causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the descriptor generation system.

[0057] According to another aspect, there is provided a computer readable storage medium having stored thereon a computer readable description of a descriptor generation system as described herein, which when processed in an integrated circuit manufacturing system causes the integrated circuit manufacturing system to:

[0058] processing the computer-readable description of the descriptor generation system using a layout processing system to generate a circuit layout description of an integrated circuit embodying the descriptor generation system; and

[0059] Using the integrated circuit generation system, a descriptor generation system based on the circuit layout description is manufactured.

[0060] According to another aspect, an integrated circuit manufacturing system is provided, which is configured to manufacture a descriptor generation system as described herein.

[0061] According to another aspect, there is provided an integrated circuit manufacturing system, comprising:

[0062] a non-transitory computer-readable storage medium having stored thereon a computer-readable description of a descriptor generation system as described herein;

[0063] a layout processing system configured to process the computer-readable description to generate a circuit layout description of an integrated circuit embodying the descriptor generation system; and

[0064] An integrated circuit generation system is configured to manufacture a descriptor generation system from a circuit layout description. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0066] Figure 1A An example of a feature descriptor pattern according to BRISK is shown;

[0067] Figure 1B An example of a feature descriptor pattern according to FREAK is shown;

[0068] Figure 2 A flow chart showing a process for generating feature descriptors;

[0069] Figure 3 Examples of feature descriptor patterns are shown;

[0070] Figure 4 Shows the use Figure 3 An example of a sample array obtained with the descriptor pattern shown;

[0071] Figure 5 A flow chart showing an exemplary process for generating feature descriptors is shown;

[0072] Fig. 6A A flow chart showing an exemplary process for generating a feature descriptor based on elements in a shift array;

[0073] Figure 6B A flow chart showing an exemplary process for generating feature descriptors based on interpolating between element values ​​in an array is shown;

[0074] Figure 7 An example of a ring shape is shown;

[0075] Fig. 8A An example of an image pyramid is shown schematically;

[0076] Figure 8B Examples of descriptor pyramids and representations of descriptor patterns are schematically shown;

[0077] Fig. 9 An example of a descriptor generating apparatus is schematically shown;

[0078] Fig.10 Another example of a descriptor generating apparatus is schematically shown; and

[0079] Fig.11 It is a schematic diagram of the integrated circuit manufacturing system. DETAILED DESCRIPTION

[0080] The following description is presented by way of example to enable those skilled in the art to make and use the invention. The invention is not limited to the embodiments described herein, and various modifications to the disclosed embodiments will be apparent to those skilled in the art. The embodiments are described by way of example only.

[0081] In computer vision and image processing techniques, an image or features of an image can be compared with features of other images or other images to try to identify matches between these images or features. This is useful in many applications, including but not limited to camera calibration, 3D reconstruction, visual SLAM (simultaneous localization and mapping), image registration and / or image stitching, video stabilization, and object detection / recognition and / or tracking. In object tracking, it is necessary to know the position of the same object in different images. For example, an object at a position (x, y) in one image may have moved to a position (x', y') in another image. This movement of the object in the reference frame of the image may be due to the movement of the object itself, the movement of the effective camera position and / or orientation when shooting or generating the image, or both. The image may include an image of a real scene, a computer-generated scene, or a combination of real and computer-generated scenes. The image may form a sequence of images, such as a video sequence.

[0082] In image reconstruction or stitching, it is desirable to understand how two or more images relate to each other. For example, if a camera viewpoint has moved from one image to the next, or if there are two cameras imaging the same scene (the two cameras are spaced apart from each other), there may be an overlapping region between the two images. Comparing features in the overlapping region enables the position and orientation of the two images relative to each other to be determined. A mapping operation from one image to another can be defined. One image can be aligned with another image. Therefore, it is useful to understand how the camera position or effective camera position changes between scenes or images. Changes in the effective camera position can include lateral translation of the camera, rotation of the camera, changes in camera zoom, perspective changes, etc.

[0083] In the initial stages of image processing, it is useful to determine a feature (or more generally features) in the image. As an example, there may be dozens or hundreds of features identified in an image.

[0084] Features can be structural features in the content of an image, such as points, edges, corners, ridges, blobs, etc. Features can be detected by performing processing on a region of the image. The region can be defined around a location of interest in the image (e.g., where the feature may appear). The process of detecting features can compute an extraction of image information, as will be explained below.

[0085] When detecting features, locations such as pixels that have been identified as being of interest may be investigated. In some examples, a subset of pixels of the entire image may be investigated. In other examples, all pixels in the image may be investigated.

[0086] Feature Descriptors

[0087] Features of interest as described above (which in some examples may include all pixels in an image) can be analyzed to determine an extracted representation or characterization of the feature or the area surrounding the feature. This representation of the feature can be referred to as a feature descriptor. The feature descriptor includes information related to the location / feature analyzed when generating the feature descriptor. The information can be arranged as a feature vector. That is, the feature descriptor can include a feature vector. Arranging the information in the form of a feature vector can facilitate simple comparisons between feature descriptors of different features, such as for feature matching.

[0088] Feature descriptors can be formed in one of several ways, and some examples are provided below.

[0089] Identify a location of interest. The location may be a point in an image, such as a pixel location in the image. The location of interest may be a key point in the image, i.e., a point of interest in the image. A key point or location of interest is a point or area in the image that has a particular characteristic or that stands out in the image. Key points may be identified based on one or more key point identification or detection algorithms. For example, key points or locations of interest may be defined as corners and / or edges in an image. Corner searches and edge searches may be performed to identify locations of interest in the image. Locations of interest may be defined in any suitable manner and searched accordingly.

[0090] Feature descriptors are typically formed by extracting and processing samples from an image based on a descriptor pattern. A descriptor pattern, or sampling pattern, is a pattern of points sampled around a location of interest. The descriptor pattern extends across a region. The region typically includes the location of interest and is typically centered around the location of interest. Figure 1A An example of a feature descriptor pattern used according to the BRISK (Binary Robust Invariant Scalable Keypoints) algorithm is given in . The dots represent sampling points. The circles represent the region around each sampling point that contributes to the sampled value at each corresponding sampling point. The region around each sampling point can be processed by applying a filter (e.g., a Gaussian filter or a box filter) to the pixel values ​​within the region in order to determine the sampled value of the sampling point. Thus, the value of each pixel in the region can contribute to the sampled value at the center point of the region. For example, this can smooth out noise in the pixel values ​​where the sampling point is located. For example, the size of the circle depends on the standard deviation of the filter applied at the corresponding sampling point.

[0091] As reference Figure 1A It can be seen that the central sampling point is surrounded by other sampling points arranged in a series of concentric rings. Figure 1A In the example, four sampling rings surround the central sampling point.

[0092] Figure 1B Another exemplary sampling pattern is shown. Figure 1BThe pattern used in the FREAK (Fast Retinal Keypoints) algorithm is shown in Figure 1B In , the dots represent sampling points, and the circles represent the corresponding regions around those points that contribute to the sampled value at each sampling point. Again, these regions can be processed by applying a filter (such as a Gaussian or box filter) to the pixel values ​​within each region in order to determine the sampled value of the sampling point. Figure 1B , the central sampling point is again surrounded by additional sampling points arranged in a series of concentric rings.

[0093] Once the values ​​for the sampling points are obtained (i.e., sampled) according to the sampling pattern, comparisons can be made between these values ​​in order to determine the feature descriptors of the sampling points. These comparisons are made according to a known comparison scheme. This is to ensure repeatability and consistency in the comparisons between the values ​​of each sampling point. In some examples, paired comparisons can be made. For example, the value of a point x in a list or array of sample values ​​for a sampling point can be compared with the value of a point y in the list or array. Comparisons can be performed between two or more sample values. Linear combinations of sample values ​​can be performed. More complex operations can be performed. The number of comparisons made can be greater than (or less than or equal to) the number of sampling points. For example, a sampling point can be compared with more than one other sampling point. The output values ​​can be stored in an array, such as a linear array. The number of elements that the array has will be the same as the number of comparisons that have been made. For example, there can be 512 comparisons, thereby generating a linear array of 512 elements. The values ​​in the array represent feature descriptors in the form of feature vectors.

[0094] In some examples, the comparison may be a threshold comparison. For example, it may be determined whether the value of point x is greater than or equal to the value of point y. If true, an output value of '1' may be generated, otherwise an output value of '0' may be generated. In the case where the comparison is a threshold comparison, as in the above example, a binary string may be output. For example, if 512 pairs of sampling points are selected, the result of the pairwise comparison will be a binary string 512 bits long. Other numbers of sampling point pairs may be selected for pairwise comparison as desired.

[0095] The feature descriptor need not be binary. For example, it can be a floating point vector.

[0096] As will be appreciated, the configuration of the sampling points is as follows Figure 1A and 1B The descriptor patterns shown vary from one another. The comparisons between the extracted sample values ​​may also vary. When different patterns are applied to the same location in an image, this will give different feature descriptors. Therefore, when comparing feature descriptors at different locations to determine a match at those locations, the same descriptor pattern should be used to generate those feature descriptors.

[0097] The sample set extracted from the image may include a number of samples corresponding to the number of sampling points in the feature descriptor pattern. Thus, the sample set includes samples corresponding to values ​​extracted from the descriptor pattern. The samples may correspond to intensity values. Other pixel attribute values ​​may be used.

[0098] The process of extracting samples from an image (e.g., forming a set of samples from which feature descriptors can be generated) can involve a relatively large amount of processing (i.e., is performed at a relatively large processing cost) and can involve reading image data from memory storing image values, which can introduce significant delays.

[0099] Feature descriptors (e.g., binary strings) can be compared at low or relatively low (e.g., compared to the sample extraction process) processing cost. Comparisons between feature descriptors can be performed by calculating an exclusive OR (XOR) between feature descriptors, by calculating the Euclidean distance between feature descriptors (where the feature descriptor vectors include floating point values), or by calculating the Hamming distance between feature descriptors. Such a comparison will give an indication of the similarity of the feature descriptors and, therefore, the similarity of the corresponding features.

[0100] Based on the similarity of the feature descriptors, it can be determined whether the measured locations correspond to each other.

[0101] In cases where the object has translated laterally between the two images (due to either or both of a change in the object's position in the scene and a change in the camera's position), i.e., in cases where there is no rotational or perspective change between the images, it will be sufficient to generate feature descriptors for each image at the same orientation and length scale.

[0102] However, where there may be a rotation between the images (e.g., because the camera viewpoint is rotated between the two images), which may be due to misalignment between the two cameras used to image the scene, then simply using a descriptor pattern as described above may not allow a match to be determined, even if the locations in the images do in fact correspond to the same object or feature. In general, it is desirable to take possible rotations into account, since the transformation between images is not typically known in advance. Therefore, the generation of feature descriptors should be rotation-invariant. In other words, the generation of feature descriptors should be such as to allow matches to be determined between identical (or sufficiently similar) image portions, even when those image portions are at arbitrary angles relative to each other.

[0103] This can be done by rotating the descriptor pattern and the image relative to each other. Feature descriptors can be generated under a number of such relative rotations, and the number of such relative rotations can depend on the implementation. For example, the descriptor pattern can be rotated and used on the image under the rotation. Different rotations of the descriptor pattern can be used on the image. A series of rotations can be implemented until the full 360-degree relative rotation of the pattern and the image. This helps to ensure the best matching results. The accuracy of feature matching will generally depend on the number of single feature descriptors generated for each position in the image, wherein, in the case of using a larger number of rotation segmentation to obtain feature descriptors, higher accuracy can be obtained. For example, in the case of the descriptor pattern being rotated in 10 degree increments and generating feature descriptors for each rotation, a total of 36 feature descriptors will be generated. In the case of the descriptor pattern being rotated in 5 degree increments and generating feature descriptors for each rotation, a total of 72 feature descriptors will be generated. This is computationally expensive because it involves determining many different feature descriptors, and it may not be possible to reliably find the best matching rotation because the true rotation may be different from any single rotation for which the feature descriptor is generated. Thus, this method is not usually used in practice.

[0104] An alternative (often followed by the method) is to first sample the image position to determine some measure of the rotation of the image position. For example, the rotation measure may be the angle between the orientation of the image and the characteristic direction of the image position. For example, the characteristic direction may be an image gradient. An image gradient may be a directional change in a feature of an image position, such as a pixel attribute such as intensity or color. For example, an image gradient is a direction in which the rate of change of an attribute at that image position is greatest. There may be several image gradients at a given position, such as an intensity gradient and a color gradient. Image gradients of different attributes may not point in the same direction. In this case, the dominant gradient direction at the position in the image may be used. The dominant gradient direction may be selected as the direction of the steepest gradient (i.e., the direction with the greatest rate of change) or the gradient direction of a selected characteristic (e.g., intensity). The gradient may be a local gradient of a region of the image under test. For example, when the sampling points are distributed across a region, the image gradient of region A may be determined. The determination of image gradients is within the purview of those skilled in the art of image processing and need not be discussed in detail here. The gradient determined for a particular region of an image is the same relative to the pixels of that region, regardless of the orientation of that region in the image. Thus, when the region of the image is rotated, for example between successively acquired images, the orientation of the image gradients relative to the image will differ. Therefore, the rotation metric will differ accordingly between these images. The orientation of the image is defined by the grid of pixels that form the image. Image gradients can be used to generate rotationally invariant feature descriptors, as will now be explained.

[0105] Once the image gradient (e.g., local image gradient) has been determined, the image can be rotated so as to orient the gradient in a known direction, such as along the positive y-axis. The image can be rotated in any convenient manner. For example, the image can be rotated by determining the new pixel coordinates (i.e., the rotated pixel position) of each pixel of the unrotated image in the region of the feature descriptor pattern. Rotating the image can include determining the new pixel value at the rotated pixel position. The feature descriptor pattern can then be applied at this new orientation of the image. Alternatively, and equivalently, the feature descriptor pattern can be rotated before being applied to the unrotated image, which may result in a more efficient implementation. The orientation of the image relative to the feature descriptor pattern ensures that: although the region of the image is initially oriented, the feature descriptor pattern can be applied in a consistent manner, i.e., the same orientation relative to the image content of the region of the image. This enables a more accurate comparison of the feature descriptors to be implemented without depending on the orientation of the image features in the image. In this way, the feature descriptors should not depend on the orientation of the image or the features in the image, so they can be considered rotation invariants.

[0106] The conventional method involves two sampling processes, as shown in reference Figure 2 Will be discussed. The first sampling process samples values ​​from a region of an image to determine a rotation metric for the region of the image. The second sampling process samples values ​​from a region of the image to determine a feature descriptor for the region. The two sampling processes do not need to be the same sampling pattern. In fact, the size of the image region sampled in each process does not need to be the same. The pattern used in the initial sampling process can be called a gradient sampling pattern because it allows the gradient of the sampled region to be determined.

[0107] refer to Figure 2 , a gradient sampling pattern is applied to a desired location in image 202. Based on the sampled values, a rotation metric (e.g., gradient) for an area near the location can be determined. For example, where the gradient sampling pattern is circular and centered at the desired location, the determined gradient will be relative to the area covered by the circular sampling pattern, which is centered at the desired location. A rotation is then performed by an angle that depends on the determined rotation metric 204 (e.g., by an angle determined as the rotation metric) so that the sampled image area is consistently oriented relative to the feature descriptor pattern. The descriptor pattern is then applied to the location in image 206. Samples are extracted from the image based on the descriptor pattern, and feature descriptors are generated based on, for example, pairwise comparisons of the extracted sample values, as discussed above.

[0108] The inventors have recognized that the above method is wasteful in terms of processing cost and the amount of data read from memory (which may be referred to herein as "memory bandwidth") because two sampling processes are required to generate a rotationally invariant feature descriptor. The set of samples extracted in the first sampling process is used to determine the rotation metric of the image region and then discarded. The inventors have recognized that the samples used to determine the rotation metric can also be used to determine the rotationally invariant feature descriptor. This saves the processing cost and memory bandwidth of the second sample extraction process, which in some cases can effectively halve the processing cost and memory bandwidth associated with feature descriptor generation. Therefore, the method described herein can significantly reduce the processing cost and memory bandwidth of the technique while maintaining a good level of accuracy. Because the processing cost and memory bandwidth associated with extracting samples are high compared to the processing cost and memory bandwidth of the rest of the feature descriptor generation process, a significant reduction in overall processing cost and memory bandwidth can be achieved. For example, the processing cost of sample extraction is higher than the processing cost of comparing feature descriptors. An initial sample extraction process is performed to allow the rotation metric to be determined. Subsequent generation of feature descriptors can be performed without performing additional sample extraction. This will be explained in detail below. Thus, processing costs associated with these additional sample extractions may be saved / avoided.

[0109] During the sample extraction process, a relatively large amount of data needs to be processed and / or transmitted, such as data that allows local area filtering to be performed on each sample point. For example, where the sample point is based on a filter covering, say, 15 pixel locations, a value such as a property value (e.g., an intensity value) for each of these 15 pixels will need to be processed and / or transmitted. Transmitting this data consumes system bandwidth. In the present technique, avoiding a subsequent sample extraction process can avoid bandwidth requirements associated with such a sample extraction process.

[0110] In the case described above where the feature descriptors are to be generated using the same set of samples extracted during the image orientation process, the rotation metrics can be generated by performing a sample extraction process. The feature descriptors can then be generated without performing a sample extraction process. Thus, the bandwidth savings can approach half of the bandwidth that would otherwise be required (since only one sample extraction process is performed instead of two). Similarly, the processing cost of generating the feature descriptors can be halved or reduced, for example, by about half.

[0111] In other examples, where feature descriptors for a particular feature are to be generated at different relative orientations of the image region and the descriptor pattern, the potential savings in processing costs and / or bandwidth requirements may be greater than the savings in the example given in the paragraph above. As a simple example, where ten feature descriptors are generated at successively rotated orientations, each relating to a particular feature, only one of the possible ten sample extractions may be required. This may result in a 90% reduction in processing and / or bandwidth (or an amount close to 90%, as in practice there may be some processing and bandwidth requirements, but at much lower levels). This may therefore speed up the processing of the feature descriptors and, therefore, the matching process overall.

[0112] Figure 3 A descriptor pattern that can be used in the present technology is shown. The present technology is not limited to this descriptor pattern, as will be explained below. However, the following discussion will provide explanations using this pattern as an example.

[0113] In this descriptor pattern, a central sampling point 301 (labeled '0') is surrounded by four rings 302, 306, 310, 314. The sampling points on the innermost ring (first ring 302) are labeled '1' to '10'; the sampling points on the next innermost ring (second ring 306) are labeled '11' to '24'; the sampling points on the next innermost ring (third ring 310) are labeled '25' to '39'; and the sampling points on the outermost ring (fourth ring 314) are labeled '40' to '59'. In this example, there are 60 sampling points in total. In other examples, the number of sampling points may be different. More or fewer sampling points may be used.

[0114] The value sampled at each sampling point may be the value of the pixel common to the sampling point, such as the value of a given or predetermined pixel attribute, such as the intensity or color value (e.g., RGB or YUV) or local gradient of the pixel. In the case where the sampling point is not aligned with a single pixel, the value sampled at the sampling point may be some combination of the values ​​of two or more adjacent pixels, such as an average or interpolation. Such combinations (including averages and interpolations) may be calculated using well-known methods.

[0115] exist Figure 3 In the example, the sampling points are shown as being co-located with the pixel positions (square grid). This is not the case. In a preferred example, the sampling points are allowed to fall between the pixel positions, for example so that they can be evenly distributed around the ring. For ease of understanding, in Figure 3 The sampling points are shown to be co-located with the pixel positions. Integer positions (e.g., positions aligned with pixel positions) may be faster and / or simpler to sample. However, integer sampling positions may also result in the image being sampled at non-integer pixel positions if the feature positions themselves are non-integer.

[0116] The value sampled at each sampling point may depend on the area surrounding the corresponding sampling point. In one example, Figure 3 As shown, filters can be applied at corresponding sampling points. The circles are centered on the pixels marked '10', '20', '31' and '42', i.e., circles 304, 308, 312 and 316 represent the range of the filters. As shown, these filters are circularly symmetrical filters, but this is not the case. The filter can include a low-pass filter. The filter can include a Gaussian filter. The size of circles 304, 308, 312 and 316 can represent the standard deviation of the Gaussian filter applied at the corresponding sampling point. In another example, the filter is a box filter. The box filter can provide an approximation of the Gaussian filter; using a box filter is faster than using a Gaussian filter. Therefore, using a box filter instead of a Gaussian filter can reduce processing costs and time. The filter used, whether a Gaussian filter, a box filter or other known filters, can be selected according to system requirements, such as speed, filter behavior and / or performance and / or bandwidth. Using a filter allows sampling to take into account the local image area of ​​the sampling point itself.

[0117] Figure 3 The range of filters shown in represents the filter applied to each sample point in the ring. Only one illustrative filter range is shown for each ring. In practice, each sample point in a given ring will be sampled using the same range of filters.

[0118] about Figure 3 The center sampling point in (i.e., the pixel marked '0') does not show a circle corresponding to the filter range. In this example, the descriptor pattern is centered on a given pixel, and the expected attribute value of that pixel is used as the sample value of the center sampling point. The center sampling point can be aligned with a single pixel, as shown in the figure. However, this is not necessary because the value sampled at the center sampling point can also be obtained by local averaging, interpolation, or filtering, as described above. The remaining sampling points are not necessarily aligned with the pixel position. Therefore, although other sampling points may also be pixel-aligned, this is not necessary.

[0119] Reference again Figure 3 , the filter ranges between each ring are different in size. The filter range 304 of the first ring 302 is the smallest. The filter range 308 of the second ring 306 is the second largest. The filter range 312 of the third ring 310 is second only to the filter range of the second ring. The filter range 316 of the fourth ring 314 (i.e., the outermost ring) is the largest. The filter ranges do not need to increase radially from the center of the descriptor pattern. The filter ranges of different rings may be the same. In some examples, the filter range of the radially inner ring may be larger than the filter range of the radially outer ring.

[0120] In the descriptor pattern configuration shown, the density of sampling points decreases on successive rings away from the center sampling location. This is because the circumference of the ring increases. In the configuration shown, although the number of sampling points increases for larger rings, the average density of points still decreases due to the size of the rings and the number of sampling points. This particular relationship between the size of the rings (or the number of rings extending outward from the center sampling point) and the density of sampling points on the rings does not always hold. In other examples, different relationships are possible. For example, the density of sampling points may remain the same or substantially the same on successive rings. The density of sampling points may also increase on successive rings, proceeding from the center outward.

[0121] exist Figure 3 In the example shown, it can be seen that the range of the filters 304, 308, 312, 316 used is large enough to cover the sample points on the same ring as the sample point under consideration (which are on either side of the sample point). For example, the filter range around the sample point marked as '10' on the first ring 302 covers the adjacent sample points on the ring: the sample points marked as '1' and '9'. Similarly, the filter range around the sample point marked as '31' on the third ring 310 covers the adjacent sample points on the ring: the sample points marked as '30' and '32'. The same is true for the sample points on the second ring 306 and the fourth ring 314.

[0122] In general, the range of the filter may be sufficient to cover at least one adjacent sampling point on the same ring. In some examples, the range of the filter is sufficient to cover two adjacent sampling points on the same ring, one on either side of the sampling point under consideration.

[0123] The range of the filter does not have to cover adjacent sample points. However, this is useful in cases where the filter ranges for adjacent sample points overlap each other. This would be the case where the filter range itself covers adjacent sample points, but there could also be cases where the filter range does not cover adjacent sample points. For example, where the filter extends more than half the distance between adjacent sample points, the filter would overlap.

[0124] The inventors have realised that this is beneficial because the overlap of the filter ranges means that the values ​​sampled in this way will vary smoothly between the sample points, i.e. the sample values ​​between adjacent sample points should vary continuously. Thus, the sample values ​​will vary smoothly or continuously between sample points along the same ring (i.e. in the circumferential direction) where the filter ranges of these sample points overlap each other.

[0125] In a similar manner, the inventors have realised that it is beneficial that the ranges of the filters also overlap each other between at least two rings. This means that the sampled values ​​will vary smoothly or continuously between the sampled points of these adjacent rings (i.e. radially). Figure 3As shown, it can be seen in the figure that the filter range 312 of the filter centered at the pixel position marked '31' will overlap with the filter range of the filter centered at the pixel position marked '16': compare the range of filter 312 with the range of filter 308.

[0126] Preferably, the range of the filter is sufficient to allow the filter to overlap with filters of adjacent sampling points along the same ring and between adjacent rings. Figure 3 306, where the filter range 308 of the filter centered at the pixel position labeled '20' on the second ring 306 covers adjacent sample positions on the same (second) ring (i.e., positions labeled '19' and '21') as well as adjacent sample positions on the radially inner ring (first ring 302: positions labeled '7' and '8') and adjacent sample positions on the radially outer ring (third ring 310: positions labeled '34' and '35'). Since this filter range overlaps with other sample points, it will overlap with the filter centered on those sample points. As described above, the filter range does not need to overlap the other sample points themselves in order to overlap the filters centered on those sample points. Therefore, filters smaller than those shown in the illustrated example can still achieve these benefits.

[0127] In the above reference Figure 2 In the case of an image region, if samples are initially extracted according to a gradient sampling pattern to determine a rotation metric for an image region, and samples are then extracted according to a descriptor pattern to determine a feature descriptor, then the gradient pattern and the descriptor pattern need not be identical. In the present technique, it is convenient (although not necessary) to use a descriptor sampling pattern to extract samples from an image if the extracted sample set is to be used to determine the rotation metric, and the same extracted sample set is to be used to determine the feature descriptors.

[0128] For example, Figure 3 As shown, samples can be extracted from an image using a descriptor pattern at an initial orientation relative to the image. The extracted samples form an initial sample set. The extracted samples or initial sample set characterizes the central position of the descriptor pattern at the initial orientation.

[0129] The initial sample set can be used to determine a rotation metric relative to the sampled region of the image. The rotation metric (e.g., the dominant gradient direction of the sampled region) can be determined in any convenient manner. The rotation metric can be an angle α relative to the orientation of the sampled image. The angle can be determined by a determined gradient, such as the dominant gradient direction of the sampled region. For example, the angle α can be determined relative to the positive y-axis of the image, such that in the case where the gradient points along the positive x-axis, the angle can be determined to be 90 degrees, and in the case where the gradient points along the negative x-axis, the angle can be determined to be 270 degrees, and so on.

[0130] In the example shown, there are 60 sampling points in the descriptor pattern. An array can be formed including the extracted samples. In this example, the array includes 60 elements. Figure 4 , the array can be a linear array consisting of elements sampled from the positions labeled '0' to '59'. Note that the array will not contain the numbers 0 to 59; Figure 4 The labels in the table show the correspondence between array elements and descriptor patterns. Array element '0' includes Figure 3 The sample value extracted from the position marked as '0' in , and similarly for the rest of the array elements. An array element may include a combination of values, such as more than one value. For example, such an example is that the extracted value is an RGB color value. An array element may include multiple values ​​in the RGB value, i.e., the values ​​of the red, green, and blue channels.

[0131] Please note, Figure 4 The linear array shown in is separated by five lines to show the correspondence with the central sampling position and the four rings. This can also be done by Figure 4 The number displayed in each array element is the same as Figure 3 This split of the arrays is for illustrative purposes only and does not imply that separate arrays are required.

[0132] As discussed herein, it is possible to generate rotationally invariant feature descriptors, i.e., feature descriptors that characterize locations in a consistent manner regardless of rotation of an image region, without sampling an image a second or subsequent times. Instead, feature descriptors can be generated based on an initial sample set or samples extracted at an initial orientation (e.g., using a descriptor pattern). In this way, the sample extraction process only needs to be performed once. This saves processing and saves memory bandwidth because subsequent extraction processes are not required. Feature descriptors are generated from the initial sample set to account for the angle α (rotation metric). Thus, feature descriptors can be generated so that the feature descriptors themselves are (at least approximately) the same as what would occur if a second sampling process was performed at a rotated orientation relative to the image.

[0133] Feature descriptors can be generated by sampling from an initial sample set, such as from an array storing the initial sample set, rather than from the image itself. At least two methods are possible. In one method, elements of the array storing the initial sample set can be shifted to generate a modified array (e.g., a shifted array). In another method, interpolation can be performed on the elements of the array to generate a modified array (e.g., an interpolated array). Combinations of these methods are possible. For example, elements of the modified array can be used in pairwise comparisons to generate feature descriptors. These methods are described in more detail below.

[0134] A modified array based on the feature descriptors can be generated by sampling from an array that includes the initial sample set. No additional filtering needs to be performed on the values ​​of this array (although it can be done). The array can be stored in system memory, such as locally to a feature descriptor processor (i.e., a processor configured to perform feature descriptor generation and / or matching). Such an array can be small (in the example shown, it only needs to store 60 elements) and therefore will not use much memory space. The memory usage of storing the array is likely not more important than the benefits including those discussed above.

[0135] Once a feature descriptor has been generated based on the modified array, it can be used in descriptor comparisons. The feature descriptors can be stored; intermediate values, such as the contents of the array, need not be stored. Discarding these intermediate values ​​can save memory space. In some examples, intermediate values ​​are not required to generate feature descriptors for additional features in the image. However, if desired, additional feature descriptors for the same feature locations in the image can be generated by further sampling from the array comprising the initial sample set, thereby forming additional modified arrays, and corresponding feature descriptors can be based on these additional modified arrays.

[0136] Now refer to Figure 5 A process for generating feature descriptors is described. The process optionally includes extracting an initial sample set 502 using a descriptor pattern at a location in an image. The initial sample set may be stored 504. The initial sample set may be stored in an array. A rotation metric for a sampled image region is determined in step 506. The rotation metric is determined based on the initial sample set (i.e., elements of the array). The rotation metric is typically the angle between a characteristic direction (e.g., a gradient) of the image region and the image orientation. Determining how the characteristic direction of the image region relates to a known direction (e.g., a positive y-axis) allows the descriptor of the image region to be rotationally invariant, i.e., the orientation can be determined and interpreted in a consistent manner during the descriptor generation process regardless of the original orientation of the region in the image. This allows for a more consistent (i.e., rotationally invariant) comparison of image regions. In some examples, a sample set representing an image location in an image is determined based on the rotation metric determined at 506 and the initial sample set 508. The initial sample set may form an array. Samples may be extracted from the array and processed (e.g., by performing interpolation) to form a modified array. The elements of the modified array may form a sample set that characterizes the image location in a rotationally invariant manner.The process generates a feature descriptor based on the determined sample set 510 (eg, based on the modified array).

[0137] Shift array elements

[0138] In one approach, elements of an array may be shifted (e.g., periodically) to generate a modified array based on which a feature descriptor may be generated, e.g., by the pairwise comparison method discussed herein. The shift of the elements may, for example, represent (or approximately represent) a relative rotation between the descriptor pattern and the image, e.g., a rotation corresponding to a rotation metric. The elements may be shifted along a portion of the array corresponding to the ring in which the elements are located. For example, the center sampling point will be the same for all rotations (because in the example shown, the descriptor pattern will be rotated around the center sampling point). Thus, the center sampling point will not shift. Thus, denoting the array comprising the initial set of sampling points as "initial [0:59]", and denoting the array representing the modification (or shift) of the relative rotation as "rotation [0:59]", it can be seen that:

[0139] Initial[0] = Rotation[0]

[0140] The remaining elements are shifted in a circular manner along their corresponding rings. Figure 4 In the example shown, the first ring 302 is represented by initial [1:10]. The second ring 306 is represented by initial [11:24]. The third ring 310 is represented by initial [25:39]. The fourth ring 314 is represented by initial [40:59]. The cyclic shift elements in these rings mean that, for a single shift (the rings are treated separately for illustration):

[0141] Ring 1: Rotation [1:10] = Initial [10, 1:9]

[0142] Ring 2: Rotation [11:24] = Initial [24, 11:23]

[0143] Ring 3: Rotation [25:39] = Initial [39, 25:38]

[0144] Ring 4: Rotation [40:59] = Initial [59, 40:58]

[0145] This means that the available rotations of the descriptor pattern can be quantized to generate a feature descriptor. Again, taking each ring individually, the first ring has 10 possible rotations, the second ring has 14 possible rotations, the third ring has 15 possible rotations, and the third ring has 20 possible rotations.

[0146] Therefore, a single shift in each ring corresponds to a different angular rotation of the descriptor pattern: Figure 3In the example of , each shift in the first, second, third, and fourth rings along an element corresponds to an approximate angular rotation of 36, 26, 24, and 18 degrees, respectively. The angular rotations are at least approximate because the sample points may not be exactly aligned along the circle, and the sample points along each ring may not be equiangularly spaced around the ring. In practice, it is desirable to effectively rotate each ring by the same angle. Shifting elements along the array (corresponding to the same angular rotation for each ring) will lead to more accurate results. However, the inventors have found that even in cases where different effective angular rotations are affected between the rings, the generated feature descriptors are still able to usefully determine feature matches in the image. Therefore, this method, which can be performed with reduced processor load, is found to be beneficial.

[0147] In one example, the number of sampling points in each ring can be an integer multiple of the number of sampling points in the radially inner ring. Thus, the number of sampling points in the second ring can be the same as the number of sampling points in the first ring, or can be twice the number of sampling points in the first ring, and so on. Therefore, a shift in the position of one sampling point in the innermost ring will correspond to a shift in the position of one or more entire sampling points in the outer rings. Therefore, for any number of shifts in the position of the sampling points in the first ring, the positions of the sampling points of the other rings that maintain the geometric pattern may shift accordingly, i.e., the pattern will rotate as a whole, rather than being distorted.

[0148] This example restricts the number of available rotations of the sample point positions in each ring, where one of these positions is in the initial orientation. This means that the possible rotations will be quantized accordingly. The desired rotation of the descriptor pattern is given by a rotation metric, for example determined from the initial samples. A rotation can be selected from the quantized set of possible rotations by selecting one of the possible rotations that is closest to the desired rotation. In case the number of possible rotations is different on different rings, a different rotation can be selected for each ring, so that each selected rotation is the rotation that is closest to the desired rotation.

[0149] In an alternative approach, the values ​​of possible sampling points in the outer ring that may lie between the sampling point positions in these rings may be obtained by averaging or interpolating between the values ​​at the sampling points (e.g., using linear interpolation or some higher order interpolation). This may relax the requirement on the number of sampling point positions in the outer ring while also avoiding the need to quantize the possible orientations of the descriptor pattern.

[0150] In another example, to avoid quantization of available rotations due to the number of sampling point positions in the innermost ring, values ​​at possible sampling points in one or more rings may be obtained by averaging or interpolating between values ​​at the sampling point positions.

[0151] To illustrate the above, consider an example descriptor pattern comprising two rings; the inner ring has 10 sample point locations, and the outer ring has 20 sample point locations, forming a 31-element array (Example_Array[0:30]), as shown below:

[0152] Center sampling point: Example_Array[0]

[0153] Inner loop: Example_Array[1:10]

[0154] Outer loop: Example_Array[11:30]

[0155] The modified arrays can be generated by sampling from Example_Array as follows. The first modified array Array1 (effectively rotated 36 degrees relative to the initial orientation) can be generated by shifting the elements corresponding to the inner ring by one position and the elements corresponding to the outer ring by two positions:

[0156] Center sampling point: Array1[0] = Example_Array[0]

[0157] Inner loop: Array1[1:10] = Example_Array[10, 1:9]

[0158] Outer loop: Array1[11:30] = Example_Array[29:30, 11:28]

[0159] A second modified array Array2 (effectively rotated 72 degrees relative to the initial orientation) can be generated by shifting the elements corresponding to the inner ring by two positions, and shifting the elements corresponding to the outer ring by four positions:

[0160] Center sampling point: Array2[0] = Example_Array[0]

[0161] Inner loop: Array2[1:10] = Example_Array[9:10, 1:8]

[0162] Outer loop: Array2[11:30] = Example_Array[27:30, 11:26]

[0163] Reference Fig. 6AA process for generating feature descriptors based on shifting elements in an array is described. An array shift is determined from a rotation metric (e.g., a rotation metric determined from an initial sample set) to shift elements in an array 602. The array includes an initial sample set. The array elements are then shifted according to the determined array shift 604 to form a modified array. Feature descriptors can be generated based on the modified array. It is not necessary to form a modified array. In some examples, the shifting of array elements can be used to determine paired (or other) comparisons between modified array elements, and feature descriptors are generated based on these comparisons.

[0164] The third modified array Array3 (rotated 18 degrees relative to the initial orientation) can be generated by averaging the values ​​of the elements corresponding to the inner ring at the given position and the adjacent positions (i.e., interpolating at the midpoints) and shifting the elements corresponding to the outer ring by one position, as shown below:

[0165] Center sampling point: Array3[0] = Example_Array[0]

[0166] Inner ring: Array3[1:10]=

[0167] Example_Array[1 / 2(10+1), 1 / 2(1+2), 1 / 2(2+3),…, 1 / 2(9+10)]

[0168] Outer loop: Array3[11:30] = Example_Array[30, 11:29]

[0169] The process of generating feature descriptors based on interpolation of elements in an array will refer to Figure 6B Described. The interpolation performed using the values ​​of the elements in the array is determined based on a rotation metric (e.g., a rotation metric determined based on an initial sample set) 610. The array contains the initial sample set. The determined interpolation is then performed using the values ​​of the elements in the array 612. The result of the interpolation can form a modified array. The modified array can be stored at least temporarily. It can be determined that an interpolation set is to be performed, wherein the interpolation set uses each value of the elements in the array. The determined interpolation set can be performed using each value of the elements in the array. Feature descriptors can be generated based on the modified array. For example, the interpolation of each ring (e.g., representing a rotation) can be performed as an operation set. When performing the operation set, all elements of the array can be used. The number of elements in the modified array may be the same as the number of elements in the array on which the modified array is based.

[0170] These simple examples are illustrative. Other rotations are possible, as will be appreciated, and different interpolation factors may be used to determine values ​​at any position between the sample positions of the initial sample set.

[0171] Reference again Figure 3 and Figure 4 , the extracted array has 60 elements, namely initial[0:59]. There are five conceptual sample sets in the array: involving the central sampling point (initial[0]), the first ring (initial[1:10]), the second ring (initial[11:24]), the third ring (initial[25:39]) and the fourth ring (initial[40:59]). Except for the central sampling point, each of the other four conceptual sets can be called a "vector circle". In order to estimate the value based on which the feature descriptor can be generated (generating an estimate for the second sampling extraction), the value can be at least one of shifting and interpolating around the circumference of each vector circle.

[0172] For example, using an initial sample set obtained by performing an initial sample extraction, a rotation metric can be determined. This angle (e.g., in degrees or radians) can be converted to another unit based on dividing the circumference of each ring into a number of circular portions (or "arcs"). The number of circular portions can depend on the number of elements in each vector circle. Thus, in the example shown, the first ring includes 10 circular portions, the second ring includes 14 circular portions, the third ring includes 15 circular portions, and the fourth ring includes 20 circular portions.

[0173] For each vector circle, the equivalent of an additional "sample extraction" (i.e., the value estimated from the initial sample set) is obtained by finding the number of positions along the array that are shifted by a circular portion. The number of positions that are shifted by a circular portion is a real number. Linear interpolation can be used between the values ​​of two adjacent portions to approximate the expected value at the expected sample point between two adjacent sample points of the initial sample point set.

[0174] For example, refer to Figure 3 , the angular spacing between sample points in the first ring is 36 degrees (360 / 10) on average. In order to generate a modified array with an effective angular rotation of α degrees (e.g., 15 degrees), interpolation can be performed between adjacent sample points in each vector circle. For example, taking the rotated sample position corresponding to a 15 degree clockwise rotation from the initial [8], it can be seen that this will be an interpolation between the values ​​of the initial [8] and the initial [9]. The rotated sample position corresponding to a 15 degree clockwise rotation from the initial

[21] will be an interpolation between the values ​​of the initial

[21] and the initial

[22] . The rotated sample position corresponding to a 15 degree clockwise rotation from the initial

[36] will be an interpolation between the values ​​of the initial

[36] and the initial

[37] . The rotated sample position corresponding to a 15 degree clockwise rotation from the initial

[54] will be an interpolation between the values ​​of the initial

[54] and the initial

[55] .

[0175] As will be understood, reference Figure 4, arrow 402 represents the radial distance from the center of the descriptor pattern. Arrow 404 represents the angular distance along the ring.

[0176] The number of elements along each vector circle (to shift array elements along the vector circle) need not be the same for each ring. The number of elements to shift array elements may be determined for each ring based on the measure of rotation. The number of elements to shift array elements may be the number of elements corresponding to the angular rotation that is closest to the rotation measure. For example, in Figure 3 In the example of , a single element shift in the first ring corresponds to an angular rotation of approximately 36 degrees. Similarly, a single element shift in the second, third, and fourth rings corresponds to an angular rotation of approximately 26, 24, and 18 degrees. In the case where the rotation metric is determined to be a 40-degree angle, the number of elements closest to 40 degrees along each ring can be selected. This number can be selected for each ring independently of the selection of each other ring. In this example, the array elements can be shifted by one element in the first ring (corresponding to 36 degrees), two elements in the second ring (corresponding to 52 degrees, which is closer to 40 degrees than shifting one element, corresponding to 26 degrees), two elements in the third ring (corresponding to 48 degrees), and two elements in the fourth ring (corresponding to 36 degrees). The rings of the descriptor pattern can be considered to be "snapped" to the nearest angular rotation corresponding to the rotation metric.

[0177] Interpolate array elements

[0178] The following discussion will further consider interpolation. In one approach, the values ​​between the sample points in the initial sample set can be approximated on a linear segment, or on a line of a loop, connecting two adjacent sample points in the descriptor pattern. In this case, linear interpolation can be performed between two adjacent sample values ​​corresponding to the two adjacent sample points in the initial sample set.

[0179] In another approach, a higher order interpolation may be performed, such as quadratic or cubic interpolation. More generally, polynomial interpolation may be performed between two or more sample values ​​in the initial sample set to generate elements in the modified array for generating feature descriptors.

[0180] Since the number of samples considered will increase as the order of the polynomial increases, a higher order polynomial may result in an interpolation result that is closer to the actual value that the descriptor pattern seeks to sample. In this way, using a higher order polynomial may result in a more accurate interpolation result. Using a higher order interpolation may also increase the computational effort required. This may increase the size (e.g., silicon area) of the hardware used to perform the interpolation, or may increase power consumption or processing time. The order of the polynomial used in the polynomial interpolation may be selected based on at least one of the speed of obtaining the interpolation result, the size of the hardware used to perform the interpolation, the power consumption of the hardware, and the accuracy (or relative accuracy) of the interpolation result, for example based on a trade-off between desired speed and accuracy.

[0181] Any combination of methods may be used. For example, for rings with a higher circumferential point density (i.e., where the points are separated by relatively low angular spacing), interpolation may be performed along linear segments; for rings with a lower circumferential point density (i.e., where the points are separated by relatively high angular spacing), a higher order interpolation may be performed. This approach is useful because, in cases where the points are separated by low angular spacing, linear segments between adjacent points may be sufficient to approximate the image values ​​between the points, whereas in cases where the angular spacing is larger, a more expensive higher order interpolation may be required.

[0182] As described above, such interpolation between samples of the initial sample set is useful because the initial sample set can be generated by performing sample extraction using filtering of pixel values ​​near the sampling point, for example using a filter function that overlaps in a circumferential direction, such as an overlapping Gaussian filter. Therefore, if the position of the sampling point is moved, the extracted sample values ​​will also change smoothly. Therefore, it is appropriate to interpolate between these smoothly changing values ​​to obtain a modified array value for generating a feature descriptor.

[0183] In the above discussion, the values ​​of the modified array are generated by interpolating between samples of the same ring in the initial sample set. This means that the interpolation between samples is the same radial distance from the center sampling point. Interpolation can also be performed between rings. This means that the interpolation between samples is different radial distances from the center sampling point. Interpolation between rings can be between at least two rings. At least two rings are adjacent in the radial direction. Again, this is useful because the initial sample set is obtained by performing sample extraction using filtering of pixel values ​​near the sampling point, where the filtering includes filters that overlap in the radial direction.

[0184] The filter ranges do not have to overlap in both radial and circumferential directions. In the examples described herein, the filter ranges overlap in at least the circumferential direction.

[0185] In some examples, interpolation between rings can be performed between a sampling point on one ring and a sampling point on an adjacent ring. The two sampling points may be at the same (or similar) circumferential position, i.e., at the same angle with respect to the pattern.

[0186] In other examples, more than two sampling points can be considered. For example, interpolation can be performed between a sampling point on one ring (e.g., at an angular position of 5 degrees) and two sampling points on another ring (e.g., at angular positions of 0 degrees and 10 degrees). Interpolation between rings can include interpolating between a number N1 of sampling points on one ring and a number N2 of sampling points on another (radially outer) ring. One ring and the other ring can be adjacent in the radial direction. N1 and N2 can be the same number, or they can be different numbers. In some examples, N1 < N2. The angular range of sampling points on one ring and the angular range of sampling points on the other ring can be centered at the same (or similar) angular position.

[0187] For example, in the case where the rings are not true circles, performing interpolation between the rings is useful. In Figure 3 the example shown, the rings are not true circles; instead, the samples are aligned with pixel positions. Thus, the radial distances of adjacent samples within a ring from the central sampling point are different. This variation can be determined, and interpolation is performed between the rings to account for this radial variation. Thereby, the accuracy of sample estimation is improved.

[0188] Ring shape

[0189] In the examples described above, the rings are circular. However, the rings do not need to be circular. Any suitable shape can be used. For example, a polygonal ring shape can be used. In the case where the sampling points are taken from the vertices of the polygon, this will effectively be the same as a circular sampling ring (since the vertices are all on a circle). In the case where additional sampling points are taken on the polygon between the vertices, there will be a difference. In this case, the radial distance of the sampling points from the central sampling point will vary between a maximum value at each vertex and a minimum value at a midpoint between adjacent vertices.

[0190] Other ring shapes are possible. For example, a wavy or undulating ring can be used, as Figure 7 shown. The undulating ring can be rotationally symmetric. For example, Figure 7 the undulating ring shown has 18-fold rotational symmetry. By way of example, the undulating ring can have a rotational symmetry of at least 10-fold, or at least 20-fold, or at least 30-fold. A higher degree of rotational symmetry means that the undulating ring can cover an annular region with a sufficient density of sampling points. This can improve the accuracy of performing interpolation between sampling points.

[0191] In such a wavy ring, or other ring shapes that deviate from a circle, consecutive sampling points may be located at different radial distances. Using rings that include points at different radial distances allows the rings to be spaced farther apart from each other in the radial direction while still maintaining a good radial density of sampling points. For a given radial spacing of the rings, this may increase the likelihood that the radial interpolation result more closely follows the underlying value sampled at the sampling point, i.e., the radial interpolation accuracy. Thus, accuracy may be maintained while requiring fewer sampling points overall. This may result in increased speed. In some examples, multiple rings may be used to increase the density of sampling points.

[0192] The rings do not have to all be the same shape. Combinations of ring shapes are possible. The ring shapes may be used in any suitable configuration.

[0193] Generating feature descriptors from the sample set, for example by pairwise comparison, can be performed based on the descriptor pattern used, for example based on at least one of the number, shape and spacing of rings in the descriptor pattern. Depending on the descriptor pattern, the pairwise comparisons performed can be selected.

[0194] The sampling points may be equally spaced along each ring. This is not the case. The sampling points may be equally spaced along at least one ring, but not equally spaced along at least one other ring. It may be the case that the sampling points are not equally spaced along any ring.

[0195] Initial sample extraction

[0196] In the above discussion, an initial sample set is formed or obtained by sampling from an image. The pixel attribute values ​​are filtered so that if the sampling position is moved, the values ​​of the initial sample set will change smoothly.

[0197] Initial sample extraction does not need to be performed on the image. Initial sample extraction can be performed on other data (e.g. pre-filtered data). For example, already filtered values ​​can be sampled, rather than an initial sampling process that requires filtering multiple values. This reduces the amount of processing required when the descriptor generation process is running. This can be achieved by frontloading the computational work involved in filtering the image into a stage that is processed before generating the feature descriptors. This can result in a speed increase when generating the feature descriptors later.

[0198] In computer vision, when analyzing a scene, it is often not known at the outset what length scale is appropriate for describing the content of the image. For example, whether a particular feature might be expected to appear in a 4x4 pixel region (i.e., at a relatively small length scale) or in a 32x32 pixel region (i.e., at a relatively large length scale). For example, the length scale at which a feature of interest appears is often unknown. Furthermore, objects often consist of different structures at different length scales. Therefore, it is appropriate to consider features at different length scales.

[0199] An image may be represented by data that considers a plurality of different length scales. The data representing the image may include multi-scale data representing the image. The data representing the image may include scale-space data. Scale-space data is a data set at different length scales, wherein structures are successively suppressed or smoothed at progressively larger length scales. Data / structures at coarser (i.e., longer) length scales in the scale-space data may be obtained from corresponding data / structures at finer (i.e., shorter) length scales.

[0200] For example, consider a three-level dataset, where each pixel in the levels above the first level is generated by averaging the pixels in a 2x2 pixel region in the previous level. Each pixel in the third level corresponds to a region of 4 pixels in the second level and a region of 16 pixels in the first level. The structure in the 2x2 pixel region in the first level will be smoothed by the third level (the pixel values ​​in the 2x2 pixel region in the first level only account for a fraction of the pixel values ​​in the third level).

[0201] Scale-space data may include structures or features represented at length scales between a minimum length scale and a maximum length scale. Scale-space data may be obtained by performing filtering at different length scales, i.e., at a maximum length scale, at a minimum length scale, and optionally at one or more length scales between the maximum and minimum length scales.

[0202] Filtering at different length scales corresponds to sample extraction with respect to rings of different sizes. As discussed above, sampling on the radially inner rings may include filtering the pixel values ​​around the sampling point using a relatively smaller filter size. Sampling on the radially outer rings may include filtering the pixel values ​​around the sampling point using a relatively larger filter size. The smaller filter size corresponds to filtering at finer details, i.e., a smaller length scale. The larger filter size corresponds to filtering at coarser details, i.e., a larger length scale.

[0203] An example of a scale space representation of data is a pyramid, such as an image pyramid. In a pyramid, a signal or image is smoothed and subsampled multiple times. In one example, a pyramid is formed by smoothing an image with a smoothing filter. The smoothed image is usually subsampled. Subsampling can be performed along each coordinate direction. In other examples, the data of one level in the pyramid can be formed from the data of another level in a single step process. The process is repeated on the resulting image. This process can be repeated many times. Each time the process is performed, the resulting image is smaller than the previous image. The spatial sampling density of subsequent images is reduced, that is, the image resolution is reduced.

[0204] An example of an image pyramid 800 is shown in Fig. 8AThe first level (level 0) is located at the top of the pyramid (i.e., the widest part, as oriented in the figure). Note that Fig. 8A shows a 2D projection of a 3D pyramid. Each level of the pyramid represents an image, such as a 2D image; conceptually, Fig. 8A The horizontal lines shown at each level in the pyramid can be thought of as showing an "end view" of the 2D image. The first level (level 0) represents the original image without any smoothing or subsampling. Successively lower levels (i.e., levels 1 to 5) represent successively smoothed versions of the original image, with the resolution decreasing at each level down the pyramid. Such a pyramid can be formed in one of several known ways. For example, the pyramid can be formed by performing low-pass or band-pass filtering between levels. An example of a pyramid is a Gaussian pyramid. Another example of a pyramid is a Laplacian pyramid. Box filtering can also be used to generate a pyramid. The type of filter used will depend on the speed, filter behavior, and / or desired performance.

[0205] A pyramid consists of discrete levels of data that represent discrete length scales, rather than a continuum. Therefore, a pyramid is necessarily an approximation of the "full" scale spatial data representing all length scales. Data associated with length scales between those of the discrete levels of the pyramid can be obtained from the pyramid levels, such as in trilinear interpolation. A pyramid is a computationally efficient approximation of the full scale spatial representation. Therefore, using a pyramid can reduce processing cost and time.

[0206] The following discussion will be in the context of pyramids for ease of reference, but it will generally be understood that scale-space data representations may be used where appropriate.

[0207] The pyramid 800 may be pre-computed for the entire image. This allows sampling from the pyramid when it is desired to sample any particular point at a given length scale. A single sample point in the descriptor pattern may correspond to a point in the pyramid. Sample points with relatively large filter sizes will correspond to points at lower levels in the pyramid (i.e., at Fig. 8A The orientation of is towards the lower end of the pyramid, i.e., the numerically higher levels. For example, level 5 is a lower level than level 1), rather than the sampling points with relatively smaller filter sizes. Using a scale-space representation allows for a fast implementation when generating feature descriptors. Precomputing the pyramid for the entire image incurs some processing cost. Higher-level pyramids (i.e., at Fig. 8ATowards the lower end of the pyramid (i.e., numerically lower levels) (or more generally, levels with higher resolution scale-space data) in the orientation of the image pyramid, one might expect to be sampled less frequently than the lower levels (i.e., levels with lower resolution). This is because the lower levels of the pyramid correspond to sampling of positions in the outer rings of the feature descriptor pattern, where larger filters are used. Typically there are more sampled positions in the outer rings than in the inner rings. In the center of the feature descriptor pattern, only a single center sample position (301) is not filtered and will therefore be read from the highest resolution level of the image pyramid. In some cases, data from the higher levels may be rarely or not reused. However, it is likely that data will be reused at the lower levels. Therefore, pre-computing the pyramid for the entire image, that is, without knowing what length scales will be needed, can still result in an overall saving in computation. One reason for this is that in any case, the higher levels need to be computed to compute the lower levels. Using a pre-computed pyramid to downsample the data at the lower levels avoids the need to compute data at these length scales every time.

[0208] When it is desired to sample from a pyramid at a given length scale, the data in the pyramid corresponding to that length scale may be sampled. For example, where it is desired to sample from a length scale corresponding to level 2 in the pyramid, the data in the pyramid corresponding to level 2 may be sampled directly. Sampling may also be performed between discrete levels in the pyramid. For example, if it is desired to sample from a length scale between the length scales of levels 2 and 3 of the pyramid, sampling may be performed based on data at levels 2 and 3. For example, filtering (e.g., trilinear filtering) may be performed on data points at levels 2 and 3, as will be appreciated by the skilled person. Such trilinear filtering adds negligible additional processing cost to the processing, since most of the filtering work is done in advance to obtain the image pyramid itself.

[0209] The length scale for sampling from the scale space data or pyramid diagram can be selected based on the filter range (i.e., the size of the filter applied around the sampling point). If the center sampling point is sampled directly, i.e., only one pixel value is sampled, this may correspond to an unfiltered pixel, such as level 0 in the pyramid. Sampling with a filter that covers more pixels will correspond to filtered pixels, i.e., lower levels in the pyramid (levels 1, 2, 3, etc.). In the case where the filter range covers, for example, 4 pixels in one direction, the filter may correspond to level 2 in the pyramid. In the case where another filter range covers, for example, 8 pixels in one direction, the filter may correspond to level 3 in the pyramid. In an illustrative example, sampling of points located on the first ring may include sampling from a higher level in the pyramid (e.g., level 1); sampling of points located on the radially outer ring (e.g., the third ring) may include sampling from lower levels in the pyramid (e.g., Fig. 8AIn the example shown in , the filter range is sampled at one level from level 2 to level 5). This is the case where the filter range increases as the radial distance of the ring from the center increases. This need not be the case in all examples.

[0210] like Fig. 8A As shown, in pyramid 800, the horizontal position at which a value is sampled will depend on the desired position in the image at a given length scale, and the vertical position at which a value is sampled will depend on the desired length scale. In practice, since the pyramid may be a 3D pyramid, the "horizontal" position is a position in the horizontal plane.

[0211] In a similar way to viewing a scale-space representation of data as an image pyramid, the feature descriptor patterns used to sample from the pyramid can be viewed as a "descriptor pyramid". Figure 8B An example of a descriptor pyramid 810 is given in the left half of FIG. Although the levels of the descriptor pyramid may correspond to the levels in the image pyramid (the vertical spacing between two levels may be half of the previous vertical spacing), this is not necessarily the case. Figure 8B In the example shown on the left, these levels are irregularly spaced. The spacing of the levels depends on the range of the filter used in the descriptor pattern. Figure 8B The right half of shows a schematic example of a descriptor pattern 820 (with the rings shown as dashed lines), and illustrative filter ranges at each ring (solid circles). The sampling points are not shown. The correspondence between the filter ranges and the levels in the descriptor pyramid is shown by dashed arrows 830. In general, the filter sizes for successive rings in the descriptor pattern need not increase by a factor of 2 each time. Thus, the levels in the descriptor pyramid may correspond to fractional levels in the image pyramid. The order of the rings (moving radially outward from the center of the descriptor pattern) need not correspond to the order of the levels in the descriptor pyramid. The order of the rings will correspond to the order of the levels in the pyramid, with the filter ranges at radially outer rings becoming larger. In some examples, filters at sampling points on different rings may have the same range. In some examples, the range of the filter at sampling points on radially outer rings may be smaller than the range of the filter at sampling points on radially inner rings.

[0212] For a circular sampling pattern, Figure 3 and Figure 8B An example of which is shown in the right half of , the descriptor pyramid may be referred to as a “descriptor cone.” The term “descriptor pyramid” is used herein, but it will be understood that, at least conceptually, a descriptor pyramid may represent other shapes in three dimensions.

[0213] The descriptor pyramid can be conceptually placed within the image pyramid to indicate the area of ​​the image pyramid that corresponds to the footprint of the feature descriptor pattern and from which the initial sample set is obtained. As the data in the image pyramid is sampled to obtain the initial sample set, the descriptor pyramid can be thought of as moving around the image pyramid. This is indicated by the dashed and dotted lines representing different descriptor pyramids within the image pyramid (the use of dashed and dotted lines helps Fig. 8A ) is schematically shown for clarity in the image pyramid. As shown, the descriptor pyramid will appear larger at higher levels of the image pyramid. This is because the length scale (corresponding to the filter size) between the top and bottom of the descriptor pyramid changes relative to each other. Regardless of the position of the descriptor pyramid within the image pyramid, the number of levels in the image pyramid that it covers may be the same.

[0214] Sampling at the finest level from the pyramid involves effectively placing the descriptor pyramids so that the highest level of each pyramid is aligned, e.g. Fig. 8A Horizontally moving the descriptor pyramid (or strictly speaking, moving the 3D representation of the descriptor pyramid in the horizontal plane of the 3D representation of the image pyramid) will change the location in the image where the descriptor pattern is applied, i.e., the center location of the descriptor pattern. It is not necessary to place the descriptor pyramid so that the highest level (or indeed any level) of the descriptor pyramid is aligned with the highest level (or any other level) of the image pyramid. This is shown as 806, which shows the descriptor pyramid being placed so that sampling is performed at a larger length scale from the scale space data (for clarity, the descriptor pyramid is shown here with dashed lines rather than dotted lines). Another exemplary position of the descriptor pyramid is shown at 808, representing sampling at an even larger length scale. In some examples, in order to magnify by a factor of 2 (i.e., to analyze scale space data at half the resolution), the descriptor pyramid can be moved down one level relative to the image pyramid (where each pixel at one level of the pyramid is obtained from a 2x2 pixel block at the next highest level).

[0215] The horizontal extent of the image pyramid represents the size of the image. The horizontal extent of the descriptor pyramid, at its widest part, represents the size (e.g., area or range) of the descriptor pattern in the image (i.e., the outer ring of sample points in the descriptor pattern, not the entire extent of the Gaussian filter ring corresponding to the sample points in the outer ring). At its narrowest part (i.e., the bottom point as shown), the horizontal extent of the descriptor pyramid represents the filtering hierarchy, where one sample represents the entire area.

[0216] Thus, moving the descriptor pyramid over the image pyramid (corresponding to sampling from locations across the image) can be used to search for image feature matches at a given length scale. This is useful, for example, when an object moves past the camera. Moving the descriptor pyramid downward relative to the image pyramid (corresponding to sampling at a larger length scale) can be used to search for image feature matches, such as at a given location in the image, at a larger length scale. This is useful, for example, when an object moves directly away from the camera. Typically, a combination of horizontal and vertical movements of the descriptor pyramid relative to the image pyramid will be performed. This combination of horizontal and vertical movements allows both changes in position and changes in length scale to be considered when performing feature matching. The feature matching process can start from the highest level resolution of the scale space data (i.e., by placing the descriptor pyramid so that its highest level is aligned with level 0 of the image pyramid), but this is not necessarily the case.

[0217] This approach allows the same descriptor pattern and size to be used when sampling the data, regardless of the length scale at which the data is sampled. This allows consistent sampling to be performed. Greater sampling consistency can allow for more accurate comparisons of the resulting feature descriptors.

[0218] When searching for an object in an image, the position of the descriptor pyramid within the image pyramid can be selected based on motion estimation, for example by estimating the possible motion of the object from known positions in different images. For example, the position of the descriptor pyramid can correspond to the last known position of the object, for example in the previous image. The position of the descriptor pyramid can correspond to the movement of the last known position. For example, if it is known or believed that the object moves at a speed of 10 pixels per frame in the positive x direction, then the descriptor pyramid can be positioned so that a feature descriptor pattern about the position (x+10, y) is applied in the current image if the object is known to be at position (x, y) in the previous image. A search area can be defined to which the descriptor pattern is applied. To give some examples, the search area can be centered on the last known position of the object, the estimated position of the object, etc. The search area can be offset from the last known position of the object, the offset depending on, for example, the estimated movement of the object.

[0219] As described, feature descriptors may be generated at different length scales in the scale space data. Each feature descriptor may be compared, i.e., with respect to different length scales. The comparison may include a measure of the likelihood of the feature at the corresponding length scale. The comparison may indicate that there are matching features at the corresponding position and length scale.

[0220] When the measure of likelihood is greater (or less) for a greater likelihood, the maximum (or minimum) value in the result set can be used to identify the most likely length scale at which the feature resides, or the portion of the scale-space data that represents that length scale; in general, turning points can be used in the measure of likelihood. The identified length scale can be a length scale in a discrete set of length scales that includes the scale-space data. The identified length scale can be between length scales in the discrete set of length scales that include the scale-space data. Applying feature detectors at different length scales can be useful to identify the most likely length scale at which the feature resides or represents. When generating a feature descriptor for the feature, sampling the scale-space data can include sampling the scale-space data at the identified length scale, or interpolating between length scales to insert to either side of the identified length scale.

[0221] The process may include sampling from a scale space representation of the image to generate a rotation metric, and one or more additional samplings from the scale space representation of the image to generate a feature descriptor. The process may include sampling from the scale space representation to generate an array, generating a rotation metric from the array, and generating a feature descriptor from the array and the generated rotation metric. The process may include sampling from the scale space representation to generate a first feature descriptor for a first location in the image, and sampling from the scale space representation to generate a second feature descriptor for a second location in the image. In practice, it is likely that many features in an image will be detected for matching with features in other images. Therefore, the scale space representation of a particular image may be reused significantly. This justifies the additional processing cost of pre-generating the scale space representation, and depending on the number of features to be considered, may result in overall processing and memory bandwidth savings.

[0222] The second image may be represented by a second scale-space dataset. The process may include sampling from the second scale-space dataset to generate a feature descriptor for the second image. The process may include: sampling from the second scale-space dataset to generate a third feature descriptor for a third position, the third position being located in the second image; and sampling from the second scale-space dataset to generate a fourth feature descriptor for a fourth position, the fourth position being located in the second image.

[0223] At least one of the feature descriptors for the original image (e.g., the first and / or second descriptors) can be compared to at least one of the feature descriptors for the second image (e.g., the third and / or fourth descriptors) to identify a match or a most likely match between the feature descriptors. This allows a match to be identified between a location in the original image and a location in the second image.

[0224] Reference now Fig. 9 and Fig.10 , which shows in schematic form an apparatus for implementing the above-mentioned system and process, which apparatus may generally be referred to as a descriptor generation system or a descriptor generator. Fig. 9 , an apparatus or descriptor generation system is shown at 900. Image data is provided at 902. This can be in the form of an input for receiving image data, a memory for storing image data, etc. A sampling unit 904 samples the image data 902. The sampling unit is configured to extract sample values ​​from the image data. The extracted sample values ​​are stored in a memory 906. The extracted sample values ​​can be stored in the form of an initial sample set 908, for example as an array. A feature descriptor generator 910 is provided for generating feature descriptors. The feature descriptor generator can receive the extracted samples directly from the sampling unit 904 (as shown in 912), or it can receive samples from the memory 906 (as shown in 914). The feature descriptor generator 910 generates feature descriptors 918 based on the initial sample set. The feature descriptor generator can be configured to determine a rotation metric about a sampled area of ​​the image based on the initial sample set. In some examples, the rotation metric can be determined in other ways, such as at the sampling unit 904. The feature descriptor generator generates a rotation-invariant feature descriptor based on the initial sample set and the determined rotation metric.

[0225] The feature descriptor generator includes a sampling unit 916. The sampling unit 916 is configured to sample from the initial sample set 908 stored in the memory 906. The sampling unit 916 is configured to shift at least one of the elements of the initial sample set and interpolate between the elements of the initial sample set to form a modified array. The feature descriptor generator is configured to output a feature descriptor 918.

[0226] Reference now Fig.10 , an apparatus or descriptor generation system is shown at 1000. Image data is provided at 1002. This may be in the form of an input to receive image data, a memory to store image data, etc. An image processor is provided at 1004. The image processor is configured to generate scale space data representing the image data 1002 at a plurality of length scales. The scale space data 1006 is stored in a memory 1008.

[0227] A feature descriptor generator 1010 is provided for generating feature descriptors. The feature descriptor generator comprises a sampling unit 1012. The sampling unit 1012 is configured to sample the scale space data 1006. The sampling unit 1012 may sample the scale space data 1006 to obtain an initial sample set. The initial sample set may be stored in a memory 1014. The sampling unit may be configured to subsequently sample from at least one of the scale space data 1006 and the initial sample set stored in the memory 1014.

[0228] The feature descriptor generator 1010 is configured to generate a feature descriptor 1016 based on the sampled scale-space data and / or the sampling set of initial samples, as described above. The feature descriptor generator may be configured to determine a rotation metric about the image sampling region based on the initial sample set (e.g., at the sampling unit 1012). In an embodiment, the sampling unit 1012 of the feature descriptor generator 1010 is configured to shift at least one of the elements of the initial sample set and interpolate between the elements of the initial sample set based on the rotation metric to form a modified array. The modified array may be at least temporarily stored at the memory 1014. The feature descriptor generator generates a rotationally invariant feature descriptor based on the modified array. The feature descriptor generator 1010 is configured to output a feature descriptor 1016. Once the feature descriptor has been generated, the modified array may be discarded.

[0229] Fig. 9 and Fig.10 The descriptor generation system in is shown as comprising a plurality of functional blocks. This is merely schematic and is not intended to define a strict division between different logical elements of such an entity. Each functional block may be provided in any suitable manner. It is to be understood that the intermediate values ​​formed by the descriptor generation system described herein need not be physically generated by the descriptor generation system at any point, and may represent only logical values ​​that conveniently describe the processing performed by the descriptor generation system between its input and output.

[0230] The descriptor generation system described herein can be embodied in hardware on an integrated circuit. The descriptor generation system described herein can be configured to perform any method described herein. Generally, any of the above functions, methods, techniques or components can be implemented in software, firmware, hardware (e.g., fixed logic circuits) or any combination thereof. The terms "module", "function", "component", "element", "unit", "block" and "logic" can be used herein to generally represent software, firmware, hardware or any combination thereof. In the case of software implementation, a module, function, component, element, unit, block or logic represents a program code that performs a specified task when executed on a processor. The algorithms and methods described herein can be performed by one or more processors that execute code, which causes (multiple) processors to perform the algorithm / method. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disk, flash memory, hard disk storage, and other memory devices that can use magnetic, optical and other technologies to store instructions or other data and can be accessed by a machine.

[0231] As used herein, the terms computer program code and computer readable instructions refer to any kind of executable code for a processor, including code expressed in machine language, interpreted language, or scripting language. Executable code includes binary code, machine code, bytecode, code that defines an integrated circuit (e.g., a hardware description language or netlist), and code expressed in programming language code such as C, Java, or OpenCL. For example, executable code can be any kind of software, firmware, script, module, or library that, when properly executed, processed, interpreted, compiled, executed in a virtual machine or other software environment, causes a processor of a computer system that supports the executable code to perform the tasks specified by the code.

[0232] A processor, computer or computer system may be any kind of device, machine or special purpose circuit, or collection or part thereof, which has processing capabilities enabling it to execute instructions. A processor may be any kind of general or special purpose processor, such as a CPU, GPU, system on chip, state machine, media processor, application specific integrated circuit (ASIC), programmable logic array, field programmable gate array (FPGA), etc. A computer or computer system may include one or more processors.

[0233] It is also intended to include software that defines the hardware configuration described herein, such as HDL (Hardware Description Language) software for designing integrated circuits or for configuring programmable chips to achieve desired functions. That is, a computer-readable storage medium having computer-readable program code encoded thereon in the form of an integrated circuit definition data set may be provided, which, when processed in an integrated circuit manufacturing system, configures the system to manufacture a descriptor generation system that is configured to perform any of the methods described herein or include any of the apparatus described herein. For example, the integrated circuit definition data set may be an integrated circuit description.

[0234] At an integrated circuit manufacturing system, a method of manufacturing a descriptor generation system as described herein may be provided. An integrated circuit definition data set may be provided that, when processed at the integrated circuit manufacturing system, causes the method of manufacturing a descriptor generation system to be performed.

[0235] The integrated circuit definition data set may take the form of computer code, for example as a netlist; code for configuring a programmable chip; as a hardware description language that defines an integrated circuit at any level, including as register transfer level (RTL) code, as a high-level circuit representation such as Verilog or VHDL, and as a low-level circuit representation such as OASIS (RTM) and GDSII. A higher-level representation (e.g., RTL) that logically defines an integrated circuit may be processed at a computer system that is configured to generate a manufacturing definition of the integrated circuit in the context of a software environment that includes definitions of circuit elements and rules for combining those elements to generate a manufacturing definition of the integrated circuit so defined by the representation. It is typically the case that software is executed on a computer system to define a machine, and one or more intermediate user steps (e.g., providing commands, variables, etc.) may be required to cause a computer system configured to generate a manufacturing definition of an integrated circuit to execute code that defines the integrated circuit so as to generate a manufacturing definition of the integrated circuit.

[0236] Now about Fig.11 An example of processing an integrated circuit definition data set in an integrated circuit manufacturing system to configure the system as a manufacturing descriptor generation system is described.

[0237] Fig.11 An example of an integrated circuit (IC) manufacturing system 1102 is shown that is configured to manufacture a descriptor generation system as described in any of the examples herein. Specifically, the IC manufacturing system 1102 includes a layout processing system 1104 and an integrated circuit generation system 1106. The IC manufacturing system 1102 is configured to receive an IC definition data set (e.g., defining a descriptor generation system as described in any of the examples herein), process the IC definition data set, and generate an IC (e.g., which embodies a descriptor generation system as described in any of the examples herein) based on the IC definition data set. The processing of the IC definition data set configures the IC manufacturing system 1102 to manufacture an integrated circuit embodying a descriptor generation system as described in any of the examples herein.

[0238] The layout processing system 1104 is configured to receive and process an IC definition data set to determine a circuit layout. Methods for determining a circuit layout based on an IC definition data set are known in the art and may, for example, involve synthesizing RTL code to determine a gate-level representation of a circuit to be generated, such as in terms of logic components (e.g., NAND, NOR, AND, OR, MUX, and flip-flop components). The circuit layout may be determined based on the gate-level representation of the circuit by determining location information for the logic components. This may be done automatically or with user involvement to optimize the circuit layout. When the layout processing system 1104 has determined the circuit layout, it may output the circuit layout definition to the IC generation system 1106. The circuit layout definition may be, for example, a circuit layout description.

[0239] As is known in the art, IC generation system 1106 generates an IC according to a circuit layout definition. For example, IC generation system 1106 may implement a semiconductor device fabrication process for generating an IC, which may involve a multi-step sequence of photolithography and chemical processing steps during which electronic circuits are gradually created on a wafer made of semiconductor material. The circuit layout definition may take the form of a mask that may be used in a photolithography process to generate an IC according to the circuit definition. Alternatively, the circuit layout definition provided to IC generation system 1106 may take the form of a computer readable code that IC generation system 1106 may use to form a mask suitable for use in generating an IC.

[0240] The different processes performed by the IC manufacturing system 1102 may be implemented, for example, all in one location by one party. Alternatively, the IC manufacturing system 1102 may be a distributed system such that some processes may be performed at different locations and may be performed by different parties. For example, some of the stages of the following steps may be performed in different locations and / or by different parties: (i) synthesizing RTL code representing an IC definition data set to form a gate-level representation of a circuit to be generated, (ii) generating a circuit layout based on the gate-level representation, (iii) forming a mask based on the circuit layout, and (iv) using the mask to fabricate the integrated circuit.

[0241] In other examples, processing an integrated circuit definition data set at an integrated circuit manufacturing system may configure the system to manufacture a descriptor generation system without processing the IC definition data set to determine a circuit layout. For example, the integrated circuit definition data set may define a configuration of a reconfigurable processor (e.g., an FPGA), and processing of the data set may configure the IC manufacturing system to generate a reconfigurable processor having the defined configuration (e.g., by loading the configuration data into the FPGA).

[0242] In some embodiments, the integrated circuit manufacturing definition data set, when processed in the integrated circuit manufacturing system, can cause the integrated circuit manufacturing system to generate the device as described herein. Fig.11 Configuring an integrated circuit manufacturing system in the manner described may enable the apparatus described herein to be manufactured.

[0243] In some examples, the integrated circuit definition data set may include software that runs on or in conjunction with hardware defined at the data set. Fig.11 In the example shown, the IC generation system can also be configured by the integrated circuit definition data set to load firmware onto the integrated circuit or otherwise provide program code to the integrated circuit for use with the integrated circuit when manufacturing the integrated circuit according to the program code defined in the integrated circuit definition data set.

[0244] The processing system described herein may be embodied in an integrated circuit in the form of hardware. The processing system described herein may be configured to perform any of the methods described herein.

[0245] The embodiments of the concepts described in the present application in devices, apparatuses, modules and / or systems (and in the methods implemented herein) can produce performance improvements when compared with known embodiments. The performance improvements can include one or more of increased computing performance, reduced waiting time, increased throughput and / or reduced power consumption. During the manufacture of such devices, apparatuses, modules and systems (e.g., in integrated circuits), performance improvements can be weighed against physical implementations to improve manufacturing methods. For example, performance improvements can be weighed against layout area to match the performance of known implementations but using less silicon. For example, this can be accomplished by reusing functional blocks in a serial manner or sharing functional blocks between elements of devices, apparatuses, modules and / or systems. On the contrary, the concepts of producing improvements (e.g., reduced silicon areas) in physical implementations of devices, apparatuses, modules and systems described in the present application can be weighed against improved performance. For example, this can be accomplished by manufacturing multiple instances of modules within a predefined area budget.

[0246] The embodiments of the concepts described in the present application in devices, apparatuses, modules and / or systems (and in the methods implemented herein) can produce performance improvements when compared with known embodiments. The performance improvements can include one or more of increased computing performance, reduced waiting time, increased throughput and / or reduced power consumption. During the manufacture of such devices, apparatuses, modules and systems (e.g., in integrated circuits), performance improvements can be weighed against physical implementations to improve manufacturing methods. For example, performance improvements can be weighed against layout area to match the performance of known implementations but using less silicon. For example, this can be accomplished by reusing functional blocks in a serial manner or sharing functional blocks between elements of devices, apparatuses, modules and / or systems. On the contrary, the concepts of producing improvements (e.g., reduced silicon areas) in physical implementations of devices, apparatuses, modules and systems described in the present application can be weighed against improved performance. For example, this can be accomplished by manufacturing multiple instances of modules within a predefined area budget.

[0247] The applicant hereby discloses each individual feature described herein in isolation and any combination of two or more such features, as long as such feature or combination can be performed based on this specification as a whole in view of the common general knowledge of a person skilled in the art, regardless of whether such feature or combination of features solves any problem disclosed herein. In view of the above description, it will be obvious to a person skilled in the art that various modifications can be made within the scope of the present invention.

Claims

1. A descriptor generation system for generating feature descriptors for locations in an image, thereby performing descriptor matching when analyzing the image, the descriptor generation system being configured to: obtaining scale space data representing the image, the scale space data comprising pre-filtered representations of the image at multiple length scales; the descriptor generation system having a feature descriptor generator, the feature descriptor generator being configured to: A set of samples representing locations in the image is determined by sampling the scale space data representing the image using a sampling unit according to the descriptor pattern of the feature descriptor, wherein: the descriptor pattern being used to determine length scales at which samples are to be sampled from the pre-filtered representation of the image; and The feature descriptor is generated according to the determined sample set.

2. The descriptor generation system according to claim 1, wherein: The feature descriptor generator is also configured to: if the determined length scale does not match one of the length scales of the pre-filtered representation of the image, sampling the scale-space data includes: interpolating between data at levels in the scale-space data associated with length scales above and below the determined length scale.

3. The descriptor generation system according to claim 1 or 2, wherein: The pre-filtering of the image means that it has been filtered using one or more of a low pass filter, a Gaussian filter and a box filter.

4. The descriptor generation system according to claim 1 or 2, wherein: The positions in the image are one or more of the following: a point in the image; The pixel location in the image; and Key points in the image.

5. The descriptor generation system according to claim 1 or 2, wherein: The feature descriptor generator is configured to sample the scale-space data in the following manner: analyzing portions of the scale-space data representing locations at different length scales to determine a likelihood metric for each analyzed portion, the likelihood metric representing the likelihood of a feature at the corresponding length scale; determining a portion of the scale-space data that results in a turning point in the measure of likelihood; Based on the determined portion, a length scale is determined at which the scale-space data is sampled to determine the sample set.

6. The descriptor generation system according to claim 1 or 2, wherein: The descriptor generation system is configured to place the relatively smaller scale-space representation within the relatively larger scale-space representation based on the identified length scale.

7. The descriptor generation system according to claim 6, wherein: The relatively smaller scale-space representation comprises a descriptor pyramid and the relatively larger scale-space representation comprises an image pyramid.

8. The descriptor generation system according to claim 1 or 2, wherein: The descriptor generation system is configured to identify a location in the image based on one or more location identification or detection algorithms, and wherein the descriptor generation system is configured to identify at least one of a location and a length scale in the scale-space data associated with the location identified in the image.

9. The descriptor generation system according to claim 8, wherein: The feature descriptor generator is configured to sample the scale-space data by one or more of: According to the length scale identified; at a level in the scale-space data associated with the identified length scale; and By interpolating between data at levels in the scale-space data associated with length scales above and below the identified length scale.

10. The descriptor generation system according to claim 1 or 2, wherein: The data in the scale-space data that have been filtered at different length scales correspond to filtered samples in the descriptor pattern extracted at different radial distances from the center of the descriptor pattern.

11. The descriptor generation system according to claim 1 or 2, wherein: The descriptor pattern includes at least one ring surrounding a location in the image, wherein the at least one ring is one of a circle, a wavy circle, and a polygon.

12. The descriptor generation system according to claim 1 or 2, wherein: The descriptor generation system is configured as one or more of the following: The determined sample set is stored in an array, and the feature descriptor generator is configured to: generate the feature descriptor based on the determined sample set by forming a modified array; A rotation metric is determined for a location in the image, the rotation metric describing an angle between an orientation of the image and a characteristic direction in the image at the location, and the feature descriptor is generated based on the determined rotation metric.

13. The descriptor generation system according to claim 12, wherein: The descriptor generation system is configured to form the modified array by one or more of: shifting elements of at least a portion of the array by an amount of positions in a corresponding portion of the array, the amount of positions being determined based on the determined rotation metric; as well as Interpolation is performed between two or more samples in the determined sample set.

14. The descriptor generation system according to claim 13, wherein: The two or more samples in the determined sample set have been obtained from one or more rings of the descriptor pattern.

15. The descriptor generation system according to claim 14, wherein: The descriptor generation system is configured to form the modified array by interpolating between two or more samples in the determined sample set along a portion of a shape of a ring to which the two or more samples correspond.

16. The descriptor generation system according to claim 13, wherein: The two or more samples between which the interpolation is performed in the determined sample set have been obtained from adjacent rings in the descriptor pattern.

17. The descriptor generation system according to claim 13, wherein: The two or more samples in the determined sample set include N1 samples from a first ring and N2 samples from a second ring, where N1 ≤ N2, wherein the first ring is radially inside the second ring.

18. The descriptor generation system according to claim 12, wherein: The descriptor generation system is configured to discard the modified array once the feature descriptor has been generated.

19. A computer-implemented method for generating feature descriptors for locations in an image for use in performing descriptor matching when analyzing the image, the method comprising: obtaining scale-space data representing the image, the scale-space data comprising pre-filtered representations of the image at a plurality of length scales; determining a set of samples representing locations in the image by sampling the scale-space data representing the image based on a descriptor pattern of the feature descriptors, wherein the descriptor pattern is used to determine length scales at which samples are to be sampled from the pre-filtered representation of the image; as well as The feature descriptor is generated according to the determined sample set.

20. A computer-readable storage medium having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed on a computer system, the computer system: obtaining scale-space data representing an image, the scale-space data comprising pre-filtered representations of the image at a plurality of length scales; A set of samples representing locations in the image is determined by sampling the scale-space data representing the image according to a descriptor pattern of a feature descriptor, wherein: the descriptor pattern being used to determine length scales at which samples are to be sampled from the pre-filtered representation of the image; as well as The feature descriptor is generated according to the determined sample set.

Images