A feature point extraction method

Abstract

The embodiment of the present application provides a feature point extraction method, and relates to the technical field of image processing. The feature point extraction method comprises the following steps: determining at least one effective sub-region of an original image, wherein the at least one effective sub-region is determined according to at least one effective sub-region on a low-resolution image, and the low-resolution image is an image obtained by performing down-sampling processing on the original image; and performing feature point extraction on each effective sub-region on the original image to obtain feature points of the original image. Since the low-resolution image has fewer pixels compared with the original image, the speed of determining the effective sub-region on the low-resolution image is higher, and therefore the efficiency of performing feature point extraction on the original image is improved.

Description

Technical Field

[0001] The present application relates to a method for extracting feature points, and belongs to the technical field of image processing. Background Art

[0002] Feature point extraction is a common technique in image processing, widely used in smart terminals, vehicle cameras, security monitoring, and other fields. While the recent rise in high-resolution (4K / 8K) and high-frame-rate (90FPS / 120FPS) displays has improved user experience, it has also brought numerous challenges to image processing, such as lengthy feature point extraction times and low accuracy.

[0003] In existing techniques, the original image is first cropped, typically in the center or four corners, to obtain a reduced-size original image. For example, the four edges of an original image are cropped, retaining only the pixels in the center. Feature points are then extracted from the cropped image using a fixed threshold. This existing technique has the following disadvantages: it is difficult to determine the valid area during cropping, which may result in an insufficient number of extracted feature points. Summary of the Invention

[0004] The present application provides a feature point extraction method that, by determining the effective sub-region of the original image, effectively solves the problem of excessive processing time due to the large amount of computation required to extract feature points in the process of extracting feature points from high-resolution images or high-frame video streams. Furthermore, by adjusting the extraction threshold for feature point extraction in the effective sub-region, the problem of feature point proliferation is reduced, thereby improving the accuracy of feature point extraction. To achieve the above objectives, the present application provides the following technical solutions:

[0005] In a first aspect, an embodiment of the present application provides a method for extracting feature points, comprising the following steps:

[0006] Determine at least one valid sub-region on the original image, where the at least one valid sub-region is determined based on at least one valid sub-region on the low-resolution image, where the low-resolution image is an image obtained by downsampling the original image;

[0007] Feature points are extracted from each valid sub-region on the original image to obtain feature points of the original image.

[0008] Since the low-resolution image has fewer pixels than the original image, it is faster to determine the valid sub-region on the low-resolution image. Therefore, it is also faster to determine the valid sub-region on the original image based on the valid sub-region on the low-resolution image, thereby improving the efficiency of feature point extraction on the original image.

[0009] In one possible implementation, determining at least one valid sub-region on the original image includes:

[0010] determining at least one valid sub-region on the low-resolution image;

[0011] Backward mapping is performed on at least one valid sub-region on the low-resolution image to obtain at least one valid sub-region of the original image, wherein one valid sub-region on the low-resolution image corresponds to one valid sub-region of the original image.

[0012] Since at least one valid sub-region on the low-resolution image is mapped to at least one valid sub-region of the original image through backward mapping, the accuracy of feature point extraction of the original image can be guaranteed.

[0013] In one possible implementation, determining at least one valid sub-region on the low-resolution image includes:

[0014] Dividing the low-resolution image into a plurality of sub-regions;

[0015] performing feature point extraction on the low-resolution image to obtain feature points of the low-resolution image, where the feature points of the low-resolution image are distributed within the plurality of sub-regions;

[0016] The first K sub-regions containing the largest number of feature points among the plurality of sub-regions are used as at least one valid sub-region of the low-resolution image, where K is a positive integer greater than or equal to 2.

[0017] Since the top K sub-regions containing the largest number of feature points among the several sub-regions are used as at least one valid sub-region of the low-resolution image, the problem of insufficient number of feature points can be avoided as much as possible, and at the same time, the feature point information of the entire image can be retained as much as possible.

[0018] In one possible implementation, the feature point extraction method further includes:

[0019] Determine an extraction threshold for each valid sub-region on the original image;

[0020] Extracting feature points from each valid sub-region on the original image includes:

[0021] Feature points are extracted from each valid sub-region on the original image according to an extraction threshold of each valid sub-region on the original image.

[0022] In one possible implementation, determining the extraction threshold of each sub-region on the original image includes:

[0023] Determining an extraction threshold for each valid sub-region in the low-resolution image;

[0024] The extraction threshold of each valid sub-region in the low-resolution image is used as the extraction threshold of each valid sub-region in the original image, wherein the extraction threshold of a valid sub-region in the low-resolution image corresponds to the extraction threshold of a valid sub-region on the original image.

[0025] In one possible implementation, determining an extraction threshold for each valid sub-region in the low-resolution image includes:

[0026] Comparing the sum of the number of feature points in all valid sub-regions of the low-resolution image with a preset expected value of feature points;

[0027] If the sum of the number of feature points is greater than the expected value of the feature points, adjusting the fixed threshold according to the number of feature points in each valid sub-region of the low-resolution image to determine an extraction threshold for each valid sub-region;

[0028] If the sum of the number of feature points is less than or equal to the expected value of feature points, the extraction threshold of all valid sub-regions of the low-resolution image is set to the fixed threshold.

[0029] Since the extraction threshold of each valid sub-region of the low-resolution image can be adjusted according to the sum of the number of feature points in all valid sub-regions, the problem of long extraction time due to too many feature points can be effectively avoided.

[0030] In a possible implementation, the fixed threshold is a threshold for extracting feature points from the low-resolution image.

[0031] In a second aspect, an embodiment of the present application further provides a feature point extraction device, comprising:

[0032] a determination module, configured to determine at least one valid subregion on the original image, wherein the at least one valid subregion is determined based on at least one valid subregion on a low-resolution image, wherein the low-resolution image is an image obtained by downsampling the original image;

[0033] The extraction module is used to extract feature points from the effective sub-regions on each original image to obtain the feature points of the original image.

[0034] In one possible implementation, the determination module is specifically configured to:

[0035] determining at least one valid sub-region on the low-resolution image;

[0036] Backward mapping is performed on at least one valid sub-region on the low-resolution image to obtain at least one valid sub-region of the original image, wherein one valid sub-region on the low-resolution image corresponds to one valid sub-region of the original image.

[0037] In one possible implementation, the determination module is specifically configured to:

[0038] Dividing the low-resolution image into a plurality of sub-regions;

[0039] performing feature point extraction on the low-resolution image to obtain feature points of the low-resolution image, where the feature points of the low-resolution image are distributed within the plurality of sub-regions;

[0040] The first K sub-regions containing the largest number of feature points among the plurality of sub-regions are used as at least one valid sub-region of the low-resolution image, where K is a positive integer greater than or equal to 2.

[0041] In one possible implementation, the extraction module further includes:

[0042] A threshold module, used for determining a threshold value of each valid sub-region on the original image;

[0043] The extraction module is specifically used to:

[0044] Feature points are extracted from each valid sub-region on the original image according to a threshold value of each valid sub-region on the original image.

[0045] In one possible implementation, the threshold module is specifically configured to:

[0046] determining a threshold value for each valid sub-region in the low-resolution image;

[0047] The threshold of each valid sub-region in the low-resolution image is used as the threshold of each valid sub-region in the original image, wherein the threshold of a valid sub-region in the low-resolution image corresponds to the threshold of a valid sub-region in the original image.

[0048] In one possible implementation, the threshold module is specifically configured to:

[0049] Comparing the sum of the number of feature points in all valid sub-regions of the low-resolution image with a preset expected value of feature points;

[0050] If the sum of the number of feature points is greater than the expected value of the feature points, adjusting the fixed threshold according to the number of feature points in each valid sub-region of the low-resolution image to determine a threshold for each valid sub-region;

[0051] If the sum of the number of feature points is less than or equal to the expected value of the feature points, the threshold of all valid sub-regions of the low-resolution image is set to the fixed threshold.

[0052] In a possible implementation, the fixed threshold is a threshold used for extracting feature points from the low-resolution image.

[0053] A third aspect: The present application also provides a feature point extraction device, including:

[0054] at least one processor;

[0055] At least one memory stores a computer program, and when the computer program is executed by the at least one processor, the feature point extraction method as described in any one of the first aspects is implemented.

[0056] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing instructions, wherein the storage medium stores a computer program or instructions, which, when executed on a computer, enables the computer to implement the feature point extraction method as described in any one of the first aspects.

[0057] In a fifth aspect, an embodiment of the present application further provides a computer program product, which, when executed on a computer, enables the computer to execute any one of the methods in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A feature point extraction method provided in an embodiment of the present application;

[0059] Figure 2 A schematic diagram of a system architecture provided in an embodiment of the present application;

[0060] Figure 3 A schematic diagram of the real-time video denoising process provided in an embodiment of the present application;

[0061] Figure 4 An implementation method for extracting feature points from low-resolution images provided in an embodiment of the present application;

[0062] Figure 5 An implementation method for dividing a low-resolution image into sub-regions is provided in an embodiment of the present application;

[0063] Figure 6 An implementation method for determining effective sub-regions of a low-resolution image provided in an embodiment of the present application;

[0064] Figure 7 A schematic structural diagram of a feature point extraction device provided in an embodiment of the present application;

[0065] Figure 8 A schematic diagram of the structure of another feature point extraction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the objectives, technical solutions and advantages of this application more clear, this application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0067] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0068] First, some terms in this application are explained to facilitate understanding by those skilled in the art.

[0069] 1. Image downsampling: Image downsampling, also known as subsampling, is a technical solution used to proportionally reduce the target image to generate a thumbnail of the corresponding image. For example, downsampling an image of size M x N by a factor of s results in an image of size (M / s) x (N / s), where M is the pixel length of the image, N is the pixel width of the image, and s is the downsampling ratio. Since the downsampled image contains fewer pixels, feature point extraction on the downsampled image is faster.

[0070] 2. Feature Points: Feature points are points where the grayscale value of an image changes dramatically, or points with large curvature on an image edge (i.e., the intersection of two edges). Image feature points reflect the essential characteristics of an image and can be used to identify target objects in the image. Image matching can be achieved by matching feature points.

[0071] 3. Image alignment: Image alignment involves extracting and matching the feature points of two images. Then, using techniques such as affine transformation or perspective transformation, the transformation parameters between the two images are obtained. In this application, the feature points on the two aligned images have a one-to-one correspondence.

[0072] 4. Original Image: Original images are mainly divided into two types: RGB and CMYK. RGB original images are composed of three different color components: red, green, and blue. CMYK original images are composed of four color components: cyan (C), magenta (M), yellow (Y), and black (K). CMYK original images are mainly used in the printing industry.

[0073] 5. Mapping: Scaling, rotating, translating, transforming, and other operations on an image require establishing a mapping rule between the pixel coordinates of the original image and the pixel coordinates of the target image, so that a corresponding relationship is established between the coordinates of the two images, thereby assigning a value to each pixel of the target image.

[0074] The mapping from the original image to the target image is called forward mapping; the mapping from the target image to the original image is called backward mapping. This backward mapping inevitably involves the problem of some pixels having floating-point coordinates when mapped back to the original image, while the pixel values ​​of the image are only defined at integer coordinates. Finally, image interpolation methods are used to assign the values ​​of these pixels. Common interpolation algorithms include nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation. Because image interpolation methods are state-of-the-art, they are not further elaborated in the examples of this application.

[0075] It can be understood that in the embodiment of the present application, the target image is a low-resolution image, that is, the mapping from the original image to the low-resolution image is called forward mapping, and the mapping from the low-resolution image to the original image is called backward mapping.

[0076] In a preferred embodiment, the original image can be high resolution (2K, 4K), high frame (60 / 90 / 120FPS), or a combination of high resolution and high frame rate; using the feature point extraction method of the present application on the original image, the feature point extraction speed is faster and the extracted feature points are more accurate than the prior art.

[0077] Figure 1 The present invention provides a feature point extraction method. This feature point extraction method mainly determines the valid sub-region of the original image and dynamically adjusts the feature point extraction threshold. This method can quickly extract feature points while ensuring the accuracy of feature points. Specifically, the feature point extraction method includes the following steps:

[0078] Step S101 : downsampling the original image to obtain a low-resolution image.

[0079] Specifically, the input image has a size of L x W, where L is the pixel length of the original image and W is the pixel width of the original image. According to the size of the original image, a downsampling operation is selectively performed. By setting the size of a desired image and comparing it with the size of the original image, it is used to determine whether the original image needs to be downsampled. If the size of the original image is smaller than the size of the desired image, the downsampling operation is not performed. If the size of the original image is larger than the size of the desired image, the downsampling operation is performed; the original image is downsampled s times, and the original image is proportionally reduced to a low-resolution image of size (L / s) x (W / s), that is, every s points in each row and column of the original image are taken to reconstruct a low-resolution image. It is worth noting that s is a positive integer greater than 2.

[0080] Step S102 : extracting feature points from the low-resolution image to obtain feature points of the low-resolution image.

[0081] Specifically, a fixed threshold is set, which can be used in all feature point extraction algorithms. Feature points are extracted from the downsampled low-resolution image based on the fixed threshold to obtain the coordinate information of each feature point. The feature points include but are not limited to edge areas or corner areas.

[0082] The feature point extraction operation of this application is further illustrated using ORB (Oriented Fast and Rotated BRIEF) as an example. It is worth noting that the feature point extraction algorithm used is ORB, which is only used as an example to more clearly explain this application. The feature point extraction algorithm can be one of SIFT, SURF, HARRIS, ORB, and optical flow, and is not specifically limited here.

[0083] The feature point extraction operation is to determine whether each pixel is a feature point based on the image grayscale value around the pixel.

[0084] First, the low-resolution image is converted from a color image to a grayscale image.

[0085] Next, we randomly select a pixel as a candidate feature point, draw a circle with the candidate feature point as the center and x pixels as the radius, and determine the grayscale value of each pixel on the circle one by one. It is worth mentioning that x is a positive integer greater than or equal to 2.

[0086] Then, the grayscale value of each pixel on the circle is subtracted from the grayscale value of the candidate feature point to obtain the absolute value of the subtraction. If the absolute value exceeds a fixed threshold thrNormal, it is considered that the difference between the pixel on the circle and the candidate feature point is too large.

[0087] Next, count how many pixels on the circle differ significantly from the candidate feature point. If there are enough pixels in the area surrounding the candidate point that differ significantly from the candidate point in grayscale value, the candidate point is considered a feature point. Otherwise, the candidate point is not considered a feature point.

[0088] Finally, traverse all pixels on the low-resolution image and extract all feature points and their pixel coordinates. Figure 4 , which shows the low-resolution image after feature points are extracted.

[0089] It should be noted that Figure 4 The feature point distribution of the low-resolution image shown is only an example; in practice, different low-resolution images have different numbers and distributions of feature points.

[0090] Step S103 : determining a valid sub-region of the low-resolution image based on the feature points on the low-resolution image.

[0091] Specifically, the low-resolution image is divided into M x N subregions, each containing a number of feature points. For an image with a length of L and a width of W, the subregions are divided into M x N subregions, where each subregion has a length of L / M and a width of W / N. All subregions are traversed, and the number of feature points in each subregion of the low-resolution image is determined. The top K subregions with the largest number of feature points are selected as valid subregions.

[0092] In order to make this step clearer and easier to understand, a specific example is used to further illustrate it. For example, if M=7, N=10, and K=10 are set. That is, the length of the low-resolution image with a length of L / M and a width of W / N is evenly divided into 7 equal parts, and the width of the low-resolution image is evenly divided into 10 equal parts. If the length and width of the low-resolution image cannot be divided by 10 and 7 respectively, that is, one of the length L / M or the width W / N of the size of each sub-region is not an integer, it can be solved by rounding the coordinates of each sub-region after segmentation. To obtain a low-resolution image with sub-regions, you can refer to Figure 5 .

[0093] The low-resolution image consists of 70 sub-regions. Traverse all 70 sub-regions and determine the number of feature points in each sub-region. Select the first 10 sub-regions with the largest number of feature points as valid sub-regions. Figure 6 , Figure 6 The sub-regions covered by shadows in the figure represent the top 10 sub-regions with the largest number of feature points, that is, the valid sub-regions of the low-resolution image.

[0094] It should be noted that Figure 5 The division method shown is only an example; in practice, the values ​​of M, N, and K can be adjusted to a certain extent, for example, M, N, and K are positive integers greater than or equal to 2, and no specific limitation is made here.

[0095] Step S104: determining an extraction threshold corresponding to each valid sub-region of the low-resolution image.

[0096] Specifically, a feature point expected value maxKpCnt is set, and it is determined whether the sum of the number of feature points in all valid sub-regions of the low-resolution image, totalKpCnt, is greater than maxKpCnt. Based on the determination result, the threshold thrAdptive of each valid sub-region of the low-resolution image is adjusted.

[0097] If totalKpCnt is greater than maxKpCnt, it means that the extraction threshold of all valid sub-regions of the low-resolution image is set too low, resulting in too many feature points extracted from the low-resolution image. Therefore, it is necessary to adjust the extraction threshold thrAdptive of each valid sub-region of the low-resolution image. The specific adjustment method is as follows:

[0098]

[0099] Wherein, thrMax is a set maximum extraction threshold, which must be greater than the fixed threshold thrNormal and less than 255. In a preferred embodiment, the value of thrMax is set to twice that of thrNormal.

[0100] Wherein, CellKpCnt is the number of feature points in the single valid sub-region that needs to be adjusted, and thrNormal is the fixed threshold set in step S102.

[0101] If the total number of feature points in the valid area, totalKpCnt, is less than or equal to maxKpCnt, it indicates that the extraction threshold thrAdptive is set reasonably. The extraction threshold thrAdptive is set to the fixed threshold thrNormal. The specific adjustment method is as follows:

[0102] thrAdptive=thrNormal

[0103] Step S105 , determining the valid sub-region and the corresponding extraction threshold of the original image according to the valid sub-region and the corresponding extraction threshold of the low-resolution image.

[0104] Specifically, first, the coordinates of the boundary pixels of each valid sub-region of the low-resolution image and the extraction threshold of the valid sub-region are read in step S104.

[0105] Then, the coordinates of the boundary pixels of each valid sub-region of the low-resolution image and the extraction threshold corresponding to the valid sub-region are backward mapped, that is, the coordinates of the boundary pixels of the valid sub-region are mapped from the low-resolution image to the original image, and the coordinates of the boundary pixels of each valid sub-region of the original image and the extraction threshold corresponding to the valid sub-region are obtained for use in feature point extraction on the original image.

[0106] It is worth mentioning that the effective sub-regions of the low-resolution image correspond one-to-one with the effective sub-regions of the original image.

[0107] Step S106: extract feature points from the original image to obtain feature points of the original image.

[0108] Specifically, the extraction threshold corresponding to the valid sub-region of the original image is used as the extraction threshold in the ORB feature point extraction method, and the ORB feature point extraction method is used to extract feature points from each valid sub-region on the original image to obtain the feature points of the original image.

[0109] According to the feature point extraction method disclosed in the embodiments of the present application, by determining the number and distribution of feature points in a low-resolution image, the effective sub-region of the low-resolution image is determined. This is then used to determine the effective sub-region of the original image. This effectively reduces the computational complexity of extracting feature points while ensuring the accuracy of feature point coordinate information. Furthermore, dynamically adjusting the extraction threshold of the effective sub-region based on the number of feature points within the effective sub-region of the low-resolution image effectively controls the number of feature points within a reasonable range, effectively avoiding an overabundance of feature points and the problem of excessively long feature point extraction times.

[0110] Provide an introduction to the general scenarios to which the technical solution of this application is applicable.

[0111] Scenario: Real-time video denoising

[0112] The application scenario of this application is to use a feature point extraction method to align images between each frame of a real-time video stream on smart terminal devices such as mobile phones.

[0113] Figure 2 1 is a schematic diagram of the system architecture of an embodiment provided by this application. The system architecture comprises a real-time video stream 1001, a feature point extraction module 1002, an inter-frame matching / alignment module 1003, and a noise reduction / anti-shake processing module 1004. Steps S101-S106 of the feature point extraction method can be implemented by the feature extraction module 1002.

[0114] The real-time video streaming module 1001 is used to read each frame from the video stream. Video streams can be divided into the following two forms: 1. Compressed stream data, called the coded stream. 2. Uncompressed stream data, which is the decoded stream data, called the raw stream. In network transmission scenarios, the video stream is the compressed coded stream. In video playback scenarios, the video stream is the decoded raw stream.

[0115] The feature point extraction module 1002 is used to extract feature points and additional feature information for each feature point, such as direction, scale, and feature vector, from each frame of the image. The extraction method used by the feature point extraction module 1002 can be SIFT, SURF, HARRIS, ORB, or an optical flow method, which is not specifically limited here. In the specific embodiment of the present application, the extraction method used by the feature point extraction module 1002 is ORB.

[0116] The inter-frame matching / alignment module 1003 is used to analyze the feature points of each frame with those of the previous frame. Specifically, the feature point extraction module 1002 first obtains the distribution position of each feature point in the frame before the current frame, and then uses the feature point extraction module 1002 to obtain the new distribution position of each feature point in the current frame. The translation and rotation information of each feature point in the current frame and each feature point are calculated, and the offset of the frame image is determined based on the translation and rotation information of each feature point.

[0117] The noise reduction / anti-shake processing module 1004 is used to perform noise reduction processing on each frame of the image. When the accumulated inter-frame offset exceeds a certain threshold, the noise reduction processing is performed on the image of the frame.

[0118] The processed real-time video stream 1005 is used to output the final real-time video stream after noise reduction.

[0119] The specific process of this scenario can be referred to Figure 3 .

[0120] Step S201: Input video stream.

[0121] Specifically, a real-time video stream is obtained in the smart terminal device, and the real-time video stream is decoded to obtain image data of each frame.

[0122] Step S202: Acquire the first frame or the next frame.

[0123] Specifically, the image data of the first frame is acquired after step S201 ; or the image data of the next frame is acquired after steps S206 and S211 .

[0124] Step S203: Determine whether it is the first frame

[0125] Specifically, it is determined whether the current frame is the first frame. If the current frame is the first frame, step S204 is executed; if the current frame is not the first frame, step S207 is executed.

[0126] Step S204: executing all steps S101 to S106 of the feature point extraction method on the first frame.

[0127] Specifically, all steps S101 to S106 of the feature point extraction method are performed on the first frame, sequentially including: S101, downsampling the original image to obtain a low-resolution image. S102, performing a first feature point extraction on the low-resolution image to obtain a feature point distribution. S103, determining a valid sub-region of the original image based on the feature point distribution. S104, determining an adaptive extraction threshold for the valid sub-region. S105, mapping the adaptive extraction threshold corresponding to the valid sub-region back to the original image. S106, performing a second feature point extraction on the original image to obtain a final feature point result.

[0128] Step S205: Save the coordinates of the valid sub-region of the current frame and the adaptive extraction threshold corresponding to the valid sub-region.

[0129] Specifically, the coordinates of the valid sub-region of the current frame and the adaptive extraction threshold corresponding to the valid sub-region are saved.

[0130] Step S206: Reset the total inter-frame offset to zero, δ=0.

[0131] Specifically, the total amount of inter-frame offset is reset to zero δ=0.

[0132] Step S207: determining feature points on the current frame image according to the valid sub-region of the previous frame and the adaptive extraction threshold.

[0133] Specifically, the feature points on the current frame image are determined based on the valid sub-region of the previous frame and the adaptive extraction threshold.

[0134] Step S208: determining the inter-frame offset variation Δδ based on the feature points of the current frame and the previous frame.

[0135] Specifically, the new position of the feature points of the previous frame in the current frame is found, and the offset of each feature point is calculated. The offset of each feature point is comprehensively calculated to obtain the inter-frame offset change Δδ.

[0136] In an existing comprehensive calculation for determining the inter-frame offset variation, the inter-frame offset variation Δδ is obtained by taking the average of the offsets of all feature points.

[0137] Step S209: Accumulate the inter-frame offset variation δ=δ+Δδ.

[0138] Specifically, the offset variation Δδ calculated in step S208 is added to the total inter-frame offset δ to obtain the accumulated total inter-frame offset δ.

[0139] Step S210: Determine whether there is a next frame

[0140] Specifically, it is determined whether there is a next frame. If there is a next frame, step S211 is executed; if there is no next frame, step S213 is executed.

[0141] Step S211: Determine whether δ exceeds a set threshold.

[0142] Specifically, it is determined whether δ exceeds a set threshold. If δ exceeds the set threshold, step S212 is executed; if δ does not exceed the set threshold, step S202 is executed.

[0143] Step S213: Output video stream. Figure 7 and Figure 8 The following provides a schematic diagram of a possible image processing device for the embodiments of the present application. These image processing devices can be used to achieve the beneficial effects of the above method embodiments.

[0144] like Figure 7 As shown, the feature point extraction device 700 includes a determination module 710, an extraction module 720 and a threshold module 730. The feature point extraction device 700 is used to implement the above Figure 1 The function of the method embodiment shown in FIG is as follows; the determination module 710 is used to execute S101-S105; the extraction module 720 is used to execute S106. For a more detailed description of the above-mentioned determination module 710, extraction module 720 and threshold module 730, please refer directly to Figure 1 The relevant description in the method embodiment shown is directly obtained and will not be repeated here.

[0145] It should be understood that the above division of the units of the feature point extraction device 700 is a division of logical functions.

[0146] Figure 8 Schematic diagram of the hardware structure of the feature point extraction device according to an embodiment of the present application. Figure 8 The feature point extraction device 800 shown includes a memory 801 , a processor 802 , a communication interface 803 and a bus 804 . The memory 801 , the processor 802 and the communication interface 803 are connected to each other via the bus 804 .

[0147] The memory 801 may be a ROM, a static storage device, or a RAM. The memory 801 may store programs. When the programs stored in the memory 801 are executed by the processor 802, the processor 802 and the communication interface 803 are used to perform the various steps of the image processing method of the embodiment of the present application.

[0148] The processor 802 can be a general-purpose CPU, microprocessor, ASIC, GPU or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the image processing device of the embodiment of the present application, or to execute the image processing method of the method embodiment of the present application.

[0149] The processor 802 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the image processing method of the embodiment of the present application may be completed by hardware integrated logic circuits in the processor 702 or software instructions.

[0150] The processor 802 may also be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. It may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor. The steps of the methods disclosed in the embodiments of this application may be directly implemented as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium well-established in the art, such as a random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or register. The storage medium is located in the memory 801, and the processor 802 reads the information in the memory 801 and, in conjunction with its hardware, completes the functions required to be performed by the units included in the image processing device of the embodiments of this application, or executes the image processing method of the method embodiments of this application.

[0151] The communication interface 803 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the apparatus 800 and other devices or a communication network. For example, the image to be processed can be obtained through the communication interface 703.

[0152] The bus 804 may include a path for transmitting information between various components of the device 800 (eg, the memory 801 , the processor 802 , and the communication interface 803 ).

[0153] It should be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0154] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0155] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0156] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0157] In this application, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0158] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0159] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0160] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0161] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0162] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0163] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0164] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0165] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A feature point extraction method, characterized in that: include: Determine at least one valid subregion on the original image, where the at least one valid subregion is determined based on at least one valid subregion on a low-resolution image, where the low-resolution image is an image obtained by downsampling the original image; Extracting feature points from each valid sub-region of the original image to obtain feature points of the original image; The determining of at least one valid sub-region on the original image includes: Dividing the low-resolution image into a plurality of sub-regions; performing feature point extraction on the low-resolution image to obtain feature points of the low-resolution image, where the feature points of the low-resolution image are distributed within the plurality of sub-regions; Selecting first K subregions containing the largest number of feature points among the plurality of subregions as at least one valid subregion of the low-resolution image, where K is a positive integer greater than or equal to 2; Backward mapping is performed on at least one valid subregion on the low-resolution image to obtain at least one valid subregion of the original image, wherein one valid subregion on the low-resolution image corresponds to one valid subregion of the original image.

2. The feature point extraction method according to claim 1, wherein: Also includes: Determining an extraction threshold for each valid sub-region on the original image; Extracting feature points from each valid sub-region on the original image includes: Feature points are extracted from each valid sub-region on the original image according to an extraction threshold of each valid sub-region on the original image.

3. The feature point extraction method according to claim 2, wherein: Determining the extraction threshold of each valid sub-region on the original image includes: Determining an extraction threshold for each valid sub-region in the low-resolution image; The extraction threshold of each valid sub-region in the low-resolution image is used as the extraction threshold of each valid sub-region in the original image, wherein the extraction threshold of a valid sub-region in the low-resolution image corresponds to the extraction threshold of a valid sub-region on the original image.

4. The feature point extraction method according to claim 3, wherein: Determining the extraction threshold of each valid sub-region in the low-resolution image includes: Comparing the sum of the number of feature points in all valid sub-regions of the low-resolution image with a preset expected value of feature points; If the sum of the number of feature points is greater than the expected value of feature points, adjusting the fixed threshold according to the number of feature points in each valid sub-region of the low-resolution image to determine an extraction threshold for each valid sub-region; If the sum of the number of feature points is less than or equal to the expected value of feature points, the extraction threshold of all valid sub-regions of the low-resolution image is set to the fixed threshold.

5. The feature point extraction method according to claim 4, characterized in that: The fixed threshold is a threshold used to extract feature points from the low-resolution image.

6. A feature point extraction device, characterized in that: include: a determination module, configured to determine at least one valid sub-region on the original image, wherein the at least one valid sub-region is determined based on at least one valid sub-region on a low-resolution image, wherein the low-resolution image is an image obtained by downsampling the original image; An extraction module, configured to extract feature points from a valid sub-region of each original image to obtain feature points of the original image; The determining module is specifically configured to: Dividing the low-resolution image into a plurality of sub-regions; performing feature point extraction on the low-resolution image to obtain feature points of the low-resolution image, where the feature points of the low-resolution image are distributed within the plurality of sub-regions; Selecting first K subregions containing the largest number of feature points among the plurality of subregions as at least one valid subregion of the low-resolution image, where K is a positive integer greater than or equal to 2; Backward mapping is performed on at least one valid subregion on the low-resolution image to obtain at least one valid subregion of the original image, wherein one valid subregion on the low-resolution image corresponds to one valid subregion of the original image.

7. The feature point extraction device according to claim 6, characterized in that: Also includes: A threshold module, used to determine a threshold value of each valid sub-region on the original image; The extraction module is specifically used for: Feature points are extracted from each valid sub-region on the original image according to a threshold value of each valid sub-region on the original image.

8. The feature point extraction device according to claim 7, characterized in that: The threshold module is specifically used for: determining a threshold value for each valid sub-region in the low-resolution image; The threshold of each valid sub-region in the low-resolution image is used as the threshold of each valid sub-region in the original image, wherein the threshold of a valid sub-region in the low-resolution image corresponds to the threshold of a valid sub-region on the original image.

9. The feature point extraction device according to claim 8, characterized in that: The threshold module is specifically used for: Comparing the sum of the number of feature points in all valid sub-regions of the low-resolution image with a preset expected value of feature points; If the sum of the number of feature points is greater than the expected value of feature points, adjusting the fixed threshold according to the number of feature points in each valid sub-region of the low-resolution image to determine a threshold for each valid sub-region; If the sum of the number of feature points is less than or equal to the expected value of feature points, the thresholds of all valid sub-regions of the low-resolution image are set to the fixed threshold.

10. The feature point extraction device according to claim 9, characterized in that: The fixed threshold is a threshold used to extract feature points from the low-resolution image.

11. A feature point extraction device, characterized in that: include: at least one processor; At least one memory stores a computer program, and when the computer program is executed by the at least one processor, the feature point extraction method according to any one of claims 1 to 5 is implemented.

12. A computer-readable storage medium comprising instructions, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 5.

13. A computer program product comprising instructions, which, when run on a computer, causes the computer to perform the method according to any one of claims 1 to 5.

Images