Image feature point matching method and device

Abstract

The present application provides an image feature point matching method and device, which relates to the field of image processing and is used to improve the matching efficiency of feature points. The method comprises: storing N feature point sets in a first image in a first buffer in a preset order, wherein the first image comprises N blocks, and each block corresponds to a feature point set; for a feature point to be matched in a second image, determining M blocks from the N blocks of the first image according to the position of the feature point to be matched in the second image, wherein the second image comprises N blocks; obtaining a feature point set of each of the M blocks from the first buffer; matching the feature point set of each of the M blocks block by block according to the feature point to be matched, so as to match at least one matching feature point from the feature point set of the M blocks.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image feature point matching method and device. Background Art

[0002] Image feature point matching is widely used in multiple application scenarios such as image detection, image recognition, and image tracking. Image information is described and characterized by extracting image feature points, and then the similarity of two images is calculated by feature point matching, providing a basis for subsequent image tracking, image detection, and image recognition applications. In the scenario of feature information description, it is a common method to describe the image by feature points and then perform image matching based on feature points.

[0003] Traditional image feature point matching methods usually use a brute force search method to perform brute force exhaustive matching on the entire feature point set to cover all matching feasibility, that is, for a feature point to be matched in the reference image, all feature points in the current image are used for matching one by one, and then the best feature point is selected as the matching feature point that matches the feature point to be matched. However, the above method requires a long matching time and a large storage space, so the matching efficiency is low. Summary of the invention

[0004] The embodiments of the present application provide an image feature point matching method and device for improving the efficiency of feature point matching. To achieve the above purpose, the embodiments of the present application adopt the following technical solutions:

[0005] In a first aspect, a method for matching image feature points is provided, the method comprising: storing N feature point sets in a first image in a first buffer in a preset order, the first image comprising N blocks, each block corresponding to a feature point set, and N being a positive integer; for feature points to be matched in a second image, determining M blocks from the N blocks of the first image according to positions of the feature points to be matched in the second image, the second image comprising N blocks, and M being an integer greater than or equal to 1; obtaining a feature point set of each of the M blocks from the first buffer; matching the feature point set of each of the M blocks block by block according to the feature points to be matched, so as to match at least one matching feature point from the feature point sets of the M blocks. In the above technical solution, the N feature point sets in the first image are stored in the first buffer in a preset order, which can improve the efficiency of subsequently obtaining the feature point set of each block in the M blocks. At the same time, M blocks are determined from the N blocks of the first image according to the positions of the feature points to be matched in the second image, which can effectively reduce the matching range. Subsequently, when the feature point set of each block in the M blocks is matched block by block according to the feature points to be matched, the matching can be performed in units of the feature point set of one block. The feature point sets of the M blocks are matched block by block through pipeline operations, thereby reducing the memory consumption and matching time in the feature point matching process, thereby improving the efficiency of image feature point matching.

[0006] In a possible implementation of the first aspect, the N blocks of the first image correspond one-to-one to the N blocks of the second image, and the corresponding two blocks have the same shape and size. In the above possible implementation, the accuracy of determining the M blocks in the first image can be improved, that is, the accuracy of determining the effective matching range can be improved, thereby reducing unnecessary feature point matching operations.

[0007] In a possible implementation of the first aspect, before storing the N feature point sets in the first image in the first buffer in a specified order, the method further includes: dividing the first image into N blocks according to the size of the first image and a preset division rule, and determining the feature point set of each block in the N blocks of the first image; wherein the preset division rule includes at least one of the following rules: division that fits the edge of the image, division that does not fit the edge of the image, division by area, division by regular shape, and division by irregular shape. In the above possible implementation, the flexibility and diversity of dividing the first image into N blocks can be improved.

[0008] In a possible implementation of the first aspect, the method further includes: determining a position index corresponding to the feature point set of each block according to the number of feature points included in each feature point set in the N feature point sets and a preset order, the position index including a starting position index and / or an ending position index, and the position index corresponding to the feature point set of a block is used to index the position of the feature point set of the block in the first buffer. In the above possible implementation, a simple and effective method for indicating the position of each feature point set is provided, thereby improving the efficiency of subsequently obtaining the feature point sets of M blocks.

[0009] In a possible implementation of the first aspect, M blocks are determined from N blocks of the first image according to the position of the feature point to be matched in the second image, including: determining a matching area according to the position of the feature point to be matched in the second image and a matching radius, or according to the position of the feature point to be matched in the second image, a matching radius and a preset offset; and selecting M blocks located in the matching area from the N blocks of the first image according to a matching block selection strategy. In the above possible implementation, the accuracy of determining the M blocks in the first image can be improved, that is, the accuracy of determining the effective matching range can be improved, thereby reducing unnecessary feature point matching operations.

[0010] In a possible implementation of the first aspect, matching the feature point set of each block in M ​​blocks block by block according to the feature point to be matched, so as to match at least one matching feature point from the feature point set of the M blocks, includes: for the M blocks, determining the matching information between each feature point in the feature point set of each block and the feature point to be matched block by block; and determining the feature point in the feature point set of each block that meets the preset conditions as the matching feature point according to the matching information. In the above possible implementation, an effective way of matching at least one matching feature point from the feature point set of the M blocks is provided.

[0011] In a possible implementation manner of the first aspect, the matching information includes: a descriptor distance and / or a matching angle difference. In the above possible implementation manner, a simple and effective matching information is provided.

[0012] In a possible implementation of the first aspect, a feature point that meets a preset condition means that matching information of the feature point meets at least one of the following: best descriptor distance, second-best descriptor distance, best matching angle difference, or second-best matching angle. In the above possible implementation, an effective method for determining matching feature points is provided.

[0013] In a possible implementation of the first aspect, the feature point set of each block in M ​​blocks is matched block by block according to the feature points to be matched, so as to match at least one matching feature point from the feature point set of M blocks, and further includes: filtering the matching feature points matched from the feature point set of each block according to the matching information and a preset screening strategy to obtain at least one matching feature point; wherein the preset screening strategy includes at least one of the following strategies: repeated matching screening, angle difference histogram screening, descriptor distance screening, or block matching statistical screening. In the above possible implementation, the flexibility and diversity of screening at least one matching feature point can be improved.

[0014] In a possible implementation of the first aspect, when a matching feature point in the first image corresponds to a first feature point to be matched and a second feature point to be matched in the second image, the method further includes: determining that the matching feature point in the first image corresponds to a feature point to be matched with the best matching information among the first feature point to be matched and the second feature point to be matched. In the above possible implementation, the problem of the same matching feature point corresponding to at least two feature points to be matched can be avoided, thereby solving the problem of feature point mismatching.

[0015] In a second aspect, an image feature point matching device is provided, the device comprising: a storage unit, used to store N feature point sets in a first image in a first buffer in a preset order, the first image comprising N blocks, each block corresponding to a feature point set, and N being a positive integer; a determination unit, used to determine M blocks from the N blocks of the first image according to the positions of the feature points to be matched in the second image for the feature points to be matched of the second image, the second image comprising N blocks, and M being an integer greater than or equal to 1; an acquisition unit, used to acquire the feature point set of each block in the M blocks from the first buffer; and a matching unit, used to match the feature point set of each block in the M blocks block by block according to the feature points to be matched, to obtain at least one matching feature point.

[0016] In a possible implementation manner of the second aspect, the N blocks of the first image correspond one-to-one to the N blocks of the second image, and the corresponding two blocks have the same shape and size.

[0017] In a possible implementation of the second aspect, the determination unit is further used to: divide the first image into N blocks according to the size of the first image and a preset division rule, and determine a set of feature points for each block in the N blocks of the first image; wherein the preset division rule includes at least one of the following rules: division that fits the edge of the image, division that does not fit the edge of the image, division by area, division by regular shape, and division by irregular shape.

[0018] In a possible implementation of the second aspect, the determination unit is further used to: determine the position index corresponding to the feature point set of each block based on the number of feature points included in each feature point set in the N feature point sets and a preset order, the position index including a starting position index and / or an ending position index, and the position index corresponding to the feature point set of a block is used to index the position of the feature point set of the block in the first buffer.

[0019] In a possible implementation manner of the second aspect, the determination unit is further used to: determine the matching area according to the position of the feature point to be matched in the second image and the matching radius, or according to the position of the feature point to be matched in the second image, the matching radius and a preset offset; and select M blocks located within the matching area from the N blocks of the first image according to the matching block selection strategy.

[0020] In a possible implementation of the second aspect, the matching unit is further used to: for the M blocks, determine the matching information between each feature point in the feature point set of each block and the feature point to be matched block by block; and based on the matching information, determine the feature point in the feature point set of each block that meets the preset conditions as the matching feature point.

[0021] In a possible implementation manner of the second aspect, the matching information includes: a descriptor distance and / or a matching angle difference.

[0022] In a possible implementation manner of the second aspect, the feature point that meets the preset condition means that the matching information of the feature point satisfies at least one of the following: the best descriptor distance, the second best descriptor distance, the best matching angle difference, or the second best matching angle.

[0023] In a possible implementation of the second aspect, the matching unit is further used to: filter the matching feature points matched by the feature point set of each block according to the matching information and the preset filtering strategy to obtain at least one matching feature point; wherein the preset filtering strategy includes at least one of the following strategies: repeated matching filtering, angle difference histogram filtering, descriptor distance filtering, or block matching statistical filtering.

[0024] In a possible implementation of the second aspect, when a matching feature point in the first image corresponds to a first feature point to be matched and a second feature point to be matched in the second image, the determination unit is further used to determine that the matching feature point in the first image corresponds to the feature point to be matched with the best matching information among the first feature point to be matched and the second feature point to be matched.

[0025] In a third aspect, a device for matching image feature points is provided, the device comprising: an input interface, an operator, and a first buffer; wherein the input interface is used to receive N feature point sets in a first image, the first image comprising N blocks, each block corresponding to a feature point set, and N being a positive integer; the operator is used to store the N feature point sets in the first image in the first buffer in a preset order; the operator is also used to determine M blocks from the N blocks of the first image according to the position of the feature points to be matched in the second image, the second image comprising N blocks, and M being an integer greater than or equal to 1, for the feature points to be matched in the second image; the feature point set of each block in the M blocks is obtained from the first buffer; the operator is also used to match the feature point set of each block in the M blocks block by block according to the feature points to be matched, so as to match at least one matching feature point from the feature point set of the M blocks. Optionally, the device further comprises: an output interface, used to output at least one matching feature point.

[0026] In a possible implementation manner of the third aspect, the N blocks of the first image correspond one-to-one to the N blocks of the second image, and the corresponding two blocks have the same shape and size.

[0027] In a possible implementation of the third aspect, the operator is further used to: divide the first image into N blocks according to the size of the first image and a preset division rule, and determine a set of feature points for each block in the N blocks of the first image; wherein the preset division rule includes at least one of the following rules: division that fits the edge of the image, division that does not fit the edge of the image, division by area, division by regular shape, and division by irregular shape.

[0028] In a possible implementation of the third aspect, the operator is also used to: determine the position index corresponding to the feature point set of each block based on the number of feature points included in each feature point set in the N feature point sets and a preset order, the position index including a starting position index and / or an ending position index, and the position index corresponding to the feature point set of a block is used to index the position of the feature point set of the block in the first buffer.

[0029] In a possible implementation of the third aspect, the operator is further used to: determine the matching area according to the position of the feature point to be matched in the second image and the matching radius, or according to the position of the feature point to be matched in the second image, the matching radius and a preset offset; and select M blocks located within the matching area from the N blocks of the first image according to the matching block selection strategy.

[0030] In a possible implementation of the third aspect, the operator is also used to: for M blocks, determine the matching information between each feature point in the feature point set of each block and the feature point to be matched block by block; based on the matching information, determine the feature point in the feature point set of each block that meets the preset conditions as the matching feature point.

[0031] In a possible implementation manner of the third aspect, the matching information includes: a descriptor distance and / or a matching angle difference.

[0032] In a possible implementation manner of the third aspect, the feature point that meets the preset condition means that the matching information of the feature point satisfies at least one of the following: the best descriptor distance, the second best descriptor distance, the best matching angle difference, or the second best matching angle.

[0033] In a possible implementation of the third aspect, the operator is further used to: filter the matching feature points matched by the feature point set of each block according to the matching information and the preset filtering strategy to obtain at least one matching feature point; wherein the preset filtering strategy includes at least one of the following strategies: repeated matching filtering, angle difference histogram filtering, descriptor distance filtering, or block matching statistical filtering.

[0034] In a possible implementation of the third aspect, when a matching feature point in the first image corresponds to a first feature point to be matched and a second feature point to be matched in the second image, the operator is also used to: determine that the matching feature point in the first image corresponds to the feature point to be matched with the best matching information among the first feature point to be matched and the second feature point to be matched.

[0035] In another aspect of the present application, a readable storage medium is provided, in which instructions are stored. When the readable storage medium is run on a device, the device executes the image feature point matching method provided by the first aspect or any possible implementation of the first aspect.

[0036] In another aspect of the present application, a computer program product is provided. When the computer program product is run on a computer, the computer executes the image feature point matching method provided by the first aspect or any possible implementation manner of the first aspect.

[0037] It can be understood that the device, computer storage medium or computer program product of any of the image feature point matching methods provided above are used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0039] Figure 2 A schematic diagram of a method for matching image feature points provided in an embodiment of the present application Figure 1 ;

[0040] Figure 3 A storage diagram of a plurality of feature point sets provided in an embodiment of the present application Figure 1 ;

[0041] Figure 4 A storage diagram of a plurality of feature point sets provided in an embodiment of the present application Figure 2 ;

[0042] Figure 5 A schematic diagram of a method for matching image feature points provided in an embodiment of the present application Figure 2 ;

[0043] Figure 6 A schematic diagram of a regular shape and an irregular shape provided in an embodiment of the present application;

[0044] Figure 7 A schematic diagram of the distribution of N blocks provided in an embodiment of the present application;

[0045] Figure 8 A schematic diagram of the distribution of M blocks provided in an embodiment of the present application;

[0046] Fig. 9 A schematic diagram of the structure of an image feature point matching device provided in an embodiment of the present application Figure 1 ;

[0047] Fig.10 A schematic diagram of the structure of an image feature point matching device provided in an embodiment of the present application Figure 2 . DETAILED DESCRIPTION

[0048] In the present application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, ab, ac, bc or abc, where a, b and c can be single or multiple. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, in the embodiments of the present application, the words "first", "second" and the like do not limit the quantity and execution order.

[0049] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0050] Figure 1 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application, the image processing device can be a mobile phone, a tablet computer, a computer, a laptop computer, a video camera, a camera, a wearable device, a vehicle-mounted device, or a terminal device, etc. For the convenience of description, the above-mentioned devices are collectively referred to as image processing devices in this application. The embodiment of the present application is described by taking the image processing device as a mobile phone as an example, and the mobile phone includes: a memory 101, a processor 102, a sensor component 103, a multimedia component 104, an audio component 105, and a power component 106, etc.

[0051] Combine the following Figure 1 A detailed introduction to the various components of the mobile phone:

[0052] The memory 101 can be used to store data, software programs and modules; it mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function, such as a sound playback function, an image playback function, etc.; the data storage area can store data created according to the use of the mobile phone, such as audio data, image data, phone book, etc. In addition, the mobile phone may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0053] The processor 102 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire device. By running or executing software programs and / or modules stored in the memory 101, and calling data stored in the memory 101, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. In the embodiment of the present application, the processor 102 can integrate an application processor (Application Processor, AP) and an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC); wherein the AP mainly processes the operating system, user interface and application program, etc., and the ASIC is a dedicated integrated chip developed specifically for a certain function, such as ASIC can be used to process the related processes of image feature point matching in the embodiment of the present application. It can be understood that the above-mentioned ASIC may not be integrated into the processor 102.

[0054] In addition, the processor 102 may further include other hardware circuits or accelerators, such as field programmable gate arrays or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor 102 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc.

[0055] The sensor component 103 includes one or more sensors for providing various aspects of status evaluation for the mobile phone. The sensor component 103 may include an optical sensor, such as a CMOS or CCD image sensor, for detecting the distance between an external object and the mobile phone, or used in imaging applications, i.e., as a component of a camera or a camera head. In addition, the sensor component 103 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor, and the acceleration / deceleration, orientation, opening / closing state of the mobile phone, the relative positioning of components, or the temperature change of the mobile phone, etc., can be detected by the sensor component 103.

[0056] The multimedia component 104 provides a screen of an output interface between the mobile phone and the user, and the screen can be a touch panel, and when the screen is a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide and gestures on the touch panel. The touch sensor can not only sense the boundary of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In addition, the multimedia component 104 also includes at least one camera, for example, the multimedia component 104 includes a front camera and / or a rear camera. When the mobile phone is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0057] The audio component 105 can provide an audio interface between the user and the mobile phone. For example, the audio component 105 can include an audio circuit, a speaker, and a microphone. The audio circuit can convert the received audio data into an electrical signal and transmit it to the speaker, which converts it into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit and converted into audio data, and then outputs the audio data to be sent to, for example, another mobile phone, or outputs the audio data to the processor 102 for further processing.

[0058] The power supply component 106 is used to provide power to various components of the mobile phone. The power supply component 106 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power in the mobile phone.

[0059] Although not shown, the mobile phone may also include a wireless fidelity (WiFi) module, a Bluetooth module, etc., which will not be described in detail in the present embodiment. Those skilled in the art will understand that Figure 1 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0060] Figure 2 A flowchart of an image feature point matching method provided in an embodiment of the present application is provided. The execution subject of the method may be an image processing device, specifically a unit having processing and storage functions in the image processing device, such as the above Figure 1 ASIC in the processor 102 shown. Figure 2 , the method includes the following steps.

[0061] S201: Store N feature point sets in a first image in a first buffer according to a preset order, where the first image includes N blocks, and each block corresponds to a feature point set.

[0062] The first image may include N blocks, N is a positive integer, and the sizes and shapes of the N blocks in the first image may be the same or different. Each block corresponds to a feature point set, which may refer to a set of feature points in the block. The first image includes N blocks, each block corresponds to a feature point set, so that the first image has N feature point sets, that is, the N feature point sets correspond one-to-one to the N blocks of the first image.

[0063] In addition, the preset order can be set in advance by those skilled in the art according to actual needs, for example, the preset order includes row order, column order, Z-order, X-order or other order. Figure 3 As shown, the N feature point sets in the first image are stored in the first buffer in row order, column order and Z-order respectively. It should be noted that, Figure 3 In the description, it is taken that the first image includes 16 blocks (ie, N=16), and the sizes and shapes of the 16 blocks are the same, and the feature point set of each block can be represented by the label in the block.

[0064] Specifically, when the N feature point sets in the first image are stored in the first buffer in a preset order, the N feature point sets may be stored in the first buffer continuously in the preset order, or the N feature point sets may be stored in the first buffer non-continuously in the preset order; wherein the feature points in the same feature point set are stored continuously. Figure 3 Taking the row-by-row storage as shown in the example, the feature point sets of 16 blocks in the first image can be stored in the first buffer as follows: Figure 4 As shown, Figure 4 (a) in the figure takes the continuous storage method of 16 feature point sets as an example. Figure 4 (b) in FIG. 1 is used as an example to explain the non-continuous storage method of a set of 16 feature points.

[0065] Optionally, when storing the N feature point sets in the first image in the first buffer, the position index corresponding to each feature point set may also be determined. Specifically, based on the number of feature points included in each feature point set in the N feature point sets and the preset order, the position index corresponding to the feature point set of each block is determined, and the position index may include a starting position index (used to indicate the starting position in the first buffer) and / or an ending position index (used to indicate the ending position in the first buffer). Optionally, if the position index corresponding to the feature point set of a block includes a starting position index, the starting position of the feature point set of the block in the first buffer can be determined by the starting position index, and the ending position can be calculated by the starting position index and the number of feature points included in the feature point set; if the position index corresponding to the feature point set of a block includes an ending position index, the starting position of the feature point set of the block in the first buffer can be calculated by the ending position index and the number of feature points included in the feature point set, and the ending position can be determined by the ending position index; if the position index corresponding to the feature point set of a block includes a starting position index and an ending position index, the starting position of the feature point set in the first buffer can be determined by the starting position index, and the ending position can be determined by the ending position index.

[0066] Exemplarily, the number of feature points included in the feature point set of a certain block is NUM. If the starting position index corresponding to the feature point set is Add_k, the ending position index of the feature point set in the first buffer is (Add_k+NUM-1); if the ending position index corresponding to the feature point set is Add_w, the starting position index of the feature point set in the first buffer is (Add_w-NUM+1). Exemplarily, if the first image includes m blocks and the number of feature points in the m blocks is NUM{n_1, n_2, ..., n_m) as an example, and they are stored in row order, then relative to the starting address of the first buffer, the starting address of the m blocks can be ADDR_start{0, n_1, n_1+n_2, ..., n_1+n_2+...+n_m-1}, and the ending address of the m blocks can be ADDR_end{-1+n_1, -1+n_1+n_2, ..., -1+sum(NUM)}.

[0067] Optionally, after determining the position index corresponding to each feature point set, the image processing device may also store the position index corresponding to the feature point set of each block in the first image. For example, the position index corresponding to the feature point set of each block in the first image stored by the image processing device includes: a block identifier and a position index, and the block identifier is used to indicate the block corresponding to the position index.

[0068] Typically, feature points are extracted from the first image, and the extracted feature points are stored in disorder. For example, feature points of different blocks may be stored adjacently, and multiple feature points of the same block are stored in a dispersed manner, which will result in a long search time and addressing calculation time when reading multiple feature points of the same block later. In the embodiment of the present application, N feature point sets are stored in an orderly manner, that is, multiple feature points belonging to the same block are stored continuously, and feature point sets of different blocks are stored in a certain order, thereby improving the efficiency of subsequent feature point reading.

[0069] Further, before S201, see Figure 5 The method may further include: S200.

[0070] S200: Divide the first image into N blocks according to the size of the first image and a preset division rule, and determine a set of feature points of each of the N blocks of the first image.

[0071] Among them, the preset division rules can be set in advance by technical personnel in this field according to actual conditions. For example, the preset division rules may include at least one of the following rules: division that fits the image edge, division that does not fit the image edge, uniform division, uneven division, division by area, division by regular shape, and division by irregular shape. Division that fits the image edge may refer to the division of the complete image, in which case the size of the image itself is used as the size of the divided area; division that does not fit the image edge refers to the division of areas other than the edge area in the image, and the edge area may include areas corresponding to one or more edges of the image, in which case the size of the divided area is smaller than the size of the image itself; uniform division may refer to the size and shape of the divided blocks being the same; uneven division may refer to the size and shape of the divided blocks being different; division by area may refer to different divisions of different areas in the image, such as different shapes or sizes of blocks corresponding to different areas; regular shape division refers to the shapes of the N blocks obtained by dividing the image being regular shapes, such as Figure 6 The rectangle, square, circle, hexagon, rhombus or triangle shown in (a) in FIG. 1 are irregular shapes. Irregular shape division means that the shapes of the N blocks obtained by dividing the image are irregular shapes, such as Figure 6 An ellipse, a cross, a trapezoid, an L-shape, an X-shape or a ring as shown in (b) in FIG.

[0072] For example, Figure 7 The above preset division rules are illustrated by way of example. Figure 7 The preset division rule corresponding to (a) in the figure may be: division in line with the image edge + uniform division; Figure 7 The preset division rule corresponding to (b) in the figure may be: division that does not fit the image edge + uniform division + regular shape division; Figure 7 The preset division rule corresponding to (c) in the figure may be: division according to image edge + division according to region + division according to regular shape; Figure 7 The preset division rule corresponding to (d) in the figure may be: division according to image edge + division according to regular shape.

[0073] It should be noted that the above-mentioned division rules are only exemplary, and other division rules may be included in actual application. Figure 6 The regular shapes and irregular shapes shown are also exemplary and do not constitute limitations on the embodiments of the present application.

[0074] In addition, determining the feature point set of each block in the N blocks of the first image may include: the image processing device extracts the feature points in the first image, and determines multiple feature points belonging to the same block according to the position of each feature point and the position range of each block in the N blocks of the first image, that is, obtaining the feature point set of each block. Among them, the feature point extraction can adopt any one of the following algorithms: Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Feature (SURF), Oriented FAST and Rotated BRIEF (ORB), Difference of Gaussian (DOG), Learned Invariant Feature Transform (LIFT), etc. The feature point extraction of the first frame image can be feature points extracted by multi-layer scaling at different scaling ratios, and the corresponding coordinates can be based on the mapping back to the first image. The common information recorded in the extracted feature points may include, but is not limited to, coordinates (x, y), the scaling layer (octave) where the current feature point is located, response score value (response), scaling ratio or scaling factor (scale), angle value (angle), descriptor (descriptor), or index (index) and other information. In addition, the extracted feature points can be used directly or screened using different screening methods, such as octree screening, etc., which is not specifically limited in the embodiments of the present application.

[0075] Furthermore, when determining the feature point set of each block in the first image, the image processing device may also count and store block statistical information of each block. For example, the block statistical information of a block may include one or more of the following information: a block identifier (used to uniquely identify a block, such as a block serial number, etc.), the number of feature points, response score information (such as a maximum response score value, an average response score value, or a minimum response score value, etc.), and feature point distribution information (such as an octave-based distribution, a block center point coordinate-based distribution, a distribution map of response score value distribution, etc.).

[0076] It should be noted that the above process of dividing the first image into N blocks and the process of extracting the feature points of the first image may be in any order. When the first image is divided into N blocks first and then the feature points of the first image are extracted, the block statistical information of each block can be counted in the process of extracting the feature points, so that the image processing device does not need to wait until all the feature points of the first image are extracted before performing statistics, thereby reducing the waiting time and the time for re-reading the feature points of the first image.

[0077] S202: For a feature point to be matched in the second image, select M blocks from N blocks in the first image according to the position of the feature point to be matched in the second image, where M is an integer greater than or equal to 1.

[0078] The size of the second image is the same as that of the first image, and the second image and the first image may have the same or similar features. For example, the first image and the second image may be two adjacent frames in the same video stream, or two frames of images of the same and similar size. Optionally, the second image includes N blocks and corresponds one-to-one with the N blocks of the first image, and the shapes, sizes and positions of the two corresponding blocks may be the same. Each block in the second image may have multiple feature points to be matched, and the multiple feature points to be matched in each block may be multiple feature points extracted from the block.

[0079] In addition, the position of the feature point to be matched in the second image corresponds to its corresponding position in the first image, that is, it can be considered that the position of the feature point to be matched in the second image is the same as its corresponding position in the first image. For example, the first image and the second image are represented by the same coordinate system, and the position of the feature point to be matched is the same in the two coordinate systems.

[0080] Specifically, when the positions of the same or similar features in the first image and the second image differ slightly or do not change, the matching area can be determined according to the position of the feature point to be matched in the second image and the matching radius. For example, in the first image, the position of the feature point to be matched in the first image is taken as the center of the circle (or referred to as the selected center point) and the matching radius is taken as the radius, and the circular area determined is used as the matching area. Alternatively, when the positions of the same or similar features in the first image and the second image differ significantly, the initial matching area can also be determined according to the position of the feature point to be matched in the second image, the matching radius and the preset offset. For example, in the first image, the position of the feature point to be matched in the first image after being offset by the preset offset is taken as the center of the circle and the matching radius is taken as the radius, and the circular area determined is used as the matching area. Afterwards, according to the matching block selection strategy, M blocks located in the matching area are selected from the N blocks of the first image, for example, the block identifier corresponding to each of the M blocks is determined. Optionally, the matching block selection strategy can be a rectangular block set, an X-shaped block set, a cross-shaped block set or a diamond block set.

[0081] It should be noted that the above matching radius, preset offset, and matching block selection strategy can be set in advance by a person skilled in the art according to actual conditions or dynamically set during the matching process, such as according to the shape, size, and the above preset division rules of the block where the feature points to be matched are located; or according to relevant experience, or using adaptive configurations such as matching algorithms, etc. For blocks or regions where different feature points to be matched are located, the same matching radius, preset offset, or matching block selection strategy can be set, or different matching radius, preset offset, or matching block selection strategy can be set, and the embodiments of the present application do not specifically limit this.

[0082] Exemplary, combined Figure 3 Assume that the position of the feature point to be matched in the first image is the center of the circle and the matching radius is the radius. The matching area is determined as follows: Figure 8 As shown, when the above matching block selection strategy is a rectangular block set, the selected M blocks are as follows Figure 8 As shown in (a) in FIG. 1 ; when the matching block selection strategy is a cross-shaped block set, the selected M blocks are as follows: Figure 8 As shown in (b) in FIG. 1 ; when the matching block selection strategy is an X-type block set, the selected M blocks are as follows Figure 8 As shown in (c) in . Figure 4 The center of the circle is located at the block numbered 10 in the first image as an example for explanation. Figure 8 The circle in the area represents the matching area mentioned above.

[0083] S203: Acquire a feature point set of each of the M blocks from the first buffer.

[0084] Specifically, the image processing device obtains the position index corresponding to the feature point set of each block in the M blocks from the storage space storing the position index corresponding to the feature point set of each block according to the block identifier of each block in the M blocks. Furthermore, according to the position index corresponding to the feature point set of each block in the M blocks, the feature point set of each block is obtained from the first buffer. Optionally, the feature point set of a block can be read at one time, or the feature point set of a block can be read one by one continuously; or, multiple feature points can be read in parallel from the feature point set of a block at one time, or feature points of different blocks can be read in parallel from the feature point sets of multiple blocks in the M blocks, etc. (when reading in parallel, subsequent steps can also be executed in parallel).

[0085] S204: Matching the feature point set of each block in the M blocks one by one according to the feature point to be matched, so as to match at least one matching feature point from the feature point set of the M blocks.

[0086] For M blocks, the feature point to be matched can be used to match the feature point set of each block in the M blocks one by one (i.e., matching is performed in units of blocks), and then at least one matching feature point is matched from the feature point set of the M blocks, and each matching feature point of the at least one matching feature point obtained by matching can form a matching pair with the feature point to be matched. In the matching process, the feature point set of the block and the feature point to be matched can be matched using the best matcher algorithm, the KNN matching algorithm, or the left-right bidirectional matching algorithm. Among them, the best matching algorithm can refer to selecting a most similar feature point from the feature point set of the block as the matching feature point of the feature point to be matched for the feature point to be matched. The KNN (n=2) matching algorithm can refer to selecting n (for example, n=2) most similar matching points from the feature point set of the block as the matching feature points of the feature point to be matched for the feature point to be matched. The left-right bidirectional matching algorithm may refer to selecting a most similar feature point (expressed as feature point Q(x, y)) from the feature point set of the block for the feature point to be matched (expressed as feature point P(x, y)), and at the same time, for the feature point Q(x, y), using the same method to select a most similar feature point (expressed as feature point P'(x, y)) from the first image, if P'(x, y) is consistent with P(x, y), then the feature point Q(x, y) is determined to be the matching feature point of the feature point to be matched P(x, y), otherwise the matching is invalid; the left-right bidirectional matching algorithm needs to perform the matching process twice, so the matching cost is relatively large. Among the above three matching algorithms, the number of at least one matching feature point obtained by the best matching algorithm is the same as the number of M blocks, the number of at least one matching feature point obtained by the KNN matching algorithm is greater than (at this time, n≥2) or equal to (at this time, n=1) the number of M blocks, and the number of at least one matching feature point obtained by the left-right bidirectional matching algorithm is less than or equal to the number of M blocks.

[0087] It should be noted that the image processing device may also use other different matching algorithms during matching. The above only uses the best matching algorithm, KNN matching algorithm and left-right bidirectional matching algorithm as examples for illustration, and does not constitute a limitation on the embodiments of the present application.

[0088] Furthermore, for the feature point set of each block in the M blocks, during the matching process, the image processing device may also determine the matching information between each feature point in the feature point set and the feature point to be matched, and the matching information may include: descriptor distance and / or matching angle difference. Afterwards, the image processing device determines the feature point in the feature point set that meets the preset conditions as the matching feature point based on the matching information, where the feature point that meets the preset conditions may refer to the matching information of the feature point that meets at least one of the following: the best descriptor distance, the second best descriptor distance, the best matching angle difference, or the second best matching angle.

[0089] The descriptor distance may be Manhattan distance, Euclidean distance, Hamming distance, etc. Taking the descriptor des1 (x1, x2, x3, ..., xn) and the descriptor des2 (y1, y2, y3, ..., yn) of two matching feature points as an example, the Manhattan distance (D1) may be calculated as shown in the following formula (1), the Euclidean distance (D2) may be calculated as shown in the following formula (2), and the Hamming distance (D3) may be calculated as shown in the following formula (3). The XOR in the above formula (3) is an exclusive OR operator of a logical operation.

[0090] D1=|x1-y1|+|x2-y2|+…+|xn-yn| (1)

[0091] D2=((x1-y1)^2+(x2-y2)^2+…+(xn-yn)^2)^0.5 (2)

[0092] D3=XOR(x1,y1)+XOR(x2,y2)+…+XOR(xn,yn) (3)

[0093] In addition, the best descriptor distance can be used to indicate the distance between the most similar descriptors of two feature points. For example, for the above-mentioned Manhattan distance, Euclidean distance or Hamming distance, the best descriptor distance can refer to the smallest non-negative value among all descriptor distances calculated, and the corresponding matching feature point can be called the best matching feature point. Correspondingly, the second-best descriptor distance can refer to the descriptor distance that is second only to the best descriptor distance among all descriptor distances corresponding to the feature point set, and the corresponding matching feature point can be called the second-best matching feature point.

[0094] The matching angle difference may refer to the angle difference between two matching feature points, and the angle of each feature point can be obtained when the feature point is extracted. The best matching angle difference may refer to the minimum value among all angle differences corresponding to the feature point set, and the corresponding matching feature point may be called the best matching feature point; the second best matching angle may refer to the matching angle difference among all angle differences corresponding to the feature point set that is second only to the best matching angle difference, and the corresponding matching feature point may be called the second best matching feature point.

[0095] Optionally, during the matching process, the image processing device may also store the feature point matching results (ie, the above-mentioned matching pair information), the matching status of the feature points and related statistical information, etc.

[0096] Specifically, in the process of matching the feature point set of each block in the M blocks, the image processing device may store the feature point matching result in the second buffer, and the second buffer may include: one or more of the index number of the feature point to be matched, the index number of the best matching feature point, the best descriptor distance, the best matching angle difference, etc. Optionally, the second buffer may also include: the index number of the second best matching feature point, the second best descriptor distance, the second best matching angle difference, etc.

[0097] For each feature point in the feature point set, the image processing device can also store the matching status of the feature point in a status register, which may include: the index number of the feature point, the index number of the block where the feature point is located, the index number of the feature point to be matched that matches the feature point, the index number of the block where the feature point to be matched that matches the feature point is located, one or more of the information such as the best descriptor distance, the best matching angle difference, etc.

[0098] For the feature point set, the image processing device can also store the statistical information of the matching angle difference corresponding to the feature point set (e.g., angle difference histogram) in the angle difference statistical register. For example, the angle difference statistical register evenly divides the [0, 360] degree angle into m (m is a positive integer) intervals, each interval is 360 / m degrees, and counts the number of matching angle differences belonging to each interval range in the m intervals (e.g., m=12, i.e., evenly divided into 12 intervals, each interval is 30 degrees, and the m interval ranges can be [0, 30), [30, 60), ..., [330, 360)). If a matching angle difference is a negative value, the matching angle difference can be added with 360 to become a positive number. For example, if the matching angle difference between a certain matching feature point and the feature point to be matched is 45 degrees, then 45 degrees belongs to the interval [30, 60), the number of feature points corresponding to [30, 60) is increased by 1, and the matching feature point belongs to the interval [30, 60), which can be represented by the angle interval index (for example, serial number) corresponding to [30, 60), such as 45 / (360 / 12) rounded up = 2, that is, the matching feature point belongs to the interval with angle serial number 2.

[0099] For each of the M blocks, the image processing device may also store the block matching statistics of the block in a block statistics register, which is mainly used to store block information and block matching results. The block statistics register may include: the index number of the block where the feature point to be matched is located, the index number of the block in the first image, the number of valid matching points, the mean / maximum / minimum value of the best descriptor distance corresponding to the matching feature points in the block, etc. In practical applications, the image processing device may also add or delete various information in the block statistics register according to the needs of subsequent applications, and the embodiments of the present application do not specifically limit this.

[0100] Furthermore, for all the feature points to be matched in the second image, the above steps may be used to obtain matching feature points corresponding to each feature point to be matched, thereby achieving feature point matching between the first image and the second image.

[0101] Optionally, when a matching feature point of a block in the first image corresponds to both the first feature point to be matched and the second feature point to be matched in the second image, it can be determined that the matching feature point in the first image corresponds to the feature point to be matched with the best matching information among the first feature point to be matched and the second feature point to be matched, that is, the matching feature point is used as the matching feature point of the feature point to be matched with the best matching information.

[0102] In actual application, the image processing device can be processed in an assembly line manner, and the above-mentioned second buffer also stores corresponding information in an assembly line manner. The above-mentioned status register will also record the matching status of each feature point. The embodiment of the present application can use the following method to deal with the above-mentioned repeated matching problem, thereby avoiding further modification of the memory that has been written.

[0103] Specifically, if the best matching feature point (expressed as Q(x, y)) is obtained after the feature point to be matched (expressed as P(x, y)), and Q(x, y) is marked in the above-mentioned status register as not yet matched, then the match between the feature point to be matched P(x, y) and Q(x, y) is valid, and the relevant information of the match between P(x, y) and Q(x, y) is written into the above-mentioned status register. If the best matching feature point Q(x, y) and the best descriptor distance (expressed as dist0) are obtained after the feature point P(x, y) to be matched is matched, Q(x, y) is marked as matched in the above-mentioned status register, and the matched point is the feature point K(x, y)\the best descriptor distance is dist1, then the best matching distances dist1 and dist0 are compared, if dist0>=dist1, P(x, y) and Q(x, y) are not optimal and are discarded; if dist0<dist1, P(x, y) and Q(x, y) are better and need to be retained, the best matching point corresponding to the feature point Q(x, y) in the above-mentioned status register is updated to P(x, y), and the best descriptor is updated to dist0 and other statistical information, and the relevant information of the matching between P(x, y) and Q(x, y) is written into the above-mentioned status register.

[0104] In addition, after matching all feature points to be matched in the second image, the image processing device may also update the above-mentioned status register and angle difference statistics register to avoid the problem of register information being inconsistent with the actual situation due to processing the above-mentioned repeated matching problem.

[0105] Furthermore, in S204, the image processing device may also filter the matching feature points in each block of the first image according to the matching information and a preset filtering strategy to obtain at least one matching feature point. The preset filtering strategy includes at least one of the following strategies: repeated matching filtering, angle difference histogram filtering, block descriptor distance filtering, or block matching statistics filtering.

[0106] For all matching pairs obtained after matching the first image and the second image (i.e., matching pairs formed by each feature point to be matched in the second image and at least one matching feature point obtained by matching), there may be a problem of mismatching in the matching results due to factors such as similarity in image textures, so the image processing device may screen all the matching pairs obtained. The following is an example of a matching pair, and each matching pair may be screened using the same method until all matching pairs are screened.

[0107] Specifically, the process of the image processing device using the repeated matching screening strategy can be: read each matching pair information (i.e., feature point matching result) from the second buffer respectively (e.g., read from the starting address), for example, the matching pair information read is the matching result of the feature point P(x, y) in the second image and the feature point Q(x, y) in the first image, that is, the image processing device queries the above-mentioned status register according to each matching pair information, if the matching result obtained by the query is consistent with the matching pair information, then the current matching pair is determined to be valid, otherwise the current matching pair is determined to be invalid, for example, the feature point obtained by querying the feature point Q(x, y) is P'(x, y), if P'(x, y) is consistent with P(x, y), then P(x, y) and Q(x, y) are determined to be valid, otherwise P(x, y) and Q(x, y) are determined to be invalid. When it is determined that P(x, y) and Q(x, y) are invalid, the image processing device can mark them as invalid in the second buffer, or directly delete the relevant information from the second buffer. Since the above-mentioned status register is updated after the above-mentioned duplicate matching problem is processed, the information in the above-mentioned status register is the latest and there is no duplicate matching pair.

[0108] The principle of the above-mentioned angle difference histogram screening is that if the objects in the two images undergo similar spatial transformations, the angle space distribution of the correct matching pairs should be similar, so the distribution should be similar, and abnormal angle difference distribution can be considered as an incorrect match. Specifically, the process of the image processing device using the angle difference histogram screening strategy can be: reading the statistical information of the matching angle difference corresponding to the feature point set of each block in the first image from the above-mentioned angle difference statistical register, analyzing the waveform distribution of the statistical information, and obtaining the angle sequence number set {angle1, angle2, ..., anglek} corresponding to the maximum k (k≤m, k is a positive integer) peaks, which represents the most likely angle difference distribution space; obtaining a valid w (1<=w<=anglek}) from {angle1, angle2, ..., anglek} according to prior information, big data analysis, machine learning and other methods. k, w are positive integers) peak sets ANGLE{bangle_1, bangle_2, ..., bangle_m}, or dynamically input from the outside, the peak set ANGLE is used to screen matching pairs; each matching pair information (i.e., feature point matching result) is read from the second buffer respectively, and the feature point matching result includes the corresponding angle interval index bangle; if the angle interval index bangle belongs to the above ANGLE{bangle_1, bangle_2, ..., bangle_m} set, it is determined that the current matching pair is valid, otherwise it is determined that the current matching pair is invalid. When it is determined that the current matching pair is invalid, the image processing device can mark it as invalid in the second buffer, or directly delete the relevant information from the second buffer.

[0109] The principle of block descriptor distance screening is that if the objects in the two images undergo similar spatial transformations, the blocks where the correct matching pairs are located should also be similar, so the block descriptor distances should be similar, and abnormal block descriptor distances can be considered as incorrect matches. Specifically, the process of the image processing device using the block descriptor distance screening strategy can be: reading the statistical information of the best descriptor distance corresponding to the feature point set of each block in the first image from the above-mentioned block statistical register, and determining the decision thresholds BDIST{dist_1, dist_2, ..., dist_f} of F blocks according to the mean / maximum / minimum of the best descriptor distance in the statistical information, or the mean / maximum / minimum of the best descriptor distance and the number of valid matching feature points of each block, or obtaining it based on prior information, big data analysis, machine learning and other methods, or obtaining it by external dynamic input; reading each matching pair information (i.e., feature point matching result) from the second buffer respectively, and the feature point matching result includes the best descriptor distance bdist of the current matching pair and the block identifier blk_i corresponding to the second image; querying the decision threshold corresponding to blk_i in the above-mentioned BDIST{dist_1, dist_2, ..., dist_f} as dist_i, if bdist<=dist_i, then the current matching pair is valid, otherwise the current matching pair is invalid. When it is determined that the current matching pair is invalid, the image processing device may mark it as invalid in the second buffer, or directly delete the relevant information from the second buffer.

[0110] The principle of the above block matching statistical screening is that the objects in the two images undergo similar spatial transformations. The feature points that were originally in adjacent areas in the first image should also be in adjacent areas in the second image after similar transformations. Therefore, the effectiveness of the matching pairs can be analyzed by analyzing the aggregation of the feature points of each block in the first image mapped to the feature points of the second image. That is, the feature points in the same block in the first image should also be concentrated in similar areas in the second image. Matching pairs with abnormal distributions with a large spatial span can be considered as incorrect matches. Specifically, the process of the image processing device using the block descriptor distance screening strategy can be: reading the block identifier of each matching corresponding block in the first image and the block identifier in the second image from the above-mentioned status register; reading the number of valid matching feature points of each block of the first image from the above-mentioned block statistics register, assuming that there are A blocks, the number of valid matching feature points of the A blocks are NUM{b_1, b_2, ..., b_a} respectively, and the total number of points is B; obtaining the block identifier set corresponding to the valid matching feature points in each block of the first image in the second image, obtaining the distribution block identifier array BLKNUM[B], and determining the best matching corresponding to each block of the first image in the second image according to the distribution block identifier array BLKNUM[B]. Matching block and next best matching block (optionally, stored in the above block statistics register); determine each matching pair information (i.e., feature point matching result) from the above second buffer, the feature point matching result includes the best descriptor distance bdist of the current matching pair, the block identifier blk_i corresponding to the first image, and the block identifier blk_j corresponding to the second image; query the above block statistics register according to the read matching pair information to obtain blk_bestindex and blk_best2index corresponding to blk_i, blk_bestinde represents the best matching block corresponding to blk_i, and blk_best2index represents the next best matching block corresponding to blk_i. Select the effective block range strategy:

[0111] 1) Take blk_bestindex as the effective matching block strategy: if blk_j is equal to blk_bestindex, the current matching pair is valid, otherwise the current matching pair is invalid;

[0112] 2) The effective block strategy is formed by blk_bestindex and blk_best2index: For example, the block set BLK_SET covered by the line of blk_bestindex and blk_best2index is a valid set. If blk_j falls within the block set BLK_SET, the current matching pair is valid, otherwise the current matching pair is invalid;

[0113] 3) Multi-block strategy centered on the blk_bestindex block: A set BLK_SET consisting of rectangular blocks (rectangular block size can be set), cross blocks, X-shaped blocks, etc. centered on the blk_bestindex block. If blk_j falls within the block set BLK_SET, the current matching pair is valid, otherwise the current matching pair is invalid.

[0114] It should be noted that the above only uses different screening strategies as examples for explanation. In actual applications, the image processing device can also use two or more of the above screening strategies to screen at the same time, or use other screening strategies to screen. By screening based on the above different information, the correct matching rate can be improved while ensuring a high screening efficiency.

[0115] Furthermore, the image processing device may also output the information of the second buffer and each register to output the final matching result. If the best match is performed, the relevant result of the best match is output, and the relevant result of the second best match may not be output; if the best match and the second best match are performed, the relevant result of the best match and the relevant result of the second best match may be output. Optionally, the image processing device may also output the matching result during the above screening process, for example, directly output a matching pair when it is determined to be a valid matching pair, and not output a matching pair when it is determined to be an invalid matching pair.

[0116] It should be noted that when the method of the present application is executed by ASIC, the ASIC can match the feature points to be matched with the feature point set of each block in the M blocks of the first image in a pipeline manner, thereby effectively saving storage resources in the ASIC and improving the matching efficiency of the feature points.

[0117] The above mainly introduces the method for screening local feature points provided by the embodiment of the present application from the perspective of the image processing device. It can be understood that in order to realize the above functions, the image processing device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the network elements and algorithm steps of each example described in the embodiments disclosed in this document, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0118] The embodiment of the present application can divide the functional modules of the device for screening local feature points (e.g., image processing equipment) according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0119] In the case of dividing each functional module into corresponding functional modules, Fig. 9 A schematic diagram of the structure of an image feature point matching device involved in the above embodiment is shown, and the device includes: a storage unit 901, a determination unit 902, an acquisition unit 903 and a matching unit 904. Among them, the storage unit 901 is used to support the device to execute S201 in the above method embodiment; the determination unit 902 is used to support the device to execute S202 and S200 in the above method embodiment; the acquisition unit 903 is used to support the device to execute S203 in the above method embodiment; the matching unit 904 is used to support the device to execute S204 in the above method embodiment. All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0120] Fig.10 FIG. 1 is a schematic diagram of the structure of another image feature point matching device involved in an embodiment of the present application. The device may be a processor in an image processing device (for example, Figure 1 The device comprises an input interface 1001, an operator 1002 and a buffer 1003, wherein the buffer 1003 comprises a first buffer. Optionally, the device may further comprise an output interface 1004.

[0121] The input interface 1001 may be used to support the device to receive N feature point sets in the first image, the size of the first image, the matching radius, the preset offset, the first image, and the second image. In some feasible embodiments, the information received by the input interface 1001 may be stored in an external memory, such as Figure 1 In the memory 101 of the image processing device shown in the figure; in some feasible embodiments, there may be only one input interface or multiple input interfaces.

[0122] The operator 1002 can be used to support the device to execute S200-S204 in the above method embodiment, and / or other technical processes described in this document. For specific descriptions, please refer to the relevant descriptions in the above method embodiment, and the embodiments of this application will not be repeated here.

[0123] The buffer 1003 can be used to cache the relevant information that needs to be cached during the execution of the above method embodiment by the operator 1002, such as the position index of the arranged set of N feature points in the first image, the matching information between the feature points in the first image and the feature points to be matched, and the feature points that meet the preset conditions, etc. The buffer 1003 can also include the second buffer, the state register, the angle difference statistics register, etc. in the above method embodiment.

[0124] The output interface 1004 can be used to output at least one matching feature point matched from the feature point sets of the M blocks. In some feasible embodiments, there can be only one output interface or multiple output interfaces. In some feasible embodiments, the processing result output by the output interface 1004 can be sent to an external memory for storage, or sent to another processing flow for further processing.

[0125] The various components of the above-mentioned image feature point matching device provided in the embodiment of the present application are respectively used to implement the functions of the corresponding steps of the above-mentioned image feature point matching method. Since each step has been described in detail in the above-mentioned image feature point matching method embodiment, it will not be repeated here.

[0126] In an embodiment of the present application, N feature point sets in the first image are stored in a first buffer in a preset order, which can improve the efficiency of subsequently acquiring the feature point set of each block in the M blocks. At the same time, M blocks are determined from the N blocks of the first image according to the positions of the feature points to be matched in the second image, which can effectively reduce the matching range. When the feature point set of each block in the M blocks is subsequently matched block by block according to the feature points to be matched, the matching can be performed in units of the feature point set of one block. The feature point sets of the M blocks are matched block by block through pipeline operations, thereby reducing memory consumption and matching time in the matching process and improving the efficiency of feature point matching.

[0127] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0129] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0130] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for the terminal to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks or optical disks.

[0131] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for matching image feature points, characterized in that: The method comprises: Store N feature point sets in a first image in a first buffer in a preset order, wherein the first image includes N blocks, each block corresponds to a feature point set, and N is a positive integer; For a feature point to be matched in the second image, a matching area is determined according to a position of the feature point to be matched in the second image and a matching radius, or according to the position of the feature point to be matched in the second image, a matching radius and a preset offset, and M blocks located in the matching area are selected from N blocks of the first image, where the second image includes N blocks, and M is an integer greater than or equal to 1; Acquire a feature point set of each of the M blocks from the first buffer; The feature point set of each block in the M blocks is matched block by block according to the feature point to be matched, so as to match at least one matching feature point from the feature point set of the M blocks.

2. The method according to claim 1, characterized in that The N blocks of the first image correspond one-to-one to the N blocks of the second image, and the two corresponding blocks have the same shape and size.

3. The method according to claim 1, characterized in that Before storing the N feature point sets in the first image in the first buffer in a preset order, the method further includes: According to the size of the first image and preset division rules, the first image is divided into N blocks, and a set of feature points of each block in the N blocks of the first image is determined; wherein the preset division rules include at least one of the following rules: division that fits the edge of the image, division that does not fit the edge of the image, division by area, division by regular shape, and division by irregular shape.

4. The method according to claim 1, characterized in that: The method further comprises: According to the number of feature points included in each feature point set of N feature point sets and the preset order, the position index corresponding to the feature point set of each block is determined, and the position index includes a starting position index and / or an ending position index. The position index corresponding to the feature point set of a block is used to index the position of the feature point set of the block in the first buffer.

5. The method according to any one of claims 1 to 4, characterized in that: The step of matching the feature point set of each block in the M blocks one by one according to the feature point to be matched, so as to match at least one matching feature point from the feature point set of the M blocks, comprises: For the M blocks, determining, block by block, matching information between each feature point in the feature point set of each block and the feature point to be matched; According to the matching information, feature points that meet preset conditions in the feature point set of each block are determined as matching feature points.

6. The method according to claim 5, characterized in that The matching information includes: a descriptor distance and / or a matching angle difference.

7. The method according to claim 6, characterized in that The feature point that meets the preset condition means that the matching information of the feature point meets at least one of the following: the best descriptor distance, the second best descriptor distance, the best matching angle difference, or the second best matching angle.

8. The method according to claim 5, characterized in that The step of matching the feature point set of each block in the M blocks one by one according to the feature point to be matched, so as to match at least one matching feature point from the feature point set of the M blocks, further includes: According to the matching information and the preset screening strategy, the matching feature points matched by the feature point set of each block are screened to obtain the at least one matching feature point; wherein the preset screening strategy includes at least one of the following strategies: repeated matching screening, angle difference histogram screening, descriptor distance screening, or block matching statistics screening.

9. The method according to claim 5, characterized in that When a matching feature point in the first image corresponds to a first feature point to be matched and a second feature point to be matched in the second image, the method further includes: It is determined that the matching feature point in the first image corresponds to a feature point to be matched with the best matching information among the first feature point to be matched and the second feature point to be matched.

10. An image feature point matching device, characterized in that: The device comprises: A storage unit, configured to store N feature point sets in a first image in a first buffer in a preset order, wherein the first image includes N blocks, each block corresponds to a feature point set, and N is a positive integer; a determining unit, configured to determine, for a feature point to be matched of a second image, a matching area according to a position of the feature point to be matched in the second image and a matching radius, or according to the position of the feature point to be matched in the second image, a matching radius and a preset offset, and select M blocks located in the matching area from N blocks of the first image, wherein the second image includes N blocks, and M is an integer greater than or equal to 1; An acquiring unit, configured to acquire a feature point set of each of the M blocks from the first buffer; The matching unit is used to match the feature point set of each block in the M blocks block by block according to the feature point to be matched, so as to obtain at least one matching feature point.

11. The device according to claim 10, characterized in that The N blocks of the first image correspond one-to-one to the N blocks of the second image, and the two corresponding blocks have the same shape and size.

12. The device according to claim 10, characterized in that The determining unit is further configured to: According to the size of the first image and preset division rules, the first image is divided into N blocks, and a set of feature points of each block in the N blocks of the first image is determined; wherein the preset division rules include at least one of the following rules: division that fits the edge of the image, division that does not fit the edge of the image, division by area, division by regular shape, and division by irregular shape.

13. The device according to claim 10, characterized in that The determining unit is further configured to: According to the number of feature points included in each feature point set of N feature point sets and the preset order, the position index corresponding to the feature point set of each block is determined, and the position index includes a starting position index and / or an ending position index. The position index corresponding to the feature point set of a block is used to index the position of the feature point set of the block in the first buffer.

14. The device according to any one of claims 10 to 13, characterized in that: The matching unit is further used for: For the M blocks, determining, block by block, matching information between each feature point in the feature point set of each block and the feature point to be matched; According to the matching information, feature points that meet preset conditions in the feature point set of each block are determined as matching feature points.

15. The device according to claim 14, characterized in that The matching information includes: a descriptor distance and / or a matching angle difference.

16. The device according to claim 15, characterized in that The feature point that meets the preset condition means that the matching information of the feature point meets at least one of the following: the best descriptor distance, the second best descriptor distance, the best matching angle difference, or the second best matching angle.

17. The device according to claim 14, characterized in that The matching unit is further used for: According to the matching information and the preset screening strategy, the matching feature points matched by the feature point set of each block are screened to obtain the at least one matching feature point; wherein the preset screening strategy includes at least one of the following strategies: repeated matching screening, angle difference histogram screening, descriptor distance screening, or block matching statistics screening.

18. The device according to claim 14, characterized in that When a matching feature point in the first image corresponds to a first feature point to be matched and a second feature point to be matched in the second image, the determining unit is further configured to: It is determined that the matching feature point in the first image corresponds to a feature point to be matched with the best matching information among the first feature point to be matched and the second feature point to be matched.

19. An image feature point matching device, characterized in that: The device comprises: an input interface, an operator and a first buffer; wherein, The input interface is used to receive N feature point sets in a first image, where the first image includes N blocks, each block corresponds to a feature point set, and N is a positive integer; The operator is used to store the N feature point sets in the first image in the first buffer according to a preset order; The operator is further used to determine, for a feature point to be matched of the second image, a matching area according to a position of the feature point to be matched in the second image and a matching radius, or according to a position of the feature point to be matched in the second image, a matching radius and a preset offset, and select M blocks located in the matching area from N blocks of the first image, where the second image includes N blocks, and M is an integer greater than or equal to 1; and obtain a feature point set of each of the M blocks from the first buffer; The operator is further configured to match the feature point set of each block in the M blocks one by one according to the feature point to be matched, so as to match at least one matching feature point from the feature point set of the M blocks.

20. The device according to claim 19, characterized in that The N blocks of the first image correspond one-to-one to the N blocks of the second image, and the two corresponding blocks have the same shape and size.

21. The device according to claim 19, characterized in that The operator is further used for: According to the size of the first image and preset division rules, the first image is divided into N blocks, and a set of feature points of each block in the N blocks of the first image is determined; wherein the preset division rules include at least one of the following rules: division that fits the edge of the image, division that does not fit the edge of the image, division by area, division by regular shape, and division by irregular shape.

22. The device according to claim 19, characterized in that The operator is further used for: According to the number of feature points included in each feature point set of N feature point sets and the preset order, the position index corresponding to the feature point set of each block is determined, and the position index includes a starting position index and / or an ending position index. The position index corresponding to the feature point set of a block is used to index the position of the feature point set of the block in the first buffer.

23. The device according to any one of claims 19 to 22, characterized in that The operator is further used for: For the M blocks, determining, block by block, matching information between each feature point in the feature point set of each block and the feature point to be matched; According to the matching information, feature points that meet preset conditions in the feature point set of each block are determined as matching feature points.

24. The device according to claim 23, characterized in that The matching information includes: a descriptor distance and / or a matching angle difference.

25. The device according to claim 24, characterized in that The feature point that meets the preset condition means that the matching information of the feature point meets at least one of the following: the best descriptor distance, the second best descriptor distance, the best matching angle difference, or the second best matching angle.

26. The device according to claim 24, characterized in that The operator is further used for: According to the matching information and the preset screening strategy, the matching feature points matched by the feature point set of each block are screened to obtain the at least one matching feature point; wherein the preset screening strategy includes at least one of the following strategies: repeated matching screening, angle difference histogram screening, descriptor distance screening, or block matching statistics screening.

27. The device according to claim 24, characterized in that When a matching feature point in the first image corresponds to a first feature point to be matched and a second feature point to be matched in the second image, the operator is further configured to: It is determined that the matching feature point in the first image corresponds to a feature point to be matched with the best matching information among the first feature point to be matched and the second feature point to be matched.

Images