Rotation angle test based robust similarity transform estimation method and electronic apparatus supporting thereof

Abstract

A method for estimating a similarity transformation between a first image and a second image in an electronic device comprises: receiving a matching pair subset comprising a plurality of matching pairs determined based on correspondence relationships among at least some of a plurality of feature points included in each of the first image and the second image to estimate the similarity transformation; and verifying the validity of the matching pair subset, wherein the step of verifying the validity of the matching pair subset comprises: verifying a first vector and a second vector based on at least some of the feature points verified based on the matching pair subset on the first image; verifying a third vector corresponding to the first vector and a fourth vector corresponding to the second vector on the second image based on the matching pair subset; and verifying a first value regarding the difference between a first rotation angle formed by the first vector and the third vector and a second rotation angle formed by the second vector and the fourth vector. A similarity transformation estimation method is disclosed, comprising: a step of verifying the validity of matching pairs corresponding to the first vector, the second vector, the third vector, and the fourth vector by comparing a threshold value verified based on the third vector and the fourth vector with the first value; and a step of verifying the validity of a subset of matching pairs based on whether the matching pairs are valid.

Description

Technology Field

[0001] The present invention relates to an electronic device and method for estimating similarity transformations, and more specifically, to an electronic device and method for estimating similarity transformations between different images. Background Technology

[0002] Estimating the transformation between two images is one of the long-standing problems in the field of computer vision. Similarity transforms and projective transforms can be used for transformations; however, when the size of the object of interest within an image is small, it is frequently approximated using a similarity transform that includes at least one of translation, rotation, or scaling transformations. This is because assuming an overly general transformation can lead to overfitting in the transformation estimation process. Fields utilizing similarity transforms include image stitching, 3D pose reconstruction of facial images, and satellite image registration.

[0003] A widely used approach for estimating image transformations is to match multiple local feature points and, based on the matching results, follow the framework of RANSAC (Random Sample Consensus) to exclude outliers and select the set of matching pairs deemed most suitable.

[0004] In this document, the term "matching pair" refers to a pair of corresponding feature points when it is determined that a feature point in one image corresponds to a feature point in another image.

[0005] RANSAC can be understood as a method to find sampling that meets defined conditions by iteratively sampling a sufficiently small dataset required for estimating model parameters, rather than examining all data to remove outliers in model estimation.

[0006] Conventional methods, such as RANSAC, that extract a suitable set of matching pairs to estimate similarity transformations assume a situation where there are a sufficiently large number of valid matching pairs among the matching pairs between input images. However, when the number of feature points within an image is small (e.g., when the size of the object where the feature points are distributed is small), there is a high probability that the majority of the matching pairs are incorrect matches. Furthermore, there is a high probability of feature point matching failure when the modalities of the two images differ (e.g., between visible light and infrared images) or when similarity is low (e.g., between daytime and nighttime images). In such cases, it is important to exclude outliers. This is because even a slight additional matching error can drastically alter the results of the transformation estimation.

[0007] Among some existing methods, there are techniques to eliminate outliers by assuming that the distribution of inliers and outliers follows a specific distribution (e.g., Gaussian distribution); however, these methods have a high probability of failure if the distribution of matched pairs does not follow such a specific distribution.

[0008] From the perspective of processing speed, conventional methods that remove outliers based on the error of the result obtained from the estimated transformation may be unsuitable in environments with computational constraints because pseudo-inverse matrix calculation is required for transformation estimation.

[0009] Deep learning-based methods require a sufficient set of training images, and additional training may be required if the modality of the input images differs from that of the training images. Furthermore, since there is a risk of performance degradation during this process, it may be difficult to apply if there is a shortage of training images or if the characteristics of the actual images and the training images differ.

[0010] For other prior art, refer to KR102144394B1 or US20220116646A1. The problem to be solved

[0011] According to the method of the present invention, an electronic device can exclude outliers of matching pairs.

[0012] According to the method of the present invention, an electronic device can reject a subset of matching pairs containing outliers and determine that a subset of matching pairs not containing outliers is valid through a preliminary inspection of a subset of matching pairs based on a rotation angle.

[0013] According to the method of the present invention, an electronic device can determine whether an outlier is included in a subset of matching pairs.

[0014] According to the method of the present invention, an electronic device can select an appropriate subset of matching pairs for similarity transformation estimation even when there is no or insufficient set of training images.

[0015] According to the method of the present invention, an electronic device can estimate a similarity transformation based on a subset of matching pairs determined to be valid.

[0016] The technical problems to be solved by the present invention are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which the present invention belongs from the description below. means of solving the problem

[0017] Various embodiments may provide a method for estimating a similar transformation and an electronic device that supports the same.

[0018] A method for estimating a similarity transformation between a first image and a second image in an electronic device according to various embodiments disclosed in this document comprises: receiving a matching pair subset comprising a plurality of matching pairs determined based on a correspondence relationship among at least some of the plurality of feature points included in each of the first image and the second image for estimating the similarity transformation; and verifying the validity of the matching pair subset, wherein the step of verifying the validity of the matching pair subset comprises: verifying a first vector and a second vector based on at least some of the feature points verified based on the matching pair subset on the first image; verifying a third vector corresponding to the first vector and a fourth vector corresponding to the second vector on the second image based on the matching pair subset; and verifying a first value regarding the difference between a first rotation angle formed by the first vector and the third vector and a second rotation angle formed by the second vector and the fourth vector. The method may include a step of verifying the validity of matching pairs corresponding to the first vector, the second vector, the third vector, and the fourth vector by comparing the threshold value verified based on the third vector and the fourth vector with the first value; and a step of verifying the validity of the matching pair subset based on whether the matching pairs are valid.

[0019] In an embodiment, the step of verifying the validity of the matching pair subset may include the step of determining the matching pair subset as invalid without needing to consider other first vectors, second vectors, third vectors, and fourth vectors that can be verified for the matching pair subset when the first value calculated for any first vector, second vector, third vector, and fourth vector that can be verified for the matching pair subset is greater than the threshold value.

[0020] In an embodiment, the step of verifying the validity of the matching pair subset may include determining that the matching pair subset is valid if the first value calculated for all combinations of the first vector, the second vector, the third vector, and the fourth vector that can be verified for the matching pair subset is less than or equal to the threshold value.

[0021] In an embodiment, the threshold value may be determined based on the ratio of the second value to the magnitude of the third vector and the ratio of the second value to the magnitude of the fourth vector.

[0022] In an embodiment, the second value is defined as the product of the maximum allowable position error in the estimation for the similarity transformation and the first constant, and the first constant may be defined as a positive real value greater than 0 and less than or equal to 2.

[0023] In an embodiment, the similarity transformation estimation method may include: a step of sampling a subset of matching pairs from a whole set of matching pairs; a step of verifying the validity of the subset of matching pairs; a step of, if the subset of matching pairs is valid, verifying the goodness of fit of the subset of matching pairs and similarity transformation estimation information including the value of one or more parameters regarding the similarity transformation between the first image and the second image based on the subset of matching pairs; a step of updating the optimal subset of matching pairs and the optimal similarity transformation estimation information to the subset of matching pairs and the similarity transformation estimation information based on the goodness of fit of the subset of matching pairs; a step of verifying the cumulative number of samplings; and a step of sampling a second subset of matching pairs that is different from the subset of matching pairs within the whole set of matching pairs based on the cumulative number of samplings.

[0024] In an embodiment, the similarity transformation estimation method may further include the step of outputting the optimal matching pair subset and the optimal similarity transformation estimation information based on the cumulative number of samplings.

[0025] In an embodiment, the fitness of the matched pair subset is determined based on the loss function value of the matched pair subset, and the loss function value can be determined based on the first value.

[0026] In an embodiment, the method may further include the step of checking the size of a matching pair subset; and, only if the size of the matching pair subset is 3 or more, checking the first vector and the second vector, and omitting validation if the size of the matching pair subset is less than 3.

[0027] An electronic device according to the various embodiments disclosed in this document comprises: a memory; The apparatus includes a processor, wherein the processor receives a subset of matching pairs comprising a plurality of matching pairs determined based on the correspondence of at least some of the plurality of feature points included in each of the first image and the second image to estimate a similarity transformation between the first image and the second image, and is configured to verify the validity of the subset of matching pairs, and verifies a first vector and a second vector based on at least some of the feature points verified based on the subset of matching pairs on the first image, verifies a third vector corresponding to the first vector and a fourth vector corresponding to the second vector on the second image based on the subset of matching pairs, verifies a first value regarding the difference between a first rotation angle formed by the first vector and the third vector and a second rotation angle formed by the second vector and the fourth vector, and compares the first value with a threshold value verified based on the third vector and the fourth vector to verify the validity of the matching pairs corresponding to the first vector, the second vector, the third vector, and the fourth vector. It can be configured to verify and check the validity of the subset of the matched pairs based on the validity of the matched pairs.

[0028] A computer-readable non-transient recording medium storing a program for executing on a computer a method for estimating a similarity transformation according to various embodiments disclosed in this document, wherein the method comprises: receiving a subset of matching pairs including a plurality of matching pairs determined based on correspondence relationships among at least some of a plurality of feature points included in each of a first image and a second image for estimating a similarity transformation; and verifying the validity of the subset of matching pairs, wherein the step of verifying the validity of the subset of matching pairs comprises: verifying a first vector and a second vector based on at least some of the feature points verified based on the subset of matching pairs on the first image; verifying a third vector corresponding to the first vector and a fourth vector corresponding to the second vector on the second image based on the subset of matching pairs; and verifying a first value regarding the difference between a first rotation angle formed by the first vector and the third vector and a second rotation angle formed by the second vector and the fourth vector. The method may include a step of verifying the validity of matching pairs corresponding to the first vector, the second vector, the third vector, and the fourth vector by comparing the threshold value verified based on the third vector and the fourth vector with the first value; and a step of verifying the validity of the matching pair subset based on whether the matching pairs are valid. Effects of the invention

[0029] According to the method of the present invention, an electronic device can exclude outliers without assuming the position error distribution of matching pairs to be a Gaussian distribution, a uniform distribution, etc., in an environment where a majority of matching pairs may fail to match and fast similarity transformation estimation is required.

[0030] According to the method of the present invention, an electronic device can reject a subset of matching pairs containing outliers as quickly as possible through a preliminary inspection of a subset of matching pairs based on a rotation angle, and determine that a subset of matching pairs not containing outliers is valid.

[0031] According to the method of the present invention, an electronic device can quickly determine whether an outlier is included in a subset of matching pairs in a hardware environment where the amount of computation is constrained.

[0032] According to the method of the present invention, an electronic device can determine the most appropriate subset of matching pairs among subsets of matching pairs determined to be valid, and for this determination, provides a loss function independent of the feature point matching algorithm.

[0033] According to the method of the present invention, since the electronic device can determine a threshold value required for outlier determination as a geometrically valid value based on an acceptable matching position error target value, there is an advantage in that the invention is easy to implement in detail.

[0034] According to the method of the present invention, it is possible to perform rapid similarity transformation estimation while providing sufficiently robust similarity transformation estimation.

[0035] The effects obtainable from the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description below. Brief explanation of the drawing

[0036] FIG. 1 is a diagram illustrating the configuration of an electronic device according to various embodiments. FIG. 2 is a diagram illustrating a similar transformation estimated by an electronic device according to various embodiments. Figure 3 is a diagram illustrating feature points. FIG. 4 is a drawing for explaining the first vector, second vector, third vector and fourth vector according to various embodiments. FIG. 5 is a drawing for explaining the first rotation angle and the second rotation angle according to various embodiments. FIG. 6 is a diagram illustrating threshold values ​​according to various embodiments. FIGS. 7a and 7b are flowcharts of an operation in which an electronic device according to various embodiments verifies the validity of a matching pair subset based on a first vector, a second vector, a third vector, and a fourth vector. FIG. 8 is an overall operation flowchart of an electronic device according to various embodiments verifying the validity of a matching pair subset. FIG. 9 is a flowchart of an operation in which an electronic device according to various embodiments determines whether there is a subset of matching pairs for correctly estimating a similarity transformation between images, and if the subset of matching pairs exists, calculates an optimal subset of matching pairs and optimal similarity transformation estimation information. FIG. 10 is a diagram showing the average error value by method according to one embodiment. FIG. 11 is a diagram illustrating the actual rejection rate according to the mismatch rate by method according to one embodiment. FIG. 12 is a diagram illustrating the execution time according to the mismatch ratio by method according to one embodiment. FIG. 13 is a diagram illustrating the execution time according to the size of the matching pair subset or the sampling size according to one embodiment of the method. Specific details for implementing the invention

[0037] The following embodiments are combinations of the components and features of various embodiments in a predetermined form. Each component or feature may be considered optional unless otherwise explicitly stated. Each component or feature may be implemented in a form not combined with other components or features. Additionally, various embodiments may be constructed by combining some components and features. The order of operations described in various embodiments may be changed. Some components or features of one embodiment may be included in another embodiment, or may be replaced with corresponding components or features of another embodiment.

[0038] In the description of the drawings, procedures or steps that could obscure the essence of the various embodiments were not described, nor were procedures or steps that can be understood by a person of ordinary knowledge in the relevant technical field described.

[0039] Throughout the specification, when a part is described as "comprising" or "including" a component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "...part," "...unit," and "module" as used in the specification refer to a unit that performs at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software. Additionally, "one (a or an)," "one," "the," and similar related terms may be used in the context describing various embodiments (particularly in the context of the following claims) in both singular and plural forms, unless otherwise indicated in the specification or clearly contradicted by the context.

[0040] Hereinafter, preferred embodiments according to various examples will be described in detail with reference to the accompanying drawings. The detailed description disclosed below, together with the accompanying drawings, is intended to describe exemplary embodiments of various examples and is not intended to represent the only embodiment.

[0041] In addition, specific terms used in various embodiments are provided to aid in understanding the various embodiments, and the use of such specific terms may be modified in other forms within the scope of not departing from the technical concept of the various embodiments.

[0042] At this time, it will be understood that each block of the process flow diagrams and combinations of the flow diagrams can be executed by computer program instructions. Since these computer program instructions can be loaded into the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, the instructions executed through the processor of the computer or other programmable data processing equipment create means to perform the functions described in the flow diagram block(s). Since these computer program instructions can also be stored in computer-available or computer-readable memory that can be directed toward the computer or other programmable data processing equipment to implement the function in a specific way, the instructions stored in computer-available or computer-readable memory can also produce a manufactured item containing the means of instruction to perform the function described in the flow diagram block(s). Since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that perform a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer can also provide steps for executing the functions described in the flowchart block(s).

[0043] Additionally, each block may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). It should also be noted that in some alternative execution examples, the functions mentioned in the blocks may occur out of order. For instance, two blocks described in succession may actually be executed substantially simultaneously, or the blocks may be executed in reverse order according to their corresponding functions.

[0044] FIG. 1 is a drawing illustrating the configuration of an electronic device (101) according to various embodiments.

[0045] Referring to FIG. 1, a method for estimating a similarity transformation according to various embodiments may be implemented in various types of devices. For example, the similarity transformation estimation method may be implemented in an electronic device (101). In other words, the electronic device (101) may perform operations according to various embodiments of the present invention based on the similarity transformation estimation method implemented in the electronic device (101). Meanwhile, the similarity transformation estimation method according to various embodiments is not limited to that shown in FIG. 1 and may be implemented in a wider variety of electronic devices and servers.

[0046] A similarity transformation estimation method according to various embodiments may include various modules for operation. The modules included in the similarity transformation estimation method may be computer code or one or more instructions implemented to enable a physical device (e.g., electronic device (101)) in which the similarity transformation estimation method is implemented (or included in the physical device) to perform a specified operation. In other words, the physical device in which the similarity transformation estimation method is implemented may store a plurality of modules in memory in the form of computer code, and when the plurality of modules stored in memory are executed, the plurality of modules may cause the physical device to perform specified operations corresponding to the plurality of modules.

[0047] Referring to FIG. 1, the electronic device (101) may include an input / output unit (210), a communication unit (220), a storage unit (230), and a processor (240).

[0048] The input / output unit (210) may be various interfaces or connection ports that receive user input or output information to the user. The input / output unit (210) may include an input module and an output module, wherein the input module receives user input from the user. User input may be in various forms, including key input, touch input, and voice input. Examples of input modules capable of receiving such user input include traditional keypads, keyboards, and mice, as well as touch sensors that detect user touch, microphones that receive voice signals, cameras that recognize gestures through image recognition, proximity sensors that include at least one of an illuminance sensor or an infrared sensor that detects user approach, motion sensors that recognize user movements through accelerometers or gyroscopes, and various other input means that detect or receive various forms of user input. The input module according to the embodiment of the present disclosure may include at least one of the devices listed above. Here, the touch sensor may be implemented as a piezoelectric or capacitive touch sensor that detects touch through a touch panel or touch film attached to a display panel, or as an optical touch sensor that detects touch by an optical method. In addition, the input module may be implemented in the form of an input interface (USB port, PS / 2 port, etc.) that connects an external input device receiving user input, instead of a device that detects user input itself. Furthermore, the output module may output various types of information. The output module may include at least one of a display that outputs video, a speaker that outputs sound, a haptic device that generates vibration, and various other forms of output means. In addition, the output module may be implemented in the form of a port-type output interface that connects the individual output means described above.

[0049] For example, an output module in the form of a display can display text, still images, and videos. The display may include at least one of a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flat panel display (FPD), a transparent display, a curved display, a flexible display, a 3D display, a holographic display, a projector, and other various types of devices capable of performing image output functions. Such a display may also be in the form of a touch display integrated with the touch sensor of the input module.

[0050] The communication unit (220) can communicate with other devices. Therefore, the electronic device (101) can transmit and receive information with other devices through the communication unit. For example, the electronic device (101) can perform communication with one another or with other devices using the communication unit.

[0051] Here, communication, that is, the transmission and reception of data, can be performed via wired or wireless means. To this end, the communication unit may be composed of a wired communication module that connects to the internet, etc., through a LAN (Local Area Network), a mobile communication module that connects to a mobile communication network via a mobile communication base station to transmit and receive data, a short-range communication module that uses a communication method of the WLAN (Wireless Local Area Network) family such as Wi-Fi or a communication method of the WPAN (Wireless Personal Area Network) family such as Bluetooth or Zigbee, a satellite communication module that uses a GNSS (Global Navigation Satellite System) such as GPS (Global Positioning System), or a combination thereof.

[0052] Storage (230) can store various types of information. Storage can store data temporarily or semi-permanently. For example, the storage of the electronic device (101) may store an operating system (OS) for operating the electronic device (101), data for hosting a website, or data regarding a program or application (e.g., a web application) for generating Braille. In addition, storage can store modules in the form of computer code as described above.

[0053] Examples of storage (230) may include a hard disk drive (HDD), a solid state drive (SSD), flash memory, ROM (Read-Only Memory), and RAM (Random Access Memory). These storage may be provided as an internal type or a removable type.

[0054] The processor (240) controls the overall operation of the electronic device (101). To this end, the processor (240) performs computation and processing of various information and can control the operation of the components of the electronic device (101). For example, the processor (240) may execute a program or application for similarity transformation estimation. The processor (240) may be implemented as a computer or a similar device depending on hardware, software, or a combination thereof. In terms of hardware, the processor (240) may be implemented in the form of an electronic circuit that processes electrical signals to perform control functions, and in terms of software, it may be implemented in the form of a program that drives the hardware processor (240). Meanwhile, unless otherwise specifically mentioned in the following description, the operation of the electronic device (101) may be interpreted as being performed by the control of the processor (240). That is, when modules implementing the aforementioned similarity transformation estimation method are executed, the modules may be interpreted as the processor (240) controlling the electronic device (101) to perform the following operations.

[0055] In summary, various embodiments may be implemented through various means. For example, various embodiments may be implemented by hardware, firmware, software, or a combination thereof.

[0056] In the case of implementation by hardware, the method according to various embodiments may be implemented by one or more ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors, controllers, microcontrollers, microprocessors, etc.

[0057] In the case of implementation by firmware or software, the method according to various embodiments may be implemented in the form of modules, procedures, or functions that perform the functions or operations described below. For example, software code may be stored in memory and executed by a processor. The memory may be located inside or outside the processor and may exchange data with the processor by various known means.

[0058] Hereinafter, various embodiments are described in more detail based on the technical concept described above. The previously explained contents may be applied to the various embodiments described below. For example, undefined operations, functions, terms, etc. in the various embodiments described below may be performed and explained based on the previously explained contents.

[0059] In the following description, various embodiments are described on the premise that the electronic device (101) performs a similar conversion estimation operation.

[0060] FIG. 2 is a diagram illustrating a similar transformation estimated by an electronic device according to various embodiments.

[0061] Referring to FIG. 2, the electronic device (101) can estimate a similarity transformation (203) between two different images. Referring to FIG. 2, two different images, namely image A (201) and image B (202), may be two different images that are in a similarity transformation relationship. For example, image B (202) may be an image that has been similarly transformed from image A (201). The similarity transformation (203) may be performed through at least some of translation, rotation, and scaling transformations of the images. For example, image B (202) may be the result of at least some of translation, rotation, and scaling transformations being performed on image A (201). The similarity transformation (203) may be performed based on one or more parameter values ​​for similarly transforming image A (201), which is the subject of the similarity transformation, into image B (202). For example, the similarity transformation (203) can be understood as a similarity transformation matrix containing the values ​​of one or more parameters for the similarity transformation. One or more parameters for the similarity transformation may include at least some of the parameters related to translation (e.g., amount of movement in the horizontal and vertical axis directions), parameters related to rotation (e.g., rotation angle), and parameters related to proportion (e.g., magnification ratio). The electronic device (101) can estimate the similarity transformation (203) between two images based on inputs for two images (Image A, Image B). The estimation of the similarity transformation (203) can be understood as deriving one or more parameter values ​​related to the similarity transformation. In the following description, the image can be understood as a single image unit, and Image A (201), which serves as the reference for the similarity transformation, can be referred to as the first image, and the similarly transformed Image B (202) can be referred to as the second image.

[0062] Figure 3 is a diagram illustrating feature points.

[0063] Referring to FIG. 3, an electronic device (101) can identify one or more feature points (301) included in an image (300). A feature point (301) refers to at least one point that is a characteristic of the image, and may be referred to as a feature point, key point, or interesting point. The electronic device (101) may use various algorithms to extract feature points of the image (300). The electronic device (101) may extract feature points using information about multiple pixels included in the image (300). For example, the electronic device (101) may extract feature points based on the difference in pixel values ​​between adjacent pixels. Feature points (301) may be identified based on corner points or extracted as the center point of a part of the image (300) identified based on various algorithms.

[0064] In this specification, among multiple feature points included in two images in a similarity transformation relationship (e.g., the first image (Image A) and the second image (Image B) of FIG. 2), those that correspond to each other are grouped together and described as a matching pair. Here, the meaning of correspondence is that feature point a of the first image and feature point b of the second image represent the same point within the image. From another perspective, the meaning of correspondence is as follows: Feature points a and b are considered to correspond if the location on the second image obtained by transforming the location of feature point a of the first image using a known actual similarity transformation between the two images becomes the location of feature point b. In this specification, the set of corresponding pairs of feature points among the set of feature points of the first image (Image A) and the set of feature points of the second image (Image B) is described as a matching pair set. However, since the present invention is based on the premise of estimating a similarity transformation, a 'matching pair' may refer to two points where two feature points are presumed to correspond to each other. The electronic device (101) can extract multiple feature points from each of two different images and can extract a set of matching pairs that correspond to each other among the feature points of each of the two images. In this case, the meaning of extracting a set of matching pairs can be understood as proposing a set of matching pairs that are presumed to match each other. The electronic device (101) can extract a set of matching pairs based on various algorithms. The set of all extracted matching pairs can be understood as the entire set of matching pairs, and a set of matching pairs that includes at least some of them can be understood as a subset of matching pairs. The reason for extracting a subset of matching pairs is that the entire set of matching pairs may contain incorrectly estimated correspondence results for the proposed set of matching pairs, so the purpose is to exclude outliers and correctly estimate the similarity transformation using only the subset of matching pairs sufficient for similarity transformation estimation.Various methods, including random sampling, can be used to extract a subset of matching pairs from the entire set of matching pairs.

[0065] FIG. 4 is a drawing for explaining the first vector, second vector, third vector and fourth vector according to various embodiments.

[0066] FIG. 5 is a drawing for explaining the first rotation angle and the second rotation angle according to various embodiments.

[0067] Referring to FIG. 4, an example of multiple feature points on image A (201) (e.g., first image) and image B (202) (e.g., second image) can be seen.

[0068] Referring to the example in FIG. 4, a plurality of feature points on image A (201) , , , and There may exist, and corresponding feature points on image B (202). , , , and There may be. In this case, multiple feature points on image A (201) and multiple matching pairs on image B (202) may correspond to each other in order to form matching pairs. For example, and A matching pair ( , ) can be constructed. At this time, feature points on image B (202) , , , and It can be understood as an estimated feature point obtained by a separate feature point extraction and matching (correspondence or matching) module. That is, the matching pairs verifiable by the electronic device (101) to perform the similar estimation method according to the present invention refer to estimated matching pairs in which errors may exist.

[0069] At this time, a vector can be defined through the feature points identified on image A (201). For example, feature points feature points A vector pointing toward can be defined, and feature points feature points from A vector pointing toward can be defined, and feature points feature points from A vector pointing toward can be defined. Similarly, a vector can be defined through the feature points identified on image B (202). For example, feature points feature points A vector pointing toward can be defined, and feature points feature points from A vector pointing toward can be defined, and feature points feature points from A vector pointing toward A can be defined. Each of these vectors can be described as corresponding to one another by the correspondence relationship between matching pairs. For example, the vector of image A (201). is the vector of image B (202) It corresponds to.

[0070] Meanwhile, feature points on image B (202) , , , and An ideal virtual feature point that matches without error , , , and It can be assumed that... In image B (202), the positional error between these virtual matching pairs and the actual matching pairs can be referred to as the matching position error. For example, the actual extracted feature points is an ideal feature point pair and It can have a matching position error of that amount, and likewise silver class It can have a matching position error of that amount.

[0071] Referring to FIG. 5, vectors on image A (201) and corresponding vectors on image B (202) can form a rotation angle. For example, vector and vector rotation angle between can be formed, and a vector and vector rotation angle between can be formed, and a vector and vector Rotation angle between It can be formed. Here, the rotation angle can be identified as the amount of rotation in a specified rotation direction (e.g., counterclockwise) to align the direction of a vector on image A with the direction of a vector on image B.

[0072] Furthermore, to determine whether the correspondence between feature points is correct from the perspective of similarity transformation, the difference between the two rotation angles can be examined. For example, the rotation angle class The difference value of, It can be investigated. If this difference value is excessively large, the vector and the corresponding vector The rotation transformation and vector represented by and the corresponding vector Since this implies that the rotational transformations represented by are different, similarity transformations cannot be correctly estimated from the feature points constituting these vectors. That is, in order to correctly estimate similarity transformations, the difference between the two rotational angles must be smaller than a certain threshold so that both pairs of vectors refer to the same rotational transformation. In the present invention, the rotational angle class A threshold value to check whether the difference value is sufficiently small It is defined as. Likewise, the rotation angle and A threshold value to check whether the difference value is sufficiently small It can be defined as.

[0073] FIG. 6 is a diagram illustrating threshold values ​​according to various embodiments.

[0074] Referring to Fig. 6, the magnitude of the set maximum position error A method for setting a threshold value based on can be described. A vector defined by the actual extracted feature points on image B (202). Wow, vector A vector defined by ideal feature points corresponding to In the relationship, vector and vector The angle formed by can be defined as the rotation angle error. In this case, for the above rotation angle error to be maximized, the direction of the position error of the i-th feature point of image B is a vector It must be perpendicular to, and at the same time, the direction of the position error of the j-th feature point of image B is also a vector It must be perpendicular to. That is, centered on each ideal feature point, the set maximum position error magnitude Assuming a circle with radius, these circles and vector It can be understood that the maximum rotation angle error occurs when they are in contact. Such vector Translating it can induce the geometric maximum rotation angle error.

[0075] The derivation of the threshold can be performed as follows. Assume that the i-th, j-th, k-th, and l-th matching pairs are selected from a subset of matching pairs. Here, i, j, k, and l are each integers greater than or equal to 1 that are less than or equal to the number of matching pairs included in the subset of matching pairs, and j and k may be the same or different from each other. From this, a first vector ( ) and a second vector pointing from a feature point of the first image (Image A) corresponding to the k-th matching pair to a feature point of the first image (Image A) corresponding to the l-th matching pair ( ) can be verified. Likewise, the third vector ( ) and a fourth vector pointing from a feature point of the second image (Image B) corresponding to the k-th matching pair to a feature point of the second image corresponding to the l-th matching pair ( You can check ).

[0076] Then, the first vector ( ) and the third vector( The first rotation angle formed by ) ) and the second vector( ) and the fourth vector( The second rotation angle formed by ) Regarding ), the first value, which represents the error in the dimension of the rotation angle and the difference value of the rotation angle, can be expressed by the following mathematical formula.

[0077]

[0078] In mathematical formula 1, is the first rotation angle, is the second rotation angle, and R is the actual rotation angle between image A and image B. is the error of the first rotation angle caused by the positional error of the feature points constituting the i-th matching pair and the j-th matching pair, represents the error of the second rotation angle (matching error in the dimension of rotation angle) caused by the positional error of the feature points constituting the k-th matching pair and the l-th matching pair, and represents the first value. As the position error becomes smaller, the rotation angle error decreases, and thus the first value becomes smaller. Therefore, when the first value is less than or equal to the threshold value, the electronic device (101) can confirm that the matching pairs corresponding to the selected vectors are valid.

[0079] First value ( The threshold value corresponding to ) ) can be explained through the following mathematical formula.

[0080]

[0081] Through the operation of absolute values, the maximum value that the first value can have is the first rotation angle ( ) error( ) absolute value and second rotation angle( ) error( It is less than or equal to the sum of the absolute values ​​of ).

[0082] The geometric maximum value of each rotation angle error can be explained by the following mathematical formula from the description of Fig. 6.

[0083]

[0084] At this time is the maximum allowable position error, meaning the maximum allowable value in the position dimension of the alignment error. and are the third vectors on the second image, respectively ( ) and the fourth vector( It means the size of ).

[0085] By introducing a first constant into the above mathematical formula 3, the first value is as follows ( The threshold value corresponding to ) ) can be defined.

[0086]

[0087] At this time is the first rotation angle ( ) and second rotation angle( The first value derived according to ) It refers to a threshold value corresponding to ), and c corresponds to a first constant. In this case, c can have a positive real value greater than 0 and less than or equal to 2. According to one embodiment, based on the first constant (c), a first value ( The strictness of the threshold value of ) can be controlled. In this case, since the geometrically acceptable maximum rotation angle error corresponds to the threshold value when the first constant has a value of 2, the first constant (c) can have a maximum magnitude of 2. According to one embodiment, the first value ( ) is the threshold( If it is less than or equal to ) the first rotation angle ( ) and second rotation angle( Each matching pair corresponding to ), that is, the i, j, k, and l-th matching pairs of the matching pair subset, can be verified as mutually valid. The smaller the value of the first constant (c), the more strictly the validity of the matching pairs (e.g., the i, j, k, and l-th matching pairs) subject to verification is judged. That is, and c are both parameters that can be determined when implementing the invention, and In the case of , it is necessary to set it so as not to be too low or too high depending on the performance limit of the feature point extraction and matching (correspondence or matching) module, and c may preferably be set to a value of 1 to 1.5 when using the method of the present invention alone, and may preferably be set to a value of 2 when combined with other existing matching frameworks.

[0088] Below, based on the above mathematical formulas, the operation flow of an electronic device according to various embodiments verifying the validity of a matching pair subset is described.

[0089] FIGS. 7a and 7b are flowcharts of an operation in which an electronic device according to various embodiments verifies the validity of a matching pair subset.

[0090] In operation S710, the electronic device (101) can identify a subset of matching pairs. The electronic device (101) can identify matching pairs that are in a corresponding relationship among the feature points of each of the first image and the second image, and can identify a subset of matching pairs that includes at least some of the total matching pairs. The electronic device (101) can identify any one of the possible subsets of matching pairs. According to one embodiment, the electronic device (101) receives an input regarding a subset of matching pairs and can identify the subset of matching pairs.

[0091] In operation S720, the electronic device (101) can verify the validity of a subset of matching pairs. The validity of a subset of matching pairs can be understood as whether the subset of matching pairs can be used to correctly estimate a similarity transformation between a first image and a second image. That is, the electronic device (101) can estimate a similarity transformation based on a valid subset of matching pairs, and an invalid subset of matching pairs cannot be used for similarity transformation estimation.

[0092] Referring to FIG. 7b, the operation S720 for verifying the validity of a subset of matching pairs may include at least some of the operations S721, S722, S723, S724, and S725.

[0093] In operation S721, the electronic device (101) can identify a first vector and a second vector in the first image. In one embodiment, for any matching pairs selected from among a plurality of matching pairs included in the matching pair subset identified in operation S710 (e.g., the i, j, k, and lth matching pairs, where j and k may be the same), the electronic device (101) can identify at least two vectors (e.g., a first vector ( ) and the second vector( You can check ))

[0094] In operation S722, the electronic device (101) has at least two vectors in the second image (e.g., a third vector ( ) and the fourth vector( ...can be verified. Each vector is a vector on the second image corresponding to at least two vectors identified in the first image. At this time, the third and fourth vectors are determined based on the correspondence relationship between the first vector, the second vector, and the feature points of the matching pair subset. For example, for each of the two feature points on the first image corresponding to the first vector, two feature points on the second image are identified according to the correspondence relationship between matching pairs in the matching pair subset, and the vector connecting the two identified feature points is the third vector. In the same way, the fourth vector corresponds to the second vector. That is, the third vector corresponds to the first vector, and the fourth vector corresponds to the second vector.

[0095] In operation S723, the electronic device (101) can determine a first value based on a first rotation angle and a second rotation angle. The rotation angle is the angle formed by a vector on the first image and a vector on the second image corresponding to the vector on the first image. For example, the first rotation angle is the angle formed by the first vector and the third vector, and the second rotation angle is the angle formed by the second vector and the fourth vector. According to one embodiment, the electronic device (101) can determine the amount of rotation in a designated rotation direction (e.g., counterclockwise) to align the direction of the vector on the first image with the direction of the vector on the second image as the rotation angle. The first value can be determined based on the difference between the first rotation angle and the second rotation angle. According to one embodiment, the first value may be the absolute value of the difference between the first rotation angle and the second rotation angle.

[0096] In operation S724, the electronic device (101) can verify the validity of the vectors. The validity of the vectors can be understood as whether the matching pairs corresponding to the vectors verified in operations S721 and S722 can be used to correctly estimate the similarity transformation. The electronic device (101) can verify, based on a first value, whether the matching pairs corresponding to the selected vectors are valid matching pairs. According to one embodiment, the electronic device (101) can verify the validity of the vectors based on the result of comparing the first value verified in S723 with a threshold value. The threshold value can be calculated according to Equation 4.

[0097] In operation S725, the electronic device (101) verifies the validity of a matching pair subset based on the validity of the selected vectors. The electronic device (101) may reject (determine as invalid) the matching pair subset if the vectors selected by operations S721 and S722 are verified as invalid by operation S724. The electronic device (101) may determine that the matching pair subset is invalid if any of the first values ​​verified based on all combinations of selectable vectors of the matching pair subset are greater than the corresponding threshold value. The electronic device (101) may reject the corresponding matching pair subset if, while repeating operations S721 through S724 with only the selected vectors changed, a combination of invalid vectors is verified at least once.

[0098] In an embodiment, the electronic device (101) can confirm a subset of matching pairs as valid when all matching pairs identified from all selectable combinations of vectors are confirmed to be valid. For example, if the electronic device (101) determines that the i, j, k, and lth matching pairs of a subset of matching pairs are valid (corresponding feature points match each other), it can select matching pairs other than the selected matching pairs (e.g., m, n, o, and pth matching pairs) and repeat operations such as S721 to S724 to confirm the validity of the selected matching pairs. The electronic device (101) can confirm a subset of matching pairs as valid when the validity of all matching pairs included in the subset of matching pairs is confirmed.

[0099] FIG. 8 is a flowchart of an operation in which an electronic device according to various embodiments verifies the validity of a matching pair subset.

[0100] In operation S801, the electronic device (101) can identify a subset of matching pairs. Operation S801 may include at least a portion of operation S710 of FIG. 7a. In operation S801, the electronic device (101) can identify a subset of matching pairs. The electronic device (101) can identify matching pairs that are in a corresponding relationship among the feature points of each of the first image and the second image, and can identify a subset of matching pairs that includes at least some of the total matching pairs. The electronic device (101) can identify any one of the possible combinations of matching pair subsets. According to one embodiment, the electronic device (101) receives an input for a subset of matching pairs and can identify the subset of matching pairs. In operation S801, the electronic device (101) can initialize the loss function value. That is, the electronic device (101) can set and store the value of the loss function to 0 or a specified default value.

[0101] In operation S802, the electronic device (101) may select at least some matching pairs belonging to a subset of matching pairs identified in S801. Operation S802 may correspond to at least some of operations S721 and S722 of FIG. 7B. That is, it corresponds to the operation of identifying the first vector and the second vector of operation S721 and the operation of identifying the third vector and the fourth vector of S722, and specifically corresponds to the operation of selecting a plurality of matching pairs to form the first vector, the second vector, the third vector, and the fourth vector. The operation of selecting a matching pair in operation S802 can be understood as the operation of selecting a feature point on the first image and a feature point on the second image that correspond as matching pairs belonging to a subset of matching pairs identified in S801.

[0102] In operation S803, the electronic device (101) can verify the validity of the matching pairs selected in operation S802. The electronic device (101) can determine at least two rotation angles (e.g., first rotation angles ( ), second rotation angle( )) check, and the value regarding the difference between the above at least two rotation angles (e.g., the first value ( )) and a threshold value defined based on selected vectors (e.g., Validity can be verified based on comparison with ). In this case, the threshold value is the vector magnitude on the second image forming the rotation angles, as described above in Equation 4, and the set maximum position error ( It can be determined based on at least some of ) and the first constant c.

[0103] In operation S804, the electronic device (101) may reject a subset of matching pairs if it is determined (S803) that the selected matching pairs are invalid.

[0104] In operation S805, the electronic device (101) can output information that the matching pair subset is invalid in response to the matching pair subset being rejected.

[0105] In operation S806, if the electronic device (101) confirms that the selected match pairs are valid (S803), it may add the loss function value corresponding to the match pairs selected in S802 to the currently accumulated loss function value. The loss function value corresponding to the selected match pairs may be defined based on the first vector, second vector, third vector, and fourth vector identified from the selected match pairs. Specifically, the first vector on the first image ( ), second vector( ) and the third vector on the second image ( ) and the fourth vector( The first rotation angle derived by ) ) and second rotation angle( In the example of ), the first value ( It can be determined based on ). As shown in the mathematical formula below, the first value regarding the i, j, k, and l-th matching pairs ( Based on ), the loss function value to be accumulated in operation S806 ( ) is the first value( ) or the first value( It can be defined as the square of ).

[0106]

[0107] In operation S807, the electronic device (101) can check whether the validity of all matching pairs belonging to the matching pair subset has been verified. If the electronic device (101) confirms that there are no uninvestigated matching pairs, it may proceed to operation S808; otherwise, it may return to operation S802, select at least some other matching pairs included in the matching pair subset in addition to the matching pairs whose validity has been verified, and perform operations S802, S803 to S804, and S805 based on their validity, or repeat operations S802, S803 to S806.

[0108] In operation S808, the electronic device (101) can return the loss function value accumulated by operation S806 up to that point as the final loss function value of the matching pair subset. Thus, the final loss function value can be defined as F as shown in the following mathematical formula.

[0109]

[0110] FIG. 9 is a flowchart of an operation in which an electronic device according to various embodiments determines whether there is a subset of matching pairs for correctly estimating a similarity transformation between images, and if the subset of matching pairs exists, calculates an optimal subset of matching pairs and optimal similarity transformation estimation information.

[0111] In operation S901, the electronic device (101) may input a whole set of matching pairs. The electronic device (101) may define a plurality of corresponding feature points among a plurality of feature points of the first image and the second image as matching pairs, and may input all matching pairs identified in the first image and the second image as a whole set of matching pairs. In operation S901, the electronic device (101) may initialize a variable regarding the existence of a matching solution (e.g., 'Solution Exist' variable) to False. The 'Solution Exist' variable may be understood as a variable representing information regarding whether a valid subset of matching pairs exists among the set of matching pairs.

[0112] In operation S902, the electronic device (101) can sample a subset of matched pairs (K). The electronic device (101) can sample a subset of matched pairs (K), which is at least part of the entire set of matched pairs, through an algorithm or random sampling that extracts a specified subset of matched pairs (K). According to one embodiment, the electronic device (101) can set the size of the subset of matched pairs (K) to 3 or more.

[0113] In operation S903, the electronic device (101) can check whether the sampled matching pair subset (K) is valid. This can be done through operations S801 to S808 of FIG. 8. Based on the validity information and loss function value obtained as a result of the operation, the validity of the sampled matching pair subset (K) and the final loss function value can be determined. The electronic device (101) can proceed to operation S907 if the matching pair subset (K) is invalid, and proceed to operation S904 if it is valid.

[0114] In operation S904, the electronic device (101) can verify similarity transformation estimation information (T) based on the sampled matching pair subset (K). The electronic device (101) can estimate the similarity transformation between the first image and the second image based on the matching pair subset (K). In operation S904, since the sampled matching pair subset (K) is confirmed to be valid (S903), the electronic device (101) can set (update) the value of the 'Solution Exist' variable, which is a variable regarding whether a matching solution exists, to 'True'.

[0115] In operation S905, the electronic device (101) can determine the goodness of fit of a sampled matching pair subset (K). The goodness of fit may be determined based on at least some of the final loss function value and the similarity transformation estimation information (T), which is the result of the similarity transformation estimation. For example, the goodness of fit may be the final loss function value of the sampled matching pair subset (K) itself, or it may be determined based on error information determined based on the similarity transformation estimation information (T), or it may be a combination of the error information and the final loss function value.

[0116] In operation S906, the electronic device (101) can update the best matching pair subset (Best K) and the best similarity transformation estimation information (Best T). That is, the electronic device (101) can update the best similarity transformation information (Best T) and the best matching pair subset (Best K) by comparing the fitness of the current best matching pair subset (Best K) with the fitness of the sampled matching pair subset (K). Based on the comparison between the fitnesss, the electronic device (101) can maintain the existing best matching pair subset (Best K) or change the best matching pair subset (Best K) to the sampled matching pair subset (K).

[0117] In S907, the electronic device (101) can check whether the accumulated number of samples is greater than or equal to a specified value. If the number of samples repeatedly accumulated by operations S902 and S907 is greater than or equal to the specified value, the electronic device (101) determines that the number of samples is sufficient and proceeds to operation S908; if it is less than the specified value, it determines that the accumulated number of samples is insufficient and returns to operation S902. For example, the electronic device (101) can return to operation S902 to sample the matching pair subset again to identify the second matching pair subset, and repeat a series of operations S903, S904, S905, and S906 based on the second matching pair subset. The electronic device (101) can determine whether to proceed from operation S907 to operation S908 or to operation S902 based on a comparison between the specified value and the number of samples already sampled (accumulated number of samples). The electronic device (101) may accumulate and store the number of times sampling is performed each time sampling is repeated. A designated value that is compared with this accumulated number of samplings may be the total number of times the electronic device (101) determines to sample a subset of matching pairs. According to one embodiment, the electronic device (101) may determine the total number of times to sample a subset of matching pairs. The electronic device (101) may determine the total number of times to sample based on the total number of feature points on the image. For example, if there are constraints on computation time, it may be inefficient to validate all subsets of matching pairs. According to one embodiment, the electronic device (101) may determine the total number of times to sample based on at least some of prior information regarding the size of the subset of matching pairs (the number of matching pairs included in the subset of matching pairs), the total number of matching pairs, and the ratio of matching pairs expected to be correctly matched relative to the total number of matching pairs.

[0118] In operation S908, the electronic device (101) may return at least some of the following: the best matching pair subset (Best K), the best similarity transformation estimation information (Best T), and the final loss function value, which is the loss function value of the best matching pair subset (Best K) that is finally identified. At this time, the Best K and Best T values ​​may represent the final Best K and Best T that are updated according to the repetition of operation S906 by continuously sampling the matching pair subset (S902). At this time, the electronic device (101) may output information regarding the existence of a matching solution, Best T, and Best K based on the 'Solution Exist' variable value. If the 'Solution Exist' variable value is 'True', the electronic device (101) may output information that a matching solution exists and at least some of Best K and Best T, and if the variable value is 'False', it may output only information that a matching solution does not exist.

[0119] According to one embodiment, the electronic device (101) can perform each of the operations of FIG. 7a, FIG. 7b, FIG. 8 and FIG. 9 based on the number of feature points in the matching pair subset. For example, since at least two vectors, i.e., at least three feature points, must be selected per image to perform the method of the present invention, if fewer than three matching pairs are included in the matching pair subset, or if fewer than three matching pairs are included in the entire matching pair set, the electronic device (101) can perform a simpler similarity transformation estimation method without proceeding with the operations according to one embodiment proposed in the present invention, i.e., the operations of FIG. 7a and FIG. 7b and FIG. 8.

[0120] FIG. 10 is a diagram showing the average error value by method according to one embodiment.

[0121] FIG. 11 is a diagram illustrating the actual rejection rate according to the mismatch rate by method according to one embodiment.

[0122] FIG. 12 is a diagram illustrating the execution time according to the mismatch ratio by method according to one embodiment.

[0123] FIG. 13 is a diagram illustrating the execution time according to the sampling size of each method according to one embodiment.

[0124] Referring to FIGS. 10, 11, 12, and 13, the results of comparing the similarity transformation estimation method according to the present invention with other conventional methods can be seen. FIGS. 10, 11, 12, and 13 are the results of analyzing the performance of a method that generates a virtual test image set to find similarity transformations and outputs a valid matching pair subset.

[0125] Specifically, the coordinates of 10 feature points are virtually generated, and this corresponds to the result of an experiment on a similarity transformation that simultaneously includes rotation within the range of +30 to -30 degrees and translational movement within the ranges of +10 pixels and -10 pixels. The virtual feature points of the first image are generated in a 2D space of 80x80 using arbitrary integer coordinate values ​​so as not to overlap, and the feature points of the second image, which is the other image obtained by transforming according to the set true similarity transformation, are generated including errors; for matching pairs corresponding to outliers, integer errors were added following a uniform distribution in the range of ±2 to ±11, while for correct matching pairs, integer errors were added following a uniform distribution in the range of up to ±1. Referring to Fig. 10, the maximum positional error to be judged as an inlier accordingly ( The value of ) was set to 1 (pixel), and the ratio of mismatched pairs existing in the input set of matched pairs was tested in a total of 11 categories ranging from 0 to 1.0 (100%) in 0.1 intervals (mismatched ratio = 1 - (inlier ratio)). For each mismatched ratio value, experiments were conducted using a total of 28,000 randomly generated test cases (a total of 11 × 28,000 experiments). Execution time was measured using an Intel Xeon CPU E5-1660 @ 3.70GHz processor. The experiment was configured by setting the number of matched pairs selected from the subset of matched pairs at a time to 4, and the comparison method (conventional method) is as follows.

[0126] Method 1 ('sample set+1' method): Estimate a similarity transformation using selected matching pairs, and the locations of feature points on one image belonging to the set of selected matching pairs, transformed by the estimated similarity transformation, are all at maximum position error from the locations of corresponding feature points on the other image ( At a distance of ) or less, and simultaneously

[0127] There exists at least one matching pair not included in the selected set of matching pairs, so that the position transformed by the estimated similarity transformation has a maximum position error from the corresponding feature point position on the other image ( A method for determining that selected matching pairs are valid when they are within a distance of ) or less. In other words, a method for determining that a set of selected matching pairs is valid when not only the selected matching pairs but also at least one of the unselected matching pairs is in a correspondence relationship that matches the estimated similarity transformation.

[0128] Second method ('sample set only' method): Similar to the first method, but without checking other matching pairs that are not included in the selected set of matching pairs. In other words, if the selected matching pairs have a correspondence relationship that corresponds to the estimated similarity transformation, the selected set of matching pairs is determined to be valid without needing to consider matching pairs that do not belong to the selected set of matching pairs.

[0129] Referring to FIG. 10, the average error according to the first method ('sample set+1' method) is described as the first average error, the average error according to the second method ('sample set only' method) is described as the second average error, and the average error according to the method of the present invention is described as the third average error. The average error according to the mismatch ratio based on the method according to the present invention was similar to the second method ('sample set only' method) at 0.05 or less in the 0 to 0.5 range, and similar to the 0.25 or less in the 0.6 to 1.0 range. For reference, the 0.7 to 1.0 range is a range where there are no matching pairs within the allowable error, so a high matching error may occur.

[0130] Referring to Figure 11, the actual rejection rate of the algorithm according to the mismatch rate (the rate at which the algorithm completely rejected similarity transformation estimation out of 28,000 total test cases) is shown. For reference, in this experiment, the total number of feature points is 10 and the size of the matching pair subset (K) is fixed at 4, so it is desirable for the rejection rate to be close to 0 when the mismatch rate is equal to or less than 0.6.

[0131] Referring to FIG. 11, the rejection rate (11) according to the first method, the rejection rate (12) according to the second method, and the rejection rate (13) according to the method of the present invention can be seen. The rejection rate according to the method of the present invention was less than 0.01 when the mismatch rate was less than 0.6, and rejection occurred at a rate of about 0.13 when it was 0.6, and it can be seen that more than half of the test cases were rejected at 0.7. In this experiment, since the total number of feature points is 10 and the size of the matching pair subset (K) is fixed at 4, ideally, rejection should occur close to 100 percent starting from 0.7, but the reason this is not actually the case is that there are various cases of matching pairs that can produce similarity transformations, so even if some outliers are included, when looking only at the sample set itself, it can be a valid matching pair in terms of geometric similarity or distance error with the model. The method according to the invention proposed in this document showed a rejection rate similar to the second method ('sample set only' method), and it was found that the first method ('sample set+1' method) had an excessive number of rejections because the rate of incorrect rejection exceeded 0.05 when the mismatch rate was 0.5.

[0132] Referring to FIG. 12, the change in execution time according to the mismatch rate is shown. Referring to FIG. 12, the execution time (21) according to the first method ('sample set+1' method), the execution time (22) according to the second method ('sample set only' method), the execution time (23) according to the method of the present invention, and additionally the execution time (24) according to the conventional RANSAC framework can be observed. The first method ('sample set+1' method) did not show a significant reduction in execution time because it requires calculating similarity transformation results for unselected matching pairs to verify the validity of the selected set of matching pairs. The second method ('sample set only' method) calculates similarity transformations only for the set of matching pairs selected in the pre-check, so from the point where the mismatch rate becomes greater than 0.3, sufficient rejection occurs, thereby shortening the execution time. However, it can be seen that the method proposed by the present invention shows a significantly reduced execution time compared to both the first and second methods. The method proposed by the present invention can significantly reduce execution time because validation is performed with fewer operations than similarity transformation matrix estimation.

[0133] Figure 13 shows the change in execution time according to the number of match pairs selected during sampling (size of the subset of selected match pairs). Referring to Figure 11, the log value of the execution time according to the first method (31), the log value of the execution time according to the second method (32), and the log value of the execution time according to the method of the present invention (33) can be observed. This experiment was conducted by simulating the match error of all points to a maximum of ±1 in order to measure the execution time while excluding the effect of the reduction in execution time due to rejection in the RANSAC framework. Looking at these results, it can be seen that when following the method of the invention proposed in this document, even if the number of match pairs selected is increased from 5 to 9, the time to examine the selected match pairs is significantly shorter compared to the first method. Specifically, it can be confirmed that the execution speed improves by approximately 2 times when 4 match pairs are selected and by approximately 1.4 times when 9 are selected.

[0134] The embodiments according to the present disclosure described above may be implemented in the form of program instructions that can be executed through various computer components and may be recorded on a computer-readable recording medium or a non-transient recording medium. The computer-readable recording medium or a non-transient recording medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the computer-readable recording medium or a non-transient recording medium may be those specifically designed and configured for the present invention, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media or a non-transient recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The above hardware device or electronic device may be configured to operate as one or more software modules to perform processing according to the present disclosure, and vice versa.

[0135] The present embodiment may be represented by functional block configurations and various processing steps. These functional blocks may be implemented by various numbers of hardware, software, or combinations thereof that execute specific functions. For example, the embodiment may employ integrated circuit configurations such as memory, processing, logic, look-up tables, etc., capable of executing various functions by the control of one or more microprocessors or other control devices. Similar to how components may be implemented as software programming or software elements, the present embodiment may be implemented in programming or scripting languages ​​such as C, C++, Java, assembler, etc., including various algorithms implemented as data structures, processes, routines, or combinations of other programming configurations. Functional aspects may be implemented as algorithms executed on one or more processors. Additionally, the present embodiment may employ prior art for electronic configuration, signal processing, data processing, or combinations thereof.

Claims

Claim 1 A method for estimating a similarity transformation between a first image and a second image in an electronic device, comprising the step of receiving a subset of matching pairs including a plurality of matching pairs determined based on a correspondence relationship of at least some of the plurality of feature points included in each of the first image and the second image to estimate the similarity transformation; The method includes a step of verifying the validity of the matching pair subset, wherein the step of verifying the validity of the matching pair subset comprises: a step of verifying a first vector and a second vector based on at least some of the feature points identified based on the matching pair subset on the first image; a step of verifying a third vector corresponding to the first vector and a fourth vector corresponding to the second vector on the second image based on the matching pair subset; a step of verifying a first value regarding the difference between a first rotation angle formed by the first vector and the third vector and a second rotation angle formed by the second vector and the fourth vector; and a step of verifying the validity of the matching pairs corresponding to the first vector, the second vector, the third vector, and the fourth vector by comparing a threshold value identified based on the third vector and the fourth vector with the first value. A similarity transformation estimation method comprising the step of verifying the validity of a subset of the matched pairs based on the validity of the matched pairs, wherein the threshold value is determined based on the ratio of the second value to the magnitude of the third vector and the ratio of the second value to the magnitude of the fourth vector. Claim 2 A similarity transformation estimation method according to claim 1, wherein the step of verifying the validity of the matching pair subset comprises the step of determining the matching pair subset as invalid without needing to consider other first vectors, second vectors, third vectors, and fourth vectors that can be verified for the matching pair subset when the first value calculated for any first vector, second vector, third vector, and fourth vector that can be verified for the matching pair subset is greater than the threshold value. Claim 3 A similarity transformation estimation method according to claim 1, wherein the step of verifying the validity of the matching pair subset comprises the step of determining that the matching pair subset is valid if the first value calculated for all first vectors, second vectors, third vectors, and fourth vectors that can be verified for the matching pair subset is less than or equal to the threshold value. Claim 4 delete Claim 5 A method for estimating a similarity transformation according to claim 1, wherein the second value is defined as the product of the maximum allowable position error in the estimation of the similarity transformation and a first constant, and the first constant is defined as a positive real value greater than 0 and less than or equal to 2. Claim 6 A method for estimating a similarity transformation between a first image and a second image in an electronic device, comprising the step of receiving a subset of matching pairs including a plurality of matching pairs determined based on a correspondence relationship of at least some of the plurality of feature points included in each of the first image and the second image to estimate the similarity transformation; The method includes a step of verifying the validity of the matching pair subset, wherein the step of verifying the validity of the matching pair subset comprises: a step of verifying a first vector and a second vector based on at least some of the feature points identified based on the matching pair subset on the first image; a step of verifying a third vector corresponding to the first vector and a fourth vector corresponding to the second vector on the second image based on the matching pair subset; a step of verifying a first value regarding the difference between a first rotation angle formed by the first vector and the third vector and a second rotation angle formed by the second vector and the fourth vector; and a step of verifying the validity of the matching pairs corresponding to the first vector, the second vector, the third vector, and the fourth vector by comparing a threshold value identified based on the third vector and the fourth vector with the first value. The method comprises the step of verifying the validity of the matching pair subset based on the validity of the matching pairs, and the similarity transformation estimation method comprises: the step of sampling the matching pair subset from the entire set of matching pairs; the step of verifying the validity of the matching pair subset; if the matching pair subset is valid, the step of verifying the goodness of fit of the matching pair subset and similarity transformation estimation information including the value of one or more parameters regarding the similarity transformation between the first image and the second image based on the matching pair subset; the step of updating the optimal matching pair subset and the optimal similarity transformation estimation information to the matching pair subset and the similarity transformation estimation information based on the goodness of fit of the matching pair subset; and the step of verifying the cumulative sampling count.A similarity transformation estimation method further comprising the step of sampling a second subset of matching pairs that is different from the subset of matching pairs within the entire set of matching pairs based on the cumulative sampling count. Claim 7 In claim 6, the similarity transformation estimation method further comprises the step of outputting the optimal matching pair subset and the optimal similarity transformation estimation information based on the cumulative number of samplings. Claim 8 A similarity transformation estimation method according to claim 6, wherein the fitness of the above-mentioned matching pair subset is determined based on the loss function value of the above-mentioned matching pair subset, and the loss function value is determined based on the above-mentioned first value. Claim 9 A similarity transformation estimation method according to claim 1, further comprising: a step of checking the size of a matching pair subset; and a step of checking the first vector and the second vector only when the size of the matching pair subset is 3 or more, and omitting validation when the size of the matching pair subset is less than 3. Claim 10 In electronic devices, memory; The system includes a processor, wherein the processor receives a subset of matching pairs comprising a plurality of matching pairs determined based on correspondence relationships among at least some of a plurality of feature points included in each of the first image and the second image to estimate a similarity transformation between the first image and the second image, and is configured to verify the validity of the subset of matching pairs, and verifies a first vector and a second vector based on at least some of the feature points verified based on the subset of matching pairs on the first image, and verifies a third vector corresponding to the first vector and a fourth vector corresponding to the second vector on the second image based on the subset of matching pairs, verifies a first value regarding the difference between a first rotation angle formed by the first vector and the third vector and a second rotation angle formed by the second vector and the fourth vector, and compares the first value with a threshold value verified based on the third vector and the fourth vector to verify the validity of the matching pairs corresponding to the first vector, the second vector, the third vector, and the fourth vector. An electronic device configured to verify and verify the validity of a subset of the matched pairs based on the validity of the matched pairs, wherein the threshold value is determined based on the ratio of the second value to the magnitude of the third vector and the ratio of the second value to the magnitude of the fourth vector. Claim 11 A computer-readable non-transient recording medium storing a program for executing on a computer a method for estimating a similarity transformation between a first image and a second image, wherein the method comprises the step of receiving a subset of matching pairs including a plurality of matching pairs determined based on correspondence relationships among at least some of a plurality of feature points included in each of the first image and the second image for estimating a similarity transformation; The method includes a step of verifying the validity of the matching pair subset, wherein the step of verifying the validity of the matching pair subset comprises: a step of verifying a first vector and a second vector based on at least some of the feature points identified based on the matching pair subset on the first image; a step of verifying a third vector corresponding to the first vector and a fourth vector corresponding to the second vector on the second image based on the matching pair subset; a step of verifying a first value regarding the difference between a first rotation angle formed by the first vector and the third vector and a second rotation angle formed by the second vector and the fourth vector; and a step of verifying the validity of the matching pairs corresponding to the first vector, the second vector, the third vector, and the fourth vector by comparing a threshold value identified based on the third vector and the fourth vector with the first value. A non-transient recording medium comprising the step of verifying the validity of a subset of the matching pairs based on the validity of the matching pairs, wherein the threshold value is determined based on the ratio of the second value to the magnitude of the third vector and the ratio of the second value to the magnitude of the fourth vector.

Images