Key point matching method and device based on wide-view eye fundus image

Abstract

The invention discloses a key point matching method and device based on a wide-view eye fundus image, and the method comprises the steps: determining a first overlapping region between a first image and a second image based on a matching result of a first key point set and a second key point set; determining a plurality of first matching point pairs based on the first overlapping region; determining a plurality of first transformation matrixes based on the plurality of first matching point pairs; for each first transformation matrix, determining a corresponding target point pair from the plurality of first matching point pairs by using the first transformation matrix; and determining a transformation relation between the first image and the second image based on the first transformation matrix corresponding to the maximum total number of the target point pairs. The method can effectively solve the problem of deformation of the wide-view eye fundus image during registration, and avoids the situation of missing key point pairs, thereby improving the accuracy of key point matching between the first image and the second image.

Description

Technical Field

[0001] The present application relates to the field of medical image processing, and in particular to a key point matching method and device based on wide-field fundus images. Background Art

[0002] The commonly used key point matching methods are not suitable for wide-field fundus images because wide-field fundus images have a large field of view, the eyeball is a unique ellipsoidal structure, and there is a large optical distortion. Therefore, the traditional RANSAC (Random Sample Consensus) + projection transformation method cannot accurately find key point pairs for wide-field fundus images.

[0003] Since the ellipsoidal structure and optical distortion of the eyeball can be expressed by a binary quadratic equation in a two-dimensional coordinate system, the transformation relationship between wide-field fundus image pairs basically conforms to the binary quadratic transformation. However, if the binary quadratic transformation method is directly used, it is difficult to directly match all key point pairs due to the deformation of the wide-field fundus image, so the key point pairs of the wide-field fundus image cannot be found well, which leads to misalignment in the subsequent splicing. Summary of the invention

[0004] The present application provides a key point matching method and device based on wide-field fundus images, aiming to improve the key point matching accuracy of wide-field fundus images.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] A key point matching method based on a wide-field fundus image, comprising:

[0007] Based on the matching results of the first key point set and the second key point set, determining a first overlapping area between the first image and the second image; the first key point set includes key points of the first image; the second key point set includes key points of the second image; the first image and the second image are both wide-field fundus images of the same eyeball; the matching results include key point pairs having a matching relationship;

[0008] Based on the first overlapping area, a plurality of first matching point pairs are determined; the first matching point pairs include a first key point and a second key point having a matching relationship; the first key point is a key point of a first image area, and the second key point is a key point of a second image area; the first image area is a partial image area in the first image corresponding to the first overlapping area, and the second image area is an image area in the second image corresponding to the first image area;

[0009] Determine a plurality of first transformation matrices based on the plurality of first matching point pairs;

[0010] For each of the first transformation matrices, determining a corresponding target point pair from a plurality of the first matching point pairs using the first transformation matrix;

[0011] Based on a first transformation matrix corresponding to the largest total number of target point pairs, a transformation relationship between the first image and the second image is determined; the transformation relationship represents a key point matching result between the two images.

[0012] Optionally, determining a transformation relationship between the first image and the second image based on a first transformation matrix corresponding to a maximum total number of target point pairs includes:

[0013] The first transformation matrix corresponding to the largest total number of target point pairs is determined as the target transformation matrix;

[0014] transforming the second image using the target transformation matrix to obtain a third image;

[0015] Determining a second overlapping area between the first image and the third image based on a matching result of the first key point set and a third key point set; the third key point set includes key points of the third image;

[0016] Based on the second overlapping area, a plurality of second matching point pairs are determined; the second matching point pairs include a third key point and a fourth key point having a matching relationship; the third key point is a key point of a third image area, and the fourth key point is a key point of a fourth image area; the third image area is a partial image area in the first image corresponding to the second overlapping area, and the fourth image area is an image area in the third image corresponding to the third image area;

[0017] Determine a plurality of second transformation matrices based on the plurality of second matching point pairs;

[0018] For each of the second transformation matrices, determine a corresponding target point pair from a plurality of second matching point pairs using the second transformation matrix;

[0019] Determining a transformation relationship between the first image and the third image based on a second transformation matrix corresponding to a maximum total number of target point pairs;

[0020] According to the inverse transformation of the target transformation matrix, the transformation relationship between the first image and the third image is corrected to obtain the transformation relationship between the first image and the second image.

[0021] Optionally, determining a first overlapping area between the first image and the second image based on the matching result of the first key point set and the second key point set includes:

[0022] Matching the first key point set and the second key point set to obtain a first key point pair;

[0023] Eliminate erroneous key point pairs from the obtained plurality of first key point pairs to obtain second key point pairs;

[0024] Using a random sampling consensus algorithm, selecting a third key point pair from the plurality of second key point pairs obtained;

[0025] Based on the screened plurality of third key point pairs, a first overlapping area between the first image and the second image is determined.

[0026] Optionally, determining a first overlapping area between the first image and the second image based on the plurality of third key point pairs obtained by screening includes:

[0027] By screening the plurality of third key point pairs obtained, calculating the affine transformation correspondence relationship between the first image and the second image;

[0028] Using the affine transformation correspondence, an affine transformation is performed on the second image to obtain a first overlapping area between the first image and the second image.

[0029] Optionally, determining a plurality of first matching point pairs based on the first overlapping area includes:

[0030] Determining a first image area and a second image area based on the first overlapping area;

[0031] Performing bidirectional matching on key points of the first image region and the second image region to obtain a fourth key point pair;

[0032] A random sampling consistency algorithm is used to screen out a plurality of first matching point pairs from the plurality of fourth key point pairs obtained.

[0033] Optionally, determining a plurality of first transformation matrices based on the plurality of first matching point pairs includes:

[0034] Performing multiple random sampling on the plurality of first matching point pairs to obtain multiple sampling results; the sampling results include n first matching point pairs, where n is a positive integer greater than or equal to 6;

[0035] For each sampling result, the least square method is used to calculate the rotation and scaling parameters of the image rigid transformation;

[0036] It is determined whether the rotation scaling parameter meets the preset conditions. If so, a first transformation matrix is ​​calculated using the sampling result and a binary quadratic transformation equation.

[0037] Optionally, for each of the first transformation matrices, determining a corresponding target point pair from a plurality of the first matching point pairs using the first transformation matrix includes:

[0038] For each of the first transformation matrices, using the first transformation matrix, calculating a theoretical point corresponding to a key point in the first matching point pair;

[0039] For each of the first matching point pairs, if the distance between the key point in the first matching point pair and the corresponding theoretical point is within a preset range, the first matching point pair is determined as the target point pair corresponding to the first transformation matrix.

[0040] A key point matching device based on a wide-view fundus image, comprising:

[0041] An overlapping area determination unit is used to determine a first overlapping area between a first image and a second image based on a matching result of a first key point set and a second key point set; the first key point set includes key points of the first image; the second key point set includes key points of the second image; the first image and the second image are both wide-field fundus images of the same eyeball; the matching result includes key point pairs having a matching relationship;

[0042] a matching point pair determination unit, configured to determine a plurality of first matching point pairs based on the first overlapping area; the first matching point pairs include a first key point and a second key point having a matching relationship; the first key point is a key point of a first image area, and the second key point is a key point of a second image area; the first image area is a partial image area in the first image corresponding to the first overlapping area, and the second image area is an image area in the second image corresponding to the first image area;

[0043] a transformation matrix determining unit, configured to determine a plurality of first transformation matrices based on a plurality of the first matching point pairs;

[0044] a target point pair determining unit, configured to determine, for each of the first transformation matrices, a corresponding target point pair from a plurality of the first matching point pairs using the first transformation matrix;

[0045] The transformation relationship determination unit is used to determine the transformation relationship between the first image and the second image based on the first transformation matrix corresponding to the largest total number of target point pairs; the transformation relationship represents the key point matching result between the two images.

[0046] A storage medium comprises a stored program, wherein the program is executed by a processor to execute the key point matching method based on a wide-field fundus image.

[0047] An electronic device, comprising: a processor, a memory and a bus; the processor and the memory are connected via the bus;

[0048] The memory is used to store programs, and the processor is used to run the programs, wherein the program executes the key point matching method based on wide-field fundus images when the processor runs the program.

[0049] The technical solution provided by the present application determines the first overlapping area between the first image and the second image based on the matching results of the first key point set and the second key point set. Based on the first overlapping area, multiple first matching point pairs are determined. Based on the multiple first matching point pairs, multiple first transformation matrices are determined. For each first transformation matrix, the corresponding target point pair is determined from the multiple first matching point pairs using the first transformation matrix. Based on the first transformation matrix corresponding to the largest total number of target point pairs, the transformation relationship between the first image and the second image is determined. The present application determines the corresponding multiple first transformation matrices based on the overlapping area between the first image and the second image, and selects the first transformation matrix corresponding to the largest total number of target point pairs from the multiple first transformation matrices as the transformation relationship between the first image and the second image, which effectively solves the problem of deformation of wide-field fundus images during registration and avoids the situation of missing key point pairs, thereby effectively improving the accuracy of key point matching between the first image and the second image. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A flowchart of a key point matching method based on a wide-field fundus image provided in an embodiment of the present application;

[0052] Figure 2 A flowchart of another key point matching method based on wide-field fundus images provided in an embodiment of the present application;

[0053] Figure 3 A flowchart of another key point matching method based on wide-field fundus images provided in an embodiment of the present application;

[0054] Figure 4 A flowchart of another key point matching method based on wide-field fundus images provided in an embodiment of the present application;

[0055] Figure 5 A flowchart of another key point matching method based on wide-field fundus images provided in an embodiment of the present application;

[0056] Figure 6 A flowchart of another key point matching method based on wide-field fundus images provided in an embodiment of the present application;

[0057] Figure 7 A schematic diagram of the architecture of a key point matching device based on a wide-field fundus image provided in an embodiment of the present application;

[0058] Figure 8 A schematic diagram of the effect of key point matching provided in an embodiment of the present application;

[0059] Fig. 9 A schematic diagram of another key point matching effect provided in an embodiment of the present application;

[0060] Fig.10 A schematic diagram of another key point matching effect provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0063] like Figure 1As shown, it is a flowchart of a key point matching method based on wide-field fundus images provided in an embodiment of the present application, including the following steps.

[0064] S101: Determine a first overlapping area between a first image and a second image based on matching results of a first key point set and a second key point set.

[0065] Among them, the first key point set includes the key points of the first image, the second key point set includes the key points of the second image, the first image and the second image are both wide-field fundus images of the same eyeball, and the matching results include key point pairs with a matching relationship. The key point pairs include two key points, one of which belongs to the first image, and the other key point belongs to the second image.

[0066] In some examples, the first image may be considered as a reference image of the wide-field fundus, and the second image may be considered as an image to be registered of the wide-field fundus.

[0067] It is understandable that the first key point set can be obtained by performing key point detection (or feature point extraction) on the first image, and similarly, the second key point set can be obtained by performing key point detection (or feature point extraction) on the second image.

[0068] In some examples, the key point detection method adopted in the embodiments of the present application includes but is not limited to: SIFT (Scale-invariant feature transform), SUFR (Speeded Up Robust Features), ORB (Oriented Fast and Rotated Brief), etc.

[0069] It should be noted that after obtaining the first key point set and the second key point set, a coarse match between the first image and the second image may be performed based on the first key point set and the second key point set to obtain a first overlapping area between the first image and the second image.

[0070] Optionally, the implementation process of determining the first overlapping area between the first image and the second image based on the matching result of the first key point set and the second key point set can be found in Figure 2 Steps shown and explanations of the steps.

[0071] S102: Determine a plurality of first matching point pairs based on the first overlapping area.

[0072] The first matching point pair includes a first key point and a second key point that have a matching relationship, the first key point is a key point of the first image area, and the second key point is a key point of the second image area. The first image area is a partial image area in the first image corresponding to the first overlapping area, and the second image area is an image area in the second image corresponding to the first image area.

[0073] In some examples, the first image area and the second image area can be determined based on a first overlapping area between the first image and the second image, key point detection can be performed on the first image area to obtain a corresponding key point set A, key point detection can be performed on the first image area to obtain a corresponding key point set B, and finally key point matching can be performed on key point set A and key point set B to obtain multiple first matching point pairs.

[0074] Optionally, the implementation process of determining a plurality of first matching point pairs based on the first overlapping area between the first image and the second image can be found in Figure 3 Steps shown and explanations of the steps.

[0075] S103: Determine a plurality of first transformation matrices based on a plurality of first matching point pairs.

[0076] The type of the first transformation matrix is ​​a binary quadratic transformation matrix.

[0077] Optionally, based on multiple first matching point pairs, the implementation process of determining multiple first transformation matrices can be found in Figure 4 Steps shown and explanations of the steps.

[0078] S104: For each first transformation matrix, determine a corresponding target point pair from a plurality of first matching point pairs using the first transformation matrix.

[0079] For each first transformation matrix, the implementation process of determining the corresponding target point pair from multiple first matching point pairs using the first transformation matrix can be found in Figure 5 Steps shown and explanations of the steps.

[0080] S105: Determine a transformation relationship between the first image and the second image based on a first transformation matrix corresponding to a maximum total number of target point pairs.

[0081] Among them, the transformation relationship represents the key point matching result between the two images.

[0082] It should be noted that the total number of target point pairs corresponding to each first transformation matrix can be counted, and based on the total number of target point pairs, the first transformation matrix corresponding to the largest total number of target point pairs can be selected from multiple first transformation matrices to determine the transformation relationship between the first image and the second image.

[0083] Generally speaking, determining the transformation relationship between the first image and the second image based on the first transformation matrix corresponding to the largest total number of target point pairs can effectively solve the problem of deformation of wide-field fundus images during registration and avoid the situation of missing key point pairs, thereby effectively improving the correctness of the transformation relationship between the first image and the second image.

[0084] Optionally, the problem of large deformation near the edge of the overlapping area between two wide-field fundus images can also be solved by using the first transformation matrix corresponding to the largest total number of target point pairs. This solution can be found in Figure 6 The steps are shown and their explanations.

[0085] In some examples, based on the transformation relationship obtained in the process shown in S101-S105, the key point matching effect of the first image and the second image is achieved, which can be seen in Fig. 9 As shown, specifically, the key point coordinates of the second image (the image to be registered) can be multiplied by the first transformation matrix shown in the transformation relationship to obtain the key point matching result.

[0086] The process shown in S101-S105 above determines multiple first transformation matrices based on the first overlapping area between the first image and the second image, and selects the first transformation matrix corresponding to the largest total number of target point pairs from the multiple first transformation matrices as the transformation relationship between the first image and the second image. This can effectively solve the problem of deformation of wide-field fundus images during registration and avoid the situation of missing key point pairs, thereby effectively improving the accuracy of key point matching between the first image and the second image.

[0087] like Figure 2 As shown, it is a flow chart of another key point matching method based on wide-field fundus images provided in an embodiment of the present application, including the following steps.

[0088] S201: Match the first key point set and the second key point set to obtain a first key point pair.

[0089] After obtaining the first image and the second image, key point detection is performed on the first image according to a preset key point detection algorithm to obtain a first key point set, and key point detection is performed on the second image to obtain a second key point set. After obtaining the first key point set and the second key point set, the first key point set and the second key point set are matched to obtain a first key point pair.

[0090] In some examples, the present embodiment may adopt a brute force matching method based on Hamming distance to achieve key point matching (ie, matching between the first key point set and the second key point set).

[0091] S202: Eliminate erroneous key point pairs from the obtained multiple first key point pairs to obtain second key point pairs.

[0092] Among them, eliminating erroneous key point pairs from the multiple first key point pairs obtained can reduce the error rate of coarse matching. Specifically, the implementation method of eliminating erroneous key point pairs from the first key point pairs includes but is not limited to GMS (Grid-based Motion Statistics, grid-based motion statistics algorithm).

[0093] Generally speaking, GMS is a fast and extremely robust image feature matching algorithm that aims to eliminate feature mismatches through statistical methods and improve matching quality and speed. The core idea of ​​the GMS algorithm is to use the smoothness of motion as a statistic and calculate it through matching of local areas. This method converts high matching numbers into high matching quality and provides a real-time, super-powerful matching system. GMS can convert high-quality matching numbers into high-quality matching results by using motion smoothness as a statistic to match local areas.

[0094] It can be understood that by using GMS to remove erroneous key point pairs from the obtained multiple first key point pairs to obtain the second key point pairs, the first key point pairs can be purified to reduce the error rate of key point matching.

[0095] S203: Using a random sampling consistency algorithm, select a third key point pair from the obtained multiple second key point pairs.

[0096] Among them, RANSAC is an iterative method for estimating mathematical model parameters from a data set containing outliers. The RANSAC algorithm randomly selects sample subsets from the data set in an iterative manner, estimates model parameters based on these subsets, and then evaluates the deviation of all data from the model, thereby identifying data points that conform to the model (inliers) and data points that do not conform to the model (outliers).

[0097] It can be understood that by using RANSAC to screen the third key point pair from the multiple second key point pairs obtained, the second key point pairs can be purified again, thereby improving the accuracy of the key point matching between the first image and the second image.

[0098] S204: Determine a first overlapping area between the first image and the second image based on the plurality of third key point pairs obtained by screening.

[0099] Among them, after screening and obtaining multiple third key point pairs, the affine transformation correspondence between the first image and the second image can be calculated through the multiple third key point pairs, and then the affine transformation correspondence is used to perform affine transformation on the second image (image to be registered), thereby completing a rough match between the first image and the second image, and obtaining a first overlapping area between the first image and the second image.

[0100] In some examples, the second image is affine transformed using the affine transformation correspondence to achieve key point matching between the first image and the second image. The effect of the key point matching can be seen in Figure 8 As shown, specifically, the key point matching result can be obtained by multiplying the key point coordinates of the second image by the matrix shown in the affine transformation correspondence relationship.

[0101] In the process shown in S201-S204 above, GMS and RANSAC are used to screen the first key point pair multiple times to obtain a third key point pair with higher accuracy as the first overlapping area between the first image and the second image, thereby improving the accuracy of key point matching between the first image and the second image.

[0102] like Figure 3 As shown, it is a flow chart of another key point matching method based on wide-field fundus images provided in an embodiment of the present application, including the following steps.

[0103] S301: Determine a first image area and a second image area based on a first overlapping area.

[0104] Among them, the first image and the second image can be aligned based on the first overlapping area to obtain an aligned image, and the aligned image can be grid-divided according to a preset grid division ratio to determine the target grid corresponding to the overlapping area, and then the image area corresponding to the target grid is obtained from the first image and marked as the first image area, and the image area corresponding to the target grid is obtained from the second image and marked as the second image area, and the second image area corresponds to the first image area.

[0105] In some examples, the preset grid division ratio can be set by a technician according to actual conditions, for example, it can be set to 20*20, that is, the aligned image is divided into 400 grids.

[0106] S302: Perform bidirectional matching on key points of the first image region and the second image region to obtain a fourth key point pair.

[0107] Among them, key point detection can be performed on the first image area and the second image area respectively to obtain key points of the first image area and the second image area. In addition, bidirectional matching of the key points of the first image area and the second image area can effectively improve the matching accuracy of the fourth key point pair.

[0108] In some examples, bidirectional matching is a conventional feature point matching method. The principle is: assuming there are two feature point sets A and B, the general brute force matching is to only calculate the distance between each feature point in A and all points in B (vector feature distance), and select the point with the closest distance as the matching point, that is, to obtain the key point pair. Bidirectional matching is to calculate the distance between each feature point in B and all points in A, select the point with the closest distance as the matching point, and retain the point pairs with the same calculation results twice to improve the stability of matching.

[0109] S303: Utilizing a random sampling consistency algorithm, select a plurality of first matching point pairs from the obtained plurality of fourth key point pairs.

[0110] Among them, by using a random sampling consistency algorithm to select multiple first matching point pairs from the obtained multiple fourth key point pairs, the matching correctness of the first matching point pairs can be further improved.

[0111] The process shown in S301-S303 above can use bidirectional matching and random sampling consistency algorithm to achieve key point matching between the first image area and the second image area to obtain multiple first matching point pairs with high matching accuracy.

[0112] like Figure 4 As shown, it is a flow chart of another key point matching method based on wide-field fundus images provided in an embodiment of the present application, including the following steps.

[0113] S401: Perform multiple random sampling on multiple first matching point pairs to obtain multiple sampling results.

[0114] The sampling result includes n first matching point pairs, where n is a positive integer greater than or equal to 6. Specifically, after all first matching point pairs are obtained, n first matching point pairs are randomly sampled from all first matching point pairs to achieve one random sampling, and the random sampling is repeated multiple times to obtain corresponding multiple sampling results.

[0115] In a possible implementation manner, it should be ensured that the first matching point pairs included in multiple sampling results are not completely the same.

[0116] In some examples, a plurality of first matching point pairs are randomly sampled each time, and the number of first matching point pairs shown in the obtained sampling result may be set to 6.

[0117] In a possible implementation manner, the total number of first matching point pairs is 100, and 200 random samplings may be performed respectively, and each sampling result obtained includes 6 first matching point pairs.

[0118] It should be noted that in order to ensure that the sampling results can be used to calculate the matrix of the binary quadratic transformation later, it is necessary to ensure that the sampling results contain at least 6 first matching point pairs, so n is a positive integer greater than or equal to 6. In addition, the number of random sampling can be set by the technician according to the actual situation, for example, 2000 random samplings can be performed.

[0119] S402: For each sampling result, the least square method is used to calculate the rotation and scaling parameters of the image rigid transformation.

[0120] The rotation and scaling parameters include rotation angle parameters and scaling parameters.

[0121] In some examples, the image rigid transformation formula is shown as (1).

[0122]

[0123] In formula (1), k represents the scaling parameter, θ represents the rotation angle parameter, t1 represents the translation parameter in the X-axis direction, t2 represents the translation parameter in the Y-axis direction, x and y represent the coordinates of the key point, and newx and newy represent the coordinates of the key point after rigid transformation.

[0124] S403: Determine whether the rotation scaling parameter meets the preset conditions. If so, calculate a first transformation matrix according to the sampling result and the binary quadratic transformation equation.

[0125] The preset condition may be that the rotation angle meets a preset angle threshold, and the scaling amount meets a preset range threshold.

[0126] In some examples, the preset angle threshold may be set to [-30°, 30°], and the preset zoom range threshold may be set to [0.5, 1.5].

[0127] It should be noted that if the rotation scaling parameter corresponding to the sampling result does not meet the preset conditions, the sampling result will be deleted and re-sampling will be performed.

[0128] The expression of the binary quadratic transformation equation can be seen in formula (2).

[0129]

[0130] In formula (2), a0 represents the translation amount in the X-axis direction, a1 and a2 represent the primary transformation parameters in the X-axis direction, a3, a4 and a5 represent the secondary transformation parameters in the X-axis direction, b0 represents the translation amount in the Y-axis direction, b1 and b2 represent the primary transformation parameters in the Y-axis direction, b3, b4 and b5 represent the secondary transformation parameters in the Y-axis direction, x and y represent the key point coordinates in the first matching point pair shown in the sampling result, and newx and newy represent the key point coordinates after the binary secondary transformation.

[0131] The matrix is ​​calculated by using the coordinates of the first matching point pair that meets the preset conditions through the binary quadratic transformation equation This is the first transformation matrix.

[0132] The process shown in S401-S403 above can utilize multiple sampling results of multiple first matching point pairs to obtain multiple first transformation matrices, providing objective reference data for subsequent key point matching, thereby effectively improving the objectivity of key point matching between the first image and the second image.

[0133] like Figure 5 As shown, it is a flow chart of another key point matching method based on wide-field fundus images provided in an embodiment of the present application, including the following steps.

[0134] S501: For each first transformation matrix, use the first transformation matrix to calculate a theoretical point corresponding to a key point in a first matching point pair.

[0135] Among them, the key point can be selected to be located in the second image. For example, for each first matching point pair (P1, P2), the key point P2 located in the second image is substituted into the binary quadratic transformation equation to obtain the theoretical point P2' corresponding to the key point P2 (specifically the coordinates of the theoretical point, P2' is located in the first image).

[0136] Alternatively, the key point can be selected to be located in the first image. First, for each of the first transformation matrices, the inverse matrix of the first transformation matrix is ​​used to calculate the theoretical point P1' corresponding to the key point P1 in the first matching point pair, and the theoretical point P1' is located in the second image.

[0137] S502: For each first matching point pair, determine whether the distance between the key point in the first matching point pair and the corresponding theoretical point is within a preset range.

[0138] If the distance between the key point in the first matching point pair and the corresponding theoretical point is within the preset range, S503 is executed; if the distance between the key point in the first matching point pair and the corresponding theoretical point is not within the preset range, S504 is executed.

[0139] It should be noted that the first matching point pair contains two key points. When at least one of the two key points has a distance from its corresponding theoretical point within a preset range, it can be determined that the distance between the key point in the first matching point pair and its corresponding theoretical point is within the preset range.

[0140] S503: Based on the first matching point pair, determine the target point pair corresponding to the first transformation matrix.

[0141] S504: Determine that the first matching point pair is not the target point pair corresponding to the first transformation matrix.

[0142] The process shown in S501-S504 above can determine the number of target point pairs corresponding to each first transformation matrix, thereby providing a reliable reference for key point matching between the first image and the second image.

[0143] like Figure 6 As shown, it is a flow chart of another key point matching method based on wide-field fundus images provided in an embodiment of the present application, including the following steps.

[0144] S601: Determine the first transformation matrix corresponding to the largest total number of target point pairs as the target transformation matrix.

[0145] S602: transform the second image using the target transformation matrix to obtain a third image.

[0146] Here, the second image is transformed using the target transformation matrix to obtain the third image, which is a conventional technical implementation of image transformation and will not be described in detail here.

[0147] It should be noted that the purpose of transforming the second image using the target transformation matrix is ​​to obtain the third image, and the third image is used to replace the second image to solve the problem of large deformation near the edge of the image overlapping area, so that the key point matching effect between the first image and the second image is better.

[0148] S603: Determine a second overlapping area between the first image and the third image based on the matching results of the first key point set and the third key point set.

[0149] The third key point set includes key points of the third image.

[0150] In some examples, key point detection (or feature point extraction) may be performed on the third image to obtain a third key point set.

[0151] It should be noted that the implementation process and execution principle of determining the second overlapping area between the first image and the third image based on the matching results of the first key point set and the third key point set are the same as the implementation process and execution principle of S101, which can be referred to and will not be repeated here.

[0152] In a possible implementation, after obtaining the first key point set and the third key point set, a coarse match between the first image and the third image may be performed based on the first key point set and the third key point set to obtain a second overlapping area between the first image and the third image.

[0153] S604: Determine a plurality of second matching point pairs based on the second overlapping area.

[0154] The second matching point pair includes a third key point and a fourth key point having a matching relationship, the third key point is a key point of the third image area, and the fourth key point is a key point of the fourth image area. The third image area is a partial image area in the first image corresponding to the second overlapping area, and the fourth image area is an image area in the third image corresponding to the third image area.

[0155] It should be noted that the implementation process and execution principle of determining multiple second matching point pairs based on the second overlapping area between the first image and the third image are the same as the implementation process and execution principle of S102, which can be referred to and will not be described again.

[0156] S605: Determine multiple second transformation matrices based on multiple second matching point pairs.

[0157] The type of the second transformation matrix is ​​a binary quadratic transformation matrix.

[0158] It should be noted that the implementation process and execution principle of determining multiple second transformation matrices based on multiple second matching point pairs are the same as the implementation process and execution principle of S103, which can be referred to and will not be described in detail.

[0159] S606: For each second transformation matrix, determine a corresponding target point pair from a plurality of second matching point pairs using the second transformation matrix.

[0160] Among them, for each second transformation matrix, the implementation process and execution principle of using the second transformation matrix to determine the corresponding target point pair from multiple second matching point pairs are the same as the implementation process and execution principle of S104, which can be referred to and will not be repeated here.

[0161] S607: Determine a transformation relationship between the first image and the third image based on the second transformation matrix corresponding to the largest total number of target point pairs.

[0162] Among them, the total number of target point pairs corresponding to each second transformation matrix can be counted, and according to the total number of target point pairs, the second transformation matrix corresponding to the largest total number of target point pairs is selected from each second transformation matrix to determine the transformation relationship between the first image and the third image.

[0163] In some examples, key point matching of the first image and the third image is achieved based on the transformation relationship obtained by the process shown in S601-S607. Specifically, the key point coordinates of the third image are multiplied by the second transformation matrix shown in the transformation relationship to obtain the key point matching result.

[0164] S608: According to the inverse transformation of the target transformation matrix, the transformation relationship between the first image and the third image is corrected to obtain the transformation relationship between the first image and the second image.

[0165] Among them, according to the inverse transformation of the target transformation matrix, the transformation relationship between the first image and the third image is corrected, which is essentially to inversely transform the third image into the second image to obtain the transformation relationship between the first image and the second image.

[0166] In the above process S601-S608, the transformation relationship between the first image and the third image is used as a reference to determine the transformation relationship between the first image and the second image, so as to solve the problem of large deformation near the edge of the overlapping area of ​​the images by using the third image instead of the second image. For specific key point matching effects, see Fig.10 As shown, the negative impact caused by the deformation of the edge position of the image can be reduced, so that the registration effect of the transformation relationship between the first image and the second image is better.

[0167] like Figure 7 As shown, it is a schematic diagram of the architecture of a key point matching device based on a wide-field fundus image provided in an embodiment of the present application, including the units shown below.

[0168] The overlapping area determination unit 100 is used to determine the first overlapping area between the first image and the second image based on the matching results of the first key point set and the second key point set; the first key point set includes the key points of the first image; the second key point set includes the key points of the second image; the first image and the second image are both wide-field fundus images of the same eyeball; the matching results include key point pairs that have a matching relationship.

[0169] The matching point pair determination unit 200 is used to determine multiple first matching point pairs based on the first overlapping area; the first matching point pairs include a first key point and a second key point that have a matching relationship; the first key point is a key point of the first image area, and the second key point is a key point of the second image area; the first image area is a partial image area in the first image corresponding to the first overlapping area, and the second image area is an image area in the second image corresponding to the first image area.

[0170] The transformation matrix determining unit 300 is used to determine a plurality of first transformation matrices based on a plurality of first matching point pairs.

[0171] The target point pair determining unit 400 is configured to determine, for each first transformation matrix, a corresponding target point pair from a plurality of first matching point pairs using the first transformation matrix.

[0172] The transformation relationship determination unit 500 is used to determine the transformation relationship between the first image and the second image based on the first transformation matrix corresponding to the largest total number of target point pairs; the transformation relationship represents the key point matching result between the two images.

[0173] Optionally, the transformation relationship determination unit 500 is specifically used to: determine the first transformation matrix corresponding to the largest total number of target point pairs as the target transformation matrix; use the target transformation matrix to transform the second image to obtain a third image; based on the matching results of the first key point set and the third key point set, determine the second overlapping area between the first image and the third image; the third key point set includes the key points of the third image; based on the second overlapping area, determine multiple second matching point pairs; the second matching point pair includes a third key point and a fourth key point that have a matching relationship; the third key point is a key point of the third image area, and the fourth key point is a key point of the fourth image area ; The third image area is a partial image area in the first image corresponding to the second overlapping area, and the fourth image area is an image area in the third image corresponding to the third image area; based on multiple second matching point pairs, multiple second transformation matrices are determined; for each second transformation matrix, the corresponding target point pairs are determined from the multiple second matching point pairs using the second transformation matrix; based on the second transformation matrix corresponding to the largest total number of target point pairs, the transformation relationship between the first image and the third image is determined; according to the inverse transformation of the target transformation matrix, the transformation relationship between the first image and the third image is corrected to obtain the transformation relationship between the first image and the second image.

[0174] Optionally, the overlapping area determination unit 100 is specifically used to: match the first key point set and the second key point set to obtain first key point pairs; eliminate erroneous key point pairs from the multiple first key point pairs obtained to obtain second key point pairs; use a random sampling consistency algorithm to screen third key point pairs from the multiple second key point pairs obtained; and determine the first overlapping area between the first image and the second image based on the multiple third key point pairs obtained by screening.

[0175] Optionally, the overlapping area determination unit 100 is specifically used to: calculate the affine transformation correspondence between the first image and the second image by screening the multiple third key point pairs obtained; and perform an affine transformation on the second image using the affine transformation correspondence to obtain a first overlapping area between the first image and the second image.

[0176] Optionally, the matching point pair determination unit 200 is specifically used to: determine the first image area and the second image area based on the first overlapping area; perform bidirectional matching on the key points of each of the first image area and the second image area to obtain a fourth key point pair; and use a random sampling consistency algorithm to screen out multiple first matching point pairs from the multiple fourth key point pairs obtained.

[0177] Optionally, the transformation matrix determination unit 300 is specifically used to: perform multiple random sampling on multiple first matching point pairs to obtain multiple sampling results; the sampling results include n first matching point pairs, where n is a positive integer greater than or equal to 6; for each sampling result, using the least squares method, calculate the rotation and scaling parameters of the image rigid transformation; determine whether the rotation and scaling parameters meet preset conditions, and if so, use the sampling results and the binary quadratic transformation equation to calculate the first transformation matrix.

[0178] Optionally, the target point pair determination unit 400 is specifically used to: for each first transformation matrix, use the first transformation matrix to calculate the theoretical point corresponding to the key point in the first matching point pair; for each first matching point pair, if the distance between the key point in the first matching point pair and the corresponding theoretical point is within a preset range, then the first matching point pair is determined as the target point pair corresponding to the first transformation matrix.

[0179] Each of the above-mentioned units determines multiple first transformation matrices based on the first overlapping area between the first image and the second image, and selects the first transformation matrix corresponding to the largest total number of target point pairs from the multiple first transformation matrices as the transformation relationship between the first image and the second image. This can effectively solve the problem of deformation of wide-field fundus images during registration and avoid the missing of key point pairs, thereby effectively improving the accuracy of key point matching between the first image and the second image.

[0180] The present application also provides a computer-readable storage medium, which includes a stored program, wherein the program executes the key point matching method based on wide-field fundus images provided by the present application.

[0181] The present application also provides an electronic device, including: a processor, a memory, and a bus. The processor and the memory are connected via a bus, the memory is used to store programs, and the processor is used to run the programs, wherein the key point matching method based on wide-field fundus images provided by the present application is executed when the programs are run.

[0182] In addition, the functions described above in the embodiments of the present application may be at least partially performed by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0183] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0184] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A key point matching method based on wide field of view fundus image, characterized in that: include: Determining a first overlapping area between the first image and the second image based on the matching results of the first key point set and the second key point set; The first set of key points includes key points of the first image; The second key point set includes key points of the second image; the first image and the second image are both wide-field fundus images of the same eyeball; the matching result includes key point pairs having a matching relationship; Based on the first overlapping area, a plurality of first matching point pairs are determined; the first matching point pairs include a first key point and a second key point having a matching relationship; The first key point is a key point of a first image area, and the second key point is a key point of a second image area; The first image area is a partial image area in the first image corresponding to the first overlapping area, and the second image area is an image area in the second image corresponding to the first image area; Determine a plurality of first transformation matrices based on the plurality of first matching point pairs; For each of the first transformation matrices, determining a corresponding target point pair from a plurality of the first matching point pairs using the first transformation matrix; Determining a transformation relationship between the first image and the second image based on a first transformation matrix corresponding to a maximum total number of target point pairs; The transformation relationship represents the key point matching result between the two images.

2. The method according to claim 1, characterized in that Determining a transformation relationship between the first image and the second image based on a first transformation matrix corresponding to the largest total number of target point pairs includes: The first transformation matrix corresponding to the largest total number of target point pairs is determined as the target transformation matrix; transforming the second image using the target transformation matrix to obtain a third image; Determining a second overlapping area between the first image and the third image based on a matching result of the first key point set and a third key point set; the third key point set includes key points of the third image; Based on the second overlapping area, a plurality of second matching point pairs are determined; the second matching point pairs include a third key point and a fourth key point having a matching relationship; the third key point is a key point of a third image area, and the fourth key point is a key point of a fourth image area; the third image area is a partial image area in the first image corresponding to the second overlapping area, and the fourth image area is an image area in the third image corresponding to the third image area; Determine a plurality of second transformation matrices based on the plurality of second matching point pairs; For each of the second transformation matrices, determine a corresponding target point pair from a plurality of second matching point pairs using the second transformation matrix; Determining a transformation relationship between the first image and the third image based on a second transformation matrix corresponding to a maximum total number of target point pairs; According to the inverse transformation of the target transformation matrix, the transformation relationship between the first image and the third image is corrected to obtain the transformation relationship between the first image and the second image.

3. The method according to claim 1, characterized in that Determining a first overlapping area between the first image and the second image based on the matching results of the first key point set and the second key point set includes: Matching the first key point set and the second key point set to obtain a first key point pair; Eliminate erroneous key point pairs from the obtained plurality of first key point pairs to obtain second key point pairs; Using a random sampling consensus algorithm, selecting a third key point pair from the plurality of second key point pairs obtained; Based on the screened plurality of third key point pairs, a first overlapping area between the first image and the second image is determined.

4. The method according to claim 3, characterized in that Determining a first overlapping area between the first image and the second image based on the plurality of third key point pairs obtained by screening includes: By screening the plurality of third key point pairs obtained, calculating the affine transformation correspondence relationship between the first image and the second image; Using the affine transformation correspondence, an affine transformation is performed on the second image to obtain a first overlapping area between the first image and the second image.

5. The method according to claim 1, characterized in that Determining a plurality of first matching point pairs based on the first overlapping area includes: Determining a first image area and a second image area based on the first overlapping area; Performing bidirectional matching on key points of the first image region and the second image region to obtain a fourth key point pair; A random sampling consistency algorithm is used to screen out a plurality of first matching point pairs from the plurality of fourth key point pairs obtained.

6. The method according to claim 1, characterized in that Determining a plurality of first transformation matrices based on the plurality of first matching point pairs includes: Performing multiple random sampling on the plurality of first matching point pairs to obtain multiple sampling results; the sampling results include n first matching point pairs, where n is a positive integer greater than or equal to 6; For each sampling result, the least square method is used to calculate the rotation and scaling parameters of the image rigid transformation; It is determined whether the rotation scaling parameter meets the preset conditions. If so, a first transformation matrix is ​​calculated using the sampling result and a binary quadratic transformation equation.

7. The method according to claim 1, characterized in that For each of the first transformation matrices, determining a corresponding target point pair from a plurality of the first matching point pairs using the first transformation matrix includes: For each of the first transformation matrices, using the first transformation matrix, calculating a theoretical point corresponding to a key point in the first matching point pair; For each of the first matching point pairs, if the distance between the key point in the first matching point pair and the corresponding theoretical point is within a preset range, the first matching point pair is determined as the target point pair corresponding to the first transformation matrix.

8. A key point matching device based on wide-field fundus images, characterized in that: include: an overlapping region determining unit, configured to determine a first overlapping region between the first image and the second image based on a matching result of the first key point set and the second key point set; The first set of key points includes key points of the first image; The second key point set includes key points of the second image; the first image and the second image are both wide-field fundus images of the same eyeball; the matching result includes key point pairs having a matching relationship; A matching point pair determining unit, configured to determine a plurality of first matching point pairs based on the first overlapping area; the first matching point pairs include a first key point and a second key point having a matching relationship; The first key point is a key point of a first image area, and the second key point is a key point of a second image area; The first image area is a partial image area in the first image corresponding to the first overlapping area, and the second image area is an image area in the second image corresponding to the first image area; a transformation matrix determining unit, configured to determine a plurality of first transformation matrices based on a plurality of the first matching point pairs; a target point pair determining unit, configured to determine, for each of the first transformation matrices, a corresponding target point pair from a plurality of the first matching point pairs using the first transformation matrix; a transformation relationship determining unit, configured to determine a transformation relationship between the first image and the second image based on a first transformation matrix corresponding to a maximum total number of target point pairs; The transformation relationship represents the key point matching result between the two images.

9. A storage medium, characterized in that: The storage medium includes a stored program, wherein the program, when executed by a processor, executes the key point matching method based on a wide-field fundus image according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: processor, memory, and bus; The processor is connected to the memory via the bus; The memory is used to store programs, and the processor is used to run programs, wherein the program, when run by the processor, executes the key point matching method based on wide-field fundus images according to any one of claims 1 to 7.

Images