Training method of image matching model
By constructing a supervised training method based on reprojection error in ophthalmic microsurgery, the training and evaluation problems of existing models in the absence of depth information and geometric priors are solved, and high-precision ophthalmic surgical navigation is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JIAXING ZHITONG TECH CO LTD
- Filing Date
- 2025-12-18
- Publication Date
- 2026-04-10
AI Technical Summary
Existing deep learning matching models lack depth information and geometric priors, making them unsuitable for effective training and evaluation in ophthalmic microsurgery navigation. This results in insufficient image matching accuracy and robustness, failing to meet the high-precision requirements of ophthalmic microsurgery.
By acquiring multiple pairs of training images, calculating the true matching matrix, and constructing label sampling point pairs based on the reprojection error of the sampling points and backprojection sampling points, the initial image matching model is trained under supervision, avoiding reliance on depth information and geometric priors, thus improving the applicability of the training method.
It enables the construction of high-quality supervised labels in the absence of depth information and geometric priors, improving the training and evaluation accuracy of image matching models in ophthalmic microscopy scenarios and meeting the high-precision requirements of ophthalmic surgical navigation.
Smart Images
Figure CN121837818A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a training method for an image matching model. Background Technology
[0002] In recent years, with the development of deep learning technology, neural network-based image matching algorithms have achieved remarkable results in fields such as 3D reconstruction, visual navigation, autonomous driving, and augmented reality. These matching models predict pixel correspondences between images by extracting local or global features from two images, providing crucial support for tasks such as pose estimation, structure-from-motion (SfM), and stereo matching.
[0003] In ophthalmic microsurgery, the operating space is extremely limited, the tissue structures are delicate, and the requirements for precision are extremely high. Any slight deviation can affect the surgical outcome or even lead to serious complications. To improve the safety and accuracy of surgery, microsurgical navigation systems have gradually become a research hotspot in recent years. This system aims to achieve real-time positioning and dynamic tracking of the intraoperative microscopic field of view through image matching and registration technology, thereby assisting doctors in accurately identifying the surgical area and planning the operating path.
[0004] However, existing deep learning matching models are typically trained on natural scene datasets with multi-view images, depth images, and precise camera parameters (intrinsic and extrinsic parameters), such as MegaDepth and ScanNet. In such datasets, geometric prior information can be used to precisely map pixels from one image to another through projection transformation, thereby obtaining high-quality pixel-level matching labels to achieve supervised training.
[0005] Because ophthalmic microsurgery navigation systems demand extremely high accuracy and robustness in image registration, directly applying the aforementioned training methods based on natural scenes presents significant challenges. Microscopic imaging systems are characterized by a single viewpoint and complex illumination variations, making it impossible to obtain sufficient geometric constraints through multi-view imaging. Furthermore, existing natural scene datasets differ significantly from ophthalmic microsurgery scenes in terms of imaging mechanisms, illumination conditions, and texture distribution, resulting in poor model transfer performance. Simultaneously, existing methods lack high-quality supervisory labels that accurately reflect the true imaging characteristics of ophthalmic microsurgery, further limiting the accuracy and stability of existing image matching models in ophthalmic surgical navigation tasks.
[0006] Therefore, how to construct high-quality supervised labels in the absence of depth information and geometric priors, and how to effectively train and evaluate image matching models in ophthalmic microscopy scenes to achieve high-precision navigation in ophthalmic surgical scenes, has become a key problem that urgently needs to be solved in this field.
[0007] It should be noted that the above description of the background technology is only for the purpose of providing a clear and complete explanation of the technical solutions of the present invention and facilitating understanding by those skilled in the art. It should not be assumed that the above technical solutions are known to those skilled in the art simply because they have been described in the background section of the present invention. Summary of the Invention
[0008] The purpose of this invention is to solve the problem of how to construct high-quality supervised labels in the absence of depth information and geometric priors, so as to achieve effective training and evaluation of image matching models in ophthalmic microscopy scenes and realize high-precision navigation in ophthalmic surgical scenes.
[0009] To achieve the above objectives, the present invention provides a training method for an image matching model, comprising:
[0010] Multiple training image pairs and an initial image matching model are obtained. Each training image pair consists of a reference image and a training image, and the training image is determined based on the reference image. The true matching matrix corresponding to the training image pair is calculated based on the reference image and the training image.
[0011] Sampling points are planned in the reference image according to a preset sampling strategy. The sampling points are projected into the training image based on the true matching matrix to obtain projection points. Then, the projection points are projected into the reference image based on the inverse of the true matching matrix to obtain back-projection sampling points.
[0012] A consistency detection is performed on the sampling points and the back-projection sampling points, the reprojection error between the sampling points and the back-projection sampling points is calculated, and the sampling points and back-projection sampling points whose reprojection error satisfies the first preset error threshold are taken as label sampling point pairs.
[0013] The initial image matching model is trained under supervision based on the label sampling points to obtain the target image matching model.
[0014] As a further improvement of the present invention, after obtaining the target image matching model, the training method further includes:
[0015] The target image matching model is evaluated based on the true matching matrix to assess the accuracy and / or error metrics of the target matching model.
[0016] As a further improvement of the present invention, the evaluation of the target image matching model based on the true matching matrix includes:
[0017] The reference image and the training image are input into the target image matching model to obtain multiple first prediction points and multiple second prediction points that correspond to and match the reference image and the training image, respectively.
[0018] The first predicted point is subjected to an affine transformation using the true matching matrix to obtain multiple true points on the training image.
[0019] The pixel error is calculated based on the Euclidean distance between the second predicted point and the real point, and the second predicted point corresponding to the pixel error satisfying the second preset error threshold is taken as the ideal matching point.
[0020] The average matching rate is calculated based on the number of ideal matching points and the number of second predicted points, and the accuracy index of the target matching model is evaluated based on the average matching rate.
[0021] As a further improvement of the present invention, the evaluation of the target image matching model based on the true matching matrix includes:
[0022] The Euclidean norm between the matching matrix predicted by the target image model and the true matching matrix is calculated to evaluate the error index of the target matching model based on the Euclidean norm.
[0023] As a further improvement of the present invention
[0024] The acquisition of multiple pairs of training images includes:
[0025] A video sequence from a real scene is obtained, and the video sequence is sequentially cropped at a first fixed interval frame to obtain multiple reference images;
[0026] Determine the current frame of the current reference image, and based on the current frame, extract the video sequence at a second fixed interval to obtain the training image corresponding to the current reference image;
[0027] The reference image and the training image constitute a training image pair;
[0028] The step of calculating the true matching matrix corresponding to the training image pair based on the reference image and the training image includes:
[0029] The reference image and the training image are matched using a preset matching algorithm to obtain a preset number of initial matching point pairs;
[0030] Perform geometric consistency verification on the initial matching point pairs, remove abnormal matching point pairs from the initial matching point pairs, and obtain the target matching point pairs;
[0031] Based on the target matching point pair, calculate the affine transformation matrix between the reference image and the training image, and use the affine transformation matrix as the true matching matrix.
[0032] As a further improvement of the present invention
[0033] The acquisition of multiple pairs of training images includes:
[0034] A video sequence from a real scene is obtained, and the video sequence is then segmented sequentially at a third fixed interval frame to obtain multiple reference images;
[0035] An affine transformation is performed on the reference image to generate a training image corresponding to the reference image, and the reference image and the training image constitute a training image pair;
[0036] The step of calculating the true matching matrix corresponding to the training image pair based on the reference image and the training image includes:
[0037] The matrix used when performing an affine transformation on the reference image is used as the true matching matrix.
[0038] As a further improvement of the present invention, the step of planning sampling points in the reference image according to a preset sampling strategy includes:
[0039] A sampling grid is generated on the reference image according to a preset grid strategy, and uniform grid sampling is performed within the sampling grid according to a preset step size to obtain the sampling points.
[0040] As a further improvement of the present invention, the formula for calculating the reprojection error is as follows:
[0041]
[0042] Where x0 and y0 refer to the x-coordinate and y-coordinate of the sampling point, respectively. ' 0 and y ' 0 refers to the x-coordinate and y-coordinate of the back-projected sampling point, respectively.
[0043] As a further improvement of the present invention, the formula for calculating the pixel error is as follows:
[0044]
[0045] Among them, X pred and y pred These refer to the x and y coordinates of the second prediction point, respectively. gt and y gt These refer to the x-coordinate and y-coordinate of the actual point, respectively.
[0046] As a further improvement of the present invention, the formula for calculating the Euclidean norm is as follows:
[0047]
[0048] Among them, H gt [i,j] represents the element in the i-th row and j-th column of the true matching matrix, H pred [i,j] represents the element in the i-th row and j-th column of the matrix predicted by the target image matching model.
[0049] Compared with the prior art, the beneficial effects of the present invention are:
[0050] By projecting sampling points from a reference image onto a training image, and then backprojecting the projected points from the training image back onto the reference image, label sampling point pairs are obtained based on the reprojection error between the sampling points and the backprojected sampling points. In this way, the initial image training model is trained using the label sampling point pairs (i.e., the label sampling point pairs are used as matching supervision labels in the training process). This avoids relying on depth information (such as camera intrinsic and extrinsic parameters, camera pose, and depth maps) to construct matching supervision labels, thereby improving the applicability of the training method. This addresses the problem of how to construct high-quality supervision labels in the absence of depth information and geometric priors, enabling effective training and evaluation of image matching models in ophthalmic microscopy scenes, and achieving high-precision navigation in ophthalmic surgical scenes. Attached Figure Description
[0051] Figure 1 This is a schematic diagram illustrating the steps of a training method for an image matching model according to the present invention.
[0052] Figure 2 This is a schematic diagram illustrating the steps of obtaining multiple pairs of training image pairs in one implementation method;
[0053] Figure 3 This is a schematic diagram illustrating the steps for calculating the true matching matrix of training image pairs based on reference images and training images;
[0054] Figure 4 This is a schematic diagram illustrating the steps for obtaining multiple pairs of training image pairs in another implementation method;
[0055] Figure 5 This is a schematic diagram illustrating the steps of evaluating a target image matching model based on the true matching matrix.
[0056] Figure 6 This is a schematic diagram of the sampling points and back-projected sampling points on the reference image and the projection points on the training image. Detailed Implementation
[0057] The present invention will be described in detail below in conjunction with the embodiments shown in the accompanying drawings. It should be noted, however, that these embodiments are not intended to limit the present invention, and any equivalent transformation or substitution in terms of function, method, or structure made by those of ordinary skill in the art based on these embodiments shall fall within the protection scope of the present invention.
[0058] Refer Figures 1 to 6 As shown, the present invention shows a specific embodiment of a training method for an image matching model (hereinafter referred to as the "training method"). This training method is used to train the image matching model in a microsurgical navigation system. By projecting the sampling points in the reference image onto the training image, and then back-projecting the projected points in the training image onto the reference image, a label sampling point pair is obtained based on the reprojection error between the sampling points and the back-projected sampling points. Thus, the initial image training model is supervised and trained based on the label sampling point pair (that is, the label sampling point pair is used as the matching supervision label (ground truth) in the training process), thereby avoiding relying on depth information (such as camera internal and external parameters, camera pose, and depth map, etc.) to construct the matching supervision label, and thus improving the applicability of the training method to solve the problem of how to construct a high-quality supervision label under the condition of lacking depth information and geometric prior, realize the effective training and evaluation of the image matching model in the ophthalmic microsurgical scene, and achieve high-precision navigation in the ophthalmic surgical scene.
[0059] Refer Figure 1 As shown, the training method includes the following steps S1 to S4.
[0060] Step S1, obtain multiple pairs of training image pairs and an initial image matching model, and calculate the true matching matrix corresponding to the training image pair based on the reference image and the training image.
[0061] Specifically, each training image pair consists of a reference image and a training image. The reference image is determined based on the video sequence in the real scene, and the training image is determined based on the reference image. The aforementioned video sequence refers to a complete video sequence recorded in the microsurgical scene during the data acquisition stage, and this video sequence is used to record the dynamic changes in the microsurgical field during the operation. In the present invention, the training image can be obtained based on the video sequence in the real scene and the current frame number in the video sequence, or can be generated by applying a random affine transformation (such as rotation, translation, scaling, etc.) to the training image. This embodiment does not make a specific limitation on this.
[0062] It should be noted that this invention does not limit the specific method for selecting the initial image matching model. For example, LoFTR (a local feature matching method based on a Transformer-based featureless detector) can be used as the baseline matching model (i.e., the initial image matching model). During the selection phase of the initial image matching model, the effects of multiple matching models are compared using test cases, and a selection is made based on the comparison results. Preferably, LoFTR is used as the initial image matching model because it has a large number of matching points, stable results, and faster inference speed compared to other models, facilitating further deployment and acceleration.
[0063] In one implementation, both the reference image and the training image are obtained by cropping video sequences. Figure 2 As shown, obtaining multiple pairs of training images includes the following steps S11 to S13.
[0064] Step S11: Obtain a video sequence from the real scene, and extract multiple reference images by sequentially cutting the video sequence at the first fixed interval frame.
[0065] Step S12: Determine the current frame of the current reference image, and extract the video sequence based on the current frame according to the second fixed interval frame to obtain the training image corresponding to the current reference image.
[0066] Step S13: The reference image and the training image constitute a training image pair.
[0067] For example, the first fixed interval frame is 20 frames, and the second fixed interval frame is 10 frames. The video sequence is sequentially cropped according to the first fixed interval frame to obtain multiple reference images. That is, the video sequence is cropped sequentially according to 20 frames, and finally, the images of the 20th, 40th, 60th, etc., are cropped sequentially, thus obtaining reference images. The current frame of the current reference image is determined, and the video sequence is cropped according to the second fixed interval frame based on the current frame to obtain training images corresponding to the current reference image. That is, the video sequence is cropped according to 10 frames based on the current frame, and finally, the images of the 30th, 50th, 70th, etc., are cropped, thus obtaining training images. The images cropped from the 20th and 30th frames form a training image pair, the images cropped from the 40th and 50th frames form a training image pair, the images cropped from the 60th and 70th frames form a training image pair, and so on.
[0068] Furthermore, participants Figure 3 As shown, calculating the true matching matrix of training image pairs based on the reference image and the training image includes the following steps S14 to S16.
[0069] Step S14: Match the reference image and the training image using a preset matching algorithm to obtain a preset number of initial matching point pairs.
[0070] It should be noted that the preset matching algorithm is implemented based on a large AI model (especially a large visual model). The large AI model refers to a feature extraction and matching model based on deep learning, which can be any algorithm in existing technologies, such as ROMA (a robust dense matching algorithm), SuperPoint (a self-supervised keypoint detection and descriptor extraction network), LoFTR or MatchFormer (a "detector-free" matching model that directly establishes dense correspondences between pixels in two images and then selects the most reliable point pairs), DINOv2, SAM, etc. This embodiment does not specifically limit this. Preferably, ROMA is used as the preset matching algorithm, which is a dense matching algorithm obtained after fine-tuning and training based on the large model DINOv2. Furthermore, the aforementioned baseline model LoFTR is a semi-dense matching algorithm, thus it can be used as a method for creating a truth matrix. The preset matching algorithm finds the correspondence between the reference image and the training image; that is, it finds the pixels in the reference image and the training image that represent the same real-world physical point. Each such pair of pixels is called an initial matching point pair.
[0071] Step S15: Perform geometric consistency verification on the initial matching point pairs, remove abnormal matching point pairs from the initial matching point pairs, and obtain the target matching point pairs.
[0072] Specifically, due to factors such as lighting variations, repetitive textures, occlusion, and viewpoint differences in large AI models, a certain proportion of initial matching point pairs will contain incorrect matches (i.e., abnormal matching point pairs). Therefore, the Random Abstract Consensus Algorithm (RANSAC) can be used to perform geometric consistency verification on the initial matching point pairs to further remove abnormal matching point pairs.
[0073] It should be noted that the above example of the Random Abstract Consensus Algorithm RANSAC can be used as an illustration, but other algorithms, such as MLESAC (Maximum Likelihood Estimation Sampling Consensus), can also be used. This embodiment does not specifically limit the specific algorithms used.
[0074] Step S16: Calculate the affine transformation matrix between the reference image and the training image based on the target matching point pairs, so as to use the affine transformation matrix as the true matching matrix.
[0075] The calculation of the affine transformation matrix between the reference image and the training image based on the target matching point pairs can be done in any of the existing technologies, and this embodiment does not specifically limit it.
[0076] In another implementation, the reference image is obtained by cropping a video sequence, and the training images are generated based on the reference image. Figure 4 As shown, obtaining multiple pairs of training images includes the following steps S17 and S18.
[0077] Step S17: Obtain the video sequence of the real scene, and extract the video sequence in sequence according to the third fixed interval frame to obtain multiple reference images.
[0078] Step S18: Perform an affine transformation on the reference image to generate a training image corresponding to the reference image, and form a training image pair with the reference image and the training image.
[0079] To further improve data diversity and model generalization ability, this implementation constructs matching samples based on single-frame images. In short, the third fixed-interval frame is one frame. Each frame of the video sequence is extracted to obtain multiple reference images. Random reflection transformations (such as rotation, translation, scaling, etc.) are applied to the reference images to generate matching images corresponding to the reference images, thereby constructing multiple pairs of matching image samples with clear geometric mapping relationships (i.e., training image pairs).
[0080] Furthermore, participants Figure 4 As shown, calculating the true matching matrix corresponding to the training image pair based on the reference image and the training image includes the following steps S19.
[0081] Step S19: Use the matrix used when performing affine transformation on the reference image as the true matching matrix.
[0082] Regardless of the method used to obtain the reference image and the corresponding training image to form the training image pair, and to calculate the real matching matrix corresponding to the training image pair, the training method disclosed in this invention is based on video sequences in real scenes. This effectively expands the number of matching samples (i.e., training image pairs) without the need for depth cameras and calibration, while ensuring that the spatial mapping relationship between each training image pair is controllable.
[0083] Furthermore, the present invention does not specifically limit the number of frames for the first fixed interval frame, the second fixed interval frame, and the third fixed interval frame. The foregoing example of 30 frames for the first fixed interval frame, 20 frames for the second fixed interval frame, and 1 frame for the third fixed interval frame is used for illustrative purposes, but this should not limit the scope of protection of the present invention.
[0084] Step S2: Plan sampling points in the reference image according to the preset sampling strategy, project the sampling points into the training image based on the true matching matrix to obtain the projection points, and then project the projection points into the reference image based on the inverse of the true matching matrix to obtain the back-projection sampling points.
[0085] Uniform network sampling is performed on the reference image at a fixed step size, generating a series of regularly distributed grid points throughout the entire reference image; these grid points are the sampling points. Figure 6 As shown, for example, with a fixed step size of 1, the generated grid points are 80*80, equivalent to one-eighth of the reference image, taking sampling point A as an example. Using the true matching matrix H... gt Projecting sampling point A(x0, y0) from the reference image onto the training image yields the corresponding projection point B(x1, y1). The projection point can be understood as the pixel in the training image that theoretically represents the same real-world physical point as the sampling point in the reference image. To verify the consistency of the projection, the inverse matrix H of the true matching matrix is used. gt -1 Projecting the projection point B(x1, y1) in the training image onto the reference image yields the back-projection sampling point A'(x1, y1) corresponding to the projection point B. ' 0, y ' 0), the back-projection sampling point can be understood as the pixel point in the reference image that theoretically represents the same real-world physical point as the projection point in the training image.
[0086] It should be noted that the aforementioned single sampling point, single projection point, and single back projection sampling point correspond one-to-one.
[0087] Step S3: Perform consistency detection on the sampling points and back-projection sampling points, calculate the reprojection error between the sampling points and back-projection sampling points, and take the sampling points and back-projection sampling points whose reprojection error meets the first preset error threshold as label sampling point pairs.
[0088] Calculate each pair of sampling points A(x0, y0) and back-projected sampling points A'(x0, y0). ' 0, y ' The reprojection error between the two points in (0) is used to perform consistency checks on each pair of sampling points and backprojection sampling points. When the reprojection error is within a first preset error threshold, the point pair (i.e., a pair of sampling points and backprojection sampling points) is considered to satisfy the bidirectional consistency condition, that is, the transformation is stable and reliable, and it is determined as a label sampling point pair. All point pairs that pass the consistency check (i.e., all point pairs whose reprojection errors are within the first preset error threshold) are retained to form the final set of label sampling point pairs.
[0089] In one implementation, the formula for calculating the reprojection error is:
[0090]
[0091] Where x0 and y0 refer to the x-coordinate and y-coordinate of the sampling point, respectively. ' 0 and y ' 0 refers to the x-coordinate and y-coordinate of the back-projected sampling point, respectively.
[0092] Step S4: Supervised training of the initial image matching model based on the label sampling points to obtain the target image matching model.
[0093] For example, during training, LoFTR (a Transformer-based featureless detector local feature matching method) is used as the baseline matching model, i.e., the initial image matching model. Each training sample consists of a training image pair and the corresponding ground truth matching matrix. During the training phase, the training image pair is input into the initial image matching model. First, a convolutional neural network is used to extract features from the reference image and the training image to obtain multi-scale feature representations. Then, a Transformer-based feature matching module is used to model global context information in the feature space of the two images, establishing cross-image feature correspondences and forming predicted matching point pairs. Under the supervision of the ground truth matching matrix (i.e., based on label sampling point pairs), the predicted matching results are constrained. Specifically, positive and negative samples are sampled in the matching space, a loss function is calculated, and this loss function is used as a supervision signal to backpropagate and optimize the model parameters. Through the above supervised training process, the model's ability to model cross-image feature correspondences is gradually improved, ultimately resulting in a target image matching model for the target scene.
[0094] In one implementation, the reference Figure 1 As shown, after obtaining the target image matching model, the training method also includes the following step S5.
[0095] Step S5: Evaluate the target image matching model based on the real matching matrix to assess the accuracy and / or error metrics of the target matching model.
[0096] The accuracy and / or error metrics of the target image matching model are evaluated, and the parameters of the target image matching model can be adjusted based on the accuracy and / or error metrics to further ensure the accuracy of the target matching model.
[0097] It should be noted that in this invention, after obtaining multiple training image pairs, the training image pairs are divided into a training set and a validation set according to a specific ratio (such as 9:1 or 4:1). The training set contains most of the microscopic video sequences and the training image pairs generated from them, and each training image pair has a corresponding true matching matrix. The validation set is used to evaluate the performance of the target image learning model and adjust hyperparameters during training, and does not participate in gradient updates. The aforementioned steps S2 to S4 are calculated using the training image pairs in the training set, and step S5 is calculated using the training image pairs in the validation set.
[0098] In one implementation, the accuracy metric of the target matching model is evaluated. (See reference...) Figure 5As shown, the evaluation of the target image matching model based on the real matching matrix includes the following steps S51 to S54.
[0099] Step S51: Input the reference image and training image into the target image matching model to obtain multiple first prediction points and multiple second prediction points that correspond to and match the reference image and training image, respectively.
[0100] Step S52: Perform an affine transformation on the first predicted point using the true matching matrix to obtain multiple true points on the training image.
[0101] Step S53: Calculate the pixel error based on the Euclidean distance between the second predicted point and the real point, and take the second predicted point corresponding to the pixel error satisfying the second preset error threshold as the ideal matching point.
[0102] Step S54: Calculate the average matching rate based on the number of ideal matching points and the number of second predicted points, so as to evaluate the accuracy index of the target matching model based on the average matching rate.
[0103] Specifically, the matching accuracy is evaluated by solving the true matching matrix for each training image pair. The reference image and training image are input into the target image matching model to obtain multiple first predicted points in the reference image and multiple second predicted points corresponding to the first predicted points in the training image. The second predicted points are the first predicted points predicted by the target image matching model. An affine transformation is performed on the first predicted points based on the true matching matrix to obtain multiple true points in the training image. The true points are the true matching matrices of the first predicted points. The Euclidean distance between the second predicted points and the true points is calculated, and the pixel error is calculated based on this. The second predicted points whose pixel errors satisfy a second preset error threshold are considered ideal matching points (i.e., points considered successfully matched). The ratio between the number of ideal matching points and the number of second predicted points, i.e., the ratio of the number of successfully matched points to the total number of transformed points, is calculated to obtain the average matching rate, thereby evaluating the accuracy index of the target matching model to reflect the accuracy of local pixel-level matching.
[0104] Furthermore, the formula for calculating pixel error is:
[0105]
[0106] Among them, X pred and y pred These refer to the x and y coordinates of the second prediction point, respectively. gt and y gt These refer to the x-coordinate and y-coordinate of the actual point, respectively.
[0107] In another implementation, the error metric of the target matching model is evaluated. Evaluating the target image matching model based on the true matching matrix includes the following step S55.
[0108] Step S55: Calculate the Euclidean norm between the matching matrix predicted by the target image model and the true matching matrix, so as to evaluate the error index of the target matching model based on the Euclidean norm.
[0109] It should be noted that, in order to measure the accuracy of the target image model in global geometric transformation, an error index is introduced. This error index is used to evaluate the accuracy of the model in global geometric transformation estimation.
[0110] Furthermore, the formula for calculating the Euclidean norm is:
[0111]
[0112] Among them, H gt [i,j] represents the element in the i-th row and j-th column of the true matching matrix, H pred [i,j] represents the element in the i-th row and j-th column of the matrix predicted by the target image matching model.
[0113] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent embodiments or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.
[0114] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
[0115] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.
Claims
1. A training method for an image matching model, characterized in that, include: Multiple training image pairs and an initial image matching model are obtained. Each training image pair consists of a reference image and a training image, and the training image is determined based on the reference image. The true matching matrix corresponding to the training image pair is calculated based on the reference image and the training image. Sampling points are planned in the reference image according to a preset sampling strategy. The sampling points are projected into the training image based on the true matching matrix to obtain projection points. Then, the projection points are projected into the reference image based on the inverse of the true matching matrix to obtain back-projection sampling points. A consistency detection is performed on the sampling points and the back-projection sampling points, the reprojection error between the sampling points and the back-projection sampling points is calculated, and the sampling points and back-projection sampling points whose reprojection error satisfies the first preset error threshold are taken as label sampling point pairs. The initial image matching model is trained under supervision based on the label sampling points to obtain the target image matching model.
2. The training method according to claim 1, characterized in that, After obtaining the target image matching model, the training method further includes: The target image matching model is evaluated based on the true matching matrix to assess the accuracy and / or error metrics of the target matching model.
3. The training method according to claim 2, characterized in that, The evaluation of the target image matching model based on the true matching matrix includes: The reference image and the training image are input into the target image matching model to obtain multiple first prediction points and multiple second prediction points that correspond to and match the reference image and the training image, respectively. The first predicted point is subjected to an affine transformation using the true matching matrix to obtain multiple true points on the training image. The pixel error is calculated based on the Euclidean distance between the second predicted point and the real point, and the second predicted point corresponding to the pixel error satisfying the second preset error threshold is taken as the ideal matching point. The average matching rate is calculated based on the number of ideal matching points and the number of second predicted points, and the accuracy index of the target matching model is evaluated based on the average matching rate.
4. The training method according to claim 2, characterized in that, The evaluation of the target image matching model based on the true matching matrix includes: The Euclidean norm between the matching matrix predicted by the target image model and the true matching matrix is calculated to evaluate the error index of the target matching model based on the Euclidean norm.
5. The training method according to claim 1, characterized in that, The acquisition of multiple pairs of training images includes: A video sequence from a real scene is obtained, and the video sequence is sequentially cropped at a first fixed interval frame to obtain multiple reference images; Determine the current frame of the current reference image, and based on the current frame, extract the video sequence at a second fixed interval to obtain the training image corresponding to the current reference image; The reference image and the training image constitute a training image pair; The step of calculating the true matching matrix corresponding to the training image pair based on the reference image and the training image includes: The reference image and the training image are matched using a preset matching algorithm to obtain a preset number of initial matching point pairs; Perform geometric consistency verification on the initial matching point pairs, remove abnormal matching point pairs from the initial matching point pairs, and obtain the target matching point pairs; Based on the target matching point pair, calculate the affine transformation matrix between the reference image and the training image, and use the affine transformation matrix as the true matching matrix.
6. The training method according to claim 1, characterized in that, The acquisition of multiple pairs of training images includes: A video sequence from a real scene is obtained, and the video sequence is then segmented sequentially at a third fixed interval frame to obtain multiple reference images; An affine transformation is performed on the reference image to generate a training image corresponding to the reference image, and the reference image and the training image constitute a training image pair; The step of calculating the true matching matrix corresponding to the training image pair based on the reference image and the training image includes: The matrix used when performing an affine transformation on the reference image is used as the true matching matrix.
7. The training method according to claim 1, characterized in that, The step of planning sampling points in the reference image according to a preset sampling strategy includes: A sampling grid is generated on the reference image according to a preset grid strategy, and uniform grid sampling is performed within the sampling grid according to a preset step size to obtain the sampling points.
8. The training method according to claim 1, characterized in that, The formula for calculating the reprojection error is: Where x0 and y0 refer to the abscissa and ordinate of the sampling point, respectively, and x'0 and y'0 refer to the abscissa and ordinate of the back-projected sampling point, respectively.
9. The training method according to claim 3, characterized in that, The formula for calculating the pixel error is: Among them, X pred and y pred These refer to the x and y coordinates of the second prediction point, respectively. gt and y gt These refer to the x-coordinate and y-coordinate of the actual point, respectively.
10. The training method according to claim 4, characterized in that, The formula for calculating the Euclidean norm is as follows: Among them, H gt [i,j] represents the element in the i-th row and j-th column of the true matching matrix, H pred [i,j] represents the element in the i-th row and j-th column of the matrix predicted by the target image matching model.