Leverage image matching and positioning method based on perspective transformation

Abstract

The application discloses a kind of lever image matching and positioning method based on perspective transformation, it is related to computer vision and image recognition technical field.It is specific to: by selecting reference picture from frame image and calculating perspective transformation matrix, combined with feature matching and instance segmentation screening, randomly sample calculation and optimize perspective transformation matrix model parameter, estimate and evaluate the effect of matching and point mapping.The application utilizes the perspective transformation matrix between the two-dimensional images of the same object multi-modal representation, based on the feature extraction and detection method of deep neural network, obtains the feature point matching of target lever image and lever scale surface area, combined with the property of vector screening matching point pair and determining perspective transformation matrix, evaluates the effect of random point set on known plane to target image.The application does not depend on single standard multi-modal representation lever, generates perspective transformation matrix dictionary, reduces the large error possibly generated by single standard operation.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image recognition technology, specifically a lever image matching and localization method based on perspective transformation, which can be applied to image matching, computer vision and other fields. Background Technology

[0002] "Exploring the Conditions for Lever Equilibrium" is an important component of high school physics experiments. Through these experiments, students can understand torque and the lever principle, and learn how to use equilibrium conditions to calculate forces and lever arms at different locations. This experiment not only enhances students' understanding of basic physics concepts but also cultivates their experimental skills and scientific inquiry spirit. Based on the exploration of lever equilibrium conditions, this experiment utilizes computer vision technology and image processing. Through image matching and perspective transformation algorithms, the lever system can be automatically identified and located, improving the efficiency and accuracy of the experiment, reducing errors caused by matching single standard image features, and providing quasi-data support for further research.

[0003] Image feature matching is a commonly used image registration method. It identifies key feature points in an image, extracts their descriptors or topological relationships, finds the correspondences between feature points, and then performs matching analysis to determine spatial transformation relationships, thereby achieving image alignment. Perspective transformation is a commonly used image geometric transformation method that projects an image onto another viewpoint while preserving the geometric structure and relative positional relationships within the image. In traditional image processing, perspective transformation matrices can convert two-dimensional images into perspective projections in three-dimensional space, enabling operations such as affine transformations and scale transformations. Perspective transformation matrices and feature matching are key technologies in computer vision, with wide applications in image registration, 3D reconstruction, and other fields. With the development of deep learning and neural networks, feature points in images can be extracted more accurately, and correspondences between images can be learned, thus improving the accuracy and robustness of feature matching.

[0004] However, several challenges and problems remain in the field of perspective transformation matrices and feature matching. For example, feature points themselves may not have sufficient discriminative power, especially in images with complex backgrounds or low contrast, potentially leading to inaccurate matching results and the inability to achieve precise matching of any point in the image. This may limit its applicability in certain application scenarios. Furthermore, since perspective transformation is a non-linear transformation, it fails when dealing with non-planar scenes in images. When the camera position changes, if the scene cannot be constrained to a plane, the perspective transformation cannot correctly map points in the image.

[0005] In summary, obtaining perspective transformation matrices and feature matching are key technologies in the field of computer vision, with significant application value in image processing, 3D reconstruction, and other areas. In particular, applications such as lever image matching and point mapping still require further research and improvement to meet the needs of different application scenarios. Summary of the Invention

[0006] To address the problem that feature matching on lever images relies on a single standard for multimodal lever representation, this invention provides a lever image matching and localization method based on perspective transformation.

[0007] This invention is achieved through the following technical solution: a lever image matching and localization method based on perspective transformation, comprising the following steps:

[0008] S1: Generation of the perspective transformation matrix dictionary for multimodal lever representation: Select multiple images of levers at different positions, angles, distances, noise levels and states from the extracted frame images as a set of reference images, calculate the perspective transformation matrix for every two images, and obtain an image transformation matrix dictionary.

[0009] S2: Filtering of matching point pairs within the lever scale surface region based on feature matching and instance segmentation: For any target lever image, feature points are matched with the reference image. The lever scale surface region with shape features is obtained through instance segmentation. Matching point pairs located within the instance segmentation polygon region are filtered based on vector calculation.

[0010] S3: Iterative optimization of perspective transformation matrix determination and point mapping: Randomly sample and match data within the set to estimate the parameters of the perspective transformation matrix model, identify and eliminate outliers far from the model, iteratively optimize to obtain a more accurate perspective transformation matrix, use homogeneous coordinates to represent a point in the lever scale plane of the reference image, and use the perspective transformation matrix to obtain the specific position of the point mapped to the target image.

[0011] S4: Evaluation of the effect of locating points on a known plane onto the target image: Using the distance and interval of the known random point set on the reference image mapped to the position on the target image according to different calculation paths, the effect of feature matching and perspective transformation between the target image and the reference image is judged to ensure that there is no excessive error affecting the subsequent further analysis of the target image.

[0012] The present invention presents a lever image matching and localization method based on perspective transformation. This method is based on perspective transformation and instance segmentation to verify the effect of feature matching and point mapping. It utilizes the perspective transformation matrix between two-dimensional images representing the multimodal representation of the same object, and uses a feature extraction and detection method based on deep neural networks to obtain feature point matching and lever scale area of ​​the target lever image. It combines the properties of vectors to filter matching point pairs and determine the perspective transformation matrix, and evaluates the effect of locating a random set of points on a known plane onto the target image.

[0013] Compared with existing technologies, the present invention has the following advantages: The lever image matching and localization method based on perspective transformation provided by the present invention does not rely on a single standard for multimodal representation of the lever, generates a perspective transformation matrix dictionary, and reduces the large errors that may be generated by a single standard operation; it uses the properties of vectors to filter matching point pairs located in the lever scale area, and excludes some outliers far from the model for random sampling data, which is conducive to obtaining more accurate model parameters; it uses the distance interval of the known random point set on the plane mapped to the position on the target image according to different calculation paths to judge the effect of feature matching and perspective transformation between the target image and the reference image, ensuring that better model parameters can be used for further analysis of the target image. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of selecting the vertex of the lever scale surface in a sub-image of a specific embodiment of the present invention.

[0015] Figure 2 This is a schematic diagram of selecting the vertex of the lever scale surface in another sub-image of the reference image for a specific embodiment of the present invention.

[0016] Figure 3 This is a schematic diagram illustrating the calculation of the perspective transformation matrix of two sub-images in a reference image according to a specific embodiment of the present invention.

[0017] Figure 4 This is a schematic diagram illustrating the calculation of matching point pairs between the target image and sub-images in a specific embodiment of the present invention.

[0018] Figure 5 This is a schematic diagram illustrating the selection of matching point pairs between the target image and the sub-image within the lever scale area, as per a specific embodiment of the present invention.

[0019] Figure 6 This is a schematic diagram showing the selection of known points on the lever scale surface in a specific embodiment of the present invention.

[0020] Figure 7 This is a schematic diagram of the point mapping of a known point through a perspective transformation matrix in a specific embodiment of the present invention. Detailed Implementation

[0021] The present invention will be further described below with reference to specific embodiments.

[0022] S1: Generation of the perspective transformation matrix dictionary for multimodal lever representation: From the extracted frame images, multiple images of the lever at different positions, angles, distances, noise levels, and states are selected as a set of reference images. A perspective transformation matrix is ​​calculated for every two images to obtain an image transformation matrix dictionary; specifically as follows:

[0023] s1-1: Use a device with video recording capabilities to record the entire experiment of "investigating the conditions for lever balance", extract static frame images from the recorded video files, and obtain multi-frame image files;

[0024] s1-2: Select N images from the extracted frame images showing the lever placed on the iron stand fulcrum in different modalities as a set of reference images. The set of reference images is denoted as K. The lever multimodalities include, but are not limited to, different distances between the lever and the camera, left or right tilt when the lever is unbalanced, degree of tilting and rotation, and different experimenters. The criteria for selecting images ensure that they can represent the multiple modalities of the lever in the experiment.

[0025] s1-3: For two images of the same lever at different positions, use cv2.getPerspectiveTransform to obtain a 3×3 perspective transformation matrix. Select four vertices of the source and target images located on the lever's scale surface, with each source point and target point corresponding one-to-one, i.e., src_points[i] corresponds to dst_points[i]. Ensure that the selected points are in the same order and can form a quadrilateral; for example... Figure 1 , Figure 2 and Figure 3 As shown, the three images are the source image and the target image, where the four vertices of the lever scale surface are selected as four corresponding point pairs, and the results are calculated and transformed. Figure 1 Medium leverage in corresponding Figure 2 The position above.

[0026] s1-4: For any two images in the reference image set K, calculate the perspective transformation matrix and store all the matrices in the dictionary Dict. The key of the dictionary is the identifier of the image pair, and the value is the corresponding perspective transformation matrix.

[0027] s1-5: For any three distinct bases in the reference image set K, if I i To I j The perspective transformation matrix is ​​matrixl i_j I j To I k The perspective transformation matrix is ​​matrixl j_k , then I i To I kperspective transformation matrix i_k The following relationships exist:

[0028] matrixl i_k =matrixl i_j ×matrixl j_k

[0029] S2: Filtering of matching point pairs within the lever scale surface region based on feature matching and instance segmentation: For any target lever image, feature points are matched with the reference image. The lever scale surface region with shape features is obtained through instance segmentation. Matching point pairs located within the instance-segmented polygonal region are then filtered based on vector calculations; specifically as follows:

[0030] s2-1: For any target image I where the lever is placed on the iron frame. lever Randomly select a sub-image I from the baseline image set K. a SuperPoint, based on a convolutional neural network (CNN), was used to obtain the target image I. lever and sub-image I a The system consists of a feature extraction network and a descriptor generation network. The feature extraction network extracts the locations of feature points from the input image using a CNN. The descriptor generation network generates a descriptor for each feature point, which is used to distinguish and match different feature points.

[0031] s2-2: LightGlue receives the target image I lever and sub-image I a The feature point coordinates and descriptor subsets are used as input. Self-attention and cross-attention units are used to adjust the matching process based on the similarity of image pairs, updating the representation of each feature point to obtain the target image I. lever and sub-image I a The set of matching point pairs M = {(p1,p1′),(p2,p2′),...,(p n ,p n ′)}, where p n and p n ′ represent the matched feature point pairs; for example Figure 4 As shown, for the target image and sub-image, SuperPoint generates feature points, LightGlue calculates the assignment matrix, discards unmatchable points and predicts matching point pairs to obtain matching point pairs for the target image and sub-image.

[0032] s2-3: Divide the extracted static frame images into training, validation, and test sets for training the instance segmentation neural network model. Manually label the lever scale surface region with polygons using labelme or cvat. Select a specific network model structure, i.e., VGG or ResNet. Use the YOLOv8 instance segmentation framework combined with the dataset to train the network weights to obtain a certain degree of intersection-over-union ratio (IoU). The IoU value is between 0.7 and 1. In this embodiment, the IoU is within the range of 0.7 to 0.85. This is used to segment and extract the lever scale surface region in the image.

[0033] s2-4: For target image I lever The set of points S = {(x1,y1),(x2,y2),...,(x1,y1),(x2,y2),...,(x2,y2),...,(x2,y1 ... n ,y n )}, target image I lever For any matching point P, the position information is (x p ,y p If (x1, y1) is point A and (x2, y2) is point B, then there is a vector... Calculate the cross product of vectors AB and AP:

[0034]

[0035] s2-5: For adjacent points in the polygon point set S, calculate the cross product sequentially using the method in step s2-4. If all cross product results have the same sign, either all positive or all negative, then point P is within the polygon segmented by the instance; otherwise, point P is not within the polygon segmented by the instance, and the target image I is deleted. lever The target image I is obtained by matching points and corresponding matching pairs that are not within the polygonal region of the lever scale plane in the instance segmentation. lever and sub-image I a New set of matching point pairs M sift ={(p1,p1′),(p2,p2′),...,(p n ,p n ′)}, Figure 5 The effect of filtering matching points within the polygonal region of the lever scale surface in the target image and the matching points corresponding to the filtered sub-images are shown. The filtered matching points are located in a plane in space, which will make the transformation matrix calculated later more accurate.

[0036] S3: Iterative optimization of perspective transformation matrix determination and point mapping: Data within a set of randomly sampled point pairs is used to estimate the parameters of the perspective transformation matrix model. Outliers far from the model are identified and eliminated. Iterative optimization yields a more accurate perspective transformation matrix. Homogeneous coordinates are used to represent a point within the lever scale plane of the reference image. The perspective transformation matrix is ​​then used to obtain the specific location of this point mapped onto the target image. Details are as follows:

[0037] s3-1: Using RANSAC, in the set of matching point pairs M sift Four points are randomly selected from the data, and the perspective transformation matrix H is estimated, where the model parameters are represented by h. ij This indicates that the target image I lever All source points (x) i ,y i Transformed into the target plane image I through matrix H a Above, calculate the transformed point (x) for each point. i ",y i ") and the actual target point (x i ′,y i The distance d of ′) i ;

[0038]

[0039] s3-2: Determine interior points based on a preset threshold ∈. If the distance is less than the threshold ∈, then it is considered... If a sufficient number of points are identified as interior points, the model and its parameters are updated. This process is repeated until the specified number of iterations is reached or a model H with the largest set of interior points is found. a Matrix H a Describes target image I lever and sub-image I a Geometric transformation relationships between them;

[0040] s3-3: Sub-image I a Any point A on the lever scale is represented by homogeneous coordinates (x, y, 1), using the target image I. lever With sub-image I a perspective transformation matrix H a Multiplying the coordinates of point A by their homogeneous coordinates gives the coordinates of point A in the target image I. lever The homogeneous coordinates (x′, y′, w) of the corresponding point B:

[0041]

[0042] s3-4: Normalize the aligned sub-coordinates (x′, y′, w) by dividing x′ and y′ by w respectively.

[0043]

[0044] Therefore, subimage I a Point A on the lever scale in image I lever The position of point B is (x′ / w, y′ / w), as shown above. Figure 6 and Figure 7 As shown, through Figure 6 and Figure 7 The perspective transformation matrix between them, choose Figure 6 Given a known set of points, obtain the mapping of that set to... Figure 7 The specific location on it.

[0045] S4: Evaluation of the effect of locating points on a known plane onto the target image: By mapping the known random point set on the reference image to the positions on the target image using different calculation paths, the effect of feature matching and perspective transformation between the target image and the reference image is judged to ensure that there are no excessive errors affecting the subsequent analysis of the target image, as detailed below:

[0046] s4-1: In sub-image I a A subset of random points is generated within the lever scale area, denoted as point set P = {(x1,y1),(x2,y2),...,(x...}. n ,y n )}, calculate sub-image I a Mapping the upper point set P to the target image I lever The position on the point set is denoted as point set P. a ={(x a1 ,y a1 ),(x a2 ,y a2 ),...,(x an ,y an )};

[0047] s4-2: Randomly select three sub-images I from the baseline image set. b1 I b2 I b3 None of the three sub-images are image I. A sub-image I b1 For example, calculate the target image I lever With sub-image I b1 The transformation matrix H b1 From steps s1-5, we know that matrixl a_b1 ×H b1 The calculated matrix H a_b1 It can also represent the target image I lever With sub-image I a The perspective transformation matrix, where matrix H is used. a_b1 Recalculate sub-image Ia Mapping the upper point set P to the target image I lever The position on the point set is denoted as point set P. b1 ={(x b1 ,y b1 ),(x b2 ,y b2 ),...,(x bn ,y bn )};

[0048] s4-3: Calculate the point set P a With point set P b1 Eulerian distance d of corresponding elements i And calculate the mean.

[0049]

[0050] s4-4: Calculate the sub-image I using steps s4-2 and s4-3 respectively. b2 and I b3 Sub-image I after action a Mapping a set of known points on the target image I lever Position on and point set P a The mean distance to corresponding elements, if the mean of three distances If all pixels are less than the preset pixel threshold ρ, then the target image I... lever With sub-image I a The feature matching and transformation matrix between them is accurate and can be used to map the obtained lever arm position onto the reference image to obtain the lever arm reading, which is used to verify the lever balance condition.

[0051] The scope of protection claimed by this invention is not limited to the specific embodiments described above. Moreover, for those skilled in the art, this invention can have various modifications and alterations. Any modifications, improvements, and equivalent substitutions made within the concept and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A lever image matching and localization method based on perspective transformation, characterized in that: Includes the following steps: S1: Generation of the perspective transformation matrix dictionary for multimodal lever representation: Select multiple images of levers at different positions, angles, distances, noise levels and states from the extracted frame images as a set of reference images, calculate the perspective transformation matrix for every two images, and obtain an image transformation matrix dictionary. S2: Filtering of matching point pairs within the lever scale surface region based on feature matching and instance segmentation: For any target lever image, feature points are matched with the reference image. The lever scale surface region with shape features is obtained through instance segmentation. Matching point pairs located within the instance segmentation polygon region are filtered based on vector calculation. S3: Iterative optimization of perspective transformation matrix determination and point mapping: Randomly sample and match data within the set to estimate the parameters of the perspective transformation matrix model, identify and eliminate outliers far from the model, iteratively optimize to obtain an accurate perspective transformation matrix, use homogeneous coordinates to represent a point in the lever scale plane of the reference image, and use the perspective transformation matrix to obtain the specific position of the point mapped to the target image. S4: Evaluation of the effect of locating points on a known plane onto the target image: Using the distance and spacing of the known random point set on the reference image mapped to the position on the target image according to different calculation paths, the effect of feature matching and perspective transformation between the target image and the reference image is judged to ensure that it will not affect the subsequent further analysis of the target image.

2. The lever image matching and localization method based on perspective transformation according to claim 1, characterized in that: Step S1 is as follows: s1-1: Use a device with video recording capabilities to record the entire experiment of "investigating the conditions for lever balance", extract static frame images from the recorded video files, and obtain multi-frame image files; s1-2: Select N images from the extracted frame images showing the lever placed on the iron stand fulcrum in different modalities as a set of reference images. The set of reference images is denoted as K. The lever multimodalities include different distances between the lever and the camera, left or right tilt when the lever is unbalanced, degree of tilting and rotation, and different experimenters. The criteria for selecting images ensure that they can represent the multiple modalities of the lever in the experiment. s1-3: For two images of the same lever at different positions, use cv2.getPerspectiveTransform to obtain a 3×3 perspective transformation matrix. Select the four vertices of the source and target images located on the lever's scale surface, with each source point and target point corresponding one-to-one. Corresponding to Ensure that the selected points are in the same order and can form a quadrilateral; s1-4: For any two images in the reference image set K, calculate the perspective transformation matrix and store all the matrices in a dictionary. In the dictionary, the keys are identifiers for image pairs, and the values ​​are the corresponding perspective transformation matrices; s1-5: For the baseline image set If any three distinct bases in the matrix, arrive The perspective transformation matrix is , arrive The perspective transformation matrix is ,So arrive perspective transformation matrix The following relationships exist: 。 3. The lever image matching and localization method based on perspective transformation according to claim 2, characterized in that: Step S2 is as follows: s2-1: For any target image where the lever is placed on the iron stand. A sub-image is randomly selected from the baseline image set K. SuperPoint was used to obtain the target images based on a convolutional neural network (CNN). and sub-image The system consists of a feature extraction network and a descriptor generation network. The feature extraction network extracts the locations of feature points from the input image using a CNN. The descriptor generation network generates a descriptor for each feature point, which is used to distinguish and match different feature points. s2-2: LightGlue receives the target image. and sub-image The feature point coordinates and descriptor set are used as input. Self-attention and cross-attention units are used to adjust the matching process based on the similarity of image pairs, updating the representation of each feature point to obtain the target image. and sub-image Matching point pair set ,in and These are the matched feature point pairs; s2-3: Divide the extracted static frame images into training, validation, and test sets for training the instance segmentation neural network model. Manually label the lever scale surface region with polygons using labelme or cvat. Select a specific network model structure, i.e., VGG or ResNet. Use the YOLOv8 instance segmentation framework combined with the dataset to train network weights with a certain degree of intersection-over-union ratio (IoU), with an IoU value of 0.7~1. This is used to segment and extract the lever scale surface region in the image. s2-4: For the target image The polygon point set of the lever scale surface obtained by instance segmentation Target image any matching point on Location information is ,like Let A be the point. Let B be a point, then we have a vector. , Calculate the cross product of vectors AB and AP: ； s2-5: For a polygon point set For adjacent points in the array, calculate the cross product sequentially using steps s2-4. If all cross product results have the same sign (either all positive or all negative), then the point... Within the polygon segmented by the instance; otherwise, the point Delete the target image if it is not within the polygon of instance segmentation. The target image is obtained by matching points and corresponding matching pairs that are not within the polygonal region of the lever scale plane in instance segmentation. and sub-image New set of matching point pairs .

4. The lever image matching and localization method based on perspective transformation according to claim 3, characterized in that: Step S3 is as follows: s3-1: Using RANSAC, in the set of matching point pairs M sift Randomly select four pairs of points and estimate the perspective transformation matrix. The model parameters are used This indicates that the target image All source points Through matrix Transformed to target plane image Above, calculate the transformed points for each point. and the actual target point distance ; ； s3-2: Based on the preset threshold Determine an interior point if the distance is less than a threshold. Then it is believed If a sufficient number of points are identified as interior points, the model and its parameters are updated. This process is repeated until the specified number of iterations is reached or a model with the largest set of interior points is found. ,matrix Describes the target image and sub-image Geometric transformation relationships between them; s3-3: Sub-image Any point A on the lever scale is represented by homogeneous coordinates (x, y, 1), using the target image. sub-image perspective transformation matrix Multiplying the coordinates of point A by their homogeneous coordinates gives the coordinates of point A in the target image. Homogeneous coordinates of point B above : ； s3-4: Align secondary coordinates To perform normalization, divide x' and y' by w respectively, i.e. ； Therefore, sub-image Point A on the lever scale in the image The position of point B above is .

5. The lever image matching and localization method based on perspective transformation according to claim 4, characterized in that: Step S4 is as follows: s4-1: In the sub-image A subset of random points is generated within the lever scale area, denoted as the point set. Calculate sub-images Mapping the upper point set P to the target image The positions on the point set are denoted as point set. ; s4-2: Randomly select three sub-images from the baseline image set. , , None of the three sub-images are images. sub-image For example, calculate the target image sub-image Transformation matrix From steps s1-5, we know that The calculated matrix It can also represent the target image sub-image The perspective transformation matrix, in this case, is used. Recalculate the sub-image Mapping the upper point set P to the target image The positions on the point set are denoted as point set. ; s4-3: Computation of point set with point set Euler distance of corresponding elements And calculate the mean. : ； s4-4: Calculate the sub-image using steps s4-2 and s4-3 respectively. and Sub-image after action Mapping a set of known points on a target image Position and point set on The mean distance to corresponding elements, if the mean of three distances All are less than the preset pixel threshold Then the target image sub-image The feature matching and transformation matrix between them is accurate and can be used to map the obtained lever arm position onto the reference image to obtain the lever arm reading, which is used to verify the lever balance condition.

Images