Lever image matching and positioning method based on perspective transformation

Abstract

The invention discloses a lever image matching and positioning method based on perspective transformation, and relates to the technical field of computer vision and image recognition. The method specifically comprises the following steps: selecting a reference picture from a frame image, calculating a perspective transformation matrix, combining feature matching and instance segmentation screening, carrying out random sampling calculation and optimizing perspective transformation matrix model parameters, and estimating and evaluating matching and point mapping effects. According to the method, a perspective transformation matrix between two-dimensional images represented by multiple modes of the same object is utilized, feature point matching of a target lever image and a lever scaleplate surface area are obtained based on a feature extraction and detection method of a deep neural network, matching point pairs are screened in combination with the property of a vector, and the perspective transformation matrix is determined; and evaluating the effect of positioning the random point set on the known plane to the target image. The method does not depend on a single-standard multi-mode representation lever, the perspective transformation matrix dictionary is generated, and large errors possibly generated by single-standard operation are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image recognition, and in particular to a lever image matching and positioning method based on perspective transformation, which can be applied to the fields of image matching, computer vision, etc. Background Art

[0002] "Exploring the conditions for lever balance" is an important part of middle school physics experiments. Through the experiment, students can understand the principles of torque and lever, and learn how to use the balance conditions to calculate the forces and lever arms at different positions. This experiment not only enhances students' understanding of basic physical concepts, but also cultivates their experimental hands-on ability and scientific inquiry spirit. Based on the lever balance condition exploration experiment, computer vision technology and image processing, through image matching and perspective transformation algorithms, the lever system can be automatically identified and located, improving the efficiency and accuracy of the experiment, reducing the errors caused by single standard image feature matching, and providing quasi-data support for subsequent further research.

[0003] Image feature matching is a commonly used image registration method. It identifies key feature points in an image, extracts their descriptors or topological relationships, finds the correspondence between feature points, and then performs matching analysis to determine the spatial transformation relationship, thereby achieving alignment between images. Perspective transformation is a commonly used image geometric transformation method that can project an image to another perspective and maintain the geometric structure and relative position relationship in the image during this process. In traditional image processing, a two-dimensional image can be converted into a perspective projection in a three-dimensional space through a perspective transformation matrix, thereby achieving operations such as affine transformation and scale transformation of the image. Perspective transformation matrix and feature matching are key technologies in the field of computer vision, and their applications are widely involved in image registration, three-dimensional reconstruction and other fields. With the development of deep learning and neural networks, feature points in images can be extracted more accurately, and the correspondence between images can be learned, thereby improving the accuracy and robustness of feature matching.

[0004] However, there are still some challenges and problems in the field of perspective transformation matrix and feature matching. For example, the feature points themselves may not have enough discrimination, especially in the presence of complex backgrounds or low contrast in the image, which may lead to inaccurate matching results and inability to achieve precise matching of any point in the image, which may limit its applicability in some application scenarios. In addition, since perspective transformation is a nonlinear transformation, it fails when processing non-planar scenes in the image. When the camera position changes, if the scene cannot be constrained to a plane, the perspective transformation cannot correctly map the points in the image.

[0005] In summary, obtaining perspective transformation matrix and feature matching are key technologies in the field of computer vision, which have important application value in image processing, 3D reconstruction, etc. Especially in the application of lever image matching and point mapping, further research and improvement are still needed to meet the needs of different application scenarios. Summary of the invention

[0006] In order to solve the problem that feature matching on a lever image relies on a single standard for multi-modal lever representation, the present invention provides a lever image matching and positioning method based on perspective transformation.

[0007] The present invention is implemented by the following technical solution: a lever image matching and positioning method based on perspective transformation, comprising the following steps:

[0008] S1: Generation of perspective transformation matrix dictionary for multimodal lever representation: multiple images of levers at different positions, angles, distances, noise levels and states are selected from the extracted frame images as a set of reference images, and the perspective transformation matrix is ​​calculated for every two images to obtain an image transformation matrix dictionary;

[0009] S2: Screening of matching point pairs in the lever scale surface area based on feature matching and instance segmentation: For any target lever image, feature point matching is performed with the reference image, and the lever scale surface area with shape features is obtained by the instance segmentation method. The matching point pairs located in the instance segmentation polygon area are screened based on vector calculation;

[0010] S3: Iterative optimization of perspective transformation matrix determination and point mapping: Randomly sample the data in the matching point pair set to estimate the perspective transformation matrix model parameters, identify and exclude outliers far from the model, iteratively optimize to obtain a more accurate perspective transformation matrix, use homogeneous coordinates to represent a point in the lever scale surface of the reference image, and use the perspective transformation matrix to obtain the specific position of the point mapped to the target image;

[0011] S4: Evaluation of the effect of positioning points on a known plane on the target image: Use the known random point set on the reference image to map to the distance spacing of the position on the target image according to different calculation paths, and judge the effect of feature matching and perspective transformation between the target image and the reference image to ensure that there will be no excessive errors that affect the subsequent further analysis of the target image.

[0012] A lever image matching and positioning method based on perspective transformation designed by the present invention is a method for verifying feature matching and point mapping effects based on perspective transformation and instance segmentation. It utilizes the perspective transformation matrix between two-dimensional images of multimodal representation of the same object, and a feature extraction and detection method based on a deep neural network to obtain feature point matching and lever scale surface area of ​​the target lever image. The matching point pairs are screened and the perspective transformation matrix is ​​determined in combination with the properties of the vector, and the effect of positioning a random point set on a known plane on the target image is evaluated.

[0013] Compared with the prior art, the present invention has the following beneficial effects: the lever image matching and positioning method based on perspective transformation provided by the present invention does not rely on a single standard multimodal characterization lever, generates a perspective transformation matrix dictionary, and reduces the large errors that may be caused by a single standard operation; uses the properties of vectors to screen matching point pairs located in the lever scale surface area, and excludes a part of abnormal values ​​far away from the model for randomly sampled data, which is conducive to obtaining more accurate model parameters; uses a known random point set on a plane to map to the distance spacing of the position on the target image according to different calculation paths, and judges the effect of feature matching and perspective transformation between the target image and the reference image, ensuring that better model parameters can be used for further analysis of the target image. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A schematic diagram of selecting the vertices of a lever scale surface in a sub-image of a reference image according to a specific embodiment of the present invention.

[0015] Figure 2 A schematic diagram of selecting the vertices of the lever scale surface of another sub-image in the reference image according to a specific embodiment of the present invention.

[0016] Figure 3 It is a schematic diagram of calculating the perspective transformation matrix of two sub-images in a reference image according to a specific embodiment of the present invention.

[0017] Figure 4 It is a schematic diagram of calculating matching point pairs between a target image and a sub-image according to a specific embodiment of the present invention.

[0018] Figure 5 It is a schematic diagram of selecting matching point pairs of a target image and a sub-image located in the lever scale surface area according to a specific embodiment of the present invention.

[0019] Figure 6 A schematic diagram of selecting known points on a portion of the lever scale surface for a specific embodiment of the present invention.

[0020] Figure 7 It is a schematic diagram of mapping known points through a perspective transformation matrix according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0021] The present invention is further described below in conjunction with specific embodiments.

[0022] S1: Generation of perspective transformation matrix dictionary for multimodal lever representation: Select multiple images of levers at different positions, angles, distances, noise levels and states from the extracted frame images as a set of reference images, calculate the perspective transformation matrix for every two images, and obtain an image transformation matrix dictionary; the details are as follows:

[0023] s1-1: Use a device with video recording function to record the entire experiment of "Exploring the Conditions of Lever Balance", extract static frame images from the recorded video files, and obtain multi-frame image files;

[0024] s1-2: Select N images of the lever placed on the fulcrum of the iron stand in different modes from the extracted frame images as a set of reference images. The reference image set is recorded as K. The multi-modality of the lever includes but is not limited to the different distances of the lever from the camera, the left or right tilt of the lever when it is unbalanced, the degree of tilt and rotation, and the different experimenters. The standard for selecting images ensures that they can represent the multiple modes of the lever in the experiment;

[0025] s1-3: For two images of the same lever at different positions, use cv2.getPerspectiveTransform to obtain a 3×3 perspective transformation matrix, select four vertices of the source image and the target image on the lever scale surface respectively, and each source point corresponds to a target point one by one, that is, src_points[i] corresponds to dst_points[i], ensuring that the selected points are in the same order and can form a quadrilateral; Figure 1 , Figure 2 and Figure 3 As shown in the figure, the three figures are the source image and the target image, and the four vertices of the lever scale surface are selected as the four point pairs corresponding to each other and after calculation and transformation. Figure 1 Medium leverage corresponds to Figure 2 The position on.

[0026] s1-4: Calculate the perspective transformation matrix for any two images in the benchmark image set K, and store all matrices in the dictionary Dict. The key of the dictionary is the identifier of the image pair, and the value is the corresponding perspective transformation matrix;

[0027] s1-5: For any three different bases in the benchmark image set K, if I i To I j The perspective transformation matrix is ​​matrixl i_j ,I j To I k The perspective transformation matrix is ​​matrixl j_k , then I i To I kThe perspective transformation matrix matrixl i_k There are the following relations:

[0028] matrixl i_k =matrixl i_j ×matrixl j_k

[0029] S2: Screening of matching point pairs in the lever scale surface area based on feature matching and instance segmentation: For any target lever image and the reference image, feature point matching is performed, and the lever scale surface area with shape features is obtained by the instance segmentation method. The matching point pairs located in the instance segmentation polygon area are screened according to the calculation of the vector; the details are as follows:

[0030] s2-1: For any target image I with a lever placed on the iron frame lever , randomly select a sub-image I in the benchmark image set K a , use SuperPoint based on convolutional neural network CNN to obtain target image I lever and sub-image I a A feature extraction network and a descriptor generation network; the feature extraction network extracts the location of feature points from the input image through CNN; the descriptor generation network generates a descriptor for each feature point to distinguish different feature points and match them;

[0031] s2-2: LightGlue accepts the target image I lever and sub-image I a The feature point coordinates and descriptor set are used as input, and the self-attention and cross-attention units are used to adjust the matching process according to the similarity of the image pair, update the representation of each feature point, and obtain the target image I lever and sub-image I a The set of matching point pairs M = {(p1,p1′),(p2,p2′),...,(p n ,p n ′)}, where p n and p n ′ are the matched feature point pairs respectively; Figure 4 As shown, for the target image and sub-image, SuperPoint generates feature points, LightGlue calculates the allocation matrix, discards the unmatched points and predicts the matching point pairs, and obtains the matching point pairs of the target image and sub-image.

[0032] s2-3: Divide the image training set, validation set, and test set used to train the instance segmentation neural network model in the extracted static frame image, and manually mark the lever scale surface area with polygons using a labeling tool. The labeling tool uses labelme or cvat, and selects a specific network model structure, namely VGG or resnet. The YOLOv8 instance segmentation framework is combined with the data set training to obtain a network weight with a certain degree of intersection and union ratio. The value of the degree of intersection and union ratio is 0.7 to 1. The degree of intersection and union ratio of this embodiment can be in the range of 0.7 to 0.85; it is used to segment and extract the lever scale surface area in the image;

[0033] s2-4: For target image I lever , the lever scale polygon point set S = {(x1,y1),(x2,y2),...,(x n ,y n )}, target image I lever Any matching point P on the p ,y p ), if (x1,y1) is point A and (x2,y2) is point B, then the vector Calculate the cross product of vector AB and vector AP:

[0034]

[0035] s2-5: For the adjacent points in the polygon point set S, calculate the cross product in the same way as step s2-4. If the results of all cross products have the same sign, all positive or all negative, then point P is within the instance segmentation polygon; otherwise, point P is not within the instance segmentation polygon, and the target image I is deleted. lever The matching points and corresponding matching pairs that are not in the lever scale polygonal area of ​​the instance segmentation are obtained to obtain the target image I lever and sub-image I a New matching point pair set M sift ={(p1,p1′),(p2,p2′),...,(p n ,p n ′)}, Figure 5 The effect of filtering the matching points located in the polygonal area of ​​the lever scale surface for the target image and the matching points corresponding to the sub-image after filtering. The filtered matching points are located in a plane in space, and the transformation matrix calculated subsequently will be more accurate.

[0036] S3: Iterative optimization of perspective transformation matrix determination and point mapping: Randomly sample the data in the matching point pair set to estimate the perspective transformation matrix model parameters, identify and exclude outliers far from the model, iteratively optimize to obtain a more accurate perspective transformation matrix, use homogeneous coordinates to represent a point in the lever scale surface of the reference image, and use the perspective transformation matrix to obtain the specific position of the point mapped to the target image; the details are as follows:

[0037] s3-1: Using RANSAC, in the matching point set M sift Randomly select four point pairs in the equation and estimate the perspective transformation matrix H, where the model parameters are represented by h ij Indicates that the target image I lever All source points (x i ,y i ) is transformed to the target plane image I through the matrix H a On the graph, calculate the transformed point (x i ″,y i ″) and the actual target point (x i ′,y i ′) i ;

[0038]

[0039] s3-2: Determine the inner point according to the preset threshold ∈. If the distance is less than the threshold ∈, it is considered If enough points are identified as inliers, the model and its parameters are updated; the above steps are repeated until the specified number of iterations is reached or a model H with the largest set of inliers is found. a , the matrix H a Describes the target image I lever and sub-image I a The geometric transformation relationship between them;

[0040] s3-3: Sub-image I a Any point A on the lever scale surface is represented as homogeneous coordinates (x, y, 1), using the target image I lever With sub-image I a The perspective transformation matrix H a Multiply it with the homogeneous coordinates of point A to get point A in the target image I lever The homogeneous coordinate position (x′, y′, w) of the corresponding point B on the graph is:

[0041]

[0042] s3-4: Align the secondary coordinates (x′, y′, w) and normalize them by dividing x′ and y′ by w, i.e.

[0043]

[0044] So the sub-image I a Point A on the lever scale surface is on image I lever The position of the corresponding point B is (x′ / w, y′ / w), such as Figure 6 and Figure 7 As shown, through Figure 6 and Figure 7 The perspective transformation matrix between Figure 6 The known partial point set on the map is mapped to Figure 7 The specific location on.

[0045] S4: Evaluation of the effect of positioning points on a known plane on the target image: Use the known random point set on the reference image to map to the distance spacing of the position on the target image according to different calculation paths, and judge the effect of feature matching and perspective transformation between the target image and the reference image to ensure that there will not be too large errors affecting the subsequent further analysis of the target image, as follows:

[0046] s4-1: In sub-image I a Generate some random points in the lever scale area, denoted as point set P = {(x1,y1),(x2,y2),...,(x n ,y n )}, calculate sub-image I a The point set P is mapped to the target image I lever The position on the point set P a ={(x a1 ,y a1 ),(x a2 ,y a2 ),...,(x an ,y an )};

[0047] s4-2: Randomly select three sub-images I from the benchmark image set b1 ,I b2 ,I b3 , the three sub-images are not image I A , with sub-image I b1 Take the target image I as an example. lever With sub-image I b1 The transformation matrix H b1 , from step s1-5, we know that matrixl a_b1 ×H b1 The calculated matrix H a_b1 It can also represent the target image I lever With sub-image I a The perspective transformation matrix, at this time, uses the matrix H a_b1 Calculate sub-image I againa The point set P is mapped to the target image I lever The position on the point set P b1 ={(x b1 ,y b1 ),(x b2 ,y b2 ),...,(x bn ,y bn )};

[0048] s4-3: Calculate the point set P a With point set P b1 The Euler distance d of the corresponding elements in i , and calculate the mean

[0049]

[0050] s4-4: Calculate the sub-image I using steps s4-2 and s4-3 respectively b2 and I b3 Sub-image I after action a The known point set on is mapped to the target image I lever The position and point set P on a The distance mean of the corresponding elements, if the three distance means are all less than the preset pixel threshold ρ, then the target image I lever With sub-image I a The feature matching and transformation matrix between them are accurate and can be used to subsequently map the acquired lever arm position to the reference image, and further obtain the lever arm reading to verify the lever balance condition.

[0051] The scope of protection claimed by the present invention is not limited to the above specific embodiments. For those skilled in the art, the present invention may have various deformations and changes. Any modifications, improvements and equivalent substitutions made within the concept and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A lever image matching and positioning method based on perspective transformation, characterized in that: The steps include: S1: Generation of perspective transformation matrix dictionary for multimodal lever representation: multiple images of levers at different positions, angles, distances, noise levels and states are selected from the extracted frame images as a set of reference images, and the perspective transformation matrix is ​​calculated for every two images to obtain an image transformation matrix dictionary; S2: Screening of matching point pairs in the lever scale surface area based on feature matching and instance segmentation: For any target lever image, feature point matching is performed with the reference image, and the lever scale surface area with shape features is obtained by the instance segmentation method. The matching point pairs located in the instance segmentation polygon area are screened based on vector calculation; S3: Iterative optimization of perspective transformation matrix determination and point mapping: Randomly sample the data in the matching point pair set to estimate the perspective transformation matrix model parameters, identify and exclude outliers far from the model, iteratively optimize to obtain an accurate perspective transformation matrix, use homogeneous coordinates to represent a point in the lever scale surface of the reference image, and use the perspective transformation matrix to obtain the specific position of the point mapped to the target image; S4: Evaluation of the effect of positioning points on a known plane on the target image: Use the known random point set on the reference image to map to the distance spacing of the position on the target image according to different calculation paths, and judge the effect of feature matching and perspective transformation between the target image and the reference image to ensure that it will not affect the subsequent further analysis of the target image.

2. The lever image matching and positioning method based on perspective transformation according to claim 1, characterized in that: Step S1 is specifically as follows: s1-1: Use a device with video recording function to record the entire experiment of "Exploring the Conditions of Lever Balance", extract static frame images from the recorded video files, and obtain multi-frame image files; s1-2: Select N images of the lever placed on the fulcrum of the iron stand in different modes from the extracted frame images as a set of benchmark images. The benchmark image set is recorded as K. The multi-modality of the lever includes different distances between the lever and the camera, left or right tilt of the lever when it is unbalanced, the degree of tilt and rotation, and different experimenters. The standard for selecting images ensures that they can represent multiple modes of the lever in the experiment; s1-3: For two images of the same lever at different positions, use cv2.getPerspectiveTransform to obtain a 3×3 perspective transformation matrix, and select the four vertices of the source image and the target image on the lever scale surface, respectively. Each source point corresponds to a target point one by one, that is, Corresponds to , make sure the selected points are in the same order and can form a quadrilateral; s1-4: Calculate the perspective transformation matrix for any two images in the benchmark image set K, and store all matrices in the dictionary In the dictionary, the key is the identifier of the image pair, and the value is the corresponding perspective transformation matrix; s1-5: For the benchmark image set If any three different bases arrive The perspective transformation matrix is , arrive The perspective transformation matrix is ,So arrive The perspective transformation matrix There are the following relations: 。 3. The lever image matching and positioning method based on perspective transformation according to claim 2, characterized in that: Step S2 is specifically as follows: s2-1: For any target image with a lever placed on the iron frame , randomly select a sub-image in the benchmark image set K , use SuperPoint based on convolutional neural network CNN to obtain the target image and sub-image A feature extraction network and a descriptor generation network; the feature extraction network extracts the location of feature points from the input image through CNN; the descriptor generation network generates a descriptor for each feature point to distinguish different feature points and match them; s2-2: LightGlue accepts the target image and sub-image The feature point coordinates and descriptor set are used as input, and the self-attention and cross-attention units are used to adjust the matching process according to the similarity of the image pairs, update the representation of each feature point, and obtain the target image. and sub-image The set of matching point pairs ,in and They are the matched feature point pairs respectively; s2-3: Divide the image training set, validation set, and test set used to train the instance segmentation neural network model in the extracted static frame image, and use the annotation tool to manually mark the lever scale surface area with polygons. The annotation tool uses labelme or cvat, and selects a specific network model structure, namely VGG or resnet. The YOLOv8 instance segmentation framework is combined with the data set training to obtain a network weight with a certain degree of intersection-over-union ratio. The value of the intersection-over-union ratio is 0.7~1; it is used to segment and extract the lever scale surface area in the image; s2-4: For the target image , the lever scale polygon point set obtained by instance segmentation , target image any matching point on , the location information is ,like For point A, is point B, then the vector , , calculate the cross product of vector AB and vector AP: ； s2-5: For polygon point set The adjacent points in the grid are calculated by following steps s2-4. If the results of all cross products have the same sign, all positive or all negative, then the point is within the instance segmentation polygon; otherwise, the point If the polygon is not within the instance segmentation, delete the target image. The matching points and corresponding matching pairs that are not in the polygonal area of ​​the lever scale face of the instance segmentation are used to obtain the target image. and sub-image New set of matching point pairs .

4. The lever image matching and positioning method based on perspective transformation according to claim 3, characterized in that: Step S3 is as follows: s3-1: Using RANSAC, in the matching point set M sift Randomly select four point pairs in and estimate the perspective transformation matrix , where the model parameters are Indicates that the target image All source points Through the matrix Transform to target plane image Calculate the transformed point of each point The actual target point Distance ; ； s3-2: According to the preset threshold Determine the inliers if the distance is less than the threshold , then it is believed that If enough points are identified as inliers, the model and its parameters are updated; the above steps are repeated until the specified number of iterations is reached or a model with the largest set of inliers is found. ,matrix Describes the target image and sub-image The geometric transformation relationship between them; s3-3: Sub-image Any point A on the lever scale surface is represented as homogeneous coordinates (x, y, 1), using the target image With sub-image The perspective transformation matrix Multiply it with the homogeneous coordinates of point A to get the coordinate of point A in the target image. The homogeneous coordinate position of the corresponding point B on : ； s3-4: Align secondary coordinates Normalize x' and y' by w, that is ； So the sub-image Point A on the lever scale surface is on the graph The position of the corresponding point B is .

5. The lever image matching and positioning method based on perspective transformation according to claim 4, characterized in that: Step S4 is specifically as follows: s4-1: In sub-image Generate some random points in the lever scale area, recorded as point set , calculate the sub-image The point set P is mapped to the target image The position on the ; s4-2: Randomly select three sub-images from the benchmark image set , , , the three sub-images are not images , with sub-image For example, calculate the target image With sub-image The transformation matrix , from step s1-5, The calculated matrix It can also represent the target image With sub-image The perspective transformation matrix is ​​used. Calculate the sub-image again The point set P is mapped to the target image The position on the ; s4-3: Calculation point set With point set The Euler distance of the corresponding elements in , and calculate the mean : ； s4-4: Use steps s4-2 and s4-3 to calculate and sub-image and Sub-image after action Map the known point set on the target image Positions and point sets on The distance mean of the corresponding elements, if the three distance means are all less than the preset pixel threshold , then the target image With sub-image The feature matching and transformation matrix between them are accurate and can be used to subsequently map the acquired lever arm position to the reference image to obtain the lever arm reading, which is used to verify the lever balance condition.

Images