Inter-view image feature matching method based on physical information neural network

Abstract

The application discloses a physical information neural network-based inter-view image feature matching method, which comprises the following steps: in an offline stage, a plurality of feature point pairs of image boundaries and corner regions are collected, linear interpolation is performed on approximate parallax fields of integral pixel points in a feature point convex hull, and a full connection network model obtained through construction is pre-trained; the pre-trained network model is globally optimized and trained by using a gray consistency assumption; and in an online stage, real-time inter-view image feature matching is performed through the globally optimized and trained full connection network model.

Description

Technical Field

[0001] This invention relates to a technique in the field of image processing, specifically a method for matching image features between viewpoints based on a physical information neural network. Background Technology

[0002] Interview feature matching refers to the process of establishing the correspondence between feature points in two or more images of the same scene acquired from different viewpoints through feature extraction and description. Interview feature matching methods based on feature descriptors suffer from sparse matching results and an inability to achieve dense matching because keypoint detection cannot cover every pixel. Interview feature matching methods based on regional grayscale correlation suffer from low accuracy in solving non-uniform disparity fields and boundary region disparity due to a lack of scientific guidance in parameter settings and missing grayscale information in boundary regions. Image feature matching methods based on global grayscale correlation suffer from low efficiency, insufficient accuracy, and even matching failure because the large disparity between images from different viewpoints makes it difficult to quickly converge to the global optimum in disparity field optimization. Summary of the Invention

[0003] This invention addresses the shortcomings of existing technologies, which can be used for image pixel classification tasks but cannot accurately track pixel displacement changes and whose matching accuracy is limited to regional features and pixel level. It also fails to solve the registration problem of disparity fields with large viewpoint distortion. This invention proposes a viewpoint image feature matching method based on a physical information neural network. Through a seed point guidance strategy, it significantly improves the convergence speed and accuracy of global optimization solutions for large disparity fields.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a viewpoint image feature matching method based on a physical information neural network. In the offline stage, several feature point pairs in the image boundary and corner regions are first collected. After performing approximate disparity field linear interpolation on the integer pixels within the convex hull of the feature points, the constructed fully connected network model is pre-trained. Then, the pre-trained network model is globally optimized and trained using the gray-level consistency assumption. In the online stage, real-time viewpoint image feature matching is performed using the globally optimized and trained fully connected network model.

[0006] Technical effect

[0007] This invention utilizes a fully connected deep neural network to establish a correspondence between the global coordinate domain and the disparity field. It extends the coordinate domain for disparity field solution to the image boundary region, ensuring that the contribution of the image boundary region to the optimization objective function is consistent with that of the internal region. This achieves consistent global feature matching accuracy and employs a seed point guidance strategy. Compared to existing technologies, this invention achieves pixel-by-pixel disparity field solution; it solves the problems of poor matching accuracy in image boundary regions in traditional disparity field solution methods, as well as the difficulty in convergence and poor convergence accuracy in the matching optimization process when the disparity is large. Attached Figure Description

[0008] Figure 1 This is a flowchart of the present invention;

[0009] Figure 2 A schematic diagram of the PINN network architecture and the pre-training and global optimization training process;

[0010] Figure 3 A schematic diagram illustrating the selection of feature points and the establishment of an approximate disparity field for seed point interpolation;

[0011] Figure 4 This is a schematic diagram of the images and ROIs used for matching in the embodiment;

[0012] Figure 5 This is a schematic diagram of the disparity field obtained by matching based on the PINN method in the embodiment;

[0013] Figure 6 This is a schematic diagram of the disparity field obtained by matching using a traditional region matching method in an example embodiment.

[0014] Figure 7 According to Figure 5 and Figure 6 A line graph plotting the disparity U and V data for y = 716 rows (line 1) and x = 716 columns (line 2);

[0015] Figure 8 To Figure 7 A magnified view of the area outlined in the middle frame. Detailed Implementation

[0016] like Figure 1 As shown in the figure, this embodiment relates to a viewpoint image feature matching method based on a physical information neural network, including:

[0017] S1: After selecting several landmark points in the boundary and corner areas of the reference image, identify each corresponding point in the target image, i.e., feature points, and record the pixel coordinates of each feature point.

[0018] The aforementioned landmarks include, but are not limited to, corners of objects, points with unique textures, etc.

[0019] The identification process can be completed by means of, but is not limited to, visual observation or automatic identification using algorithms such as SIFT.

[0020] The matching accuracy of the corresponding points only needs to reach the feature level (or region level), not the pixel level. Four or slightly more sets of point pairs should be selected.

[0021] S2: Connect the outer perimeter of the feature points as shown in the figure. Figure 3 The convex hull shown is the smallest convex polygon. For the integer pixels within the convex hull, i.e. the seed points, linear disparity field interpolation is performed to record the coordinates of the seed points on the reference image and their approximate disparity.

[0022] The minimum convex polygon is formed by sequentially connecting a ring of feature points closest to the edge of the reference image.

[0023] The approximate parallax ,in: For the disparity of feature points, The number of feature point pairs, These are the interpolation coefficients. The coordinates of the seed point, This is the approximate disparity of the seed point obtained through interpolation.

[0024] S3: Construct a fully connected neural network model with coordinate domains X and Y as inputs and disparity fields U and V as outputs. The network model includes: an input layer with two neurons, an output layer with two neurons, and a hidden layer with two layers, each with 30 neurons.

[0025] The two neurons in the input layer represent the X and Y coordinates, respectively; the two neurons in the output layer represent the U and V disparity field components, respectively.

[0026] The activation function for the hidden layer is tanh.

[0027] The fully connected network model is as follows: ,in: is the parameter vector of the fully connected network model, where NN stands for Neural Network.

[0028] S4: Construct and pre-train the pre-trained loss function: using the disparity field at the seed point output by the fully connected network model ( ) and the approximate disparity field obtained in step S2 ( The mean squared error of the model is used to pre-train the fully connected neural network model in S3.

[0029] The pre-training loss function ,in: This represents the number of seed points.

[0030] The pre-training process employs the Adam optimizer, where the pre-training loss function... Training can be stopped at this point, and the network parameters obtained are: And save the network model.

[0031] The pre-training process preferably involves inputting seed point coordinates into the fully connected neural network model. ), thus obtaining the approximate disparity field at the seed point ( ), and calculate the pre-training loss function.

[0032] S5: Construct a globally optimized loss function based on the assumption of grayscale consistency and perform global optimization training: using the network parameters obtained from pre-training... Based on this, the coordinates of all pixels within the region of interest (ROI) are used. The model is trained in a two-stage manner until the loss function converges, at which point the disparity field output by the model is saved. ).

[0033] The global optimization loss function ,in: These are the pixel coordinates within the ROI region. These are the parameters of the neural network.

[0034] The region of interest can be rectangular or irregular in shape.

[0035] The two-stage training method mentioned above refers to: first, using the Adam optimizer to train until the global optimization loss function is achieved. Once stable, the L-BFGS optimizer is applied to optimize the global loss function. Further reduce until convergence, and preserve the disparity field output by the model. ).

[0036] S6: In the online phase, image feature matching between online viewpoints is performed using the fully connected neural network model trained through global optimization in step S5.

[0037] Through practical application experiments, the inter-view image feature matching method based on physical information neural network of this invention was run on an NVIDIA RTX 3090 GPU with 24GB of video memory. The corresponding code of this invention was written in the Python 3.12.8 environment based on the PyTorch 2.8.0 library.

[0038] Taking the matching of speckle images from left and right perspectives in the digital image correlation (DIC) problem as an example, such as Figure 4(a) The left-view image is the reference image, (b) the right-view image is the target image, and the ROI region shown in (c) is selected. The simulation of the above-mentioned inter-view image feature matching method based on a physical information neural network is performed, including:

[0039] S1: Observe the images. In the areas near the four corners of both images, use the mouse to click on a set of points with larger speckle sizes as feature points to obtain the quadrilateral convex hull, such as... Figure 4 As shown by the red dot. The program records the pixel coordinates of four sets of points and calculates the disparity.

[0040] S2: Using a step size of 10 pixels, perform disparity field interpolation on integer pixel coordinates within the convex hull. This operation is accomplished by calling the `griddata` function from the `interpolate` module of the `scipy` package in Python, resulting in 14200 seed points (X). seed Y seed Approximate disparity field () ).

[0041] S3: Construct a fully connected network with input in the coordinate domain (X, Y) and output in the disparity field (U, V);

[0042] S4: Constructing the loss function

[0043] S5: Pre-training: Arrange the seed point coordinates into a 14200-row, 2-column matrix, and input it into the neural network. The loss function is defined using the Adam optimizer with a learning rate of 0.03 and a weight decay coefficient of 0.01. Other optimizer hyperparameters are set to default values. Training is performed for 3000 epochs. Reduced to approximately 0.9), and the network model was saved.

[0044] S6: Construct the loss function There are a total of 1,822,211 computation points within the ROI domain. ;

[0045] S7: Global Optimization Training: Load the network model obtained in step S5, and input the coordinates within the ROI domain ( At this point, the output corresponds to the disparity field within the ROI domain. ),by Using the loss function, the Adam optimizer was used for initial training with a learning rate of 0.001, a weight decay coefficient of 0.01, and 10,000 training epochs. Then, the L-BFGS optimizer was used for high-precision training with a learning rate of 0.01 and 10,000 training epochs. After training, the resulting disparity field was saved and visualized, as shown below. Figure 5 As shown.

[0046] As a contrast, the disparity field obtained by the traditional inter-viewpoint image feature matching method based on regional gray-level correlation is plotted on... Figure 6 , Figure 6 The region outlined in the middle box represents areas with mismatches or low matching accuracy. It can be seen that in areas with ROI boundaries, significant disparity changes, and large viewpoint distortion, this method is significantly superior to traditional methods. To further demonstrate the advantages of this invention, extraction was performed separately... Figure 5 and Figure 6 Plot a line graph of the disparity U and V data in row 716 (y = 716, line 1) and column 716 (x = 716, line 2). Figure 7 ),exist Figure 8 Will Figure 7 Detailed zoom in on certain areas.

[0047] Compared with existing technologies, this method utilizes a fully connected neural network model to globally represent the disparity field, resulting in a more continuous, smooth, and low-noise disparity field. By employing a physical information neural network method with input in the ROI coordinate domain, it more accurately solves for the disparity at the ROI boundary, without obvious noise or jagged regions. The seed point guidance strategy enables the network model to converge to a high-precision solution, and for regions with significant viewpoint distortion (such as the edges, corners, and ends of convex structures), the solution results do not exhibit abrupt changes or other anomalies. In summary, the disparity field obtained by this method is smoother and more accurate.

[0048] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A method for inter-viewpoint image feature matching based on a physical information neural network, characterized in that, In the offline stage, several feature point pairs in the image boundary and corner regions are first collected. After performing approximate disparity field linear interpolation on the integer pixels within the convex hull of the feature points, the constructed fully connected network model is pre-trained. Then, the gray-level consistency assumption is used to perform global optimization training on the pre-trained network model. In the online phase, real-time image feature matching between viewpoints is performed using a fully connected network model trained with global optimization.

2. The image feature matching method between viewpoints based on a physical information neural network according to claim 1, characterized in that, The training set used in the pre-training was obtained in the following way: S1: After selecting several landmark points in the boundary and corner areas of the reference image, identify each corresponding point in the target image, i.e., feature points, and record the pixel coordinates of each feature point; S2: Connect the outer perimeter of the feature points to form a convex hull, i.e. the smallest convex polygon. Perform disparity field linear interpolation on the integer pixels inside the convex hull, i.e. the seed points, and record the coordinates of the seed points on the reference image and their approximate disparity. The minimum convex polygon is formed by sequentially connecting points selected in the four corner regions of the reference image. The approximate parallax ,in: For the disparity of feature points, The number of feature point pairs, These are the interpolation coefficients. The coordinates of the seed point, This is the approximate disparity obtained through interpolation.

3. The image feature matching method between viewpoints based on a physical information neural network according to claim 1, characterized in that, The fully connected network model includes: an input layer with two neurons, an output layer with two neurons, and a hidden layer consisting of two layers, each with 30 neurons. The two neurons in the input layer represent the X and Y coordinates, respectively; the two neurons in the output layer represent the U and V disparity field components, respectively. The activation function for the hidden layer is tanh.

4. The inter-viewpoint image feature matching method based on a physical information neural network according to any one of claims 1-3, characterized in that, The aforementioned pre-training specifically refers to: using the disparity field at the seed point output by the network model ( ) and the approximate disparity field obtained in step S2 ( The mean squared error of the fully connected neural network model is used for pre-training. The pre-training loss function ,in: This represents the number of seed points; The pre-training process employs the Adam optimizer, where the pre-training loss function... Training can be stopped when the network parameters are denoted as follows: Save the network model.

5. The image feature matching method between viewpoints based on a physical information neural network according to claim 4, characterized in that, The pre-training process involves inputting seed point coordinates into the fully connected neural network model. ), thus obtaining the approximate disparity field at the seed point ( ), and calculate the pre-training loss function.

6. The inter-viewpoint image feature matching method based on a physical information neural network according to any one of claims 1-5, characterized in that, The aforementioned global optimization training specifically refers to: optimizing the network parameters obtained during pre-training. Based on this, the coordinates of all pixels within the region of interest (ROI) are used. The model is trained in a two-stage manner until the loss function converges, at which point the disparity field output by the model is saved. ); The global optimization loss function ,in: These are the pixel coordinates within the ROI region. These are the parameters of the neural network; The two-stage training method mentioned above refers to: first, using the Adam optimizer to train until the global optimization loss function is achieved. Once stable, the L-BFGS optimizer is applied to optimize the global loss function. Further reduce until convergence, and preserve the disparity field output by the model. ).

Images