Environment data fitting method and device based on graph neural network

Abstract

The application relates to the technical field of computer vision, and provides an environment data fitting method and device based on a graph neural network, which can break through the existing bottleneck in a targeted manner, solve the defects of missing spatial relationship and insufficient diversity of hypotheses in the prior art, and improve the accuracy and determination efficiency of an optimal model, and meets the application requirements of a real scene for a high-precision, high-robustness and high-efficiency method.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and apparatus for fitting environmental data based on graph neural networks. Background Technology

[0002] In the field of robust environmental data fitting in computer vision, the core is to estimate multiple sets of model parameters from noisy, outlier observations to support key applications such as 3D reconstruction. It has evolved from the early stage of single-model fitting to the current stage of core requirements for environmental data fitting.

[0003] Traditional methods, such as Sequential RANSAC and J-Linkage, are based on sampling, optimization, or clustering, but rely on manual rules and lack robustness and efficiency. Deep learning methods, such as Neural-Guided RANSAC and ARS-MAGSAC, improve performance by guiding hypothesis generation through neural networks, but have limitations such as missing spatial relationships and insufficient hypothesis diversity. Summary of the Invention

[0004] This invention provides an environmental data fitting method and apparatus based on graph neural networks to address the deficiencies in existing technologies.

[0005] This invention provides an environmental data fitting method based on graph neural networks, comprising:

[0006] Acquire target environment data images of the target object from multiple perspectives, and obtain the two-dimensional coordinate set of matching feature points in the target environment data images from any two perspectives;

[0007] The two-dimensional coordinate set is input into a dynamic graph convolutional neural network to obtain the sampling weights and inlier weights output by the dynamic graph convolutional neural network.

[0008] Based on the sampling weights, a minimum observation set is sampled for each candidate model, and multiple model hypotheses are generated using the minimum observation set based on the minimum solver.

[0009] Gaussian modeling is performed on multiple model assumptions to generate additional assumptions. Based on the multiple model assumptions and the additional assumptions, the interior point weights are applied to select the optimal assumption for each candidate model, forming a candidate model set.

[0010] The candidate model set is sorted and clustered to obtain the optimal model and cluster label for the target object.

[0011] According to the present invention, an environmental data fitting method based on a graph neural network is provided, wherein the dynamic graph convolutional neural network includes an initial feature extraction module, a local feature extraction module, a global feature extraction module, a feature fusion module, a spatial aggregation module, a sampling weight output module, and an interior point weight output module;

[0012] The step of inputting the two-dimensional coordinate set into a dynamic graph convolutional neural network to obtain the sampling weights and inlier weights output by the dynamic graph convolutional neural network includes:

[0013] Based on the initial feature extraction module, the initial features of the two-dimensional coordinate set are extracted;

[0014] Based on the local feature extraction module, local features of the initial feature are extracted, and based on the global feature extraction module, global features of the initial feature are extracted.

[0015] Based on the feature fusion module, a cross-attention mechanism is applied to fuse the local features and the global features to obtain fused features;

[0016] Based on the spatial aggregation module, channel attention mechanism and spatial attention mechanism are applied to spatially aggregate the fused features to obtain aggregated features;

[0017] Based on the sampling weight output module, the aggregation feature is applied to determine the sampling weight, and based on the inlier weight output module, the aggregation feature is applied to determine the inlier weight.

[0018] According to the present invention, an environmental data fitting method based on a graph neural network is provided, wherein the global feature extraction module includes multiple graph attention modules;

[0019] Each of the graph attention modules is used to extract the global features through a multi-head attention mechanism.

[0020] According to the present invention, an environmental data fitting method based on a graph neural network is provided, wherein performing Gaussian modeling on multiple model hypotheses to generate additional hypotheses includes:

[0021] Based on target environment data images from any two perspectives, calculate the first calculation error of the fundamental matrix and the second calculation error of the homography matrix. Based on the first calculation error, determine the first Jacobian matrix and based on the second calculation error, determine the second Jacobian matrix.

[0022] Based on the first Jacobian matrix, estimate the first covariance matrix of the foundation matrix, and based on the second Jacobian matrix, estimate the second covariance matrix of the homography matrix;

[0023] Based on the first covariance matrix and the second covariance matrix, Gaussian modeling is performed on multiple model assumptions to generate the additional assumptions.

[0024] According to the present invention, an environmental data fitting method based on a graph neural network is provided, wherein the additional hypotheses are generated by performing Gaussian modeling on multiple model hypotheses based on the first covariance matrix and the second covariance matrix, including:

[0025] Based on the first covariance matrix and the second covariance matrix, standard normal vectors are generated respectively;

[0026] Based on the standard normal vector, the additional assumptions of the model are modified to obtain the additional assumptions.

[0027] According to the present invention, an environmental data fitting method based on a graph neural network is provided, wherein the dynamic graph convolutional neural network is trained based on the following steps:

[0028] Acquire training samples of environmental data images of sample objects from multiple perspectives, and acquire a set of two-dimensional coordinates of matching key points in the training samples of environmental data images from any two perspectives, as well as ground truth class labels of matching key points in the training samples of environmental data images from any two perspectives.

[0029] The two-dimensional coordinate set samples are input into the initial graph convolutional neural network to obtain the output result of the initial graph convolutional neural network;

[0030] Based on the output of the initial graph convolutional neural network, the optimal model and cluster label of the sample object are determined, and the task loss is calculated based on the ground truth class label and the cluster label of the sample object. Based on the expectation of the task loss, a loss function is constructed.

[0031] The deterministic hypothesis is converted into a probabilistic form, and based on the probabilistic form, the gradient of the loss function with respect to the network parameters of the initial graph convolutional neural network is calculated;

[0032] The gradient is approximated by Monte Carlo sampling to obtain an approximate expression for the gradient. Based on the approximate expression for the gradient, the initial graph convolutional neural network is iteratively trained to obtain the dynamic graph convolutional neural network.

[0033] The present invention also provides an environmental data fitting device based on a graph neural network, comprising:

[0034] The acquisition module is used to acquire target environment data images of the target object from multiple perspectives, and to acquire the two-dimensional coordinate set of matching feature points in the target environment data images from any two perspectives.

[0035] The weight determination module is used to input the two-dimensional coordinate set into the dynamic graph convolutional neural network to obtain the sampling weights and inlier weights output by the dynamic graph convolutional neural network.

[0036] The assumption solution module is used to sample a minimum observation set for each candidate model based on the sampling weights, and to generate multiple model hypotheses by applying the minimum observation set based on the minimum solver;

[0037] The Gaussian modeling module is used to perform Gaussian modeling on multiple model hypotheses, generate additional hypotheses, and, based on the multiple model hypotheses and the additional hypotheses, apply the interior point weights to select the optimal hypothesis for each candidate model, forming a candidate model set;

[0038] The clustering module is used to sort and cluster the candidate model set to obtain the optimal model of the target object and the clustering label.

[0039] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the environmental data fitting method based on graph neural networks as described above.

[0040] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the environmental data fitting method based on graph neural networks as described above.

[0041] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the environmental data fitting method based on graph neural networks as described above.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The environmental data fitting method and apparatus based on graph neural networks provided by this invention model spatial relationships through dynamic graph convolutional neural networks, and generate additional hypotheses through Gaussian modeling, Gaussian uncertainty quantification, and hypothesis generation. It can specifically overcome existing bottlenecks and solve the defects of existing technologies such as missing spatial relationships and insufficient hypothesis diversity. It can improve the accuracy and determination efficiency of the optimal model, which meets the application requirements of real-world scenarios for high-precision, high-robustness, and high-efficiency methods. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on the drawings described below without creative effort.

[0045] Figure 1 This is one of the flowcharts illustrating the environmental data fitting method based on graph neural networks provided by this invention;

[0046] Figure 2 This is a schematic diagram of the dynamic graph convolutional neural network structure in the environmental data fitting method based on graph neural networks provided by this invention;

[0047] Figure 3 This is the second flowchart of the environmental data fitting method based on graph neural networks provided by the present invention;

[0048] Figure 4 This is a schematic diagram of the structure of the environmental data fitting device based on graph neural network provided by the present invention;

[0049] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0050] Figure label:

[0051] 41: Acquisition module; 42: Weight determination module; 43: Hypothesis solving module; 44: Gaussian modeling module; 45: Clustering module; 810: Processor; 820: Communication interface; 830: Memory; 840: Communication bus. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0053] Currently, existing environmental data fitting methods face core challenges such as difficulty in modeling spatial relationships, an imbalance between hypothesis generation efficiency and diversity, a lack of uncertainty quantification, and a difficulty in balancing universality and specialization. Based on this, this invention provides an environmental data fitting method based on graph neural networks.

[0054] Figure 1 This is a flowchart illustrating an environmental data fitting method based on a graph neural network provided in an embodiment of the present invention, as shown below. Figure 1As shown, the method includes:

[0055] S1, acquire target environment data images of the target object from multiple perspectives, and acquire the two-dimensional coordinate set of matching feature points in the target environment data images from any two perspectives;

[0056] S2, input the two-dimensional coordinate set into the dynamic graph convolutional neural network to obtain the sampling weights and inlier weights output by the dynamic graph convolutional neural network;

[0057] S3, based on the sampling weights, sample a minimum observation set for each candidate model, and based on the minimum solver, apply the minimum observation set to generate multiple model hypotheses;

[0058] S4. Perform Gaussian modeling on the multiple model assumptions to generate additional assumptions, and based on the multiple model assumptions and the additional assumptions, apply the interior point weights to select the optimal assumption for each candidate model to form a candidate model set;

[0059] S5, sort and cluster the candidate model set to obtain the optimal model and cluster label of the target object.

[0060] Specifically, the environmental data fitting method based on graph neural networks provided in this embodiment of the invention is executed by an environmental data fitting device based on graph neural networks. This device can be configured in a computer, which can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0061] First, step S1 is executed to acquire target environment data images of the target object from multiple viewpoints. Then, a scale-invariant feature transform (SIFT) operation is performed on the target environment data images from any two viewpoints to obtain a two-dimensional coordinate set of matching feature points in the target environment data images from any two viewpoints. The dimension of the two-dimensional coordinate set can be N×1×D, where N is the total number of target environment data images from each viewpoint, and D is the total dimension of the two-dimensional coordinates of the matching feature points, which is usually 4.

[0062] Then, step S2 is executed to input the two-dimensional coordinate set into the dynamic graph convolutional neural network to obtain the sampling weights and interior point weights output by the dynamic graph convolutional neural network.

[0063] The dynamic graph convolutional neural network includes an initial feature extraction module, a local feature extraction module, a global feature extraction module, a feature fusion module, a spatial aggregation module, a sampling weight output module, and an interior point weight output module.

[0064] like Figure 2As shown, after inputting the two-dimensional coordinate set into the dynamic graph convolutional neural network, the initial feature extraction module can be used to extract the initial features of the two-dimensional coordinate set. This initial feature extraction module can consist of a 1×1 convolutional layer (i.e., CONV1×1) and a batch normalization and activation layer (i.e., BN+ReLU). The 1×1 convolutional layer can be used to extract the initial features, and the batch normalization and activation layer can be used to normalize and activate the initial features.

[0065] Subsequently, the local feature extraction module is used to extract the local features of the initial features, and the global feature extraction module is used to extract the global features of the initial features.

[0066] The local feature extraction module can be a dynamic graph (DG block), which captures the feature associations and structural information of local regions in the initial features by constructing a dynamic graph structure, thereby obtaining local features.

[0067] The global feature extraction module can include multiple graph attention (GAT) blocks. The graph attention blocks can distinguish the importance of different nodes in the initial features through a multi-head attention mechanism, thereby effectively extracting feature information in the global scope and obtaining global features.

[0068] Subsequently, a feature fusion module is used to fuse local and global features using a cross-attention mechanism, resulting in fused features. The feature fusion module can include a cross-attention layer, a residual block (ResBlock), a layer normalization layer (LayerNorm), and a linear layer. The cross-attention layer enables cross-attention between local and global features, uncovering potential correlations between them; the residual block effectively alleviates the vanishing gradient problem while preserving the original information of the features; the layer normalization layer normalizes features, stabilizing the training process; and the linear layer performs linear transformations on the features, adjusting their dimensionality and representation. Finally, the fused features are output through the linear layer.

[0069] Subsequently, the spatial aggregation module is used to apply channel attention and spatial attention mechanisms to spatially aggregate and enhance the fused features, resulting in aggregated features that provide reliable spatial feature support for the probability distribution output. The spatial aggregation module can include three pooling layers, a channel attention layer, a spatial attention layer, and a residual block at three different spatial scales.

[0070] Each pooling layer aggregates features at different spatial scales to capture multi-scale spatial information. Channel attention layers and spatial attention layers highlight features from important channels and key spatial regions, respectively. Residual blocks further ensure the complete transmission of feature information. Finally, the aggregated features are output through the residual blocks.

[0071] Finally, the sampling weight output module is used to apply aggregated features to determine the sampling weights, and the inlier weight output module is used to apply aggregated features to determine the inlier weights. Both the sampling weight output module and the inlier weight output module include a 1x1 convolutional layer (i.e., CONV1×1) and a normalization layer (i.e., Normalization).

[0072] Dynamic graph convolutional neural networks (DNNs) can model spatial relationships by constructing graph structures. By fusing local and global features, they can accurately capture the global and local dependencies of geometric data, filling the gap in structured information processing.

[0073] Next, step S3 is executed, which uses sampling weights to sample the minimum observation set for each candidate model. A candidate model refers to an optional model type, which can include various geometric models such as planes, spheres, cylinders, and cones.

[0074] By utilizing sampling weights, a minimum set of observations required to construct each candidate model can be obtained from target environment data images from various perspectives. Each minimum set of observations corresponds to a candidate model. Furthermore, using a minimum solver, multiple model hypotheses for each candidate model can be generated by combining the minimum set of observations. Each model hypothesis is a candidate model instance.

[0075] Next, step S4 is executed, where Gaussian modeling is performed on multiple model hypotheses to generate multiple additional hypotheses for each candidate model. Then, using the multiple model hypotheses and additional hypotheses, interior point weights are applied to select the optimal hypothesis for each candidate model. The optimal hypotheses of each candidate model constitute the candidate model set.

[0076] Finally, step S5 is executed to sort and cluster the candidate model set to obtain the optimal model for the target object and its cluster label. This cluster label can be determined by whether each feature point in the target environment data image from different viewpoints is an interior point of the optimal model.

[0077] like Figure 3 As shown, the environmental data fitting method based on graph neural networks provided in this embodiment of the invention mainly includes the following steps:

[0078] Acquire observation data, which includes target environment data images of the target object from multiple perspectives and a set of two-dimensional coordinates of matching feature points in target environment data images from any two perspectives;

[0079] The two-dimensional coordinate set is input into the dynamic graph convolutional neural network to obtain the sampling weights and inlier weights output by the dynamic graph convolutional neural network.

[0080] Based on the minimum solver, multiple model hypotheses are generated using the minimum observation set;

[0081] The hypothesis generation process is based on multiple model assumptions; the hypothesis generation process includes Gaussian modeling of multiple model assumptions to generate additional hypotheses.

[0082] Based on multiple model assumptions and additional assumptions, a post-processing procedure is performed to obtain the optimal model and clustering label of the target object.

[0083] The post-processing process includes:

[0084] By applying interior point weights, the optimal hypothesis is selected for each candidate model, forming a candidate model set;

[0085] The candidate model set is sorted and clustered to obtain the optimal model for the target object and the cluster label.

[0086] In addition, the results of post-processing can be fed back into the hypothesis generation process. By calculating the residuals between the selected optimal model and the true model, the hypothesis generation process can be prompted to generate better additional hypotheses.

[0087] The environmental data fitting method based on graph neural networks provided in this embodiment of the invention models spatial relationships through dynamic graph convolutional neural networks, and generates additional hypotheses through Gaussian modeling, Gaussian uncertainty quantification, and hypothesis generation. It can specifically overcome existing bottlenecks and solve the defects of existing technologies such as missing spatial relationships and insufficient hypothesis diversity. It can improve the accuracy and determination efficiency of the optimal model, which meets the application requirements of real-world scenarios for high-precision, high-robustness, and high-efficiency methods.

[0088] Based on the above embodiments, the step of performing Gaussian modeling on multiple model assumptions to generate additional assumptions includes:

[0089] Based on target environment data images from any two perspectives, calculate the first calculation error of the fundamental matrix and the second calculation error of the homography matrix. Based on the first calculation error, determine the first Jacobian matrix and based on the second calculation error, determine the second Jacobian matrix.

[0090] Based on the first Jacobian matrix, estimate the first covariance matrix of the foundation matrix, and based on the second Jacobian matrix, estimate the second covariance matrix of the homography matrix;

[0091] Based on the first covariance matrix and the second covariance matrix, Gaussian modeling is performed on multiple model assumptions to generate the additional assumptions.

[0092] Specifically, after obtaining the sampling weights and interior point weights through the dynamic graph convolutional neural network, multiple model hypotheses are obtained using a minimum solver and stored as a hypothesis set. Based on this hypothesis set, Gaussian modeling is used to expand the hypothesis by incorporating uncertainty, thereby generating more additional hypotheses.

[0093] First, using target environment data images from any two perspectives, calculate the first calculation error of the fundamental matrix and the second calculation error of the homography matrix.

[0094] The first calculation error can be the square of the Sampson error, which can be expressed by the following formula:

[0095] ;

[0096] in, For Sampson error, Let be a feature point in the target environment data image from the first of any two viewpoints. For any two viewpoints, the target environment data image in the second viewpoint Matching feature points, Based on the matrix, It is an L1 norm. It is an L2 norm.

[0097] The first calculation error can provide a first-order approximate estimate of the geometric reprojection error between matching feature points under polar geometric constraints.

[0098] During algorithm performance evaluation, the true basis matrix is ​​obtained by calculating the minimum Sampson error of each pair of matched feature points on several of the most significant estimated basis matrices. With estimation of the fundamental matrix First comparison error between:

[0099] ;

[0100] in, , For the j-th estimated fundamental matrix, for The corresponding square of the Sampson error, This is the first comparison error.

[0101] The Sampson error performance index can be obtained by calculating the average error of all points within the true value:

[0102] ;

[0103] in, Here, N1 represents the Sampson error performance metric, and N1 is the number of all feature points in the target environment data image. For Iverson (in brackets), Indicates that the condition is satisfied. of , This represents the cluster label of the i-th feature point. This indicates that the i-th feature point is an interior point of the truth value. The set of all matching feature points. This represents the first comparison error corresponding to the i-th feature point.

[0104] The Sampson error performance metric can be used to quantify the degree of geometric consistency between the true matching feature points and the estimated polar geometry of the matching feature points.

[0105] The second calculation error can be determined by the square of the transformation error, which can be calculated using the following formula:

[0106] ;

[0107] in, It is a homography matrix. for and The distance between them for through Feature points in the target environment data image obtained from the second perspective after conversion. for and The distance between them for through Feature points in the target environment data image obtained from the first perspective after conversion.

[0108] In the algorithm performance evaluation process, the true homography matrix is ​​obtained by calculating the minimum propagation error of each pair of matched feature points on several of the most salient estimated homography matrices. With Estimation of Homography Matrix The second comparison error between them:

[0109] ;

[0110] in, , For the j-th estimated homography matrix, for The square of the corresponding propagation error, This is the second comparison error.

[0111] The conversion error performance index can be obtained by calculating the average error of all points within the true values:

[0112] ;

[0113] in, For conversion error performance indicators, This represents the second contrast error corresponding to the i-th feature point.

[0114] This transformation error performance metric can be used to measure the geometric accuracy of homography matrix estimation algorithms.

[0115] Subsequently, the first Jacobian matrix is ​​determined using the first calculation error, and the second Jacobian matrix is ​​determined using the second calculation error.

[0116] The first Jacobian matrix can be represented as:

[0117] ;

[0118] in, This is the first Jacobian matrix. It is the element in the i-th row and j-th column of the basic matrix F.

[0119] The second Jacobian matrix can be represented as:

[0120] ;

[0121] in, This is the second Jacobian matrix. Let be the element in the i-th row and j-th column of the homography matrix H.

[0122] Understandably, the first Jacobian matrix can accurately capture the impact of parameter changes on the first calculation error, and the second Jacobian matrix can accurately capture the impact of parameter changes on the second calculation error, laying the foundation for subsequent uncertainty estimation.

[0123] Subsequently, the first covariance matrix of the fundamental matrix is ​​estimated using the first Jacobian matrix, and the second covariance matrix of the homography matrix is ​​estimated using the second Jacobian matrix.

[0124] For the problem of estimating the covariance matrix, the lower bound of the estimated covariance matrix is ​​the Cramér-Rao lower bound. Therefore, the first covariance matrix can be the inverse of the Fisher information matrix corresponding to the first Jacobian matrix, i.e.:

[0125] ;

[0126] in, Let be the first covariance matrix. Let be the variance of the first covariance matrix corresponding to the target environment data image from the i-th viewpoint.

[0127] The second covariance matrix can be the inverse of the Fisher information matrix corresponding to the second Jacobian matrix, i.e.:

[0128] ;

[0129] in, The second covariance matrix, Let be the variance of the second covariance matrix corresponding to the target environment data image from the i-th viewpoint.

[0130] Understandably, the first and second covariance matrices not only quantify the uncertainty of parameter estimation but also capture the correlation structure between parameters, providing richer information than single-point estimation.

[0131] Finally, after obtaining the first and second covariance matrices, we can use them to perform Gaussian modeling on multiple model hypotheses, construct a multivariate Gaussian distribution model, and generate diverse, high-quality additional hypotheses through Gaussian sampling.

[0132] In this embodiment of the invention, a multivariate Gaussian distribution model with uncertainty quantification is constructed through Gaussian modeling, which can achieve dual output of geometric parameters and confidence quantification, filling the technical gap of existing schemes that can only output single-point parameters and lack uncertainty characterization.

[0133] Based on the above embodiments, the step of performing Gaussian modeling on multiple model hypotheses based on the first covariance matrix and the second covariance matrix to generate the additional hypotheses includes:

[0134] Based on the first covariance matrix and the second covariance matrix, standard normal vectors are generated respectively;

[0135] Based on the standard normal vector, the additional assumptions of the model are modified to obtain the additional assumptions.

[0136] Specifically, in the process of generating additional hypotheses through Gaussian sampling, Cholesky decomposition can be performed on the first and second covariance matrices respectively to obtain the decomposition results of the first and second covariance matrices, namely:

[0137] ;

[0138] in, This is the decomposition result of the first covariance matrix. It is the lower triangular matrix of the first covariance matrix. It is the upper triangular matrix of the first covariance matrix.

[0139] ;

[0140] in, This is the decomposition result of the second covariance matrix. It is the lower triangular matrix of the second covariance matrix. It is the upper triangular matrix of the second covariance matrix.

[0141] Subsequently, standard normal vectors are generated using the first and second covariance matrices. The first standard normal vector corresponds to the first covariance matrix. It can be represented as: , It is a unit vector. It follows a standard normal distribution. The second standard normal vector corresponds to the second covariance matrix. It can be represented as .

[0142] Finally, using the first and second standard normal vectors, multiple model assumptions are revised to obtain additional hypotheses. This revision process is the same as the Gaussian sampling process. The first additional hypothesis corresponding to the first standard normal vector can be expressed as: The second additional hypothesis corresponding to the second standard normal vector can be expressed as: .

[0143] Therefore, the resulting general hypothesis can be expressed as:

[0144] ;

[0145] in, Here, A represents the general hypothesis and A represents the model hypothesis.

[0146] The sampling implemented using standard normal vectors in this embodiment of the invention, compared with traditional random sampling, can adaptively explore according to the estimated uncertainty, sample more samples in high uncertainty regions, maintain the correlation structure between geometric parameters, generate additional hypotheses that better conform to geometric constraints, conduct more effective exploration in the parameter space, reduce the generation of invalid hypotheses, and provide a more reliable foundation for subsequent model selection and clustering.

[0147] Based on the above embodiments, the dynamic graph convolutional neural network is trained using the following steps:

[0148] Acquire training samples of environmental data images of sample objects from multiple perspectives, and acquire a set of two-dimensional coordinates of matching key points in the training samples of environmental data images from any two perspectives, as well as ground truth class labels of matching key points in the training samples of environmental data images from any two perspectives.

[0149] The two-dimensional coordinate set samples are input into the initial graph convolutional neural network to obtain the output result of the initial graph convolutional neural network;

[0150] Based on the output of the initial graph convolutional neural network, the optimal model and cluster label of the sample object are determined, and the task loss is calculated based on the ground truth class label and the cluster label of the sample object. Based on the expectation of the task loss, a loss function is constructed.

[0151] The deterministic hypothesis is converted into a probabilistic form, and based on the probabilistic form, the gradient of the loss function with respect to the network parameters of the initial graph convolutional neural network is calculated;

[0152] The gradient is approximated by Monte Carlo sampling to obtain an approximate expression for the gradient. Based on the approximate expression for the gradient, the initial graph convolutional neural network is iteratively trained to obtain the dynamic graph convolutional neural network.

[0153] Specifically, in the process of training the dynamic graph convolutional neural network, we can first obtain training samples of environmental data images of the sample object from multiple perspectives, and obtain a set of two-dimensional coordinates of matching key points in the training samples of environmental data images from any two perspectives, as well as ground truth class labels of matching key points in the training samples of environmental data images from any two perspectives.

[0154] Subsequently, the two-dimensional coordinate set samples are input into the initial graph convolutional neural network to obtain the output of the initial graph convolutional neural network.

[0155] Using the output of the initial graph convolutional neural network, the optimal model and cluster label of the sample objects are determined. The task loss is calculated using the ground truth class label and the cluster label of the sample objects. Finally, the loss function is constructed using the expectation of the task loss.

[0156] The loss function is the expected value that minimizes the task loss, and can be expressed as:

[0157] ;

[0158] in, For loss function, These are the network parameters for the initial graph convolutional neural network. Represents an estimated model instance. Represents a set of target environment data images. Represents a probability distribution. express obey . For mission losses, Indicates to The expected value of the task loss is calculated. This loss function measures the quality of the estimated model instances; a smaller value indicates a better model fit.

[0159] Subsequently, the deterministic hypothesis is converted into a probabilistic form, and based on the probabilistic form, the gradient of the loss function with respect to the network parameters of the initial graph convolutional neural network is calculated;

[0160] To enable end-to-end training of the initial graph convolutional neural network, make the initial graph convolutional neural network differentiable, and enable gradient backpropagation, the gradient is approximated by Monte Carlo sampling to obtain an approximate expression for the gradient, thereby enabling the calculation of the gradient of the loss function with respect to the network parameters.

[0161] For the j-th candidate model It follows a probability distribution , Assumptions for the j-th model: .

[0162] Based on this, from Choose hypothesis Using assumptions The approximate expression is modeled as a softmax function with a weighted number of interior points:

[0163] ;

[0164] in, It is the probability form of the output of the softmax function. It is a scaling factor that controls the ability to classify and discriminate distributions. Indicates a hypothesis The weighted number of interior points, Indicates a hypothesis The weighted number of interior points. This modeling approach enables the initial graph convolutional neural network to learn the hypothesis selection strategy in a differentiable manner.

[0165] Based on the j-th candidate model mentioned above The probability model will use the sampling hypothesis and candidate model instances joint probability Represented as:

[0166] ;

[0167] in, This represents the number of model assumptions in the sampling hypothesis. and These represent the probability distributions corresponding to M and H, respectively. This joint probability form provides the theoretical basis for subsequent gradient calculation. Based on the fundamental principles of the policy gradient method, the gradient of the loss function with respect to the network parameters can be expressed as:

[0168] ;

[0169] in, The gradient of the loss function with respect to the network parameters. Indicates to The expected loss of the task.

[0170] Since directly calculating the expectation is difficult to implement in practice, this gradient can be approximated using Monte Carlo sampling. Specifically, The probability distribution that follows is That is, .

[0171] Then, from Medium sampling Candidate model instances Each candidate model instance The probability distribution that follows is That is, .

[0172] Therefore, the approximate expression for the gradient is:

[0173] ;

[0174] in, , It is a set of cluster labels. Let be the set of truth class labels, and ME be the formula for calculating the misclassification rate, and we have:

[0175] ;

[0176] in, yes The i-th cluster label in the data. yes The i-th truth class label in the data. The brackets represent Iverson.

[0177] The misclassification rate represents the proportion of data points that are misclassified, and it is a standard evaluation criterion for estimation problems such as the fundamental matrix and homography matrix. A lower misclassification rate indicates better clustering performance.

[0178] In this embodiment of the invention, the loss function based on the misclassification rate can directly reflect the accuracy of the model's segmentation. In summary, a key characteristic of this optimization framework is the task loss. The absence of differentiability provides flexibility in the choice of loss function. Through the policy gradient method, even if the task loss itself is not differentiable, its gradient can still be effectively propagated back to the network parameters, enabling end-to-end training. This design allows the framework to adapt to a variety of different geometric estimation tasks and evaluation metrics.

[0179] Then, using an approximate expression of the gradient, the initial graph convolutional neural network is iteratively trained until the preset number of iterations or the loss function converges, thus obtaining a dynamic graph convolutional neural network.

[0180] like Figure 4 As shown, based on the above embodiments, this embodiment of the invention provides an environmental data fitting device based on a graph neural network, comprising:

[0181] The acquisition module 41 is used to acquire target environment data images of the target object from multiple perspectives, and to acquire a set of two-dimensional coordinates of matching feature points in the target environment data images from any two perspectives.

[0182] The weight determination module 42 is used to input the two-dimensional coordinate set into the dynamic graph convolutional neural network to obtain the sampling weights and inlier weights output by the dynamic graph convolutional neural network.

[0183] Assume that the solution module 43 is used to sample a minimum observation set for each candidate model based on the sampling weights, and to generate multiple model hypotheses by applying the minimum observation set based on the minimum solver;

[0184] The Gaussian modeling module 44 is used to perform Gaussian modeling on multiple model hypotheses, generate additional hypotheses, and, based on the multiple model hypotheses and the additional hypotheses, apply the interior point weights to select the optimal hypothesis for each candidate model, forming a candidate model set;

[0185] Clustering module 45 is used to sort and cluster the candidate model set to obtain the optimal model of the target object and the cluster label.

[0186] Specifically, the functions of each module in the graph neural network-based environmental data fitting device provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0187] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the environmental data fitting method based on graph neural networks provided in the above embodiments.

[0188] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0189] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the environmental data fitting method based on graph neural networks provided in the above embodiments.

[0190] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the environment data fitting method based on graph neural networks provided in the above embodiments.

[0191] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0192] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for fitting environmental data based on graph neural networks, characterized in that, include: Acquire target environment data images of the target object from multiple perspectives, and obtain the two-dimensional coordinate set of matching feature points in the target environment data images from any two perspectives; The two-dimensional coordinate set is input into a dynamic graph convolutional neural network to obtain the sampling weights and inlier weights output by the dynamic graph convolutional neural network. Based on the sampling weights, a minimum observation set is sampled for each candidate model, and multiple model hypotheses are generated using the minimum observation set based on the minimum solver. Gaussian modeling is performed on multiple model assumptions to generate additional assumptions. Based on the multiple model assumptions and the additional assumptions, the interior point weights are applied to select the optimal assumption for each candidate model, forming a candidate model set. The candidate model set is sorted and clustered to obtain the optimal model and cluster label of the target object; The dynamic graph convolutional neural network includes an initial feature extraction module, a local feature extraction module, a global feature extraction module, a feature fusion module, a spatial aggregation module, a sampling weight output module, and an interior point weight output module. The step of inputting the two-dimensional coordinate set into a dynamic graph convolutional neural network to obtain the sampling weights and inlier weights output by the dynamic graph convolutional neural network includes: Based on the initial feature extraction module, the initial features of the two-dimensional coordinate set are extracted; Based on the local feature extraction module, local features of the initial feature are extracted, and based on the global feature extraction module, global features of the initial feature are extracted. Based on the feature fusion module, a cross-attention mechanism is applied to fuse the local features and the global features to obtain fused features; Based on the spatial aggregation module, channel attention mechanism and spatial attention mechanism are applied to spatially aggregate the fused features to obtain aggregated features; Based on the sampling weight output module, the aggregation feature is applied to determine the sampling weight, and based on the inlier weight output module, the aggregation feature is applied to determine the inlier weight; The step of performing Gaussian modeling on multiple model assumptions to generate additional assumptions includes: Based on target environment data images from any two perspectives, calculate the first calculation error of the fundamental matrix and the second calculation error of the homography matrix. Based on the first calculation error, determine the first Jacobian matrix and based on the second calculation error, determine the second Jacobian matrix. Based on the first Jacobian matrix, estimate the first covariance matrix of the foundation matrix, and based on the second Jacobian matrix, estimate the second covariance matrix of the homography matrix; Based on the first covariance matrix and the second covariance matrix, Gaussian modeling is performed on multiple model assumptions to generate the additional assumptions; The additional hypotheses are generated by performing Gaussian modeling on multiple model assumptions based on the first covariance matrix and the second covariance matrix, including: Based on the first covariance matrix and the second covariance matrix, standard normal vectors are generated respectively; Based on the standard normal vector, the additional assumptions of the model are modified to obtain the additional assumptions.

2. The environmental data fitting method based on graph neural networks according to claim 1, characterized in that, The global feature extraction module includes multiple graph attention modules; Each of the graph attention modules is used to extract the global features through a multi-head attention mechanism.

3. The environmental data fitting method based on graph neural networks according to any one of claims 1-2, characterized in that, The dynamic graph convolutional neural network is trained based on the following steps: Acquire training samples of environmental data images of sample objects from multiple perspectives, and acquire a set of two-dimensional coordinates of matching key points in the training samples of environmental data images from any two perspectives, as well as ground truth class labels of matching key points in the training samples of environmental data images from any two perspectives. The two-dimensional coordinate set samples are input into the initial graph convolutional neural network to obtain the output result of the initial graph convolutional neural network; Based on the output of the initial graph convolutional neural network, the optimal model and cluster label of the sample object are determined, and the task loss is calculated based on the ground truth class label and the cluster label of the sample object. Based on the expectation of the task loss, a loss function is constructed. The deterministic hypothesis is converted into a probabilistic form, and based on the probabilistic form, the gradient of the loss function with respect to the network parameters of the initial graph convolutional neural network is calculated; The gradient is approximated by Monte Carlo sampling to obtain an approximate expression for the gradient. Based on the approximate expression for the gradient, the initial graph convolutional neural network is iteratively trained to obtain the dynamic graph convolutional neural network.

4. An environmental data fitting device based on a graph neural network, characterized in that, include: The acquisition module is used to acquire target environment data images of the target object from multiple perspectives, and to acquire the two-dimensional coordinate set of matching feature points in the target environment data images from any two perspectives. The weight determination module is used to input the two-dimensional coordinate set into the dynamic graph convolutional neural network to obtain the sampling weights and inlier weights output by the dynamic graph convolutional neural network. The assumption solution module is used to sample a minimum observation set for each candidate model based on the sampling weights, and to generate multiple model hypotheses by applying the minimum observation set based on the minimum solver; The Gaussian modeling module is used to perform Gaussian modeling on multiple model hypotheses, generate additional hypotheses, and, based on the multiple model hypotheses and the additional hypotheses, apply the interior point weights to select the optimal hypothesis for each candidate model, forming a candidate model set; The clustering module is used to sort and cluster the candidate model set to obtain the optimal model and clustering label of the target object; The dynamic graph convolutional neural network includes an initial feature extraction module, a local feature extraction module, a global feature extraction module, a feature fusion module, a spatial aggregation module, a sampling weight output module, and an interior point weight output module. The step of inputting the two-dimensional coordinate set into a dynamic graph convolutional neural network to obtain the sampling weights and inlier weights output by the dynamic graph convolutional neural network includes: Based on the initial feature extraction module, the initial features of the two-dimensional coordinate set are extracted; Based on the local feature extraction module, local features of the initial feature are extracted, and based on the global feature extraction module, global features of the initial feature are extracted. Based on the feature fusion module, a cross-attention mechanism is applied to fuse the local features and the global features to obtain fused features; Based on the spatial aggregation module, channel attention mechanism and spatial attention mechanism are applied to spatially aggregate the fused features to obtain aggregated features; Based on the sampling weight output module, the aggregation feature is applied to determine the sampling weight, and based on the inlier weight output module, the aggregation feature is applied to determine the inlier weight; The step of performing Gaussian modeling on multiple model assumptions to generate additional assumptions includes: Based on target environment data images from any two perspectives, calculate the first calculation error of the fundamental matrix and the second calculation error of the homography matrix. Based on the first calculation error, determine the first Jacobian matrix and based on the second calculation error, determine the second Jacobian matrix. Based on the first Jacobian matrix, estimate the first covariance matrix of the foundation matrix, and based on the second Jacobian matrix, estimate the second covariance matrix of the homography matrix; Based on the first covariance matrix and the second covariance matrix, Gaussian modeling is performed on multiple model assumptions to generate the additional assumptions; The additional hypotheses are generated by performing Gaussian modeling on multiple model assumptions based on the first covariance matrix and the second covariance matrix, including: Based on the first covariance matrix and the second covariance matrix, standard normal vectors are generated respectively; Based on the standard normal vector, the additional assumptions of the model are modified to obtain the additional assumptions.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the environmental data fitting method based on graph neural networks as described in any one of claims 1-3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the environmental data fitting method based on graph neural networks as described in any one of claims 1-3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the environmental data fitting method based on graph neural networks as described in any one of claims 1-3.

Images