A physical constraint geometry reconstruction method and system based on a twin system

Abstract

The application belongs to the technical field of computer three-dimensional reconstruction, and relates to a physical constraint geometry reconstruction method and system based on a twin system, which acquires multi-view images and camera poses of a target device, generates a matching feature set through feature matching, and obtains an initial three-dimensional grid model through a deep neural network after extracting spatial coordinate information. Explicit geometric constraints and their design tolerance bands are read from a digital twin system, and a soft interval loss function is used to generate a tolerance band perception loss value. Running condition data fed back by the digital twin system is input into a weight adjustment network, and dynamic weight coefficients of each constraint are output. The loss value is weighted and fused, and a visual loss is combined to form an optimization target. The gradient is updated to update the grid vertex, and the optimized model is written back to the digital twin system to update the geometric layer and the constraint verification layer. The application combines physical constraints and working condition data to realize engineering semantic level high-precision reconstruction and form a closed-loop updating mechanism.

Description

Technical Field

[0001] This invention belongs to the field of computer 3D reconstruction technology, and is a physical constraint geometric reconstruction method and system based on twin systems. Background Technology

[0002] 3D geometric reconstruction technology is used to recover the 3D surface model of an object from data such as 2D images, depth maps, or point clouds. It has wide applications in reverse engineering, industrial inspection, and digital twins. As a dynamic virtual mapping of a physical entity, one of the basic components of a digital twin is a 3D model that can reflect the geometric shape of the physical entity. The accuracy of the model directly affects the fidelity of the twin system.

[0003] Existing 3D reconstruction schemes based on multi-view images typically employ steps such as feature detection and matching, motion reconstruction, and multi-view stereo matching to generate 3D mesh models. However, existing technologies still suffer from the following technical limitations: First, the reconstruction process relies solely on the visual loss function and does not incorporate the geometric constraints explicitly defined in industrial equipment design. This results in reconstructed models that are similar in macroscopic shape, but the deviations in key geometric features exceed the design tolerance range.

[0004] Second, even if some methods attempt to incorporate geometric constraints as hard constraints into the optimization, they do not consider the allowable deviation within the design tolerance zone, i.e., they force the deviation to be zero. This is inconsistent with the engineering semantics of acceptable deviation within the tolerance zone in actual manufacturing, and it is difficult to be compatible with asymmetric tolerances or floating tolerance zones.

[0005] Third, existing methods do not utilize the real-time operating data stored in the digital twin system to dynamically adjust the tolerance range or loss weight of constraints, and there is a lack of a closed-loop update mechanism between the reconstruction results and the twin system, that is, the optimized mesh model cannot be automatically written back to the twin system to update its geometric model layer. Summary of the Invention

[0006] In view of this, in order to solve the problems mentioned in the background technology, a physical constraint geometry reconstruction method and system based on twin systems is proposed.

[0007] The objective of this invention can be achieved through the following technical solution: The first embodiment of this invention provides a physical constraint geometry reconstruction method based on a twin system, including: performing feature matching between multi-view image data of the target device and camera pose parameters to generate a matching feature set.

[0008] Spatial coordinate information is extracted from the matching feature set and deep neural network inference is performed to generate an initial 3D mesh model.

[0009] Read the design data of the target device from the digital twin system, and obtain at least the explicit geometric constraint parameters including parallelism, perpendicularity or concentricity, as well as the design tolerance zone corresponding to each explicit geometric constraint parameter.

[0010] Calculate the geometric deviation between the initial 3D mesh model and each explicit geometric constraint parameter, map the geometric deviation to the soft interval loss function according to the design tolerance zone, and generate the tolerance zone perceived loss value for each explicit geometric constraint parameter.

[0011] The operating condition data fed back from the digital twin system is input into the weight adjustment network, and the dynamic weight coefficient of each explicit geometric constraint parameter is output.

[0012] The tolerance band perceptual loss value and dynamic weight coefficient are weighted and fused together, and combined with the visual loss function to form a joint optimization objective. The gradient is backpropagated to the deep neural network to update the vertex positions of the initial 3D mesh model and generate the optimized 3D mesh model.

[0013] The optimized 3D mesh model is written back to the digital twin system to update the geometry layer and constraint verification layer of the digital twin system.

[0014] The second embodiment of the present invention provides a physical constraint geometry reconstruction system based on a twin system, including: a feature matching module, an initial model establishment module, a design constraint reading module, a tolerance-aware loss module, a dynamic weight generation module, a joint optimization module, and a model write-back module.

[0015] The feature matching module is connected to the initial model building module, the initial model building module is connected to the design constraint reading module, the design constraint reading module is connected to the tolerance-aware loss module, the tolerance-aware loss module is connected to the dynamic weight generation module, the dynamic weight generation module is connected to the joint optimization module, and the joint optimization module is connected to the model write-back module.

[0016] The feature matching module performs feature matching between the multi-view image data of the target device and the camera pose parameters to generate a matching feature set.

[0017] The initial model building module extracts spatial coordinate information from the matching feature set and performs deep neural network inference to generate an initial 3D mesh model.

[0018] The design constraint reading module reads the design data of the target device from the digital twin system, and obtains explicit geometric constraint parameters, including at least parallelism, perpendicularity or concentricity, as well as the design tolerance zone corresponding to each explicit geometric constraint parameter.

[0019] The tolerance-aware loss module calculates the geometric deviation between the initial 3D mesh model and each explicit geometric constraint parameter. Based on the design tolerance zone, it maps the geometric deviation to the soft interval loss function, generating the tolerance zone-aware loss value for each explicit geometric constraint parameter.

[0020] The dynamic weight generation module inputs the operating condition data fed back from the digital twin system into the weight adjustment network and outputs the dynamic weight coefficient of each explicit geometric constraint parameter.

[0021] The joint optimization module weights and fuses the tolerance band perceptual loss value with the dynamic weight coefficients, and combines them with the visual loss function to form a joint optimization objective. The gradient is then fed back to the deep neural network to update the vertex positions of the initial 3D mesh model and generate the optimized 3D mesh model.

[0022] The model write-back module writes the optimized 3D mesh model back to the digital twin system, updating the geometry layer and constraint verification layer of the digital twin system.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention reads explicit geometric constraints such as parallelism, perpendicularity, and concentricity from the digital twin system and their corresponding design tolerance bands, and integrates the geometric constraints into the training and vertex fine-tuning process of the deep learning network in the form of a differentiable loss function. This makes the reconstructed three-dimensional mesh model not only visually faithful, but also its key geometric features automatically fall within the allowable range specified by the design tolerance, avoiding the problem of manual correction or rework after reconstruction by traditional methods.

[0024] (2) The present invention sets a piecewise linear soft interval loss: the loss is zero when the actual geometric deviation is within the design tolerance zone, and a penalty proportional to the amount of deviation is generated only when the deviation exceeds the upper or lower tolerance limit. Compared with the existing hard constraint method, this scheme respects the allowable tolerance range in the part processing, and will not cause distortion in other areas of the model due to excessive constraints. At the same time, it can adapt to asymmetric tolerances, so that the reconstruction result more realistically reflects the actual manufacturing state of the part.

[0025] (3) The present invention accesses the working condition data fed back by the twin system, and adjusts the loss weight of each geometric constraint in real time through the weight adjustment network or mapping function, automatically relaxes the penalty intensity of the corresponding constraint, thereby generating a three-dimensional mesh model that is more in line with the current working condition, and solves the problem that the static reconstruction model cannot reflect the dynamic behavior changes of physical entities.

[0026] (4) After completing the constraint fine-tuning, the present invention writes the three-dimensional mesh model that meets the engineering semantics and its associated geometric feature parameters back to the digital twin system, updates its geometric layer and constraint information database, so that the twin system can continuously obtain the latest geometric model verified by physical rules, providing a high-fidelity geometric basis for subsequent assembly simulation, tolerance analysis, finite element simulation or condition monitoring, while avoiding the cumulative error caused by repeated reconstruction. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of the method steps of the present invention; Figure 2 This is a schematic diagram of the module connection of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Please see Figure 1 As shown, the first embodiment of the present invention provides a method for physical constraint geometry reconstruction based on a twin system, the specific steps of which are as follows: S1. Perform feature matching between the multi-view image data of the target device and the camera pose parameters to generate a matching feature set.

[0031] In this embodiment, the implementation process of step S1 is as follows: S11. Key point extraction and descriptor construction are performed on multi-view image data using feature detection operators. Specifically, scale-invariant feature transformation operators are used to perform scale space extremum detection on multi-view image data, and a Gaussian difference pyramid is constructed. Local gray-level extrema points are detected in each layer as candidate key points.

[0032] For each candidate extremum point, sub-pixel-level position refinement is performed using a second-order Taylor expansion of the scale space function: a three-dimensional offset vector is constructed. , in, Represent the horizontal coordinate (x), vertical coordinate (y), and scale factor of the candidate extreme points, respectively. The sub-pixel level correction amount, the construction of the three-dimensional offset vector can be obtained by the following formula: ; Where D represents the Gaussian difference function, Represents a three-dimensional gradient vector. This represents a 3×3 Hessian matrix.

[0033] If the absolute value of any component of the 3D offset vector in any dimension exceeds half of the unit sampling interval, the candidate extreme point is moved to the corrected integer coordinate position, and sub-pixel refinement iteration is performed again at the new position until the offset of all components is no greater than half of the unit sampling interval or the preset maximum number of iterations is reached.

[0034] After location refinement, low-contrast points and edge response points are removed from the candidate extrema points to form the final key point set. The low-contrast condition is as follows: if the absolute value of the difference of Gaussians function at the candidate extrema point after sub-pixel refinement is less than the preset low-contrast threshold, then the point is determined to be a low-contrast point and is removed. The low-contrast threshold is adaptively set according to the image quantization bit depth or noise variance, and can be exemplarily set to 0.03.

[0035] Edge response condition: Calculate the two principal curvature eigenvalues ​​of the Hessian matrix at the candidate extreme point. If the ratio of the larger eigenvalue to the smaller eigenvalue is greater than the preset edge response threshold, then the point is determined to be an edge response point and is removed. The edge response threshold is pre-configured according to the required corner point specificity, and can be exemplarily set to 10.

[0036] Centered on each key point, a neighborhood window is divided according to a preset radius. The neighborhood window is further divided into multiple sub-regions. The gradient direction histogram of each sub-region is calculated and concatenated to form a 128-dimensional vector, which serves as the descriptor for each key point.

[0037] For key points in adjacent viewpoint image pairs, a high-dimensional index structure is built for the descriptor of each key point using the Fast Approximate Nearest Neighbor Search Library (FLANN) algorithm. The index structure automatically selects a random KD tree or a hierarchical clustering tree based on the distribution of the descriptors. The index structure is used to search for the nearest neighbor descriptor and the second nearest neighbor descriptor of any key point in another image. During the search, the balance between approximate accuracy and speed is controlled by a preset number of check nodes.

[0038] The ratio of the nearest neighbor distance to the second nearest neighbor distance is calculated. When the ratio is less than a preset ratio threshold, the current nearest neighbor match is determined to be a reliable match, and an initial matching set is generated, thereby establishing feature association between image pairs with adjacent viewpoints. The ratio threshold is calibrated through offline experiments. The distribution of the ratio of the nearest neighbor to the second nearest neighbor distance of positive and negative samples is statistically analyzed on a training set with real matching labels, and the quantile that maximizes the F1 score is selected as the preset value.

[0039] S12. Combining the extrinsic parameter matrix provided by the camera pose parameters, redundant and erroneous feature point pairs are eliminated through epipolar geometric constraints.

[0040] Specifically, the essential matrix is ​​calculated based on the extrinsic parameter matrix provided by the camera pose parameters; based on the essential matrix, the epipolar constraint residuals of each pair of feature points in the initial matching set are calculated, and the residuals are defined using the symmetric epipolar distance or the Sampson error criterion; all residual values ​​are sorted in ascending order, and the median of the sorted sequence is taken as the adaptive threshold; matching point pairs with residual values ​​less than or equal to the adaptive threshold are determined as inliers and retained as feature point pairs that pass the geometric consistency check; matching point pairs with residual values ​​greater than the adaptive threshold are discarded as outliers.

[0041] S13. Triangulate the feature point pairs that pass the consistency check: Based on the camera projection matrix corresponding to the multi-view image, construct an overdetermined linear equation system for each pair of matching points, use singular value decomposition to obtain the least squares solution, and use the solution as the spatial coordinates of the three-dimensional spatial point after perspective division normalization.

[0042] Perform triangulation geometric quality checks on candidate 3D points, including at least one or a combination of the following conditions: I. Ray Angle Test: Calculate the angle between two rays pointing from the candidate 3D point to the optical centers of the two cameras respectively, obtain the distribution of ray angles of all candidate 3D points, and determine the points whose angles are located at the two ends of the distribution as low-quality points and remove them. The two ends refer to the angles being lower than the 5th percentile or higher than the 95th percentile.

[0043] II. Disparity Test: Calculate the Euclidean distance between the projected pixels of the candidate 3D point on the two images as the disparity value. Extract the focal length and baseline length from the camera calibration parameters, calculate their product, and further divide it by the maximum working distance of the target device defined in the digital twin system. The resulting quotient is used as the effective disparity lower limit. If the disparity value corresponding to a candidate 3D point is less than the effective disparity lower limit, the candidate 3D point is determined to have unreliable reconstruction quality due to the baseline being too short or the distance being too far, and is therefore discarded.

[0044] The process of obtaining the normal vector of a 3D spatial point is as follows: For the target 3D spatial point whose normal vector is to be calculated, in the generated sparse 3D point cloud, the neighborhood is defined with the target point as the center and a radius of twice the average spacing of the point cloud is set. Points in the neighborhood are collected to construct the neighborhood point set. The average value of the spatial coordinates of all points in the neighborhood point set is calculated to obtain the geometric center point. Then, the coordinates of the geometric center point are subtracted from the spatial coordinates of each point in the neighborhood point set to obtain the offset vector of each point relative to the center point.

[0045] Based on the centered offset vector, a 3x3 covariance matrix is ​​constructed. Each element of the covariance matrix is ​​obtained by calculating the average of the products of the offset vectors in different coordinate axis directions. Specifically, the diagonal elements of the matrix represent the variance of the offset vector in the horizontal, vertical, and center directions, respectively, while the off-diagonal elements represent the covariance between different coordinate axis directions.

[0046] The covariance matrix is ​​decomposed into eigenvalues ​​to obtain three eigenvalues ​​and their corresponding eigenvectors. The eigenvector corresponding to the smallest eigenvalue is taken as the initial normal vector, and the initial normal vector is adjusted for directional consistency: the average vector of the target 3D space point pointing to the optical center of each camera is calculated, and the angle between the initial normal vector and the average vector is compared. If the angle between the two is greater than 90 degrees, the initial normal vector is reversed so that the final normal vector points roughly to one side of the optical center of the camera, thereby ensuring that the normal vector faces the outside of the object or the visible surface.

[0047] The output normal vector after orientation adjustment is used as the final normal vector of the target 3D space point.

[0048] The process of obtaining semantic labels for 3D spatial points is as follows: by projecting 3D spatial points back into the CAD model coordinate system, the nearest entity surface is found, and the predefined part name of this entity is matched as the semantic label.

[0049] Each 3D point is assigned a normal vector and a semantic label to form a sparse 3D point cloud.

[0050] S14. Perform multi-view aggregation on the sparse 3D point cloud, merge 3D spatial points from all image pairs, cluster multiple points whose spatial distance is less than a preset aggregation distance threshold, and take the average of the coordinates of all points in the cluster after merging; take the weighted average of all normal vectors in the cluster, with the weight being the observation number of the corresponding point; take the label with the highest frequency in the cluster; organize the merged 3D spatial points according to a preset data structure, with each data entry containing a 3D coordinate field, a normal vector field, and a semantic label field to form a matching feature set.

[0051] S2. Extract spatial coordinate information from the matching feature set and perform deep neural network inference to generate an initial 3D mesh model.

[0052] In this embodiment, step S2 is implemented as follows: S21. Extract the spatial coordinate information of the sparse 3D point cloud from the matching feature set, denoted as the point set. Where N is the number of points in the point cloud, and the point set... Each point in the graph is treated as a node in the graph. For each node... By calculating the Euclidean distance between a node and all other points, the K nearest nodes are selected as its neighbors (K is preset as a positive integer based on the point cloud density), and then... With each neighbor node Add an undirected edge between them to construct the graph structure; S22. Construct a multi-layer graph convolutional neural network, consisting of an input layer, several graph convolutional layers, a fully connected layer, and an output layer; the aggregation function of each graph convolutional layer is defined as: ; in, For the first Layer nodes eigenvectors, For nodes The set of neighboring nodes, and The weight matrix is ​​a learnable matrix. In the form of a nonlinear function (e.g.) ); Initialize node features Set as the node's own 3D coordinates The combination of its optional attributes (such as normal vector and color value) requires the three-dimensional coordinates and selected attribute features to be standardized with zero mean and unit variance before combination, so as to eliminate the influence of dimensional differences on network feature extraction; after multiple layers of information transmission, the high-dimensional feature vector of each node is output.

[0053] S23. Based on the feature space output by the encoder, construct an implicit surface decoder; the decoder uses any query point in the space. Given the input, predict its directed distance function value in the following manner: First, find the match in the point cloud. The feature vectors of the nearest nodes are aggregated by interpolation to obtain the location-related feature embeddings.

[0054] Then, the features are embedded into the multilayer perceptron, and after layer-by-layer linear transformation and nonlinear activation, a scalar value is finally output. The sign of the scalar value indicates whether the query point is inside the object (negative value) or outside the object (positive value), and the absolute value indicates the shortest distance from the query point to the surface of the object.

[0055] The loss function during network training includes: SDF ground truth supervision of query points near the input point cloud, Eikonal regularization (ensuring gradient magnitude is 1), and unsupervised normal consistency loss, which jointly optimizes the parameters of the entire network through backpropagation.

[0056] S24. After completing implicit surface regression, determine a spatial bounding box containing all sparse 3D point clouds, and divide the bounding box into sections in the X, Y, and Z directions respectively. Each voxel unit (with preset resolution in each direction); for each voxel's eight vertices, a trained implicit surface decoder is invoked to calculate the directed distance function value at that vertex; then, the moving cube algorithm is used: traversing each voxel, based on the positive and negative combinations of the directed distance values ​​of its eight vertices, the number of triangles to be generated inside the voxel and the vertex offsets are determined from a predefined triangulation lookup table; within each voxel, the precise positions of the triangle vertices are calculated through linear interpolation, the triangle fragments generated by all voxels are summarized, and duplicate vertices and edges are removed, finally resulting in a closed triangular mesh model; this model includes a vertex index list (each vertex is identified by its sequence number in the vertex array), triangle patch connections (each patch consists of three vertex indices), and the normal vector distribution of each vertex calculated by the difference method or the average neighborhood surface normal; this model is output as the initial 3D mesh model.

[0057] S3. Read the design data of the target device from the digital twin system, and obtain at least the explicit geometric constraint parameters including parallelism, perpendicularity or concentricity, as well as the design tolerance zone corresponding to each explicit geometric constraint parameter.

[0058] In this embodiment, step S3 includes: reading the solid geometric features in the design data and identifying basic geometric elements including at least a plane, a cylindrical surface, and a central axis.

[0059] Based on the geometric consistency conditions defined in the design criterion, the basic geometric elements identified are traversed. If the normal vectors of two planes satisfy the collinearity condition, which requires the dot product of the two normal vectors to be 1, then it is determined that there is a parallelism constraint.

[0060] If the normal vector of one of the planes satisfies the orthogonality condition with the direction vector of an axis, and the orthogonality condition requires that the dot product of the plane normal vector and the axis direction vector be 0, then it is determined that there is a perpendicularity constraint.

[0061] If the central axes of two cylindrical surfaces coincide in space, and the coincidence condition requires that the shortest spatial distance between the two axes is 0, then it is determined that there is a concentricity constraint.

[0062] Read the part tolerance table associated with the geometric features of the entity. The part tolerance table is derived from the engineering drawings and is used to specify the allowable range of variation in geometric dimensions and shape during the manufacturing process. Directly obtain the design tolerance zone value of each determined constraint from the part tolerance table, including the lower and upper tolerance limits, which are used as the tolerance zone parameters corresponding to parallelism, perpendicularity, and concentricity constraints, respectively.

[0063] Among them, the tolerance zone parameter for parallelism is defined in angular units, specifying the maximum permissible deviation of the angle between the normal vectors of two planes; the tolerance zone parameter for perpendicularity is defined in angular units, specifying the maximum permissible deviation of related elements from a 90-degree angle; and the tolerance zone parameter for concentricity is defined in length units, specifying the maximum permissible distance deviation between two axes.

[0064] The explicit geometric constraint parameters and design tolerance zones obtained from the analysis are used to establish an index mapping relationship with the corresponding vertex sets in the initial 3D mesh model.

[0065] It should be noted that the collinearity, orthogonality, and coincidence conditions mentioned above all have approximate tolerances in actual operation. The tolerance values ​​are determined by the coordinate accuracy of the model. As an example, the collinearity and orthogonality conditions require that the deviation between the calculated dot product and their respective ideal planning values ​​does not exceed [a certain value]. The coincidence condition requires that the shortest spatial distance between the two axes, relative to the ideal value, does not exceed one ten-thousandth of the average radius of the two cylindrical surfaces.

[0066] S4. Calculate the geometric deviation between the initial 3D mesh model and each explicit geometric constraint parameter, map the geometric deviation to the soft interval loss function according to the design tolerance zone, and generate the tolerance zone perceived loss value for each explicit geometric constraint parameter.

[0067] Step S4, which calculates the geometric deviation between the initial 3D mesh model and each explicit geometric constraint parameter, includes: Parallelism constraint: Perform least-squares plane fitting for associated vertices and calculate the angular deviation between the unit normal vectors of the fitted plane.

[0068] Verticality constraint: Calculate the orthogonal projection components of the constrained surface and the reference axis in the initial 3D mesh model to obtain the residual value that deviates from the theoretical state by 90 degrees.

[0069] Concentricity constraint: Perform least-squares cylindrical fitting on the vertices belonging to the body of revolution in the initial 3D mesh model, and calculate the spatial position deviation between the fitted axis and the given reference axis of the digital twin system.

[0070] Normalization is performed on various deviations. Specifically, for each explicit geometric constraint, the upper limit of its corresponding design tolerance zone is used as the normalization benchmark. The values ​​of various deviations are compared with the upper limit of their corresponding design tolerance zones to obtain the normalized deviation values. The normalized deviation values ​​of all constraints are then combined into a multidimensional geometric deviation vector in a preset order.

[0071] The step S4, which generates the tolerance band perceived loss value for each explicit geometric constraint parameter, includes: setting the soft interval loss function to a piecewise linear function, wherein when the geometric deviation is less than the lower limit of the design tolerance band, the output non-zero gradient signal is equal to the lower limit of the design tolerance band minus the geometric deviation; when the geometric deviation is greater than the upper limit of the design tolerance band, the output non-zero gradient signal is equal to the geometric deviation minus the upper limit of the design tolerance band; and when the geometric deviation is within the interval defined by the upper and lower limits of the design tolerance band, zero-value loss is output.

[0072] The non-zero gradient signal or zero-value loss output is used as the tolerance band sensing loss value.

[0073] It should be noted that when the design tolerance zone is a one-sided tolerance, the other boundary can be set to negative infinity or positive infinity, and the corresponding conditions will never be triggered.

[0074] S5. Input the operating condition data fed back by the digital twin system into the weight adjustment network, and output the dynamic weight coefficient of each explicit geometric constraint parameter.

[0075] Step S5 includes: S51. Receives real-time temperature data, real-time load data, and historical deformation trend data sent by the digital twin system to form operating condition data.

[0076] Based on the target equipment's operational technology plan, obtain the maximum and minimum values ​​of various operational data. Then, normalize the currently received operational data to map it to... Interval.

[0077] Normalized temperature tensors, load tensors, and historical deformation trend tensors are constructed and concatenated along the feature dimension to obtain a feature matrix, which is then input into a pre-trained weight adjustment network.

[0078] S52. The weight adjustment network employs a feedforward neural network with a multilayer perceptron structure, where the output dimensions correspond one-to-one with the explicit geometric constraint parameters. The specific structure of the weight adjustment network is as follows: Input layer: Receives the feature matrix and flattens it into a one-dimensional feature vector, or retains its temporal structure and inputs it into the subsequent feature extraction layer.

[0079] Feature extraction layer: Two fully connected layers with rectified linear units as activation functions, the first layer having 0 hidden units. The output is The number of hidden units in the second layer is The output is .in , This is the weight matrix. , For bias vectors, To correct the form of the linear unit function.

[0080] Sensitivity Analysis Layer: This layer uses an attention mechanism to quantitatively evaluate the impact of each working condition feature dimension on geometric accuracy. Let the first... The sensitivity eigenvectors corresponding to the explicit geometric constraint parameters are: Through This is implemented by connecting independent small neural network branches or multi-head attention layers, with each branch outputting a real number as a sensitivity evaluation value, denoted as . .

[0081] Weight generation layer: This layer generates the sensitivity evaluation values. The fully connected output layer with a Softmax function is input, and dynamic weight coefficients for each explicit geometric constraint parameter are generated, expressed by the following formula: ; Where w u Let be the dynamic weight coefficient of the u-th explicit geometric constraint parameter, satisfying And w u ∈(0,1). The weight generation layer ensures that all weight coefficients are normalized, avoiding unconstrained growth of each weight that could lead to unbalanced optimization.

[0082] Learnable parameter set: The learnable parameters included in the weight adjustment network are: W1, b1, W2, b2, and all trainable parameters of the sensitivity analysis layer and the weight generation layer.

[0083] S53. The pre-training process of the weight adjustment network is divided into the data preparation stage, the annotation stage, and the model training stage.

[0084] (1) Data preparation stage: Collect multiple sets of operating data of the target equipment under diverse operating conditions. Each set of operating data includes temperature time series, load time series and corresponding historical deformation trend data, which constitute the input sample. At the same time, record the actual target geometric constraint weight configuration of the target equipment under the operating condition (or the convergent weight vector obtained through offline optimization) as training labels.

[0085] (2) Labeling stage: For each group of training samples, the optimal weight vector labeling is obtained by offline simulation optimization method: the weight of the visual loss term is fixed at 1.0, and the weights of each geometric constraint are used as adjustable parameters. Grid search is performed in the interval [0,1] with a step size of 0.1 (if there are many constraints, Bayesian optimization is used to reduce the number of searches). For each group of candidate weight vectors, the joint optimization objective is calculated, and the weight vector that minimizes the objective value is selected as the label of the sample. At the same time, to ensure that the weight vector is consistent with the actual physical meaning, prior constraints can be applied: the weight of the key constraint shall not be lower than 0.1. The key constraint is determined by the technicians according to the actual situation of the target equipment. Usually, the concentricity affecting safety is set as the key constraint.

[0086] (3) Model training phase: The training loss function is constructed using the mean squared error between the output weight vector of the weight adjustment network and the label weight vector. The gradient descent optimization algorithm is used to iteratively update the parameters of the weight adjustment network. After training converges, the parameters of the weight adjustment network are saved as a pre-trained model for online inference.

[0087] Taking a motor shaft as the target device as an example, its key geometric constraints typically include: the concentricity of the bearing seat and the journal, the perpendicularity of the shaft shoulder to the axis, and the parallelism between the bearing seats at both ends. When the real-time temperature data fed back by the digital twin system shows that the motor housing temperature rises from 25°C to 80°C, due to the thermal expansion effect, the shaft diameter increases by approximately 0.02 mm, and the inner ring of the bearing expands by approximately 0.015 mm. This causes the original design tolerance zone (e.g., concentricity ≤ 0.01 mm) to become too strict under actual high-temperature conditions. If the loss calculation is still performed based on the tolerance zone at room temperature, the optimization process will force the model to excessively tighten the geometric deviation, thus introducing unrealistic local deformations. At this point, the weight adjustment network learns from historical deformation trend data (such as shaft bending data recorded from multiple start-stop cycles) that the importance of the concentricity constraint at high temperatures should be reduced from 0.6 to 0.3, while the parallelism constraint is less affected by temperature, and its weight is slightly adjusted from 0.2 to 0.25. Through this specific mapping relationship, the dynamic weight coefficients output by the network enable the reconstructed model to appropriately relax the concentricity requirements under high-temperature conditions, thereby obtaining a geometric shape that matches the measured thermal deformation. This example is only to illustrate the physical relationship between the operating condition data and the geometric constraint weights. In actual applications, the mapping relationship is automatically learned by the network based on offline simulation annotation data, and is not limited to this example.

[0088] S54: Input the preprocessed operating condition data from step S51 into the pre-trained weight adjustment network, complete one forward calculation according to the forward propagation process in step S52, and the network output layer directly outputs the dynamic weight coefficient vector of M explicit geometric constraint parameters. All coefficients have satisfied the normalization constraints.

[0089] S6. The tolerance band perceptual loss value and the dynamic weight coefficient are weighted and fused together, and combined with the visual loss function to form a joint optimization objective. The gradient is backpropagated to the deep neural network to update the vertex positions of the initial 3D mesh model and generate the optimized 3D mesh model.

[0090] In this embodiment, step S6 includes: calculating a visual loss function, which includes at least chamfer distance loss, normal consistency loss, and contour intersection-union ratio loss.

[0091] The calculation process for each loss involved in the visual loss function is as follows: Suppose a predetermined number of points are uniformly sampled from the surface of a 3D mesh model to obtain a predicted point set. The 3D point cloud of the target object surface, obtained through multi-view stereo matching or a depth sensor, is used as the true point set.

[0092] The chamfer distance loss is defined as the mean of the sum of the squares of the bidirectional nearest neighbor distances from the predicted point set to the real point set and from the real point set to the predicted point set. Specifically, the mean of the squares of the minimum Euclidean distance from each point in the predicted point set to the real point set and the mean of the squares of the minimum Euclidean distance from each point in the real point set to the predicted point set are calculated separately, and the two are summed to obtain the chamfer distance loss.

[0093] Normal consistency loss: For each point in the predicted point set, its normal vector is calculated using principal component analysis of the local neighborhood of the grid; for the corresponding nearest neighbor point in the real point cloud, its normal vector is obtained. The cosine of the angle between the normal vectors of each pair of matched points is negative and the mean is calculated to obtain the normal consistency loss.

[0094] Contour Cross-Union Ratio (CUI) Loss: For each camera viewpoint, the 3D mesh model is projected into a 2D contour mask using a differentiable rasterizer renderer, while simultaneously obtaining the ground truth segmentation mask for that viewpoint. The CUI of the predicted mask and the ground truth mask is calculated for each viewpoint, and the natural logarithm is taken, followed by a negative value. The average of the results for all viewpoints is then used as the contour CUI loss.

[0095] Ultimately, the visual loss function output is the sum of the chamfer distance loss, the normal consistency loss, and the contour intersection-union ratio loss.

[0096] Using the visual loss function as the base term and the weighted tolerance band perceptual loss value as the physical correction term, a total loss objective function is constructed.

[0097] The partial derivatives of the total loss function with respect to the coordinates of each vertex of the 3D mesh model are calculated using an automatic differentiation mechanism to form a gradient vector.

[0098] Based on the gradient vector, a gradient-based iterative optimization algorithm is used to update the mesh vertex coordinates. Specifically, an adaptive momentum estimation optimizer is selected as the iterative update strategy. This optimizer maintains first-order moment estimates and second-order moment estimates in each iteration and adaptively adjusts the update step size of each parameter using historical gradient information. Let the vertex coordinates at the t-th iteration be... The optimizer adjusts the current gradient. and historical momentum calculation update increment Update vertex coordinates using the following formula: ; Where α(t) is the learning rate at step t, which can be set to a fixed value or adopt an exponential decay strategy.

[0099] During the iterative optimization process, the convergence conditions are set as follows: When the absolute value of the relative rate of change of the total loss function in the most recent iterations (e.g., 10 to 20 consecutive iterations) is less than a preset threshold (e.g., 10... −5 Up to 10 −4 When the condition is met, the optimization is considered to have converged.

[0100] Alternatively, the iteration can be stopped when the number of iterations reaches the preset maximum number of iterations (e.g., 1000 to 5000 times, depending on the grid size and accuracy requirements).

[0101] Once any convergence condition is met, the current vertex coordinates are output as the optimization result.

[0102] Throughout the optimization process, the topological connectivity of the initial 3D mesh model remains unchanged. This topological connectivity includes: the edge indices between vertices, the vertex order of triangle faces, and adjacency relationships. Specifically, only the (x, y, z) numerical components stored in the vertex coordinate array within the mesh data structure are updated. The vertex index array and face index array are initialized and fixed before optimization, and are not added, deleted, or reordered during the optimization phase.

[0103] For each vertex's spatial coordinate components, the following numerical correction is performed: After each iteration of optimization, the calculated new coordinates are directly assigned to the vertex's coordinate storage unit, overwriting its old value. This process is executed in parallel or sequentially for all vertices, completing one full coordinate correction.

[0104] Once the iteration converges or reaches the maximum number of iterations, the vertex coordinate set obtained from the last update is encapsulated together with the initial fixed topology index array to generate an optimized 3D mesh model. This model has the exact same number of vertices and facet connections as the initial model, with only the vertex spatial positions shifted to conform to visual and geometric constraints.

[0105] S7. Write the optimized 3D mesh model back to the digital twin system and update the geometry layer and constraint verification layer of the digital twin system.

[0106] In this embodiment, step S7 includes: performing topological verification and geometric simplification on the optimized 3D mesh model, and encapsulating it into a geometric entity object that can be recognized by the digital twin system.

[0107] The topology verification includes the following verification items: Manifold verification: Check whether each edge is shared by no more than two faces and whether the neighboring faces of each vertex form a disk topology.

[0108] Degenerate primitive verification: Remove triangular faces or edges with an area or side length smaller than the target equipment's baseline design standard to avoid zero-area or zero-length primitives affecting subsequent calculations.

[0109] Normal consistency check: Through neighborhood propagation or global correction based on volume sign, the normal direction of all facets is unified to point outward from the model. If there are facet clusters with conflicting normal directions, they are automatically flipped.

[0110] Isolated element verification: Remove isolated vertices that do not belong to any facet, as well as isolated clusters of faces that are not connected to the main model.

[0111] The geometric simplification employs a quadratic error metric edge collapse algorithm, specifically including: reducing the number of facets to 50% of the original number; the quadratic error introduced by each edge collapse must not exceed 0.001 times the length of the diagonal of the model bounding box; if it does, the edge is prohibited from collapsing; for the assembly reference edges and edges marked in the design data, the collapse cost is set to infinity, and they are retained without simplification.

[0112] The old model in the geometry layer is replaced with a new mesh model through the API interface of the digital twin system.

[0113] In the constraint verification layer, interference checks, tolerance compliance assessments, and structural stress simulations are rerun based on the updated model, and the verification results are fed back to the monitoring interface in real time.

[0114] Interference Check: A collision detection algorithm based on hierarchical bounding boxes is used to traverse and detect the new model and other device models in the environment. The output includes a list of interference volumes and minimum safe distances. If unexpected interference is found, the interference location is highlighted on the monitoring interface and an alarm is triggered.

[0115] Tolerance compliance assessment: The measured features of the new model vertices, such as the fitted position of the plane, the axis and radius of the cylinder, are compared with the nominal values ​​and design tolerance zones in the design data. The deviation of each inspected feature is calculated and whether it is within the design tolerance zone. Qualified inspected features are screened and the ratio of the number of qualified inspected features to the total number of features is used as the output pass rate for feedback.

[0116] Structural stress simulation: Material properties are assigned to the simplified model (read from the material library of the digital twin system), the design load under the current working condition is applied, and the displacement and stress distribution are solved by linear elastic finite element analysis.

[0117] If any constraint verification result is not met, model rollback is triggered: the old model saved before this optimization is restored to the geometry layer via the API interface, and a penalty signal is sent to the joint optimization module to increase the weight of the physical correction term in this optimization process. Steps S6 and S7 are re-executed until the verification passes or the maximum number of rollbacks is reached. If it still fails, the old model is kept and the optimization is marked as a failure for manual review.

[0118] The deviation data and optimization gains during this geometric reconstruction process are recorded and stored in the lifecycle history database of the digital twin system. The deviation data includes the average deviation between the model before optimization and the design nominal value. The average deviation between the optimized model and the design nominal value The optimization gain is defined as the residual deviation value for each explicit geometric constraint parameter, and the optimization gain is defined as... , , All values ​​are dimensionless.

[0119] It should be noted that the lifecycle history database provides a query interface, supporting the retrieval of historical records by time range, device ID, optimization gain threshold, and other conditions. During each retraining or fine-tuning, the weight adjustment network of the digital twin system can read all records from the database within a set period (e.g., 30 days) and use them as incremental training samples to achieve continuous model learning.

[0120] It should also be noted that the scale-invariant feature transformation algorithm, the fast approximate nearest neighbor search library (FLANN), the basic structure of graph convolutional networks, the moving cube algorithm, the adaptive momentum estimation optimizer (Adam), the direct integration method in time history analysis, the quadratic error metric edge collapse (QEM) algorithm, hierarchical bounding box collision detection, and linear elastic finite element analysis mentioned in this specific embodiment are all existing technologies or open-source algorithms known in the art. This invention does not elaborate on the internal execution details of these existing technologies, but only uses them as tool modules within the overall flow of this invention. Those skilled in the art, based on the guidance of this specification and their existing technical knowledge, can fully implement the technical solution of this invention without any creative effort.

[0121] Reference Figure 2 As shown, the second embodiment of the present invention provides a physical constraint geometry reconstruction system based on a twin system, including: a feature matching module, an initial model establishment module, a design constraint reading module, a tolerance-aware loss module, a dynamic weight generation module, a joint optimization module, and a model write-back module.

[0122] The feature matching module is connected to the initial model building module, the initial model building module is connected to the design constraint reading module, the design constraint reading module is connected to the tolerance-aware loss module, the tolerance-aware loss module is connected to the dynamic weight generation module, the dynamic weight generation module is connected to the joint optimization module, and the joint optimization module is connected to the model write-back module.

[0123] The feature matching module performs feature matching between the multi-view image data of the target device and the camera pose parameters to generate a matching feature set.

[0124] The initial model building module extracts spatial coordinate information from the matching feature set and performs deep neural network inference to generate an initial 3D mesh model.

[0125] The design constraint reading module reads the design data of the target device from the digital twin system, and obtains explicit geometric constraint parameters, including at least parallelism, perpendicularity or concentricity, as well as the design tolerance zone corresponding to each explicit geometric constraint parameter.

[0126] The tolerance-aware loss module calculates the geometric deviation between the initial 3D mesh model and each explicit geometric constraint parameter. Based on the design tolerance zone, it maps the geometric deviation to the soft interval loss function, generating the tolerance zone-aware loss value for each explicit geometric constraint parameter.

[0127] The dynamic weight generation module inputs the operating condition data fed back from the digital twin system into the weight adjustment network and outputs the dynamic weight coefficient of each explicit geometric constraint parameter.

[0128] The joint optimization module weights and fuses the tolerance band perceptual loss value with the dynamic weight coefficients, and combines them with the visual loss function to form a joint optimization objective. The gradient is then fed back to the deep neural network to update the vertex positions of the initial 3D mesh model and generate the optimized 3D mesh model.

[0129] The model write-back module writes the optimized 3D mesh model back to the digital twin system, updating the geometry layer and constraint verification layer of the digital twin system.

[0130] The above content is merely an example and illustration of the concept of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the concept of the invention or exceed the scope defined by the present invention, and all such modifications and additions should fall within the protection scope of the present invention.

Claims

1. A method for physical constraint geometry reconstruction based on twin systems, characterized in that, include: S1. Perform feature matching between the multi-view image data of the target device and the camera pose parameters to generate a matching feature set; S2. Extract spatial coordinate information from the matching feature set and perform deep neural network inference to generate an initial 3D mesh model; S3. Read the design data of the target device from the digital twin system, and obtain at least explicit geometric constraint parameters including parallelism, perpendicularity or concentricity, as well as the design tolerance zone corresponding to each explicit geometric constraint parameter; S4. Calculate the geometric deviation between the initial 3D mesh model and each explicit geometric constraint parameter, map the geometric deviation to the soft interval loss function according to the design tolerance zone, and generate the tolerance zone perceived loss value for each explicit geometric constraint parameter. S5. Input the operating condition data fed back by the digital twin system into the weight adjustment network, and output the dynamic weight coefficient of each explicit geometric constraint parameter; S6. The tolerance band perceptual loss value and the dynamic weight coefficient are weighted and fused together, and combined with the visual loss function to form a joint optimization objective. The gradient is backpropagated to the deep neural network to update the vertex positions of the initial 3D mesh model and generate the optimized 3D mesh model. S7. Write the optimized 3D mesh model back to the digital twin system and update the geometry layer and constraint verification layer of the digital twin system.

2. The physical constraint geometry reconstruction method based on a twin system according to claim 1, characterized in that, Step S1 includes: Feature detection operators are used to extract key points and construct descriptors from multi-view image data, and feature associations are established between image pairs from adjacent views. By combining the extrinsic parameter matrix provided by the camera pose parameters, redundant and erroneous feature point pairs are eliminated through epipolar geometric constraints. The feature point pairs that pass the consistency check are triangulated to generate three-dimensional spatial points, and a normal vector and semantic label are attached to each three-dimensional spatial point to form a sparse three-dimensional point cloud. Multi-view aggregation is performed on sparse 3D point clouds to form a matching feature set.

3. The physical constraint geometry reconstruction method based on a twin system according to claim 2, characterized in that, Step S2 includes: Extract spatial coordinate information of sparse 3D point clouds from the matching feature set; Spatial coordinate information is input into a graph convolution-based deep neural network. Through implicit surface regression and isosurface extraction, an initial 3D mesh model is generated, which includes a list of vertex indices, triangle patch connections, and the normal vector of each vertex.

4. The physical constraint geometry reconstruction method based on a twin system according to claim 1, characterized in that, Step S3 includes: Read the solid geometric features in the design data and identify basic geometric elements, including at least planes, cylindrical surfaces, and central axes. Based on the geometric consistency conditions defined in the design datum, the basic geometric elements identified are traversed. If the normal vectors of two planes satisfy the collinearity condition, then it is determined that there is a parallelism constraint. If the normal vector of one of the planes satisfies the orthogonality condition with the direction vector of an axis, then it is determined that there is a perpendicularity constraint. If the central axes of two of the cylindrical surfaces coincide in space, then it is determined that there is a concentricity constraint. Read the part tolerance table associated with the solid geometry features, obtain the design tolerance zone value for each determined constraint, including the lower and upper tolerance limits, which are used as the tolerance zone parameters corresponding to parallelism, perpendicularity, and concentricity constraints, respectively. The explicit geometric constraint parameters and design tolerance zones obtained from the analysis are used to establish an index mapping relationship with the corresponding vertex sets in the initial 3D mesh model.

5. The physical constraint geometry reconstruction method based on a twin system according to claim 1, characterized in that, Step S4, which calculates the geometric deviation between the initial 3D mesh model and each explicit geometric constraint parameter, includes: Parallelism constraint: Perform least-squares plane fitting for associated vertices and calculate the angular deviation between the unit normal vectors of the fitted plane; Verticality constraint: Calculate the orthogonal projection components of the constrained surface and the reference axis in the initial 3D mesh model to obtain the residual value that deviates from the theoretical state by 90 degrees; Concentricity constraint: Perform least-squares cylindrical fitting on the vertices belonging to the body of revolution in the initial 3D mesh model, and calculate the spatial position deviation between the fitted axis and the given reference axis of the digital twin system; Various deviations are normalized to generate a multi-dimensional geometric deviation vector.

6. The physical constraint geometry reconstruction method based on a twin system according to claim 1, characterized in that, Step S4, which generates the tolerance band perceived loss value for each explicit geometric constraint parameter, includes: The soft interval loss function is defined as a piecewise linear function. When the geometric deviation is less than the lower limit of the design tolerance band, the output non-zero gradient signal is equal to the lower limit of the design tolerance band minus the geometric deviation; when the geometric deviation is greater than the upper limit of the design tolerance band, the output non-zero gradient signal is equal to the geometric deviation minus the upper limit of the design tolerance band; when the geometric deviation is within the interval defined by the upper and lower limits of the design tolerance band, zero-value loss is output. The non-zero gradient signal or zero-value loss output is used as the tolerance band sensing loss value.

7. The physical constraint geometry reconstruction method based on a twin system according to claim 1, characterized in that, Step S5 includes: It receives real-time temperature data, real-time load data, and historical deformation trend data from the digital twin system to form operating condition data; The operating condition data is input into a pre-trained weight adjustment network. The weight adjustment network is used to analyze the sensitivity of the operating condition data to geometric accuracy and dynamically adjust the importance and priority of each explicit geometric constraint in the global optimization. Output a set of dynamic weight coefficient vectors that correspond one-to-one with the explicit geometric constraint parameters, used to adjust the contribution ratio of each loss term.

8. The physical constraint geometry reconstruction method based on a twin system according to claim 1, characterized in that, Step S6 includes: Calculate the visual loss function, which includes at least the chamfer distance loss, normal consistency loss, and contour intersection-over-union ratio loss; Using the visual loss function as the base term and the weighted tolerance band perceptual loss value as the physical correction term, a total loss objective function is constructed. The partial derivatives of the total loss with respect to the grid vertex coordinates are backpropagated through an automatic differentiation mechanism, and a gradient-based iterative optimization algorithm is executed. While keeping the topological connections unchanged, the coordinates of each vertex position in the initial 3D mesh model are corrected to obtain the optimized 3D mesh model.

9. The method for physical constraint geometry reconstruction based on a twin system according to claim 1, characterized in that, Step S7 includes: The optimized 3D mesh model undergoes topological verification and geometric simplification, and is then encapsulated into geometric entity objects that can be recognized by the digital twin system. The old model in the geometry layer is replaced with a new mesh model through the API interface of the digital twin system. In the constraint verification layer, the interference check, tolerance compliance assessment and structural stress simulation are rerun based on the updated model, and the verification results are fed back to the monitoring interface in real time. Record the deviation data and optimization gains during this geometric reconstruction process and store them in the lifecycle history library of the digital twin system.

10. A physical constraint geometry reconstruction system based on a twin system, characterized in that, For performing the method according to any one of claims 1 to 9, comprising: The feature matching module performs feature matching between the multi-view image data of the target device and the camera pose parameters to generate a matching feature set. The initial model building module extracts spatial coordinate information from the matching feature set and performs deep neural network inference to generate an initial 3D mesh model. The design constraint reading module reads the design data of the target device from the digital twin system, and obtains explicit geometric constraint parameters, including at least parallelism, perpendicularity or concentricity, as well as the design tolerance zone corresponding to each explicit geometric constraint parameter. The tolerance-aware loss module calculates the geometric deviation between the initial 3D mesh model and each explicit geometric constraint parameter, maps the geometric deviation to the soft interval loss function based on the design tolerance zone, and generates the tolerance zone-aware loss value for each explicit geometric constraint parameter. The dynamic weight generation module inputs the operating condition data fed back by the digital twin system into the weight adjustment network and outputs the dynamic weight coefficient of each explicit geometric constraint parameter. The joint optimization module weights and fuses the tolerance band perception loss value with the dynamic weight coefficients, and combines them with the visual loss function to form a joint optimization objective. The gradient is then fed back to the deep neural network to update the vertex positions of the initial 3D mesh model and generate the optimized 3D mesh model. The model write-back module writes the optimized 3D mesh model back to the digital twin system, updating the geometry layer and constraint verification layer of the digital twin system.

Images