Visual sparse observation and physical model fused flexible body deformation tracking method

Abstract

The invention discloses a visual sparse observation and physical model fused flexible body deformation tracking method, which comprises the following steps of: firstly, constructing a flexible body co-rotation finite element model considering geometric nonlinearity, and establishing a low-dimensional flexibility coupling model for describing a mechanical transfer relationship between a virtual actuating source and a characteristic observation point by utilizing a static condensation technology; in the real-time tracking process, a binocular vision system is used for collecting images and resolving three-dimensional coordinates of sparse feature points in a real physical world; constructing and solving a reverse optimization problem based on the low-dimensional flexibility coupling model, and calculating a virtual correction force vector capable of minimizing an observation error; and applying the virtual correction force vector to the finite element model to drive the finite element model to deform, and updating a full-field geometric node position and a tangent stiffness matrix in real time. Through deep fusion of a physical model and sparse visual observation, the problems of visual occlusion and virtual-real model deviation in flexible body assembly are effectively solved, and online real-time full-field high-precision deformation state estimation is realized.

Description

Technical Field

[0001] This invention belongs to the field of robot visual perception and physical simulation technology, specifically involving a flexible body deformation tracking method that integrates visual sparse observation and physical model. Background Technology

[0002] With the rapid development of intelligent manufacturing and precision assembly technologies, robotic systems are increasingly widely used in fields such as electronics manufacturing and aerospace, playing a particularly important role in the precision shape manipulation of flexible objects such as flexible printed circuit boards (FPCs) and cable bundles. Unlike traditional rigid workpieces, flexible bodies in such assembly scenarios are typically small in size, made of lightweight and soft materials, and prone to large-scale geometric nonlinear deformation under gravity or contact forces. In complex operating environments with limited space, to ensure the accuracy and safety of robot assembly operations on flexible bodies, it is necessary to accurately and in real-time perceive the deformation state and spatial pose of the flexible body throughout the operation process.

[0003] Currently, deformation monitoring of flexible bodies mainly relies on computer vision technology. Existing mainstream methods typically employ stereo vision reconstruction, RGB-D depth cameras, or structured light sensors to acquire relatively dense 3D point clouds or mesh models of the object's surface to reconstruct its shape. For example, a multilinear deformable body tracking method based on structure-preserving registration and iterative geometric separation (CN118823067A) uses visual algorithms to extract feature points to track multiple linear objects. However, such purely visual methods have certain limitations: First, they can only acquire geometric information of the object's visible surface, making it difficult to infer the complete shape of the object's interior or occluded areas; second, in small-scale industrial assembly scenarios, due to limitations in depth of field and field of view, ordinary visual sensors are prone to significant perspective distortion; furthermore, under the influence of complex lighting environments in industrial settings and the high reflectivity of metal components, existing feature extraction and matching algorithms struggle to operate stably, easily introducing environmental noise and false features. In the presence of incomplete data and noise interference, purely visual methods struggle to reconstruct a complete and reliable deformation field.

[0004] On the other hand, simulation methods based on physical models (such as the finite element method (FEM) and flexible beam models) can deduce the deformation behavior of flexible bodies based on the principles of continuum mechanics, and theoretically have the ability to infer full-field deformation using boundary conditions. However, this type of method also has two limitations: First, computational efficiency bottleneck. In order to accurately describe the large-scale geometric nonlinear deformation of flexible bodies, it is usually necessary to construct a high-order discrete mesh model with a large number of nodes. Directly solving the huge full-order nonlinear system in real time requires huge computational resources, and the computation time is difficult to meet the requirements of robot control systems for online real-time performance at the tens of millisecond level. Second, the uncertainty of model parameters. Since the material physical parameters and boundary conditions in the simulation model are difficult to be completely consistent with the real physical world, the open-loop physical simulation results often deviate significantly from the deformation of real objects.

[0005] To overcome the limitations of single methods, existing technologies have attempted to combine physical models with visual perception. For example, a robot global pose estimation method combining physical modeling and visual perception (CN120791851A) utilizes Cosserat Rod theory to model flexible bodies. However, most of these hybrid methods still rely on relatively dense surface observation data (such as complete contour lines or dense arrays of marker points) to constrain the physical model, and are generally only applicable to relatively simple linear flexible bodies. In actual assembly scenarios, limited by the field of view and occlusion, only sparsely distributed feature points can often be obtained. When the observation data is sparse, existing methods lack an effective mechanism to use limited information to perform closed-loop correction on complex physical models, making it difficult to eliminate parameter deviations between simulation and reality in a timely manner, which can easily lead to a significant decrease in deformation tracking accuracy.

[0006] In summary, existing technologies for full-field deformation tracking of flexible bodies face the following bottlenecks: pure vision methods struggle to acquire complete surface point clouds in complex environments and fail to reconstruct the object's full 3D shape; while pure physical model methods can theoretically calculate full-field deformation, their high computational complexity due to full-order nonlinearity makes them unsuitable for online real-time performance, and inaccurate modeling parameters lead to severe distortions between simulation and reality; existing virtual-real hybrid methods typically rely on dense observation constraints and are often limited to simple linear flexible bodies, lacking sparse driving and correction mechanisms applicable to complex flexible bodies. Therefore, there is an urgent need in this field for a technical solution that can combine sparse visual observation data with efficient physical models to achieve real-time full-field deformation tracking of complex flexible bodies with geometrically nonlinear characteristics. Summary of the Invention

[0007] This invention addresses the problems in existing flexible body deformation monitoring technologies, such as the susceptibility of pure visual methods to environmental interference and difficulty in acquiring full-field information, low computational efficiency of full-order physical simulation, and large virtual-real deviation caused by parameter uncertainties in open-loop physical models. It proposes a flexible body deformation tracking method that integrates visual sparse observation and physical model.

[0008] The above-mentioned objective of the present invention is achieved by at least one of the following technical solutions.

[0009] A method for tracking the deformation of flexible bodies by fusing visual sparse observations with a physical model includes the following steps: Step 1: Construct a flexible body co-rotation finite element model considering geometric nonlinearity. Define model feature observation points in the finite element model that correspond one-to-one with the feature points on the surface of the real flexible body, and arrange several virtual actuation sources on the surface or inside the flexible body. Step 2: Use static condensation technique to reduce the dimensionality of the full-order tangent stiffness matrix of the finite element model, and construct a low-dimensional flexibility coupling model that describes the transmission relationship between the force applied by the virtual actuation source and the displacement generated by the characteristic observation point of the model. Step 3: During real-time tracking, images of the target flexible body are acquired using a binocular vision system, and the three-dimensional coordinates of sparse feature points on the surface of the flexible body in the real physical world are calculated using a feature extraction and matching algorithm based on clustering screening and topological constraints. Step 4: Using the true 3D coordinates of the sparse feature points as the tracking target, construct and solve the inverse optimization problem based on the low-dimensional flexibility coupling model, and calculate the optimal virtual correction force vector that minimizes the positional deviation between the model feature observation points and the true sparse feature points. Step 5: Apply the optimal virtual correction force vector to the finite element model to drive the finite element model to deform in order to synchronize the physical state of the real flexible body, and update and output the full-field geometric node positions and corresponding tangent stiffness matrices of the flexible body in real time.

[0010] Furthermore, in step 1, a flexible body co-rotation finite element model considering geometric nonlinearity is constructed, specifically including: The co-rotation coordinate method is used to describe the deformation motion of finite element mesh elements, and the total displacement of the elements in the global coordinate system is decomposed into rigid body rotation components and purely elastic deformation components. The linear element stiffness matrix is ​​calculated in the local coordinate system that rotates with the element, and then mapped back to the global coordinate system through coordinate transformation to assemble the full-order tangent stiffness matrix that depends on the current geometry.

[0011] Furthermore, in step 2, a low-dimensional flexibility coupling model is constructed, specifically including: Obtain the full-order tangent stiffness matrix, rearrange the rows and columns of the matrix according to the nodal degree of freedom type, and divide it into a sub-matrix of retained degree of freedom stiffness corresponding to the virtual actuation source and the model feature observation point, and a sub-matrix of internal degree of freedom stiffness corresponding to the remaining nodes. The Schur complement method is used to operate on the block-wise matrix to eliminate internal degrees of freedom, and the reduced stiffness matrix containing only the retained degrees of freedom is calculated. Based on this, the corresponding flexibility matrix is ​​obtained.

[0012] Furthermore, constructing a low-dimensional flexibility coupling model also includes: Based on the compliance matrix Establish a description of the virtual correction force vector Deviation vector between model feature observation point position and the model feature observation point position The equation for the linear mapping relationship between them is:

[0013] in, Let be the free deformation deviation vector under the action of no virtual correction force.

[0014] Furthermore, in step 3, the binocular vision system employs dual telecentric cameras to eliminate perspective errors in small fields of view and improve measurement accuracy.

[0015] Furthermore, in step 3, the feature extraction and matching algorithm based on clustering screening and topological constraints specifically includes: Adaptive thresholding and morphological opening operations are performed on the acquired left and right images respectively to extract several candidate contours; Calculate the geometric roundness features and local region brightness ratio features of each candidate contour to construct a multidimensional feature vector; The K-Means clustering algorithm is used to perform binary classification on the multidimensional feature vectors. Based on the class center score, candidate contours belonging to environmental interference are automatically eliminated, and the true feature point contours are retained.

[0016] Furthermore, the feature extraction and matching algorithm also includes: Using the Hungarian algorithm combined with geometric topological consistency constraints, stereo matching is performed on the contours of real feature points in the left and right images, and their three-dimensional coordinates in the real physical world coordinate system are calculated by linear triangulation.

[0017] Furthermore, in step 4, constructing and solving the inverse optimization problem specifically includes: Construct a quadratic optimization objective function that includes a position tracking error term and an energy regularization term:

[0018] in, For virtual correction force vector, This is the positional observation deviation vector between the real sparse feature points and the model feature observation points. For the flexibility matrix, Let be the free deformation deviation vector under the action of no virtual correction force. The regularization coefficient is used. The objective function is solved iteratively using a numerical optimization solver, and the optimal virtual correction force vector is output when the convergence condition is met.

[0019] Furthermore, the flexible body is a deformable body with linear elasticity of material and nonlinear deformation characteristics of geometric deformation.

[0020] A computer device according to the present invention includes a memory and a processor, the memory being electrically connected to the processor, the memory storing a computer program, which, when executed by the processor, causes the processor to implement the method described herein.

[0021] The present invention provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor implements the method described herein.

[0022] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention addresses the limitations and susceptibility to interference in visual observation under complex environments. It employs a feature extraction and matching algorithm based on clustering and topological constraints, automatically eliminating environmental noise through multidimensional feature clustering. This enables accurate extraction of sparse feature points in complex backgrounds. Furthermore, by combining the continuous medium mechanical properties of an efficient physical model, the full-field geometry of the flexible body, including occluded areas, can be inferred using only sparse surface observation data. This overcomes the limitations of purely visual methods that rely on dense point clouds and struggle to reconstruct complete morphology.

[0023] 2. Achieving full-field deformation tracking with both computational efficiency and physical accuracy. This invention utilizes static condensation technology to reduce the dimensionality of the full-order finite element model to a low-dimensional constrained space composed of virtual actuation sources and observation points, constructing an efficient flexible coupling model, significantly reducing the computational scale of inverse solving, and meeting online real-time requirements; simultaneously, by constructing an inverse optimization problem including position error and energy regularization terms, the virtual correction force is solved in real time to drive the finite element model in a closed loop, effectively eliminating the virtual-real deviation caused by inaccurate model parameters, and achieving high-precision full-field state estimation. Attached Figure Description

[0024] Figure 1 The above is a flowchart of a flexible body deformation tracking method that integrates visual sparse observation and physical model as an example. Figure 2This is a schematic diagram of the hardware experimental platform structure for flexible body deformation tracking in an embodiment of the present invention; Figure 3 This is a flowchart of the feature extraction and matching method based on clustering screening and topological constraints in an embodiment of the present invention; Figure 4 This is a schematic diagram of the flexible body shape correction principle based on virtual correction force vector in an embodiment of the present invention. Detailed Implementation

[0025] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. However, it should be noted that these embodiments are merely for explaining the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Various equivalent transformations and substitutions in function, method steps, or structure made by those skilled in the art after reading this specification, based on the technical concept of the present invention, should all be included within the scope of protection of the present invention.

[0026] Please see Figure 1 , Figure 1 This is a flowchart illustrating the overall process of a flexible body deformation tracking method that integrates visual sparse observation with a physical model, according to the present invention. The method provided in this embodiment aims to achieve real-time and accurate tracking of the full-field deformation state of a small flexible body by constructing a high-fidelity finite element model and inversely driving it using sparse visual data. The method specifically includes the following steps: Step 1: Construct a flexible body co-rotation finite element model that considers geometric nonlinearity. Define model feature observation points in the finite element model that correspond one-to-one with the feature points on the surface of the real flexible body, and arrange several virtual actuation sources on the surface or inside the flexible body.

[0027] Preferably, step 1 in the embodiments of the present invention includes: (1) Discretization of the finite element model: First, a three-dimensional geometric model of the target flexible body (such as a flexible printed circuit board, FPC) is established using a physics simulation engine (e.g., the SOFA framework). Based on the geometric dimensions and material properties of the flexible body, it is discretized into a finite element mesh model composed of several nodes and elements using a meshing tool (e.g., Gmsh). In one embodiment, tetrahedral elements are preferably used for meshing to accommodate complex geometries, and regular meshing is appropriately applied in key deformation regions to ensure computational accuracy and efficiency. Of course, this invention is not limited to tetrahedral elements; other types of finite element elements can be used depending on specific application requirements.

[0028] (2) Co-rotation finite element modeling: To address the significant geometric nonlinearity (i.e., large rotation and small strain) of flexible bodies during large-scale motion, this invention employs the co-rotational coordinate method to construct a finite element model. The core idea of ​​this method is to decompose the total displacement of each finite element in the global coordinate system into rigid body rotation components and purely elastic deformation components. Specifically, for each element, a local coordinate system that rotates with its rigid body motion is established. In this local coordinate system, the element's deformation is considered as a small deformation, allowing the use of linear stress-strain relationships (such as Hooke's law) to calculate the elastic force and element stiffness matrix within the element. Subsequently, the local element stiffness matrix is ​​mapped back to the global coordinate system through coordinate transformation and assembled to obtain a full-order tangent stiffness matrix dependent on the current geometric configuration. The above modeling method can accurately describe the geometric nonlinear behavior of flexible bodies during large-scale motion while keeping the computational load under control.

[0029] (3) Boundary conditions and virtual actuation source definition: On the constructed finite element model, physical boundary conditions are set according to the real experimental environment (e.g., applying displacement constraints or fixation constraints to the fixed end of the flexible body). Simultaneously, to enable subsequent observation correction and state updates based on the finite element model, two types of key nodes are defined in the model: First, model feature observation points, which are selected from mesh nodes corresponding to physical feature points (such as marker holes, corner points) or key shape features on the surface of the real flexible body, used for subsequent calculation of the positional deviation between the model and the real state; second, virtual actuation sources, where several nodes (e.g., nodes evenly distributed along the central axis of the flexible body or in areas of significant bending) are selected on or inside the flexible body as virtual actuation sources. It should be noted that the virtual actuation sources are not driving actuators in the real physical world, but rather points of application in the simulation model used to apply virtual correction forces. By applying calculated virtual forces at these locations, the finite element model can be driven to deform to approximate the actual shape of the real flexible body.

[0030] Step 2: Use static condensation technique to reduce the dimensionality of the full-order tangent stiffness matrix of the finite element model, and construct a low-dimensional flexibility coupling model that describes the transmission relationship between the force applied by the virtual actuation source and the displacement generated by the characteristic observation point of the model.

[0031] Preferably, step 2 in this embodiment of the invention includes: (1) Stiffness matrix partitioning: At each time step, obtain the full-order tangent stiffness matrix of the finite element model under the current state. The dimension of this matrix is ​​determined by the total number of nodes in the finite element mesh, typically reaching thousands to tens of thousands of dimensions. To reduce the computational complexity of subsequent inverse solutions, the matrix is ​​adjusted according to the type of node degrees of freedom. The rows and columns are rearranged and divided into blocks. Specifically, the degrees of freedom of all nodes corresponding to the virtual actuation source and the model feature observation points are defined as the retained degrees of freedom (denoted by subscripts). The degrees of freedom of all other nodes in the model are defined as internal degrees of freedom (denoted by subscripts). The stiffness matrix after partitioning is expressed as:

[0032] in, To preserve the stiffness submatrices between degrees of freedom, The stiffness submatrix between internal degrees of freedom, and To preserve the coupling stiffness submatrix between the degrees of freedom and the internal degrees of freedom, the virtual actuation source and the model's characteristic observation points are uniformly included in the preserved degrees of freedom. This allows for the direct establishment of the transfer relationship between applying virtual forces and generating observed displacements after dimensionality reduction.

[0033] (2) Shure complement dimensionality reduction operation: Static condensation techniques are used to eliminate the large number of internal degrees of freedom. Based on the aforementioned block matrix, the reduced stiffness matrix containing only the retained degrees of freedom is calculated through Schur complement operations. The calculation formula is as follows:

[0034] In practical calculations, large-scale... Instead of matrix inversion, we use sparse matrix decomposition techniques (such as LDLT decomposition or Cholesky decomposition) to efficiently solve linear equations to achieve equivalent operations, thus balancing numerical stability and computational efficiency.

[0035] (3) Construct a low-dimensional flexibility coupling model: Obtaining the dimensionality-reduced stiffness matrix Building upon this foundation, a low-dimensional flexibility coupling model describing the force-displacement relationship is further constructed. Since the goal of inverse correction is to solve for the virtual correction force, it is necessary to establish a model based on the virtual correction force. Deviation from feature point position A direct mapping. Based on the characteristics of the system after dimensionality reduction, this linear mapping relationship can be expressed as:

[0036] in, As a low-dimensional flexibility matrix, the stiffness matrix can be reduced by... Obtained by inverting or equivalently solving the relevant linear equations, it is used to characterize the displacement response generated at the model's characteristic observation points when a unit force is applied to the virtual actuation source; To be under the action of no virtual correction force (i.e. This refers to the free deformation deviation of model feature observation points caused only by external conditions such as gravity or boundary driving. Through the above steps, the originally high-dimensional nonlinear finite element problem is transformed into a low-dimensional linear mapping problem within the current time step, laying the foundation for subsequent real-time inverse optimization.

[0037] Step 3: During real-time tracking, images of the target flexible body are acquired using a binocular vision system, and the three-dimensional coordinates of sparse feature points on the surface of the flexible body in the real physical world are calculated using a feature extraction and matching algorithm based on clustering screening and topological constraints.

[0038] Preferably, please refer to Figure 3 Step 3 of the present invention includes: (1) Binocular image acquisition and system calibration: During real-time tracking, a binocular vision system simultaneously acquires left and right view images of the flexible body under its current deformation state. Preferably, the binocular vision system employs dual telecentric cameras and matching telecentric lenses to eliminate perspective errors in small fields of view. Before tracking, a system calibration process is completed, including: internal parameter calibration and distortion correction for the left and right cameras respectively; binocular stereo calibration to obtain the relative pose relationship between the left and right cameras; and obtaining the rigid body transformation matrix of the camera coordinate system relative to the world coordinate system (i.e., the finite element coordinate system where the physical model resides) through hand-eye calibration. Based on the above calibration results, the projection matrices of the left and right cameras defined in a unified world coordinate system are constructed. and This ensures consistency between the imaging model and the physical simulation space.

[0039] (2) Image preprocessing and candidate contour extraction: Adaptive thresholding (such as the Otsu algorithm) is performed on the acquired left and right grayscale images to separate the highlighted feature regions from the background, resulting in binary images. Then, morphological opening operations are performed on the binary images to remove isolated small spots caused by noise, high-frequency textures, or metallic reflections, preserving connected regions with regular shapes and appropriate areas. Next, the contours of each connected region are traversed to initially filter out abnormal contours that are too small or too large, thus obtaining a set of candidate feature point contours. .

[0040] (3) Multidimensional feature construction and cluster screening: For each candidate contour, its geometric roundness features and local brightness distribution features are further extracted to improve the ability to distinguish between real marker points and environmental interference points. Specifically, roundness features... Calculate using the following formula:

[0041] in, For the outline area, The perimeter of the outline is represented; this index is close to 1 for an ideal circle, but decreases significantly for irregular shapes. Simultaneously, the brightness proportion of the region inside the outline is statistically analyzed. The above features are combined into a multidimensional feature vector. The K-Means clustering algorithm is used to perform unsupervised binary classification on the feature vectors of all candidate contours, and the comprehensive score of each type of center is calculated. The type with higher score (i.e., higher roundness and brightness that better matches the characteristics of feature points) is identified as the true feature point contour, and the other type is identified as the interference contour generated by metallic reflection, background texture, etc. and is removed, thus obtaining a clean set of true feature points in the left and right images.

[0042] (4) Stereo matching and consistency verification: After obtaining the true feature point sets from the left and right images, a geometric topological consistency constraint is introduced to achieve robust stereo matching. Specifically, the Euclidean distance between feature point pairs is calculated, and a local neighborhood distance descriptor is constructed to characterize the relative geometric structure of each feature point within its neighborhood. Subsequently, a cost matrix is ​​constructed using the difference in descriptors between the feature points of the left and right images as the cost, and the Hungarian algorithm is used for globally optimal bipartite graph matching. Furthermore, by combining epipolar constraints and neighborhood topological structure consistency checks, erroneous matching pairs with excessively large residuals are eliminated, ensuring that the final retained matching feature point pairs maintain consistency with the surface structure of the real flexible body in terms of spatial topological relationships.

[0043] (5) Solving for the three-dimensional coordinates of sparse feature points: For any successfully matched feature point pair, obtain its pixel coordinates in the left and right images. and Combined with the projection matrix calibrated in step (1) and Construct a system of linear equations:

[0044] in and These represent the homogeneous pixel coordinate vectors of the feature points on the left and right image planes, respectively. Let be the coordinate vector of the feature point to be solved in the world coordinate system.

[0045] The system of equations is solved directly using the least squares method to obtain the true three-dimensional coordinate vector of the feature point in the world coordinate system. .in , , These represent the x-coordinate, y-coordinate, and y-coordinate of the feature point in the real-world coordinate system, respectively. Repeating this process for all matching points yields a sparse 3D feature point cloud, which serves as the observation input for subsequent inverse optimization.

[0046] Step 4: Using the true 3D coordinates of the sparse feature points as the tracking target, construct and solve the inverse optimization problem based on the low-dimensional flexibility coupling model, and calculate the optimal virtual correction force vector that minimizes the positional deviation between the model feature observation points and the true sparse feature points.

[0047] Preferably, step 4 in this embodiment of the invention specifically includes: (1) Construction of the observation bias vector: In step 3, a sparse set of 3D feature points defined in the world coordinate system has been obtained, denoted as . ,in Represents the first in the real physical world The three-dimensional coordinate vector of each feature point The total number of valid observation points; in step 1, model feature observation points corresponding one-to-one with the real feature points have been predefined in the finite element model, and their predicted positions at the current time are denoted as . ,in, In the finite element model, the first... The three-dimensional coordinate vector of each feature observation point.

[0048] For each pair of corresponding points, calculate the current position deviation:

[0049] in, For the first The positional deviation vector of each feature point. Stack all deviations sequentially by point index to construct the global observation deviation vector:

[0050] The vector It is a dimension The column vector represents the overall observation error between the model and the actual state at the current moment.

[0051] On the other hand, without applying a virtual correction force (i.e. Under the condition of gravity, boundary constraints, and external contact, the finite element model can be solved by co-rotation finite element method to obtain a set of "free deformation" states, and the corresponding deviation of the characteristic observation points is denoted as . The aforementioned The free deformation deviation vector reflects the natural positional deviation between the model observation points and the actual sparse feature points generated solely by the open-loop prediction of the finite element model when no virtual force correction is required, providing a benchmark for subsequent inverse correction.

[0052] (2) Establishment of the inverse optimization problem based on the flexibility coupling model: In step 2, the full-order stiffness matrix was reduced in dimensionality through static condensation and Schur complement operations to obtain a low-dimensional compliance matrix describing the displacement transfer relationship between the virtual actuation source and the model's characteristic observation points. And a virtual correction force vector was established. Deviation from the location of the feature observation point Linear relationship between them:

[0053] To ensure that the model's feature observation points approximate the positions of the actual sparse feature points as closely as possible, this invention formalizes the inverse solution of the virtual correction force vector problem into a quadratic optimization problem with a regularization term:

[0054] in, For virtual correction force vector, This is the regularization coefficient. The first term of the objective function measures the residual between the model observation points after virtual force correction and the actual observation points. The second term is the energy regularization term, which is used to suppress excessive virtual forces and numerical oscillations, and improve the solution stability and robustness under noisy observations.

[0055] In a preferred embodiment, to further differentiate the reliability of observations at different feature points, a diagonal weighting matrix can be introduced before the residual term. The objective function is rewritten as follows:

[0056] in, This represents the objective function to be optimized. The observation confidence weighting matrix is ​​typically a diagonal matrix, with its diagonal elements corresponding to the observation weights of each feature point. When the observation accuracy of some feature points may decrease due to occlusion or reflection, the matrix can be reduced accordingly. The corresponding weights are used to improve the robustness of the overall estimation.

[0057] (3) Numerical solution and result output of virtual correction force vector: To meet the practical limitations of physical drives, this invention can impose amplitude constraints on the virtual correction force vector (e.g., ),in, and Let represent the lower and upper threshold vectors of the virtual correction force vector, respectively, determined by the material physical limits or simulation stability requirements of the flexible body. This transforms the optimization problem into a standard constrained quadratic programming problem. Due to the flexibility matrix... The dimension is determined by the number of virtual actuation sources and feature observation points. Compared with the original full-order finite element system, the dimension is relatively low (e.g., it can be tens of dimensions). The QP problem has a moderate size and the objective function is strictly convex, so it can be solved efficiently.

[0058] This embodiment employs a mature numerical optimization solver (such as OSQP or other quadratic programming solvers) to iteratively solve the above optimization problem at each simulation time step. When the optimization process meets the preset convergence conditions (such as the objective function descent being less than a threshold or the number of iterations reaching the upper limit), the optimal virtual correction force vector corresponding to the current time step is output. .

[0059] The obtained optimal virtual correction force vector The virtual actuation source node is assigned to the degrees of freedom of the corresponding node in the finite element mesh according to the predefined virtual actuation source node index, thus forming a virtual correction force vector applied inside the finite element model. This virtual force vector will be applied to the co-rotating finite element model in step 5, driving the finite element model to produce corresponding deformation, so that the position of the model's characteristic observation point is as close as possible to the observation position of the real flexible body, thereby realizing the inverse correction and synchronization of the full-field deformation state of the flexible body.

[0060] Step 5: Apply the virtual correction force vector to the finite element model to drive the finite element model to deform in order to synchronize the physical state of the real flexible body, and update and output the full-field geometric node positions and corresponding tangent stiffness matrices of the flexible body in real time.

[0061] Preferably, step 5 of the present invention specifically includes: (1) Virtual force loading and model state update: The optimal virtual correction force vector calculated in step 4 Virtual actuation source nodes are assigned to the corresponding virtual actuation source nodes in the finite element model according to predefined node indices, and applied to the finite element model as external loads. Let be the mapping matrix between the virtual actuation source degrees of freedom and the global degrees of freedom. Then, the nonlinear static equilibrium equation that needs to be solved in the current time step can be expressed as:

[0062] in, For node location dependent Nonlinear internal elastic force, Given known external loads such as gravity, the above equations are iteratively solved using a nonlinear solver in the simulation engine (e.g., the Newton-Raphson iteration method) to obtain the converged node position vectors. That is, the updated global geometric state.

[0063] (2) Full-field deformation and reconstruction: In this embodiment, as Figure 4 As shown, the flexible body to be tracked is a slender plate-shaped component, with its two ends fixed to clamps to form fixed boundary conditions. Three marker points in the middle of the plate surface are defined as model feature observation points closely related to the overall bending shape, used to reflect the main shape changes of the flexible body. Figure 4 (a) shows the deviation between the initial shape obtained from simulation based solely on the open-loop physical model and the actual deformation state. The upper part is the predicted shape of the finite element simulation model, and the lower part is a schematic diagram of the deformation of the actual flexible body. Figure 4 (b) shows the correction process in which the model moves toward the real shape under the action of the virtual force after the optimal virtual correction force vector is obtained, and the virtual force is applied to several virtual actuation source nodes (shown as 10 virtual driving nodes in this embodiment) that are evenly distributed along the length of the two ends of the flexible body. The arrows show the direction and relative magnitude of the force at each virtual actuation source. Figure 4 (c) shows that under virtual force drive, the model's full-field geometry is corrected to a state that is basically consistent with the observation results of the real flexible body, that is, the deformation tracking and full-field shape correction of this time step are completed.

[0064] During this process, thanks to the physical continuity and mechanical constraints of the finite element model, even in areas not observed visually (including areas between sparse feature points and occluded areas), the internal nodes will undergo deformation in accordance with physical laws under the transmission of virtual forces, thereby achieving robust completion and full-field reconstruction of occluded and observed sparse areas.

[0065] (3) Stiffness parameter update and result output: After obtaining the updated global node positions Subsequently, the full-order tangent stiffness matrix under the current configuration was recalculated and reassembled based on the co-rotation finite element theory. This matrix reflects the instantaneous mechanical properties of the flexible body under its current real deformation state. The system outputs the updated global nodal coordinates and the corresponding tangent stiffness matrix as the tracking result; if necessary, the corresponding low-dimensional flexibility matrix or deformable Jacobian matrix can be obtained again using static condensation techniques based on the new configuration for inverse solving or external control tasks in the next time step. Figure 1As shown, after completing the output of the current time step, the system determines whether to continue tracking. If the task is not finished, it returns to the dimension reduction mapping step, acquires the next frame image, and enters the solution loop for the next time step based on the currently updated finite element model state, thereby realizing continuous real-time tracking of the dynamic deformation process of the flexible body.

[0066] Furthermore, in one specific embodiment, the flexible body deformation tracking method of the present invention, which fuses visual sparse observation with a physical model, is applied to, for example... Figure 2 The flexible body deformation tracking experimental platform shown. Figure 2 This is a schematic diagram of the hardware experimental platform structure for flexible body deformation tracking in an embodiment of the present invention. It includes a binocular vision acquisition unit 1 (composed of a pair of telecentric cameras and their matching lens assemblies), micromanipulation robots (2, 5) for manipulating the flexible body to produce the desired deformation, a flexible body specimen 3 (in this embodiment, a flexible circuit board substrate polyimide flexible sheet), a clamping fixture 4 for holding and fixing both ends of the flexible body, and a high-rigidity optical platform 6 for fixing the above modules. The clamping fixture 4 is installed at the end of the robot. In this embodiment, the flexible body specimen 3 uses a flexible circuit board substrate polyimide flexible sheet.

[0067] In this embodiment, the two ends of the flexible body specimen 3 are rigidly fixed to the ends of the micromanipulation robots (2, 5) by clamps 4. By controlling the end poses of the two robots, the flexible body can undergo various deformations. The binocular vision acquisition unit 1 is fixedly mounted on the bracket and establishes a spatial correspondence with the world coordinate system pre-established on the optical platform 6 through system calibration, so as to ensure that the three-dimensional coordinates of the acquired feature points can correspond one-to-one with the coordinate system of the finite element model and its feature observation points.

[0068] In actual operation, the host computer, serving as a unified computing platform, communicates with the micro-manipulation robots (2, 5) and the binocular vision acquisition unit 1. The host computer integrates a robot motion control module, a vision processing module, and a physical simulation and shape correction module. The motion control module drives the robots (2, 5) to generate desired pose changes based on preset trajectories or operational commands. The vision processing module triggers the binocular camera 1 to synchronously acquire images at each time step and completes feature extraction and 3D reconstruction. The physical simulation and shape correction module executes the algorithm flow described in steps 1 to 5 in real time, updating and correcting the full-field deformation state of the flexible body. Based on this experimental platform, the method of this invention can continuously output physical state information such as the full-field geometric shape and tangent stiffness matrix of the flexible body even when it undergoes large geometric nonlinear deformation. This information can be used for online monitoring of shape and posture changes during the assembly process of the flexible body, and can also provide real-time feedback data for subsequent modules such as shape control, interference collision detection, and path planning.

[0069] In summary, this invention achieves high-precision online tracking of the full-field geometry of flexible bodies by deeply combining sparse visual observation with a dimensionality-reduced physical model, while ensuring real-time performance.

[0070] The detailed descriptions listed above are merely illustrative examples of feasible embodiments of the present invention and are not intended to limit the scope of protection of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that other specific forms can be adopted without departing from the spirit or essential characteristics of the present invention. For example, although this embodiment uses a flexible printed circuit board as an example, the method of the present invention is equally applicable to deformation tracking of other flexible bodies with geometrically nonlinear characteristics, such as biological tissues and cable harnesses; furthermore, although this embodiment uses a binocular telecentric vision system, other vision or sensing devices capable of providing high-precision three-dimensional feature point coordinates can also be used as alternatives. Therefore, in all respects, the above embodiments should be considered exemplary rather than restrictive. The scope of protection of the present invention is defined by the appended claims and should not be limited by the specific embodiments described in the specification. Any equivalent substitutions or modifications made within the spirit and principles of the present invention should fall within the scope of protection of the present invention.

Claims

1. A method for tracking the deformation of flexible bodies by fusing visual sparse observation with a physical model, characterized in that, Includes the following steps: Step 1: Construct a flexible body co-rotation finite element model considering geometric nonlinearity. Define model feature observation points in the finite element model that correspond one-to-one with the feature points on the surface of the real flexible body, and arrange several virtual actuation sources on the surface or inside the flexible body. Step 2: Use static condensation technique to reduce the dimensionality of the full-order tangent stiffness matrix of the finite element model, and construct a low-dimensional flexibility coupling model that describes the transmission relationship between the force applied by the virtual actuation source and the displacement generated by the characteristic observation point of the model. Step 3: During real-time tracking, images of the target flexible body are acquired using a binocular vision system, and the three-dimensional coordinates of sparse feature points on the surface of the flexible body in the real physical world are calculated using a feature extraction and matching algorithm based on clustering screening and topological constraints. Step 4: Using the true 3D coordinates of the sparse feature points as the tracking target, construct and solve the inverse optimization problem based on the low-dimensional flexibility coupling model, and calculate the optimal virtual correction force vector that minimizes the positional deviation between the model feature observation points and the true sparse feature points. Step 5: Apply the optimal virtual correction force vector to the finite element model to drive the finite element model to deform in order to synchronize the physical state of the real flexible body, and update and output the full-field geometric node positions and corresponding tangent stiffness matrices of the flexible body in real time.

2. The flexible body deformation tracking method based on the fusion of visual sparse observation and physical model according to claim 1, characterized in that, Step 1 involves constructing a flexible body co-rotation finite element model that considers geometric nonlinearity, specifically including: The co-rotation coordinate method is used to describe the deformation motion of finite element mesh elements, and the total displacement of the elements in the global coordinate system is decomposed into rigid body rotation components and purely elastic deformation components. The linear element stiffness matrix is ​​calculated in the local coordinate system that rotates with the element, and then mapped back to the global coordinate system through coordinate transformation to assemble the full-order tangent stiffness matrix that depends on the current geometry.

3. The flexible body deformation tracking method based on the fusion of visual sparse observation and physical model according to claim 1, characterized in that, Step 2 involves constructing a low-dimensional flexibility coupling model, specifically including: Obtain the full-order tangent stiffness matrix, rearrange the rows and columns of the matrix according to the nodal degree of freedom type, and divide it into a sub-matrix of retained degree of freedom stiffness corresponding to the virtual actuation source and the model feature observation point, and a sub-matrix of internal degree of freedom stiffness corresponding to the remaining nodes. The Schur complement method is used to operate on the block-wise matrix to eliminate internal degrees of freedom, and the reduced stiffness matrix containing only the retained degrees of freedom is calculated. Based on this, the corresponding flexibility matrix is ​​obtained.

4. The flexible body deformation tracking method based on the fusion of visual sparse observation and physical model according to claim 3, characterized in that, Constructing a low-dimensional flexible coupling model also includes: Based on the compliance matrix Establish a description of the virtual correction force vector Deviation vector between model feature observation point position and the model feature observation point position The equation for the linear mapping relationship between them is: in, Let be the free deformation deviation vector under the action of no virtual correction force.

5. The flexible body deformation tracking method based on the fusion of visual sparse observation and physical model according to claim 1, characterized in that, In step 3, the binocular vision system uses dual telecentric cameras to eliminate perspective errors in small fields of view and improve measurement accuracy.

6. The flexible body deformation tracking method based on the fusion of visual sparse observation and physical model according to claim 1, characterized in that, Step 3, the feature extraction and matching algorithm based on clustering screening and topological constraints, specifically includes: Adaptive thresholding and morphological opening operations are performed on the acquired left and right images respectively to extract several candidate contours; Calculate the geometric roundness features and local region brightness ratio features of each candidate contour to construct a multidimensional feature vector; The K-Means clustering algorithm is used to perform binary classification on the multidimensional feature vectors. Based on the class center score, candidate contours belonging to environmental interference are automatically eliminated, and the true feature point contours are retained.

7. The flexible body deformation tracking method based on the fusion of visual sparse observation and physical model according to claim 6, characterized in that, The feature extraction and matching algorithm also includes: Using the Hungarian algorithm combined with geometric topological consistency constraints, stereo matching is performed on the contours of real feature points in the left and right images, and their three-dimensional coordinates in the real physical world coordinate system are calculated by linear triangulation.

8. The flexible body deformation tracking method based on the fusion of visual sparse observation and physical model according to claim 4, characterized in that, Step 4, which involves constructing and solving the inverse optimization problem, specifically includes: Construct a quadratic optimization objective function that includes a position tracking error term and an energy regularization term: in, For virtual correction force vector, This is the positional observation deviation vector between the real sparse feature points and the model feature observation points. For the flexibility matrix, Let be the free deformation deviation vector under the action of no virtual correction force. The regularization coefficient is used. The objective function is solved iteratively using a numerical optimization solver, and the optimal virtual correction force vector is output when the convergence condition is met.

9. The flexible body deformation tracking method based on the fusion of visual sparse observation and physical model according to claim 1, characterized in that, The flexible body is a deformable body with linear elasticity of material and nonlinear deformation characteristics of geometric deformation.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor implements the method as described in any one of claims 1 to 9.

Images