Assembling alignment error detection method based on multi-sensor fusion and cross attention

Abstract

The present application relates to the technical field of industrial data processing, and discloses a method for detecting assembly alignment error based on multi-sensor fusion and cross attention. In view of the problems of insufficient detection accuracy and poor robustness caused by the mispositioning of heterogeneous feature spaces, the loss of micro-gap features, and the singularity of Euclidean space pose regression in the existing multi-modal detection scheme, the method comprises the following steps: obtaining two-dimensional texture images and three-dimensional point cloud data of an assembly area and constructing a spatial consistency representation; extracting surface texture features and local geometric topology features by using a multi-scale feature extraction network; performing deep coupling of two-dimensional texture image features and three-dimensional point cloud data features by using a cross attention fusion engine; strengthening the assembly gap feature response by using a differential geometric feature enhancement module; inputting the enhanced features into a high-precision error regression head integrated with a spatial flow pattern constraint, and outputting six-degree-of-freedom error parameters. The present application improves the precision and robustness of precision assembly alignment detection.

Description

Technical Field

[0001] This invention relates to the field of industrial data processing technology, specifically to a method for detecting assembly alignment errors based on multi-sensor fusion and cross-attention. Background Technology

[0002] In industrial scenarios involving the assembly of side panels for train carriages, multiple large metal sheets require welding or riveting for assembly. The uniformity of the gaps and the accuracy of edge alignment in the side panel assembly directly affect the subsequent welding quality and the overall structural strength of the carriage. Currently, gap detection in side panel assembly relies primarily on manual visual inspection or spot checks using handheld measuring tools. Manual inspection is inefficient and struggles to cover the continuous state of the entire seam. For flexible thin panels, localized micro-deformations caused by their own weight or clamp stress cannot be effectively identified by visual means, particularly micrometer-level gap fluctuations. Automated inspection schemes based on single two-dimensional vision are susceptible to interference from oil stains, oxide layers, and strong light reflections from the workshop environment on the side panel surface, leading to abrupt changes in gap width measurements. Furthermore, the perspective projection effect of two-dimensional imaging nonlinearly compresses the spatial scale of large-span seams, making it impossible to reconstruct the misalignment of the assembled surfaces in the depth direction. While 3D point cloud-based detection schemes can acquire depth information, when dealing with narrow assembly gaps, the laser beam is affected by multiple reflections from the metal surfaces on both sides of the gap, causing a sharp decrease in point cloud density and a break in the geometric continuity at the gap, making it difficult to stably extract the center trajectory of the assembly gap. The transient pose shifts caused by low-frequency mechanical vibrations and welding thermal deformation during the assembly of the carriage side panels further exacerbate the difficulty of spatial alignment of multi-source data. Existing fusion architectures with fixed calibration parameters cannot compensate for these dynamic drifts in real time, resulting in spatial misalignment of heterogeneous features during the fusion stage, leading to insufficient repeatability and reliability of the detection results.

[0003] Meanwhile, in industrial scenarios such as aero-engine blade assembly and high-density semiconductor packaging, the relative pose of components in three-dimensional Euclidean space determines the operational reliability of the mechanical system. Existing automated inspection methods rely on single-modal sensors. Industrial vision sensors are limited by optical imaging mechanisms; when faced with anisotropic reflections from metal surfaces, the edge gradient information of the acquired two-dimensional texture images is distorted, and the perspective projection process leads to nonlinear compression of spatial scale. When processing narrow assembly gaps, the three-dimensional point cloud data reconstructed by high-precision three-dimensional laser scanning sensors suffers from laser beam obstruction and multiple reflections, resulting in a sharp decrease in sampling density and high-frequency noise.

[0004] Existing multimodal fusion detection schemes employ an early channel stitching architecture. The pixel coordinate system of 2D texture images differs fundamentally from the spatial coordinate system of 3D point cloud data. Static calibration parameters cannot compensate for transient drift in sensor physical pose caused by low-frequency mechanical vibrations in industrial environments. Micrometer-level physical drift, magnified by the projection matrix, can lead to spatial misalignment of heterogeneous features in the feature map. Existing deep learning pose estimation models use a mean squared error loss function based on Euclidean distance when regressing error parameters. Linear interpolation in Euclidean space cannot accurately reflect the nonlinear kinematic trajectory of the pose matrix on the Lie group manifold, and the detection model risks outputting physically interfering poses under complex operating conditions. Summary of the Invention

[0005] The purpose of this invention is to provide an assembly alignment error detection method based on multi-sensor fusion and cross-attention, which solves the technical problems of insufficient detection accuracy and poor robustness of existing multimodal detection schemes in precision assembly alignment error detection caused by heterogeneous feature space misalignment, loss of micro gap features, and singularity of Euclidean space pose regression.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: The purpose of this invention is to provide an assembly alignment error detection method based on multi-sensor fusion and cross-attention, which solves the technical problems of insufficient detection accuracy and poor robustness of existing multimodal detection schemes in precision assembly alignment error detection caused by heterogeneous feature space misalignment, loss of micro gap features, and singularity of Euclidean space pose regression.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for detecting assembly alignment errors based on multi-sensor fusion and cross-attention is applied to detection systems including industrial vision sensors and high-precision 3D laser scanning sensors. The method is characterized by the following steps: Step 1: Obtain the two-dimensional texture image and three-dimensional point cloud data of the assembly area. Based on the joint calibration model, unify the two-dimensional texture image and three-dimensional point cloud data to the global working coordinate system and construct a spatial consistency representation of the two-dimensional texture image and three-dimensional point cloud data. Step 2: Use a multi-scale feature extraction network to extract heterogeneous features from the spatial consistency representation. The two-dimensional image branch extracts surface texture features and geometric edge features, while the three-dimensional point cloud branch extracts local geometric topological features through dynamic graph convolution operators. Step 3: Deeply couple the features of the two-dimensional texture image with the features of the three-dimensional point cloud data through a cross-modal bidirectional cross-attention fusion engine. Use the gradient information of the two-dimensional texture image to guide the boundary enhancement of the three-dimensional point cloud data, and use the depth information of the three-dimensional point cloud data to correct the perspective distortion of the two-dimensional texture image. Step 4: Use the differential geometric feature enhancement module to perform weighted processing on the fused features, and enhance the feature response at the assembly gap through the dynamic anisotropic tensor field to obtain enhanced features; Step 5: Input the enhanced features into the high-precision error regression head, and output the six-degree-of-freedom error parameters describing the assembly alignment state. The high-precision error regression head is constrained by a combined loss function during the training phase. This combined loss function integrates spatial manifold constraint terms to restrict rotation prediction to a specific orthogonal group manifold.

[0008] Furthermore, in step 1, the joint calibration model establishes a projective geometric mapping relationship. The coordinates of the global working coordinate system of the three-dimensional space point are projected onto the pixel coordinate system of the two-dimensional texture image through a rigid body transformation consisting of the intrinsic parameter matrix of the industrial vision sensor, the rotation matrix, and the translation vector. A fifth-order nonlinear distortion compensation model is then used to correct the distortion of the pixel coordinates.

[0009] The joint calibration model establishes the projective geometric mapping relationship, and its mathematical expression is:

[0010] in, As a scale factor, The x-coordinate of a pixel in a two-dimensional texture image. The vertical coordinate of a pixel in a two-dimensional texture image. This is the intrinsic parameter matrix of an industrial vision sensor. For rotation matrix, It is a translation vector. Let x be the x-coordinate of a point in three-dimensional space in the global working coordinate system. The ordinate of a point in three-dimensional space in the global working coordinate system. The depth coordinates of a point in 3D space in the global working coordinate system, with superscript. This indicates the matrix transpose.

[0011] Furthermore, in step 2, the two-dimensional image branch employs a deep residual shrinking network. This network integrates an automatic thresholding learning submodule, which uses a soft thresholding function to non-linearly truncate the features. Feature responses below the adaptive learning threshold are set to zero, while those above the threshold are retained and propagated. The mathematical expression for the soft thresholding function mapping the features is:

[0012] in, The output feature values ​​after soft thresholding. For symbolic functions, For input feature values, It is a function with maximum value. The threshold parameter is obtained through adaptive learning.

[0013] Furthermore, in step 2, the 3D point cloud branch extracts features through edge convolution operation. The edge convolution operation concatenates the original feature vector of the center point with the relative offset feature vector of the neighboring points along the channel dimension, and then updates the feature vector of the center point after nonlinear transformation and max pooling.

[0014] The mathematical expression for edge convolution is:

[0015] in, The feature vector is updated after edge convolution. For neighborhood point index, Let K be the set of K nearest neighbors of the center point. To modify the activation function of the linear unit, which is a nonlinear transformation function. The original feature vector of the center point, This is a channel-level splicing operation. is the feature vector of the neighborhood points.

[0016] Furthermore, in step 3, the cross-modal bidirectional cross-attention fusion engine includes an image-guided point cloud attention module and a point cloud-guided image attention module.

[0017] Furthermore, the mathematical expression for the image-guided point cloud attention module is:

[0018] in, For the attention output matrix, For image query matrix, It is a three-dimensional point cloud key matrix. This is a three-dimensional point cloud value matrix. For normalized exponential functions, represents the feature dimension of the key matrix.

[0019] Furthermore, the point cloud-guided image attention module uses the absolute spatial dimension information provided by the 3D point cloud data to correct the scale distortion of the 2D texture image during perspective projection.

[0020] Furthermore, in step 4, the differential geometric feature enhancement module calculates the covariance matrix in the local neighborhood of the point set, and constructs a non-Euclidean metric space based on the inverse of the local covariance matrix to generate a dynamic anisotropic tensor field.

[0021] The mathematical expression for calculating the covariance matrix within the local neighborhood of a point set using the differential geometric feature enhancement module is as follows:

[0022] in, It is a local covariance matrix. For neighborhood point index, For the set of neighborhood point indices, Spatial distance weighting coefficient, Let be the three-dimensional coordinate vector of the neighborhood points. is the centroid coordinate vector.

[0023] Furthermore, the mathematical expression for the enhancement weights guided by the dynamic anisotropic tensor field is:

[0024] in, To enhance the weighting, a natural constant is used. An exponential function with base , where is the three-dimensional coordinate vector of the neighborhood points, and is the three-dimensional coordinate vector of the center point. It is the inverse of the local covariance matrix; when When it is irreversible, Alternative ,in To preset the regularization constant, It is an identity matrix.

[0025] Furthermore, the six-degree-of-freedom error parameters in step 5 include translational deviations along the horizontal, vertical, and depth axes of the global working coordinate system, as well as rotational deviations around the horizontal, vertical, and depth axes of the global working coordinate system.

[0026] Furthermore, the mathematical expression for the spatial manifold constraint term in step 5 is:

[0027] in, For spatial manifold constraints, For matrix logarithmic mapping, To predict the rotation matrix, For a true reference rotation matrix, superscript Indicates matrix transpose. For the Frobenius norm, To predict the translation vector, As the true reference translation vector, For vectors Norm.

[0028] Furthermore, an adaptive modal confidence assessment step is included before step 3: The image confidence weight and point cloud confidence weight are calculated. When the image confidence weight is lower than the preset lower limit, the graph-guided feature space realignment module is triggered to perform spatial correction on the image features.

[0029] Furthermore, the graph-guided feature space realignment module constructs a cross-modal bipartite graph using physical spatial distance and semantic feature distance, and the mathematical expression for the edge weights is:

[0030] Among them, the connection of the first in the cross-modal bipartite graph is... The image node and the first The edge weights of each point cloud node are expressed as constants of nature. An exponential function with base 0. For the first Each image feature point is back-projected into a three-dimensional coordinate vector in the global working coordinate system. Let be the three-dimensional coordinate vector of the i-th point cloud feature point. For vectors Norm, The preset spatial scale hyperparameters, The cosine similarity function is used. For the first Feature vectors of image nodes For the first The feature vector of a point cloud node.

[0031] Furthermore, in step 2, the 3D point cloud branch is aggregated using geodesic distance-weighted features in the dynamic graph convolution operator. The mathematical expression for the geodesic distance weights is: The mathematical expression for the geodesic distance weight is:

[0032] in, For geodesic distance weighting, For the natural constant An exponential function with base and center at the th... The feature vector at each point For the neighborhood point at the th The eigenvectors at each point, where $|\cdot|{2}$ is the L{2} norm of the vector. The preset feature space scale parameters, Point cloud coordinates and point cloud coordinates in three-dimensional space The approximate geodesic distance along the surface flow pattern is a preset geometric spatial scale parameter; when the point cloud coordinates Point cloud coordinates When the angle between the local surface normal vectors is greater than a preset threshold, it is determined that the two points are separated by the assembly gap, and the geodesic distance... It is assigned a preset large value greater than the Euclidean distance.

[0033] Furthermore, in step 5, the high-precision error regression head outputs a dual quaternion. After being forced to satisfy the unit dual quaternion constraint by the normalization layer, the dual quaternion is converted into a rigid body transformation matrix. The mathematical expression of the combined loss function is:

[0034] in, For dual quaternion combination loss function, , , For balance coefficient, This is a normalized vector of quaternions with real parts. The unit quaternion vector corresponding to the actual rotation. For vectors Norm, The predicted translation vector is the actual translation vector. It is the absolute value of the inner product of two quaternions.

[0035] Compared with the prior art, the present invention has the following beneficial effects: The proposed method for detecting assembly alignment errors based on multi-sensor fusion and cross-attention in this invention constructs a cross-modal bidirectional cross-attention fusion engine to deeply couple the gradient information of two-dimensional texture images with the depth information of three-dimensional point cloud data. This achieves bidirectional synergy between texture-guided geometric boundary sharpening and geometrically constrained texture distortion correction, effectively overcoming the problems of insufficient single-modal perception capability and feature space misalignment caused by channel splicing fusion in existing technologies. This significantly improves the robustness and accuracy of assembly alignment error detection under complex working conditions.

[0036] The differential geometry feature enhancement module strengthens the feature response at the assembly gap through a dynamic anisotropic tensor field, ensuring that the geometric discontinuities of the micrometer-level gap are preserved and amplified during feature mapping. This solves the technical challenge of microscopic defect signals being easily submerged by smoothing pooling operations in precision assembly scenarios. The high-precision error regression head integrates spatial manifold constraints, limiting rotation prediction to specific orthogonal group manifolds. This avoids non-rigid deformation singular solutions generated by Euclidean linear regression, ensuring the physical kinematic rationality of the output six-degree-of-freedom error parameters. Furthermore, the adaptive modal credibility assessment and graph-guided feature space realignment mechanism perform inverse feature correction through cross-modal bipartite graphs when image credibility decreases, enhancing the method's self-healing ability under extreme conditions such as strong light reflection and local occlusion.

[0037] This invention utilizes a geodesic distance-weighted feature aggregation method to distinguish between continuous surfaces and gap edges using manifold geometric priors, achieving natural sharpening of gap features while reducing computational overhead. The dual quaternion normalization output scheme fundamentally ensures the rigid body constraint of the pose transformation matrix during the inference stage, simplifying loss function design and improving training convergence stability. These techniques work synergistically to provide a highly reliable and timely end-to-end assembly alignment error detection solution for the field of industrial precision assembly. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0039] Figure 1 This is the main flowchart of the method described in this invention.

[0040] Figure 2 This is a flowchart of the adaptive modal credibility evaluation and graph-guided feature space realignment branch of the present invention.

[0041] Figure 3 This is a flowchart of the geodesic distance weighted feature aggregation of the present invention.

[0042] Figure 4 This is one of the system operation interface diagrams when the method described in this invention is used.

[0043] Figure 5 This is the second diagram of the system operation interface when the method described in this invention is in use.

[0044] Figure 6 This is the third diagram of the system operation interface when the method described in this invention is used. Detailed Implementation

[0045] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0046] The following is in conjunction with the appendix Figures 1-6 The embodiments of the present invention will be described in detail below.

[0047] Example 1: This example discloses a method for detecting assembly alignment errors based on multi-sensor fusion and cross-attention. Its hardware sensing layer is deployed on the end effector of a flexible assembly station. The industrial vision sensor uses a global shutter complementary metal-oxide-semiconductor photosensitive element, with a pixel size set to [missing information]. The optical lens is configured as a low-distortion telecentric lens. The high-precision 3D laser scanning sensor uses a wavelength of... The blue line structured light generator reconstructs surface geometry based on the principle of triangulation. The shorter wavelength of blue light reduces the photon penetration depth of the metal surface, suppressing point cloud coordinate drift caused by subsurface scattering.

[0048] The mechanical vibrations of the on-site environment require multi-source data to coincide on a time slice. The hardware control layer sends high-level trigger pulses through a field-programmable gate array. The exposure timing of the industrial vision sensor and the scanning cycle of the high-precision 3D laser scanning sensor are locked within the same clock domain, and the time synchronization error is controlled at the microsecond level.

[0049] Spatial dimension data alignment relies on a joint calibration model.

[0050] During the system initialization phase, a checkerboard calibration board with known dimensions is placed within the field of view of the assembly station.

[0051] The system simultaneously acquires two-dimensional texture images and three-dimensional point cloud data, and extracts the two-dimensional pixel coordinates and three-dimensional spatial coordinates of the corner points of the chessboard grid.

[0052] Based on multiple sets of corresponding point pairs, the rigid transformation relationship between the optical center of the industrial vision sensor and the emission baseline of the high-precision 3D laser scanning sensor, i.e., the rotation matrix, is solved using the perspective n-point algorithm. With translation vector Intrinsic parameter matrix of industrial vision sensors The projection geometric mapping relationship between a point in three-dimensional space and the projection onto a two-dimensional pixel plane is obtained in advance using Zhang Zhengyou's calibration method. ; in, As a scale factor, The x-coordinate of a pixel in a two-dimensional texture image. The vertical coordinate of a pixel in a two-dimensional texture image. This is the intrinsic parameter matrix of an industrial vision sensor. For rotation matrix, It is a translation vector. Let x be the x-coordinate of a point in three-dimensional space in the global working coordinate system. The ordinate of a point in three-dimensional space in the global working coordinate system. The depth coordinates of a point in 3D space in the global working coordinate system are given by the superscript, which indicates matrix transpose.

[0053] Residual optical distortion at the edge of a telecentric lens violates the linear assumption of projective transformation. A fifth-order nonlinear distortion compensation model is embedded within the joint calibration model to perform inverse differentiation and rearrangement of the pixel coordinates on the imaging plane. The expressions for radial and tangential deformation processing are: ; in, The x-coordinate of the actual pixel after distortion compensation. Let x be the pixel x-coordinate of the ideal projective model. For the pixel ordinate of the ideal projective model, The first-order radial distortion coefficient, The second-order radial distortion coefficient. The third-order radial distortion coefficient. The first-order tangential distortion coefficient, The second-order tangential distortion coefficient. This is the Euclidean distance from the pixel to the optical center. The microscopic edges in the two-dimensional texture image and the geometric faults in the three-dimensional point cloud data achieve spatial overlap with sub-pixel accuracy, thus completing the construction of spatial consistency representation.

[0054] A multi-scale feature extraction network processes spatially aligned heterogeneous data streams. The network architecture is decoupled into two independent tensor evolution branches.

[0055] The input to the 2D image branch is size normalized to Two-dimensional texture image.

[0056] The feature extraction backbone network employs a deep residual shrinking network, containing 18 residual blocks, and the initial convolutional layer uses... The convolution kernel has a stride of 2 and outputs a 64-channel feature map. The subsequent four stages contain 2, 2, 2, and 2 residual blocks, with output channels of 64, 128, 256, and 512, respectively.

[0057] The deep residual shrinking network integrates an automatic thresholding learning submodule into the bypass branch of the standard residual block. This submodule compresses the spatially dimensional receptive field into a one-dimensional channel descriptor using a global average pooling layer. A two-layer fully connected network maps the channel descriptor to a scalar threshold parameter specific to each feature channel.

[0058] The backbone branches of the deep residual shrinking network apply a soft thresholding function to perform element-wise nonlinear truncation of the feature tensors: ; in, The output feature values ​​after soft thresholding. For symbolic functions, For input feature values, It is a function with maximum value. The threshold parameters are obtained through adaptive learning. Low-amplitude features characterizing the reflection of cutting fluid or stray light from the metal surface are set to zero when crossing the soft thresholding function. High-amplitude tensors characterizing the geometric edges of the assembly gaps penetrate the residual blocks with a near-identical mapping. The final output feature map has a dimension of [dimension missing]. .

[0059] The input to the 3D point cloud branch is 3D point cloud data containing absolute depth and spatial geometric topology, with dynamic point sampling. Each point contains three-dimensional coordinates. The dynamic graph convolution operator is applied to the branching of 3D point clouds. The dynamic graph convolution operator dynamically constructs a K-nearest neighbor directed graph at each level of the feature space, with K set to 20.

[0060] The dynamic graph convolution operator calculates the edge features of the central node and its neighboring nodes in the feature space: ; in, The feature vector is updated after edge convolution, and the output dimension is 128. For max pooling operation, For neighborhood point index, Let K be the set of K nearest neighbors of the center point. To modify the activation function of the linear unit, It is a nonlinear transformation function, composed of two layers of perceptrons, with a hidden layer dimension of 64. The original feature vector of the center point, This is a channel-level splicing operation. is the feature vector of the neighborhood points.

[0061] Tensor splicing operation The absolute position encoding of nodes is fused with relative geometric offset. The relative feature displacement includes information on local surface normal vector flipping and high-frequency curvature abrupt changes. Max pooling aggregates local topological features, eliminating the sensitivity to the arrangement of the input point cloud sequence. After stacking four layers of dynamic graph convolution operators, the output point-by-point feature dimension is... .

[0062] Features from 2D texture images and 3D point cloud data are fed into a cross-modal bidirectional cross-attention fusion engine after exiting the multi-scale feature extraction network. This fusion engine includes an image-guided point cloud attention module.

[0063] Image query matrix Image features through linear transformation The three-dimensional point cloud key matrix is ​​obtained. Sum matrix Each point cloud feature is transformed by a linear transformation and get.

[0064] Number of feature channels Key query dimensions Value dimension The image-guided point cloud attention module is as follows: ; in, For the attention output matrix, For image query matrix, It is a three-dimensional point cloud key matrix. This is a three-dimensional point cloud value matrix. is the normalized exponential function, and is the characteristic dimension of the bond matrix.

[0065] Matrix dot product The semantic relevance of heterogeneous modalities in the latent space is calculated. A normalized exponential function transforms the relevance into an attention weight distribution map between 0 and 1. Strong edge regions in the 2D texture image are activated with higher weight scores, forcing the feature flow of the 3D point cloud data to lock the receptive field at physical gaps. The feature mean normalization caused by physical occlusion in the gap region of the laser point cloud is inversely sharpened by the 2D optical texture.

[0066] The cross-modal bidirectional cross-attention fusion engine simultaneously activates the point cloud-guided image attention module. The absolute depth scale inherent in the 3D point cloud data is extracted as a rigid constraint to correct perspective distortion in the 2D image. Bidirectional information flow eliminates the blind spots of a single sensor. The fusion output feature dimension remains unchanged. .

[0067] Example 2: This example, based on the hardware architecture and multi-scale feature extraction network of Example 1, deploys a differential geometric feature enhancement module to address the problem of microscopic feature loss in the assembly and docking gap region. In precision assembly, deviations below the millimeter level manifest as localized minute extrusion deformations on the assembly surface, edge chamfer misalignment, or non-parallel fit. When the feature tensor crosses the pooling layer, the high-frequency features characterizing these minute deformations are assimilated by the low-frequency features of the surrounding smooth surface. The differential geometric feature enhancement module extracts the three-dimensional geometric change gradient of the local neighborhood for directional feature amplification.

[0068] The differential geometric feature enhancement module calculates the covariance matrix within the local neighborhood of the point set: ; in, It is a local covariance matrix. For neighborhood point index, The set of indices of the neighborhood points of the center point. , The spatial distance weighting coefficient is expressed as follows: , For the natural constant An exponential function with base 0. For vectors Norm, For preset bandwidth parameters, Let be the three-dimensional coordinate vector of the neighborhood points. is the centroid coordinate vector of all points in the neighborhood of the center point.

[0069] The local covariance matrix implies the geometric and topological invariants of the local surface.

[0070] Eigenvalue decomposition is performed on the local covariance matrix. The eigenvector corresponding to the smallest eigenvalue indicates the direction of the normal vector of the local surface, while the remaining eigenvalues ​​characterize the surface gradient along the principal curvature direction. The local covariance matrix of the smooth region of the assembled surface exhibits a flattened ellipsoidal distribution, while the local covariance matrix of the edge region of the docking seam exhibits a slender anisotropic distribution.

[0071] The differential geometric feature enhancement module constructs a non-Euclidean geometric metric space based on the inverse of the local covariance matrix. The enhancement weights guided by the dynamic anisotropic tensor field are: ; in, To enhance weight, For the natural constant An exponential function with base 0. Let be the three-dimensional coordinate vector of the neighborhood points. The three-dimensional coordinate vector of the center point, It is the inverse of the local covariance matrix; when When it is irreversible, Alternative The regularization constant is , where , It is an identity matrix.

[0072] Along the smooth transition direction of the curved surface, the spatial distance function is amplified by the inverse of the local covariance matrix, the exponential function output has an enhancement weight close to zero, and the smooth feature response is attenuated. At the edge of the docking seam where the normal vector undergoes a steep abrupt change, the spatial distance metric is compressed, and the seam edge features are given a feature channel amplification factor close to a constant one. The differential geometry feature enhancement module upgrades the Euclidean distance metric to a curvature metric under Riemannian geometry.

[0073] A high-precision error regression head is constructed using a multi-layer fully connected network, receiving high-dimensional feature vectors from a cross-modal bidirectional cross-attention fusion engine and a differential geometric feature enhancement module. The regression head outputs a 6-dimensional vector: the first three dimensions are translation vectors. The latter three dimensions are represented by axis angles. The axis-angle is converted into a rotation matrix using the Rodrigues transformation. .

[0074] Translational deviations belong to three-dimensional Euclidean space, while rotational deviations belong to a special orthogonal group, which is a nonlinear manifold space.

[0075] Linear interpolation of a 3D rotation matrix in Euclidean space breaks the rigid constraints of matrix orthogonality and the requirement that the determinant is equal. The high-precision error regression head accepts parameters for training via a combined loss function with embedded Lie algebraic mappings.

[0076] The spatial manifold constraint terms are: ; in, For spatial manifold constraints, This is a matrix logarithmic mapping that maps elements of a special orthogonal group to their corresponding Lie algebras. To predict the rotation matrix, For a true reference rotation matrix, superscript Indicates matrix transpose. For the Frobenius norm, To predict the translation vector, As the true reference translation vector, For vectors Norm.

[0077] The spatial manifold constraint term uses a matrix logarithmic mapping to map the relative rotation difference between the predicted rotation matrix and the true reference rotation matrix onto a Lie algebra. The Frobenius norm measures the geodesic distance of the rotation parameters in the manifold tangent space, which follows the laws of rigid body kinematics. The algorithmic model is constrained to evolve within a physically meaningful three-dimensional kinematic constraint framework during backpropagation gradient updates.

[0078] Example 3: Example 3 extends the architecture of Example 2 and introduces an adaptive modal reliability assessment mechanism and a graph-guided feature space realignment module to address the dynamic extreme working conditions that may occur in flexible assembly lines.

[0079] Flickering ambient light gratings, momentary vibrations of the robot's end effector axis, or specular reflections from the cutting fluid surface can cause transient failures in a single sensor's sensing link. Fixed-weight cross-modal fusion mechanisms can suffer systemic detection failures due to interference from low-quality modes.

[0080] An adaptive modal confidence assessment mechanism monitors in real time the local contrast statistics and high-frequency signal-to-noise ratio estimates of the feature maps output by industrial vision sensors, and monitors the local point cloud density distribution and curvature consistency variance output by high-precision 3D laser scanning sensors. It dynamically calculates the current image confidence weight and point cloud confidence weight by combining the real-time quality assessment vector and the Cramer-Rhodes lower bound.

[0081] When partial occlusion or direct strong light causes the image confidence weight to fall below a preset lower limit of 0.3, the graph-guided feature space realignment module is triggered. This module constructs a cross-modal bipartite graph using physical spatial distance and high-dimensional semantic feature distance, with edge weights as follows: ; in, Connecting nodes in a cross-modal bipartite graph The edge weights of nodes are given by natural constants. An exponential function with base 0. Let be the coordinate vector of the back projection of the 2D texture image node into 3D space, and be the coordinate vector of the 3D point cloud data node. The spatial scale parameter is set to 0.05m. The cosine similarity function is used. For image feature vectors, This represents the feature vector of the point cloud.

[0082] Two-dimensional textured image nodes, distorted by strong light interference, are on the verge of semantic feature collapse. A graph-guided feature space realignment module forces the failed image nodes to absorb the geometric and topological features of their corresponding high-confidence point cloud nodes based on the edge weight network. Spatial geometric distance excludes false matches that are physically far apart but semantically similar, while cosine similarity measure excludes outlier nodes that are physically close but cross assembly faults.

[0083] The experimental setup was constructed at the robotic automated assembly station for high-pressure compressor blades of an aero-engine. The blades are made of titanium alloy with extremely low surface roughness, exhibiting anisotropic metallic reflection characteristics. The assembly area included irregular frequency ambient lighting and fundamental frequency vibration interference transmitted by the industrial robotic arm itself.

[0084] The training dataset was constructed taking into account the optical complexity of industrial environments. To address the anisotropic reflectivity of metal surfaces, a physically based rendering engine using a bidirectional reflectance distribution function was employed to generate synthetic data. This was achieved by adjusting surface roughness parameters. and Fresnel reflection coefficient Simulates a continuous spectrum from specular reflection to diffuse reflection. A Gaussian-Poisson mixed noise model is injected to simulate laser speckle effects. The INT8 quantization calibration set is uniformly sampled from the enhanced dataset for 2000 frames according to the operating condition distribution.

[0085] The experimental comparison team incorporated two technical approaches.

[0086] Comparative Example 1: A scale-invariant feature transform algorithm is used to extract feature points from a 2D image. Then, a point-to-plane variant from the iterative nearest-point algorithm is used for 3D point cloud alignment. The iteration terminates when the relative pose change is less than 1. Or it can reach the maximum number of iterations, 50.

[0087] Comparative Example 2: The same multi-scale feature extraction backbone network as in Example 1 is used, but the cross-modal fusion part is replaced with channel-dimensional concatenation. The regression loss function is the standard mean squared error loss function. .

[0088] Embodiment 3 of the present invention adopts a complete scheme including adaptive modal confidence assessment, graph-guided realignment, differential geometry enhancement, and manifold constraint loss.

[0089] The test set covers 3000 assembly processes with actual measured laser tracker calibration values. The extracted inspection metrics include root mean square error of translation, root mean square error of rotation, average inspection time, and success rate under reflective conditions. The success rate under reflective conditions is defined as the predicted translation error being less than [value missing] under simulated cutting fluid mirror reflection conditions. And the predicted value of rotational error is less than The percentage of the total number of samples under a specific working condition.

[0090] Table 1. Comparison of assembly alignment accuracy and performance of each scheme;

[0091] Comparative Example 1 lacks the nonlinear extraction capability of deep learning for high-dimensional features, resulting in significant feature matching failures when dealing with anisotropic reflective conditions. Comparative Example 2 suffers from a lack of bidirectional feedback guidance in the cross-modal stage of heterogeneous data streams, and high-frequency illumination noise from two-dimensional texture images contaminates the joint features after channel stitching. The Euclidean mean square error loss function exhibits instability when regressing a three-dimensional rotation matrix.

[0092] Example 3: Root mean square error of horizontal axis translation compressed to... The root mean square error of the depth axis translation is The spatial manifold constraint term reshapes the loss surface of the high-precision error regression head, and the root mean square error of the horizontal axis rotation converges to... The success rate of reflective condition testing remained at [percentage missing]. The end-to-end average detection time is locked at... .

[0093] Example 4: This example provides a gap enhancement scheme that differs from the existing technology based on covariance matrix decomposition. It uses dynamic graph convolution based on geodesic distance to achieve manifold perception enhancement of assembly gap features.

[0094] Example 4 can be run independently or in combination with the aforementioned examples.

[0095] In the dynamic graph convolution operator, the Euclidean distance weights used in neighborhood aggregation for the 3D point cloud branch are replaced with geodesic distance weights in the feature space. Points on both sides of the assembly gap are relatively close in 3D Euclidean space, but are separated by the gap in the surface manifold, and the geodesic distance is greater than the Euclidean distance. This property can be used to distinguish between smooth, continuous surfaces and abrupt gap edges during feature aggregation.

[0096] Specifically as follows: Step 4-1: Construct the K-nearest neighbor graph in the feature space. For the point-by-point feature set output from the 3D point cloud branch, calculate the Euclidean distance between each point in the feature space, and select the feature with the closest feature distance for each point. Construct a directed K-nearest neighbor graph from the neighboring points.

[0097] Step 4-2: Calculate the geodesic distance weights for manifold patterns. For the center point... Its characteristic neighborhood points Define manifold geodesic distance weights : ; in, For geodesic distance weighting, For the natural constant An exponential function with base 0. The feature vector of the center point, For the feature vector of the neighborhood points, For vectors Norm, The feature space scale parameter is set to 0.5 times the standard deviation of the feature distance. Midpoint in three-dimensional space With point Approximate geodesic distance along the surface flow pattern, The geometric scale parameter is set to twice the average sampling interval of the point cloud.

[0098] Approximate calculation method for geodesic distance: If point With point The angle between the normal vectors is less than a preset threshold. Furthermore, if the Euclidean distance between two points is less than twice the local radius of curvature, the two points are considered to lie on the same continuous surface. Otherwise, it is determined that the two points are separated by assembly gaps or geometric edges, and the geodesic distance is set to... .

[0099] Step 4-3: Geodesic Weighted Feature Aggregation. In the edge convolution operation of the dynamic graph convolution, max pooling is modified to geodesic weighted pooling: ; in, For the updated feature vector, For geodesic distance weighting, To modify the activation function of the linear unit, It is a nonlinear transformation function. The original feature vector of the center point, This is a channel-level splicing operation. is the feature vector of the neighborhood points.

[0100] Geodesic distance weights differentiate between continuous surface neighborhoods and gap edge neighborhoods using manifold geometric priors. For continuous surface points on the same side of the assembly gap, the geodesic distance is approximately equal to the Euclidean distance, resulting in a weight close to 1 and sufficient feature aggregation. Points on opposite sides of the gap, while having a closer Euclidean distance, experience abrupt changes in their normal vectors, amplifying the geodesic distance and causing the weight to approach zero, thus hindering feature aggregation. Features on either side of the gap edge cannot be smoothed out, achieving gap sharpening enhancement at the feature level.

[0101] Compared with the differential geometric feature enhancement scheme described in Example 3, this embodiment has the following advantages: The time-to-time single-frame inference latency was reduced from 28.8ms to approximately 23.6ms, and the video memory usage was reduced by approximately [missing information]. .

[0102] Example 5: This example provides a pose regression scheme that differs from those based on Lie group manifold constraint loss functions, employing an end-to-end rigid body pose estimation method based on dual quaternions. Example 5 can be run independently or used in combination with the aforementioned examples.

[0103] Dual quaternions represent arbitrary rigid body transformations in three-dimensional space using eight real components, with inherent unit modulus constraints and orthogonal constraints. They are the most compact and singularity-free parameterized form of special Euclidean groups.

[0104] Step 5-1: Design the dual quaternion regression head. Modify the output dimension of the high-precision error regression head to 8 dimensions, denoted as... The first four dimensions are the real parts of the dual quaternions, representing rotations, and the last four dimensions are the dual parts, encoding translation information.

[0105] Step 5-2: Apply unit dual quaternion normalization constraints. Add a normalization layer to the regression head output and perform real part normalization. Ensure the real part is a unit quaternion, and project the dual part onto a subspace orthogonal to the real part. The normalized dual quaternion is denoted as... , As a dual unit, satisfying and .

[0106] Step 5-3: Transformation from dual quaternions to rigid body transformation matrices. Rotation matrix. From the real part Calculated using the standard quaternion to rotation matrix formula. Translation vector. Calculated using both the real part and the dual part: , To represent quaternion multiplication, for The conjugate quaternion.

[0107] Step 5-4: Loss Function Design. After normalization, the dual quaternion naturally guarantees that the output belongs to a special Euclidean group. The loss function simplifies to directly supervise the regression accuracy of each component of the dual quaternion with the translation vector: ; in, For dual quaternion combination loss function, , , The balance coefficients are set to 1.0, 1.0, and 0.5 respectively. This is a normalized vector of quaternions with real parts. The unit quaternion vector corresponding to the actual rotation. For vectors Norm, To predict the translation vector, For the true translation vector, It is the absolute value of the inner product of two quaternions, used to eliminate the ambiguity of quaternion symbols.

[0108] The unit dual quaternion normalization layer forces the network output to satisfy rigid body kinematics constraints during the inference phase. Dual quaternions encode rotations and translations as a single mathematical object with a dimension of 8, avoiding the need for axis-angle representations. and This addresses the singularity problem. The loss function only measures the distance between the predicted and true values, eliminating the need to calculate the logarithmic mapping and Frobenius norm, resulting in a shorter backpropagation gradient path.

[0109] Compared to the scheme in Example 2 based on the Lie group manifold constraint loss function, the output in this example is explicitly subject to normalization constraints during the inference stage, avoiding logarithmic mapping at rotation angles close to the normalization constraint. The numerical values ​​at that time were singular, and the training convergence speed was improved by approximately .

[0110] Example 6: This example is used for aircraft engine casings or large rotating body components with complex full-circumference geometry. Single-view sensors have inherent physical blind spots, and the protruding structures of the assembled components themselves create self-occlusion. Example 6 constructs a multi-viewpoint global topology optimization sensing array.

[0111] A multi-view global topology optimization sensing array is deployed at equal intervals on a ring-shaped assembly fixture. Each node is a photoelectric sensing node. Each node contains a set of rigidly bonded industrial vision sensors and high-precision 3D laser scanning sensors. Each node independently performs asynchronous data acquisition, and the central computing hub allocates absolute timestamps through a precise time protocol.

[0112] The core of multi-view data fusion lies in eliminating the distributed calibration accumulation error of heterogeneous coordinate systems across multiple devices. The system establishes a global nonlinear cost function based on graph optimization. The vertices of the graph optimization network are divided into two categories: The extrinsic pose matrix of each photoelectric sensing node in the global reference coordinate system and the coordinates of three-dimensional spatial physical feature points on the surface of the assembled parts. The optimized edges of the graph are composed of the observation residuals from each sensor.

[0113] The global pose graph optimization energy function is: ; Among these, the energy scalar is optimized for the global graph. This represents the total number of photoelectric sensing nodes. This represents the total number of extracted 3D physical feature points. For robust kernel functions, Let be the projective mapping function of the i-th industrial vision sensor. For the first The pose transformation matrix of each node relative to the global coordinate system. For the first The three-dimensional coordinate vector of each feature point For the first The feature point at the th ... Two-dimensional pixel observations on a sensor plane For the distance to Maharanobis, To observe the covariance matrix, For depth residual weighting factor, This represents the theoretical depth component of the feature points transformed to the camera coordinate system. The actual physical depth observation value measured by the sensor. It is the square of the Euclidean distance.

[0114] The energy function simultaneously constrains the two-dimensional reprojection error of image features and the one-dimensional depth residual of point cloud features. A robust kernel function suppresses mismatched extrema caused by metallic mirror reflections. The central computing hub uses the Levenberg-Marquardt algorithm to iteratively solve the energy function.

[0115] The multi-scale feature extraction network adds a viewpoint-aware attention layer in the spatial dimension after receiving the globally aligned data array. The viewpoint-aware attention layer evaluates the visibility and observation signal-to-noise ratio of different nodes to the current assembly gap region, and calculates the node weights. ; in, For the first The global fusion weight of each node, For learnable attention weight matrix, For the first Each node extracts high-dimensional pooled features from the region of interest during assembly. Nodes whose features degrade due to self-occlusion or local reflection are assigned fusion weights approaching zero. Features from nodes at the optimal observation angle dominate subsequent processing.

[0116] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0117] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting assembly alignment error based on multi-sensor fusion and cross-attention, characterized in that, An inspection system applicable to industrial vision sensors and high-precision 3D laser scanning sensors includes the following steps: Step 1: Obtain the two-dimensional texture image and three-dimensional point cloud data of the assembly area. Based on the joint calibration model, unify the two-dimensional texture image and three-dimensional point cloud data to the global working coordinate system and construct a spatial consistency representation of the two-dimensional texture image and three-dimensional point cloud data. Step 2: Use a multi-scale feature extraction network to extract heterogeneous features from the spatial consistency representation. The two-dimensional image branch extracts surface texture features and geometric edge features, while the three-dimensional point cloud branch extracts local geometric topological features through dynamic graph convolution operators. Step 3: Deeply couple the features of the two-dimensional texture image with the features of the three-dimensional point cloud data through a cross-modal bidirectional cross-attention fusion engine. Use the gradient information of the two-dimensional texture image to guide the boundary enhancement of the three-dimensional point cloud data, and use the depth information of the three-dimensional point cloud data to correct the perspective distortion of the two-dimensional texture image. Step 4: Use the differential geometric feature enhancement module to perform weighted processing on the fused features, and enhance the feature response at the assembly gap through the dynamic anisotropic tensor field to obtain enhanced features; Step 5: Input the enhanced features into the high-precision error regression head, and output the six-degree-of-freedom error parameters describing the assembly alignment state. The high-precision error regression head is constrained by a combined loss function during the training phase. This combined loss function integrates spatial manifold constraint terms to restrict rotation prediction to a specific orthogonal group manifold.

2. The assembly alignment error detection method based on multi-sensor fusion and cross-attention according to claim 1, characterized in that, In step 1, the joint calibration model establishes a projective geometric mapping relationship. The coordinates of the global working coordinate system of the three-dimensional space point are projected onto the pixel coordinate system of the two-dimensional texture image through a rigid body transformation consisting of the intrinsic parameter matrix of the industrial vision sensor, the rotation matrix, and the translation vector. A fifth-order nonlinear distortion compensation model is then used to correct the distortion of the pixel coordinates.

3. The assembly alignment error detection method based on multi-sensor fusion and cross-attention according to claim 1, characterized in that, In step 2, the two-dimensional image branch uses a deep residual shrinking network. The deep residual shrinking network integrates an automatic thresholding learning submodule, which performs non-linear truncation on the features through a soft thresholding function, setting the feature responses below the adaptive learning threshold to zero, while keeping the feature responses above the threshold propagated.

4. The assembly alignment error detection method based on multi-sensor fusion and cross-attention as described in claim 1, characterized in that, In step 2, the 3D point cloud branch extracts features through edge convolution operation. The edge convolution operation concatenates the original feature vector of the center point with the relative offset feature vector of the neighboring points along the channel dimension. After nonlinear transformation and max pooling, the feature vector of the center point is updated.

5. The assembly alignment error detection method based on multi-sensor fusion and cross-attention according to claim 1, characterized in that, In step 3, the cross-modal bidirectional cross-attention fusion engine includes an image-guided point cloud attention module and a point cloud-guided image attention module.

6. The assembly alignment error detection method based on multi-sensor fusion and cross-attention according to claim 5, characterized in that, The image-guided point cloud attention module performs a dot product operation between the image query matrix and the 3D point cloud key matrix, and generates an attention weight allocation map using a normalized exponential function. The 3D point cloud value matrix is ​​then weighted and summed using this weight allocation map. The mathematical expression for this summation is as follows: in, For the attention output matrix, For image query matrix, It is a three-dimensional point cloud key matrix. This is a three-dimensional point cloud value matrix. For normalized exponential functions, represents the feature dimension of the key matrix.

7. The assembly alignment error detection method based on multi-sensor fusion and cross-attention according to claim 5, characterized in that, The point cloud-guided image attention module uses the absolute spatial dimension information provided by 3D point cloud data to correct the scale distortion of 2D texture images during perspective projection.

8. The assembly alignment error detection method based on multi-sensor fusion and cross-attention according to claim 1, characterized in that, In step 4, the differential geometric feature enhancement module calculates the covariance matrix in the local neighborhood of the point set, and constructs a non-Euclidean metric space based on the inverse of the local covariance matrix to generate a dynamic anisotropic tensor field.

9. The assembly alignment error detection method based on multi-sensor fusion and cross-attention according to claim 8, characterized in that, The enhancement weights guided by the dynamic anisotropic tensor field are based on the definition of Mahalanobis distance, and their mathematical expression is as follows: in, To enhance weight, For the natural constant An exponential function with base 0. Let be the three-dimensional coordinate vector of the neighborhood points. The three-dimensional coordinate vector of the center point, with superscript Represents the transpose of a vector or matrix. It is the inverse of the local covariance matrix; when When it is not invertible, use a regularized inverse matrix. Alternative , To preset the regularization constant, To and An identity matrix of the same dimension.

10. The assembly alignment error detection method based on multi-sensor fusion and cross-attention according to any one of claims 1-9, characterized in that, In step 5, the spatial manifold constraint term maps the relative rotation difference between the predicted rotation matrix and the true reference rotation matrix to the Lie algebra space through matrix logarithmic mapping, and measures the geodesic distance using the Frobenius norm. Its mathematical expression is: in, For spatial manifold constraints, For matrix logarithmic mapping, To predict the rotation matrix, For a true reference rotation matrix, superscript Indicates matrix transpose. For the Frobenius norm, To predict the translation vector, As the true reference translation vector, For vectors Norm.

Images