Pose prediction method and system for robot cable assembly based on multi-modal fusion

Abstract

The application discloses a kind of pose prediction methods and systems based on multimodal fusion for robot cable assembly, it is related to robot vision and intelligent assembly technical field.The method includes the following steps: obtaining the RGB image and original point cloud data of the cable to be assembled;Feature extraction is carried out on the preprocessed original point cloud data;Adaptive identification is carried out on the multi-modal features using a density-based spatial clustering algorithm;Terminal recognition is carried out on the terminal cluster and the RGB image using a double-branch multi-modal neural network;A centerline extraction algorithm based on graph theory is used to calculate the centerline of the cable cluster, and the topological relationship between the terminal and the cable is established according to the centerline calculation result and the terminal pose;According to the topological relationship between the terminal and the cable, the accurate pose of the mechanical arm assembly cable process is obtained combined with the clamp cluster.The application solves the sensitivity problem of traditional clustering methods to point density changes and direction information loss.

Description

Technical Field

[0001] This invention relates to the fields of robot vision and intelligent assembly technology, and in particular to a pose prediction method and system for robot cable assembly based on multimodal fusion. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In automated cable assembly production, the identification and positioning of flexible cables remains a critical challenge. The cables themselves have small cross-sectional dimensions and long longitudinal extensions, and the material itself has a low elastic modulus, leading to continuous nonlinear deformation under gravity, clamping forces, and inertia. Furthermore, their surface coating is a monochromatic matte material with weak texture gradients. Under two-dimensional imaging conditions, the grayscale distribution is approximately uniform, and the edge gradient signal-to-noise ratio is lower than the conventional threshold of industrial cameras. This makes it impossible to establish a stable feature-coordinate mapping using traditional methods based on grayscale or texture for edge extraction, template matching, and contour fitting.

[0004] Existing 3D point cloud clustering algorithms have several limitations when applied to automated cable assembly scenarios. For example, DBSCAN relies on density reachability; when a cable sags or bends to form sparse segments, the number of core points drops instantly below the neighborhood threshold, causing the same cable to be segmented into multiple clusters. K-Means uses Euclidean distance as the sole metric; lacking directional constraints, it groups spatially adjacent but physically different clamps, cables, and terminals to the same centroid, resulting in structural misclassification. Further complicating matters, cable insulation and metal terminals have similar reflectivity in the visible light band, and their hue and saturation differences are less than the color quantization step size, making reliable segmentation impossible when relying solely on color clustering. If only depth discontinuity features are used, the overlapping of clamps and terminals, the overlapping of depth abrupt changes, and the spatial misalignment of geometric and color edges further amplify the probability of missegmentation. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a pose prediction method and system for robot cable assembly based on multimodal fusion. Through an adaptive clustering algorithm based on multimodal feature fusion, the recognition accuracy is significantly improved in high-noise and occluded environments, and the sensitivity of traditional clustering methods to changes in point density and lack of orientation information is solved.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution: The first aspect of this invention provides a pose prediction method for robot cable assembly based on multimodal fusion, comprising the following steps: Acquire the RGB image and raw point cloud data of the cable to be assembled, and preprocess the raw point cloud data and RGB image; Feature extraction is performed on the preprocessed raw point cloud data and the corresponding RGB image to obtain multimodal features; A density-based spatial clustering algorithm is used to adaptively identify multimodal features, resulting in cable clusters, terminal clusters, and fixture clusters. A dual-branch multimodal neural network is used to identify terminals from terminal clusters and RGB images to obtain terminal poses; A graph theory-based centerline extraction algorithm is used to calculate the centerline of the cable cluster, and the topological relationship between the terminal and the cable is established based on the centerline calculation results and the terminal pose. The precise pose of the robotic arm during cable assembly is obtained by combining the topological relationship between the terminals and the cables with the fixture cluster.

[0007] Further, the specific steps for preprocessing the original point cloud data and RGB images are as follows: Image denoising and contrast enhancement are performed on RGB images, and voxel downsampling, outlier removal and plane segmentation are performed on the original point cloud data.

[0008] Furthermore, the specific steps for feature extraction from the preprocessed raw point cloud data and the corresponding RGB images are as follows: Geometric, color, and statistical features are extracted from the original point cloud data and the corresponding RGB images, and a multimodal feature matrix is ​​constructed. Normalize the multimodal features.

[0009] Furthermore, the specific steps for adaptive identification of multimodal features using density-based spatial clustering algorithms are as follows: Orientation consistency constraints are set based on the composite distance of multimodal features; Adaptive clustering of different regions based on directional consistency constraints; Post-cluster processing is performed based on the adaptive clustering results.

[0010] Furthermore, the specific steps for terminal identification using a dual-branch multimodal neural network on terminal clusters and RGB images are as follows: Point cloud branch performs point cloud feature extraction on terminal clusters; The image branch extracts image features from the RGB images of the terminals; The point cloud features and image features are stitched together.

[0011] Furthermore, the specific steps for calculating the centerline of the cable cluster using a graph theory-based centerline extraction algorithm are as follows: Construct a k-nearest neighbor graph where nodes are points in a point cloud and edge weights are the Euclidean distances between nodes. Based on the node degree detection of both endpoints, the centerline point sequence is obtained through the shortest path algorithm.

[0012] A second aspect of the present invention provides a pose prediction system for robot cable assembly based on multimodal fusion, comprising: The data processing module is configured to acquire the RGB image and raw point cloud data of the cable to be assembled, and to preprocess the raw point cloud data and RGB image. The feature extraction module is configured to extract features from the preprocessed raw point cloud data and the corresponding RGB image to obtain multimodal features; The adaptive clustering module is configured to adaptively identify multimodal features using a density-based spatial clustering algorithm to obtain cable clusters, terminal clusters, and fixture clusters. The terminal recognition module is configured to use a dual-branch multimodal neural network to recognize terminal clusters and RGB images to obtain terminal pose; The topology modeling module is configured to use a graph theory-based centerline extraction algorithm to calculate the centerline of the cable cluster and establish the topological relationship between the terminal and the cable based on the centerline calculation results and the terminal pose. The pose prediction module is configured to obtain the precise pose of the robotic arm during the cable assembly process based on the topological relationship between the terminals and cables, combined with the fixture cluster.

[0013] A third aspect of the present invention provides a computer-readable storage medium storing a computer program adapted for loading by a processor and executing steps in the pose prediction method for robot cable assembly based on multimodal fusion as described in the first aspect of the present invention.

[0014] A fourth aspect of the present invention provides a computer device comprising: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the pose prediction method for robot cable assembly based on multimodal fusion as described in the first aspect of the present invention.

[0015] A fifth aspect of the present invention provides a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the pose prediction method for robot cable assembly based on multimodal fusion as described in the first aspect of the present invention.

[0016] The above one or more technical solutions have the following beneficial effects: This invention discloses a pose prediction method and system for robot cable assembly based on multimodal fusion. It acquires color images and point cloud data of the work scene using a 3D camera; an adaptive clustering algorithm based on multimodal feature fusion identifies and segments the cable clamp, cable body, and terminal areas; combines geometric features and color texture information to achieve point cloud-level multi-target classification; and extracts terminal morphological features through a convolutional neural network to identify terminal type and posture. By fusing RGB and deep point cloud features, this invention achieves highly robust identification of slender, flexible cable targets in high-noise and occluded environments, providing an accurate pose perception foundation for autonomous cable assembly by robotic arms.

[0017] This invention significantly improves recognition accuracy in high-noise and occluded environments through an adaptive clustering algorithm that integrates multimodal features, solving the sensitivity issues of traditional clustering methods to changes in point density and missing orientation information. By combining geometric features with color and texture information, it achieves multi-target classification at the point cloud level, avoiding misclassification caused by single features and improving the segmentation accuracy of cable clamps, cable bodies, and terminal areas. Convolutional neural networks are used to extract terminal morphological features, enabling accurate identification of terminal type and posture, providing a sub-millimeter-level pose perception foundation for robotic arms, and improving the efficiency and robustness of autonomous cable assembly. The overall method is applicable to complex industrial scenarios, reduces manual intervention, lowers assembly error rates, and has high practical value.

[0018] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of the pose prediction method for robot cable assembly based on multimodal fusion in Embodiment 1 of the present invention; Figure 2 This is a flowchart of the adaptive clustering process in Embodiment 1 of the present invention; Figure 3 This is a flowchart of the data processing of a dual-branch multimodal neural network in Embodiment 1 of the present invention. Detailed Implementation

[0021] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0022] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0023] Example 1: Embodiment 1 of this invention provides a pose prediction method for robot cable assembly based on multimodal fusion, used to achieve high-precision identification and positioning of the cable body, fixture, and terminals during automated cable insertion by a robotic arm. This embodiment acquires RGB images and depth point cloud data using a Mech-Mind 3D point cloud camera, fuses geometric features and color texture information, and utilizes an adaptive clustering algorithm and a deep learning network to identify the cable and terminal categories and postures, thereby providing precise pose input for the robotic arm.

[0024] like Figure 1 As shown, the specific steps include: S1: Acquire the RGB image and raw point cloud data of the cable to be assembled, and preprocess the raw point cloud data and RGB image.

[0025] S1.1: Obtain the RGB image and raw point cloud data of the cable to be assembled.

[0026] In one specific implementation, this embodiment uses an industrial-grade 3D camera to acquire RGB images of the work scene and corresponding dense point cloud data. The camera is based on the principle of structured light and aligns the RGB image with the depth map through calibration parameters to obtain the three-dimensional coordinate information corresponding to each pixel.

[0027] In order to ensure the effective input of the recognition algorithm, the collected images and point cloud data in this embodiment must meet the full coverage requirement, that is, the field of view (FOV) must completely include key operating areas such as the fixture area, terminal area, and cable area.

[0028] In this embodiment, the clamping area refers to the rigid mechanical structure used to fix one end of the cable, which typically has a regular geometric shape and a fixed spatial position. The terminal area refers to the metal connector located at the end of the cable, which has high reflectivity and contains key features (such as pins and metal plates) that require attitude estimation. The cable area refers to the flexible tubular portion connecting the various terminals, whose surface is typically made of a single-color matte insulating material and is prone to nonlinear deformation.

[0029] S1.2: Preprocess the raw point cloud data and RGB image.

[0030] In one specific implementation, the acquired raw data typically includes environmental noise, uneven illumination, and background plane interference. Therefore, this embodiment performs targeted preprocessing on the raw point cloud data and RGB images respectively. Specifically, image denoising and contrast enhancement are performed on the RGB images, and voxel downsampling, outlier removal, and plane segmentation are performed on the raw point cloud data.

[0031] 1) RGB image preprocessing: Image denoising: Gaussian filtering is used to remove thermal noise generated by industrial cameras at high gain, and to smooth image textures; Contrast Enhancement: To address the issue of weak surface texture in cables, the Limiting Contrast Adaptive Histogram Equalization (CLAHE) algorithm is employed to enhance the contrast of the highlight areas between the cable edges and metal terminals, providing high-quality input for subsequent feature extraction.

[0032] 2) Preprocessing of raw point cloud data: a) Voxel downsampling: Let the voxel size be... The value is typically 0.5 to 1 times the cable diameter. For each point within a voxel grid, its centroid is taken as the representative point, significantly reducing the number of points while preserving the geometric structure.

[0033] b) Outlier Removal: Statistical filtering is used for noise reduction. First, k-nearest neighbors are defined as the k nearest neighbors to the target point in the point cloud space. The distance from each point to its nearest neighbor is calculated. -Average distance between nearest neighbors If satisfied If a point is identified as a noise point, it is considered a noise point and removed. The mean, Standard deviation This is an empirical coefficient (taken as 1.5 to 2).

[0034] c) Plane segmentation: Fitting a plane model using the Random Sample Consensus (RANSAC) algorithm. The inner points are identified as the background of the workbench and removed, leaving only the cable, terminal and clamp areas.

[0035] S2: Extract features from the preprocessed raw point cloud data and the corresponding RGB image to obtain multimodal features.

[0036] In one specific implementation, this embodiment not only utilizes the spatial geometric information of the point cloud but also integrates the texture and color information of the RGB image. By extracting the geometric, color, and statistical features of each point cloud point, a multimodal feature matrix is ​​constructed, supporting PCA normal vector calculation and the extraction of linearity, flatness, and divergence features. The specific steps are as follows: S2.1: Extract geometric, color, and statistical features from the original point cloud data and the corresponding RGB image, and construct a multimodal feature matrix.

[0037] In one specific implementation, for each point cloud point In this embodiment, geometric, color, and statistical features are extracted, and a multimodal feature matrix is ​​constructed: .

[0038] in, Represents the point index of the point cloud. This represents the normal vector components obtained based on PCA; This represents the color vector of the corresponding pixel from the RGB image; Representing linearity, flatness, and divergence characteristics respectively, defined as: Among them, linearity Linearity is used to characterize whether a local point cloud exhibits a one-dimensional linear structure (such as a thin cable). The closer its value is to 1, the more pronounced the point cloud distribution is in a single direction. Flatness (Planarity) is used to characterize whether a local point cloud exhibits a two-dimensional planar structure (such as a fixture surface or a flat portion of a terminal). The closer its value is to 1, the more the point cloud extends in the two principal directions. Divergence Scattering is used to characterize whether a local point cloud exhibits a three-dimensional isotropic distribution (such as noise or complex corner regions), reflecting the sphericity of the local structure.

[0039] Their calculation formulas are: .

[0040] in These are the eigenvalues ​​of the point cloud covariance matrix. Physically, they represent the variance (i.e., dispersion) of the local point cloud along the three orthogonal principal directions: Corresponding to the main direction, it represents the variance of the point cloud distribution in the direction of maximum extension; The variance corresponding to the secondary principal direction; The variance corresponding to the normal direction (the direction with the flattest distribution).

[0041] S2.2: Normalize the multimodal features.

[0042] The feature matrix is ​​normalized and then subjected to adaptive clustering recognition to achieve joint clustering of color and geometric information.

[0043] S3: Adaptive identification of multimodal features is performed using a density-based spatial clustering algorithm to obtain cable clusters, terminal clusters, and fixture clusters.

[0044] In one specific implementation, this embodiment improves the existing DBSCAN algorithm by performing adaptive clustering based on directional consistency and color geometry fusion, supporting cluster merging and splitting post-processing, and realizing the output of cable clusters, terminal clusters and fixture clusters.

[0045] like Figure 2 As shown, the specific steps are as follows: S3.1: Set directional consistency constraints based on the composite distance of multimodal features.

[0046] In one specific implementation, to address the clustering instability problem of the traditional DBSCAN algorithm when point cloud density is uneven and orientation differences are large, an improved DBSCAN algorithm based on orientation consistency and color geometry fusion is proposed.

[0047] First, for point clouds Calculate the local principal direction vectors respectively and For point cloud points Define its composite distance: .

[0048] in, Point cloud The composite distance, For point clouds coordinate vector, For point clouds coordinate vector, For point clouds color vector, For point clouds color vector, These are adjustable weighting coefficients. This is the spatial scale coefficient.

[0049] Then define the direction consistency constraint: .

[0050] That is, when the absolute value of the dot product of the principal direction vectors of two adjacent points is greater than a set threshold. When considering that they belong to the same cluster, it is generally taken as... .

[0051] S3.2: Adaptive clustering of different regions based on directional consistency constraints.

[0052] In one specific implementation, to adapt to density variations in different regions, the cluster radius... Adaptive form adopted: .

[0053] in, For the first Proximity distance To be the minimum cluster radius, This is a scaling factor. The adaptive clustering principle of this formula for the three key regions in this embodiment is as follows: 1) For cable regions: Due to the small diameter and light absorption of cables, point clouds are often sparse or even locally broken. In this case, the k-nearest neighbor distance of points in the region is... The larger size leads to a larger cluster radius. Adaptive scaling. This allows the algorithm to bridge sparse gaps and successfully cluster discontinuous point clouds of the same cable into a complete cluster, avoiding over-segmentation.

[0054] 2) For the Terminal & Fixture Region: These areas are rigid entities with regular surfaces and high, uniform point cloud density. At this time... Smaller cluster radius Automatic convergence to A smaller search radius prevents terminals from sticking to the adjacent background or other cables, ensuring high precision in terminal edge segmentation.

[0055] S3.3: Perform post-cluster processing based on the adaptive clustering results.

[0056] In one specific implementation, after the initial clustering is completed, cluster merging and splitting operations are performed sequentially to correct for over-segmentation and under-segmentation issues.

[0057] S3.3.1: Perform cluster merging operation.

[0058] Specifically, iterate through all adjacent cluster pairs, and denote the geometric centroids of the current cluster and its neighboring clusters as follows: , The principal direction unit vectors are respectively , Calculate the Euclidean distance between the centroids of the clusters. Angle with the average principal direction .

[0059] , .

[0060] Only if both conditions are met and If two clusters are determined to belong to the same physical object (such as the same cable that has been blocked and cut), a merge operation is performed. The maximum allowable breakage gap (usually 2-3 times the cable diameter) is used to determine spatial proximity. The maximum permissible directional deviation (usually taken as 10°) 15° is used to determine geometric alignment consistency. For cases where only one condition is met or neither condition is met (e.g., close but perpendicular, or aligned but too far apart), they are determined to be different physical objects (e.g., intersecting cables or parallel cables), and the clusters remain independent and are not merged.

[0061] S3.3.2: Perform cluster splitting operation.

[0062] Perform a geometric analysis on each of the merged clusters to calculate the local curvature along the principal axis. A covariance matrix is ​​constructed for a point p in the point cloud and its k-nearest neighbors, and eigenvalue decomposition is performed to obtain three non-negative eigenvalues. , , Local curvature is defined using the minimum eigenvalue ratio. The calculation formula is: .

[0063] If a local curvature at a certain point is detected If the point is determined to be a geometric abrupt change point (usually corresponding to the connection between the cable root and the rigid clamp, or the point where the cable is bent), the cluster will be cut and split at this point. The curvature change threshold is set based on the reciprocal of the minimum bending radius determined by the physical properties of the cable material.

[0064] The final output is a set of point cloud indexes with semantic labels, specifically divided into: cable cluster point sets. , terminal cluster point set and fixture cluster point set This provides accurate data input for subsequent neural network recognition and topology modeling.

[0065] S4: Use a dual-branch multimodal neural network to identify terminals from terminal clusters and RGB images to obtain terminal poses.

[0066] In one specific implementation, this embodiment employs a dual-branch neural network to achieve terminal type classification, translation vector prediction, and quaternion pose regression, supporting ICP local registration refinement, such as... Figure 3 As shown in the figure. Among them, the dual-branch neural network is a structure that combines PointNet++ and ResNet-18.

[0067] S4.1: Point cloud branch performs point cloud feature extraction on terminal clusters.

[0068] The point cloud branch uses a PointNet++ network structure, taking candidate point clouds as input and outputting point cloud feature vectors. .

[0069] S4.2: Extract image features from the RGB image of the terminal in the image branch.

[0070] First, based on the terminal cluster point cloud data output by S3, the three-dimensional spatial points of the terminal cluster are projected onto a two-dimensional RGB image plane using the camera's intrinsic parameter matrix and distortion coefficients. The minimum bounding box of all projected pixels is calculated, and a certain pixel margin (e.g., 10-20 pixels) is added outward to include the terminal edge background, forming a region of interest (ROI).

[0071] Subsequently, the ROI region is cropped from the original RGB image and resized to the network input specifications (e.g., 224×224 pixels) to serve as the terminal RGB cropped image.

[0072] The image branch uses a ResNet-18 network structure. It takes the processed RGB cropped image as input, extracts texture and shape information through convolutional layers, and outputs image features. .

[0073] S4.3: Combine point cloud features and image features.

[0074] S4.3.1: Feature fusion.

[0075] Point cloud branch features (dimensionality such as 1024) and image branch features (Dimensions such as 512) Concatenate along the channel dimension to obtain the fused feature vector. .

[0076] S4.3.2: Multi-task output head for prediction.

[0077] Fusion features The input is fed into three independent fully connected layer branches (FC Heads), which are used to predict the type, location, and orientation of the terminals, respectively: a) Classification branches: This branch outputs a dimensional probability vector ( This is a preset quantity of terminal types, such as straight terminals, elbow terminals, flag terminals, etc. The output is processed using the Softmax activation function and presented in the following format: ,in Indicates that the terminal belongs to the first The confidence level of the class, take The corresponding index serves as the terminal type identification result.

[0078] b) Translational regression branch: This branch outputs a 3D residual vector. To improve prediction stability, this embodiment employs a residual learning strategy, specifically the final terminal center coordinates. The calculation formula is: .

[0079] in The input in step S3 is the geometric centroid of the terminal cluster. The network only needs to learn the tiny offset from the centroid to the actual gripping point (such as the terminal crimp), which significantly reduces the difficulty of regression.

[0080] c) Pose regression branch: This branch outputs a 4-dimensional vector. Since the rotated quaternion must satisfy the unit modulus constraint, an L2 normalization operation is applied after the output layer to obtain the final unit quaternion. : .

[0081] This quaternion uniquely represents the rotational attitude of the terminal in three-dimensional space, avoiding the gimbal lock problem in Euler angle representation.

[0082] The training loss function for a two-branch neural network is: .

[0083] in, , , These are the loss weight coefficients for classification, translation regression, and pose regression tasks, respectively, used to balance the proportion of multi-task learning. For classification loss, the cross-entropy loss function is used; and The losses are for translation and rotation, respectively, and the calculation formulas are as follows: .

[0084] In the formula, The translation vector predicted by the network. Labels for real translation vectors that are manually annotated; The unit quaternion predicted by the network. The quaternion labels are manually labeled for the actual poses.

[0085] The training dataset in this embodiment was obtained by collecting a large number of historical work samples (including different lighting, angles, and occlusion conditions) from an industrial camera at the actual assembly site, and then manually annotating them to obtain the true category and pose labels. On this basis, algorithms such as random rotation, scaling, color perturbation, and point cloud occlusion are further used to expand and enhance the collected data to improve the model's generalization ability in complex environments.

[0086] S5: A graph-based centerline extraction algorithm is used to calculate the centerline of the cable cluster, and the topological relationship between the terminal and the cable is established based on the centerline calculation results and the terminal pose.

[0087] This embodiment extracts the cable centerline based on graph theory and B-spline fitting, calculates curvature and endpoint orientation, and establishes the topological relationship between the terminal and the cable. Specifically, it includes the following steps: S5.1: A graph theory-based centerline extraction algorithm is used to calculate the centerline of the cable cluster.

[0088] S5.1.2: Construct a k-nearest neighbor graph, where nodes are point cloud points and edge weights are the Euclidean distances between nodes.

[0089] S5.1.3: Detect both endpoints based on node degree, and obtain the centerline point sequence using the shortest path algorithm.

[0090] Specifically, B-spline fitting is used to fit the centerline, and the objective function is optimized: .

[0091] In the formula, The control points of the B-spline curve to be solved are used as optimization variables, which determine the shape of the curve. For the first The three-dimensional coordinate vector of each original centerline data point; for The equation of the B-spline curve at time , Indicates parameters The corresponding point on the curve; This is the data fitting error term, representing the squared Euclidean distance between the original data points and the fitted curve; These are the smoothing constraint coefficients, used to balance fitting accuracy and curve smoothness. The equation of the curve with respect to the parameters The second derivative of the integral of the square of its modulus is given by the integral of the second derivative ... This represents the regularization term (energy term), used to suppress excessive oscillations in the curve.

[0092] Calculate curvature from fitted curve : .

[0093] In the formula: For the curve in parameters The first derivative at that point (i.e., the tangent vector); For the curve in parameters The second derivative at that point (i.e., the acceleration vector).

[0094] The point of sudden curvature corresponds to the cable bend or the terminal connection location.

[0095] Tangent vectors are calculated at the endpoints based on local PCA. With normal vector The posture is in quaternion form This indicates that it provides an initial value for the orientation of the terminal.

[0096] Subsequently, local PCA analysis was performed on the detected cable endpoints and their neighboring point clouds to calculate the tangent vectors. With normal vector And convert the local coordinate system into geometric quaternions. This geometric pose Compared with the terminal pose predicted by the neural network in step S4 There are geometric constraints. Specifically, this embodiment will... As a basis for physical consistency verification: S4 Based on image texture and overall shape, it excels at identifying the specific model and approximate orientation of terminals; Based on the extension direction of the cable skeleton, the tangential angle at the connection between the cable and the terminal can be accurately reflected.

[0097] Therefore, in subsequent steps, the tangential direction at the end of the cable is utilized. The axial component of the terminal orientation is calibrated to eliminate any axial deviations that may exist in the neural network prediction, ensuring a smooth transition between the terminal and the cable in terms of physical topology.

[0098] S5.2: Establish the topological relationship between the terminal and the cable based on the centerline calculation results and the terminal pose.

[0099] In one specific implementation, this embodiment uses both geometric distance and directional consistency constraints to logically associate the isolated terminals identified in step S4 with the extracted cable centerline. The specific matching algorithm is as follows: S5.2.1: Discriminate based on distance neighborhood.

[0100] Traverse each identified terminal (Its predicted centroid is) ) and each cable The two endpoints Calculate the Euclidean distance. : .

[0101] like Then determine the terminal With cables If there is a possible spatial connection, it is listed as a candidate matching pair. This is the physical connection threshold, for example, 5mm.

[0102] S5.2.2: Perform direction consistency verification.

[0103] To prevent false matching at cable intersections, direction verification is performed on candidate matching pairs. The tangent vector at the cable endpoints is obtained. and terminal attitude quaternions Corresponding axial vector Calculate the dot product of the two: .

[0104] like That is, the angle between two vectors Less than the tolerance threshold If the angle is 30°, then the connection is confirmed to be valid.

[0105] S5.2.1: Generate topological relationships.

[0106] Pairing that passes double verification Store the topology diagram structure. For unmatched cable endpoints, mark them as "bare wire ends"; for unmatched terminals, mark them as "floating interference items" and remove them, thereby establishing a complete cable assembly topology model.

[0107] S6: Based on the topological relationship between the terminals and cables, and combined with the fixture cluster, the precise pose of the robotic arm during the cable assembly process is obtained.

[0108] In one specific implementation, this embodiment not only needs to identify the terminal pose, but also needs to combine the position of the clamp at the fixed end to plan the collision-free path of the robotic arm. The specific steps are as follows: S6.1: Calculate the spatial reference pose of the fixture cluster.

[0109] Fixture cluster point set output by S3 The process is then performed. Since the fixture is a rigid fixing component, its geometric center is calculated using the Oriented Bounding Box (OBB) algorithm. and spindle direction Thus, a fixture coordinate system is established. This coordinate system serves as a static reference datum during the assembly process.

[0110] S6.2: Establish assembly constraint model.

[0111] Based on the terminal pose (dynamic end) obtained in S4 and the fixture pose (static beginning) obtained in S6.1, calculate the relative transformation matrix between the two. Meanwhile, the cable centerline extracted by S5 is used as a flexible deformation constraint.

[0112] S6.3: Output precise assembly pose.

[0113] By fusing the aforementioned relative transformation matrix with the flexible deformation constraint, the optimal gripping point (GraspPoint) and approach vector of the robotic arm are calculated.

[0114] If the curvature of the cable centerline is detected If the value is too large (meaning the cable is taut), adjust the approach vector to reduce the tension. Finally, the robot control system outputs comprehensive control commands, including terminal gripping posture and fixture obstacle avoidance area, to achieve high-precision assembly.

[0115] This embodiment improves the robustness of recognition in complex environments through multimodal fusion, making it suitable for automated assembly scenarios of flexible cables.

[0116] Example 2: Embodiment 2 of the present invention provides a pose prediction system for robot cable assembly based on multimodal fusion, comprising: The data processing module is configured to acquire the RGB image and raw point cloud data of the cable to be assembled, and to preprocess the raw point cloud data and RGB image. The feature extraction module is configured to extract features from the preprocessed raw point cloud data and the corresponding RGB image to obtain multimodal features; The adaptive clustering module is configured to adaptively identify multimodal features using a density-based spatial clustering algorithm to obtain cable clusters, terminal clusters, and fixture clusters. The terminal recognition module is configured to use a dual-branch multimodal neural network to recognize terminal clusters and RGB images to obtain terminal pose; The topology modeling module is configured to use a graph theory-based centerline extraction algorithm to calculate the centerline of the cable cluster and establish the topological relationship between the terminal and the cable based on the centerline calculation results and the terminal pose. The pose prediction module is configured to obtain the precise pose of the robotic arm during the cable assembly process based on the topological relationship between the terminals and cables, combined with the fixture cluster.

[0117] Example 3: Embodiment 3 of the present invention provides a computer-readable storage medium storing a computer program adapted for loading by a processor and executing the steps in the pose prediction method for robot cable assembly based on multimodal fusion as described in Embodiment 1 of the present invention.

[0118] Example 4: Embodiment 4 of the present invention provides a computer device, the device comprising: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the steps in the pose prediction method for robot cable assembly based on multimodal fusion as described in Embodiment 1 of the present invention.

[0119] Example 5: Embodiment 5 of the present invention provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in the pose prediction method for robot cable assembly based on multimodal fusion as described in Embodiment 1 of the present invention.

[0120] The steps and methods involved in Examples 2, 3, 4 and 5 above correspond to those in Example 1. For specific implementation methods, please refer to the relevant description section of Example 1.

[0121] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0122] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, an optical medium, or a semiconductor medium, etc.

[0123] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A pose prediction method for robot cable assembly based on multimodal fusion, characterized in that, Includes the following steps: Acquire the RGB image and raw point cloud data of the cable to be assembled, and preprocess the raw point cloud data and RGB image; Feature extraction is performed on the preprocessed raw point cloud data and the corresponding RGB image to obtain multimodal features; A density-based spatial clustering algorithm is used to adaptively identify multimodal features, resulting in cable clusters, terminal clusters, and fixture clusters. A dual-branch multimodal neural network is used to identify terminals from terminal clusters and RGB images to obtain terminal poses; A graph theory-based centerline extraction algorithm is used to calculate the centerline of the cable cluster, and the topological relationship between the terminal and the cable is established based on the centerline calculation results and the terminal pose. The precise pose of the robotic arm during cable assembly is obtained by combining the topological relationship between the terminals and the cables with the fixture cluster.

2. The pose prediction method for robot cable assembly based on multimodal fusion as described in claim 1, characterized in that, The specific steps for preprocessing the raw point cloud data and RGB images are as follows: Image denoising and contrast enhancement are performed on RGB images, and voxel downsampling, outlier removal and plane segmentation are performed on the original point cloud data.

3. The pose prediction method for robot cable assembly based on multimodal fusion as described in claim 1, characterized in that, The specific steps for feature extraction from the preprocessed raw point cloud data and the corresponding RGB image are as follows: Geometric, color, and statistical features are extracted from the original point cloud data and the corresponding RGB images, and a multimodal feature matrix is ​​constructed. Normalize the multimodal features.

4. The pose prediction method for robot cable assembly based on multimodal fusion as described in claim 1, characterized in that, The specific steps for adaptive identification of multimodal features using density-based spatial clustering algorithms are as follows: Orientation consistency constraints are set based on the composite distance of multimodal features; Adaptive clustering of different regions based on directional consistency constraints; Post-cluster processing is performed based on the adaptive clustering results.

5. The pose prediction method for robot cable assembly based on multimodal fusion as described in claim 1, characterized in that, The specific steps for terminal identification using a dual-branch multimodal neural network on terminal clusters and RGB images are as follows: Point cloud branch performs point cloud feature extraction on terminal clusters; The image branch extracts image features from the RGB images of the terminals; The point cloud features and image features are stitched together.

6. The pose prediction method for robot cable assembly based on multimodal fusion as described in claim 1, characterized in that, The specific steps for calculating the centerline of a cable cluster using a graph theory-based centerline extraction algorithm are as follows: Construct a k-nearest neighbor graph where nodes are points in a point cloud and edge weights are the Euclidean distances between nodes. Based on the node degree detection of both endpoints, the centerline point sequence is obtained through the shortest path algorithm.

7. A pose prediction system for robot cable assembly based on multimodal fusion, characterized in that, include: The data processing module is configured to acquire the RGB image and raw point cloud data of the cable to be assembled, and to preprocess the raw point cloud data and RGB image. The feature extraction module is configured to extract features from the preprocessed raw point cloud data and the corresponding RGB image to obtain multimodal features; The adaptive clustering module is configured to adaptively identify multimodal features using a density-based spatial clustering algorithm to obtain cable clusters, terminal clusters, and fixture clusters. The terminal recognition module is configured to use a dual-branch multimodal neural network to recognize terminal clusters and RGB images to obtain terminal pose; The topology modeling module is configured to use a graph theory-based centerline extraction algorithm to calculate the centerline of the cable cluster and establish the topological relationship between the terminal and the cable based on the centerline calculation results and the terminal pose. The pose prediction module is configured to obtain the precise pose of the robotic arm during the cable assembly process based on the topological relationship between the terminals and cables, combined with the fixture cluster.

8. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the pose prediction method for robot cable assembly based on multimodal fusion as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-6: a pose prediction method for robot cable assembly based on multimodal fusion.

10. A computer device, characterized in that, include: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the pose prediction method for robot cable assembly based on multimodal fusion as described in any one of claims 1-6.

Images