Multi-modal drainage wire clamp pose estimation method, system and device based on cross-modal comparative learning and medium

Abstract

The invention discloses a multi-modal drainage wire clamp pose estimation method, system and device based on cross-modal contrast learning and a medium, and belongs to the technical field of electric power operation, and the method comprises the steps: obtaining an RGB image and point cloud data, and carrying out the preprocessing of the RGB image and the point cloud data, and generating a data pair in spatial alignment; image semantic features, point cloud geometric features and instance features are obtained through the image and point cloud feature extraction network; mapping the image semantic features and the point cloud geometric features to a unified embedding space; performing intra-modal and inter-modal feature alignment through a contrast learning mechanism to generate cross-modal fusion features; performing instance segmentation based on the point cloud instance features and the cross-modal fusion features; boundary point pair features are extracted to be matched with the template, and candidate poses are generated through voting; and iteratively optimizing and outputting the six-degree-of-freedom pose information. According to the method, the pose estimation precision and robustness under severe conditions such as strong light reflection and structure shielding are improved, the dependence on a large amount of labeled data is reduced, and high-precision real-time pose estimation is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric power operation, in particular to a multi-modal drainage clamp pose estimation method, system, device and medium based on cross-modal contrast learning. BACKGROUND

[0002] The current technology closest to the content of the present application is mainly concentrated in the field of three-dimensional target pose estimation based on multi-modal fusion. Related research generally adopts the strategy of combining RGB images (RGB refers to the abbreviation of red: Red, green: Green, and blue: Blue, which refers to the three primary color channels of an image. In computer vision and image processing, an RGB image represents a color image composed of red, green, and blue color channels, commonly used to represent two-dimensional visual information) and point cloud information. By constructing a double-branch deep network, the semantic features of the image and the geometric structure features of the point cloud are extracted respectively, and then the two types of modal features are fused in the middle and late stages to improve the overall perception ability. For example, DenseFusion (DenseFusion is a multi-modal fusion method used to combine RGB images and point cloud data to achieve 6D pose estimation of a target. After encoding the image and the point cloud respectively, feature splicing or weighted averaging is performed in the embedding space to achieve information complementation between modalities, thereby improving the perception ability.) and other methods use image and point cloud encoding respectively, and perform feature splicing or weighted averaging in the embedding space to achieve information complementation between modalities, thereby completing the 6D pose estimation of the target.

[0003] Other representative methods such as PVN3D (PVN3D is a target pose estimation method based on deep neural networks. Its core idea is to introduce a key point supervision mechanism and a fusion module. After encoding the image and the point cloud data respectively, they are fused, and a voting mechanism is combined to predict the 6D pose of the target.) introduce a key point supervision mechanism and a fusion module. After encoding the image and the point cloud data by a deep neural network, they are fused, and a voting mechanism is combined to realize the regression of the target pose. This kind of method has achieved good performance on multiple standard three-dimensional target recognition data sets, especially in the limited texture or partially occluded scene, showing the advantage of multi-modal collaborative perception. They have high reference value in high-precision pose estimation tasks, and also constitute the mainstream direction of current multi-modal perception methods.

[0004] In addition, some researches also try to introduce contrast learning, attention mechanism or self-supervised strategy to enhance the alignment ability and discrimination ability between multi-modal features. These methods generally pay attention to the consistency learning between modalities, aiming to improve the joint modeling effect of image and point cloud through shared embedding space, and promote the development of multi-modal pose estimation algorithm towards higher generalization and practicality. This kind of technology trend provides an important theoretical basis and method inspiration for the proposal of the present application.

[0005] Traditional image methods mainly rely on RGB images for target recognition and pose inference, often using edge detection, template matching, key point extraction, and PnP (Perspective-n-Point) algorithm, etc. classic visual technology. This kind of method can achieve relatively good pose estimation effect under the ideal condition of uniform light, simple background and clear texture. But in the actual work scene of substation, the metal material of the lead wire clamp often causes unstable image feature extraction due to strong light reflection, and is limited by the dense wires in the background, equipment obstruction and other complex interference, the visibility and distinguishability of the target are greatly reduced. In addition, relying only on two-dimensional images is difficult to obtain spatial depth information, especially when the robot performs high-precision grabbing and positioning tasks, it is often difficult to provide reliable three-dimensional reference. Therefore, even if combined with deep neural networks for image feature learning and pose regression, it is still easy to fail in pose estimation in the scene of sparse point cloud and incomplete target structure.

[0006] Compared with image methods, point cloud provides three-dimensional geometric information of objects and is a widely used pose estimation method. Typical methods include ICP (Iterative Closest Point) template matching algorithm, registration algorithm based on point pairs or local geometric descriptors (such as PPF, FPFH), and deep network-based point cloud regression model (such as PointNet, DGCNN). Although the ICP method has high alignment accuracy, it depends on the initial pose estimation and is complex in iterative calculation, which is difficult to meet the online real-time operation demand. Local geometric matching method is prone to feature loss and matching error in the scene of fuzzy boundary, structure occlusion or closely stacked fittings. The deep point cloud regression method is heavily dependent on training data and requires a large number of manually labeled pose samples, which has limited generalization ability and limited adaptability to point cloud resolution, pose change range and target category. Especially in the process of high-altitude operation, point cloud is often disturbed by factors such as occlusion, wind disturbance and structure change, making the existing point cloud algorithm perform poorly in pose stability and robustness.

[0007] To overcome the above shortcomings, some studies have introduced multi-modal perception strategies, trying to combine the semantic information of images with the spatial structure advantages of point clouds to improve the comprehensive perception ability of the system for complex targets. For example, methods such as DenseFusion and PVN3D use a dual-channel network structure to process RGB images and point cloud inputs respectively, and perform feature fusion in the middle and later stages. However, these methods usually only integrate modalities based on shallow strategies such as feature stitching or attention weighting, lacking a unified embedding space and semantic consistency modeling mechanism, resulting in limited fusion quality of images and point clouds at the semantic level, and still having a significant "semantic gap" problem. In addition, these methods generally rely on high-precision RGB-point cloud labeled data for supervised training, which has high data acquisition costs and poor model transferability, making it difficult to generalize to different types of line clamp structures or substation working conditions. Moreover, multi-modal models have high demand for computing resources, causing performance bottlenecks in deployment on power robot or edge computing platforms, making it difficult to achieve stable perception and response to high-frequency dynamic scenes.

[0008] In summary, existing pose estimation methods have obvious technical shortcomings in terms of insufficient depth of multi-modal information fusion, high dependence on labeled data, weak structure discrimination ability, and poor real-time deployment, making it difficult to meet the integrated requirements of target recognition accuracy, system response speed, and job safety in 220kV and above high-voltage substation operating environments. To address these issues, there is an urgent need for a multi-modal pose estimation algorithm that has strong modal alignment capability, structure understanding capability, low data dependence, and efficient deployment.

[0009] In the live drawing line disconnection operation in high-voltage substations, the target object, the live drawing line clamp, has complex shape, fuzzy boundary, and strong reflection, and the operating environment is often accompanied by strong electromagnetic interference, high-altitude wind disturbance, and point cloud occlusion, etc. unfavorable factors, which puts high requirements on the accuracy, robustness and real-time performance of the perception system. In the existing technology, traditional single-modal-based pose estimation methods are difficult to capture both the texture semantics in the image and the three-dimensional geometric structure in the point cloud, leading to recognition failure or pose misjudgment in unstructured environments, making it difficult to meet the operation needs of robots in 220kV and above voltage level substations.

[0010] Although some studies have attempted to introduce multi-modal perception mechanisms to fuse RGB images and point cloud information, current fusion strategies are mostly limited to feature stitching or shallow attention mechanism stages, lacking deep-level modal alignment and consistency modeling. This rough fusion method not only fails to fully exploit the complementary advantages of modalities, but also introduces interference information in the case of inconsistent semantic mapping, further reducing the accuracy and stability of pose estimation. In addition, existing point cloud processing methods generally lack structure discrimination ability, making it difficult to accurately identify the edge region of the live drawing line clamp, increasing the uncertainty of subsequent grasping, alignment and installation operations.

[0011] On the other hand, most existing deep learning methods rely on a large amount of data with accurate 6D pose labels for supervised training, but it is extremely costly and inefficient to obtain large-scale and high-quality labeled data in the actual substation operation environment. The model has limited generalization ability, and often needs to be retrained when facing different types of wire clamps or different operation conditions of different sites, which seriously restricts the transferability and engineering deployment value of the algorithm.

[0012] In addition, existing algorithms are often limited by large model size, slow inference speed, high hardware resource occupation and other problems when deployed to the operation robot platform, making it difficult to realize real-time operation on the edge computing unit and unable to meet the strict requirements on response speed and safety decision-making in the process of high-voltage live working. Therefore, it is urgent to propose a multi-modal pose estimation method with high precision, high robustness and high deployability, which can fully fuse RGB image and point cloud information under weak supervision conditions, realize accurate understanding and pose perception of the target structure of the drainage wire clamp, and have the ability to run stably in the power field environment. SUMMARY

[0013] In view of the above problems, the present application is proposed.

[0014] Therefore, the technical problem solved by the present application is: how to realize deep semantic alignment of RGB image and point cloud information, robust perception in complex environment, high-precision pose estimation under weak supervision conditions through cross-modal contrast learning and unified embedding space construction, while overcoming the interference of strong light reflection, structure occlusion and other adverse working conditions on the recognition and positioning of the drainage wire clamp.

[0015] To solve the above technical problems, the present application provides the following technical solutions: a multi-modal drainage wire clamp pose estimation method based on cross-modal contrast learning, comprising the following steps,

[0016] An RGB image and point cloud data of a target drainage clip are acquired, the RGB image and the point cloud data are preprocessed to generate a spatially aligned image-point cloud data pair, the RGB image is input into an image feature extraction network to obtain image semantic features, and the point cloud data is input into a point cloud feature extraction network to obtain point cloud geometric features and point cloud instance features, the image semantic features and the point cloud geometric features are mapped to a unified embedding space through a mapping network to generate image embedding features and point cloud embedding features, cross-modal fusion features are generated through a contrast learning mechanism for intra-modal and inter-modal feature alignment based on the image embedding features and the point cloud embedding features, instance segmentation is performed on the point cloud data based on the point cloud instance features and the cross-modal fusion features, and instance point clouds of the target drainage clip are acquired, boundary point pair features are extracted from the instance point clouds, matched with a preset template point cloud, and candidate poses are generated through a voting mechanism, and six-degree-of-freedom pose information of the target drainage clip is output through iterative optimization of the candidate poses.

[0017] As a preferred scheme of the multi-modal drainage clip pose estimation method based on cross-modal contrast learning, the point cloud feature extraction network comprises a plurality of serially connected feature extraction modules, each of which is used for extracting geometric features of different levels, and the features output by the point cloud feature extraction network are input into a semantic branch and an instance branch respectively, the semantic branch generates point cloud semantic features, and the instance branch generates the point cloud instance features.

[0018] As a preferred scheme of the multi-modal drainage clip pose estimation method based on cross-modal contrast learning, the instance segmentation performed on the point cloud data based on the point cloud instance features and the cross-modal fusion features comprises the following steps: the point cloud semantic features are fused into the point cloud instance features to generate semantic-enhanced instance features, k-nearest neighbor search is performed on each point in the semantic-enhanced instance feature space to obtain a nearest neighbor feature set of each point, channel max-pooling operation is performed on the nearest neighbor feature set to generate aggregated features of each instance, and the spatial distribution of the semantic-enhanced instance features is constrained by a discriminative loss function based on the aggregated features to realize instance-level point cloud segmentation.

[0019] As a preferred scheme of the multi-modal drainage clip pose estimation method based on cross-modal contrast learning, the feature alignment in the intra-modal and inter-modal is performed through a contrast learning mechanism, including: performing data enhancement on the RGB image and the point cloud data respectively to generate multiple enhanced views;Constructing positive sample pairs and negative sample pairs, wherein the positive sample pairs include different enhanced views of the same data and matched image-point cloud pairs, and the negative sample pairs include unmatched image-point cloud pairs;The image embedding features and the point cloud embedding features are optimized through a contrast loss function, so that the distance of the positive sample pairs in the embedding space is reduced, and the distance of the negative sample pairs in the embedding space is increased.

[0020] As a preferred scheme of the multi-modal drainage clip pose estimation method based on cross-modal contrast learning, the boundary point pair features are extracted from the instance point cloud, including: identifying boundary points from the instance point cloud;Construct a point pair for the boundary points, and calculate the geometric descriptor of each point pair, wherein the geometric descriptor includes the relative position relationship and the tangential direction information between the point pairs;A hash index table is constructed based on the geometric descriptor, which is used to quickly match the scene point cloud with the template point cloud.

[0021] As a preferred scheme of the multi-modal drainage clip pose estimation method based on cross-modal contrast learning, the candidate pose is generated through a voting mechanism, including: matching the geometric descriptor of the scene point pair with the geometric descriptor of the template point cloud;For each matched point pair, calculate the transformation matrix from the template coordinate system to the scene coordinate system;Cluster and vote all the transformation matrices, and select the transformation matrix with the most votes as the candidate pose.

[0022] As a preferred scheme of the multi-modal drainage clip pose estimation method based on cross-modal contrast learning, the candidate pose is iteratively optimized, including: taking the candidate pose as an initial transformation, and transforming the template point cloud to the scene coordinate system;Establish a point correspondence between the transformed template point cloud and the instance point cloud;Iteratively update the rotation matrix and the translation vector by minimizing the distance error between the corresponding points until the error converges or the preset iteration number is reached.

[0023] The application provides a multi-modal drainage clip pose estimation system based on cross-modal contrast learning.

[0024] To solve the above technical problems, the application further provides the following technical solutions: a multi-modal drainage clip pose estimation system based on cross-modal contrast learning, comprising a data acquisition module, a feature extraction module, a feature mapping module, a contrast learning module, an instance segmentation module, a pose generation module and a pose optimization module, wherein: the data acquisition module is used to acquire an RGB image and point cloud data of a target drainage clip, pre-process the RGB image and the point cloud data, and generate a spatially aligned image-point cloud data pair; the feature extraction module is used to input the RGB image into an image feature extraction network to obtain image semantic features, and input the point cloud data into a point cloud feature extraction network to obtain point cloud geometric features and point cloud instance features; the feature mapping module is used to map the image semantic features and the point cloud geometric features to a unified embedding space through a mapping network to generate image embedding features and point cloud embedding features; the contrast learning module is used to perform intra-modal and inter-modal feature alignment through a contrast learning mechanism based on the image embedding features and the point cloud embedding features to generate cross-modal fusion features; the instance segmentation module is used to perform instance segmentation on the point cloud data according to the point cloud instance features and the cross-modal fusion features to obtain instance point clouds of the target drainage clip; the pose generation module is used to extract boundary point pair features from the instance point clouds, match the boundary point pair features with a preset template point cloud, and generate candidate poses through a voting mechanism; and the pose optimization module is used to iteratively optimize the candidate poses to output six-degree-of-freedom pose information of the target drainage clip.

[0025] The application provides a computer device, comprising a memory and a processor, and the memory stores a computer program, characterized in that the processor implements the steps of the multi-modal drainage clip pose estimation method based on cross-modal contrast learning when executing the computer program.

[0026] The application provides a computer readable storage medium, which stores a computer program, characterized in that the computer program is executed by a processor to implement the steps of the multi-modal drainage clip pose estimation method based on cross-modal contrast learning.

[0027] The application has the beneficial effects that: the application realizes deep alignment and fusion of image semantic information and point cloud geometric structure at the feature level by constructing a unified cross-modal embedding space and introducing a contrast learning mechanism. Compared with the existing shallow feature splicing or simple weighted fusion method, the application establishes semantic consistency constraints between modalities, so that the two heterogeneous data sources can enhance each other in the common representation space, effectively solving the semantic gap problem existing in the traditional multi-modal fusion. At the same time, through the instance segmentation strategy of semantic feature enhancement, the category-level semantic information is fused into the instance-level geometric representation, and the discriminative loss function is constrained to improve the target boundary recognition ability in the scene of sparse point cloud and structure occlusion. In addition, the geometric matching and voting mechanism based on boundary point pairs is adopted, which avoids the strong dependence of the traditional ICP algorithm on the initial pose, greatly improves the calculation efficiency while ensuring the matching accuracy. The overall scheme still can stably output high-precision pose information in the extreme working environment of substation strong light reflection and electromagnetic interference through multi-level feature learning and fusion strategy, and greatly reduces the demand for labeled data, providing reliable technical support for the intelligentization and automation of high-voltage live working. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0029] Figure 1 The overall flowchart of the multi-modal drainage wire clip pose estimation method based on cross-modal contrast learning provided by an embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0030] In order to make the above-mentioned purposes, features and advantages of the application more apparent and easy to understand, the specific embodiments of the application will be described in detail below with reference to the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor should be within the protection scope of the application.

[0031] Embodiment 1, refer to Figure 1 For the first embodiment of the application, the embodiment provides a multi-modal drainage wire clip pose estimation method based on cross-modal contrast learning, which comprises:

[0032] S1: obtaining the RGB image and point cloud data of the target drainage wire clip, preprocessing the RGB image and point cloud data, and generating a space-aligned image-point cloud data pair;

[0033] S2: Input the RGB image into the image feature extraction network to obtain the image semantic features, and input the point cloud data into the point cloud feature extraction network to obtain the point cloud geometric features and point cloud instance features;

[0034] S3: By mapping the semantic features of the image and the geometric features of the point cloud to a unified embedding space through a mapping network, image embedding features and point cloud embedding features are generated.

[0035] S4: Based on image embedding features and point cloud embedding features, feature alignment within and between modalities is performed through a contrastive learning mechanism to generate cross-modal fusion features;

[0036] S5: Based on the point cloud instance features and cross-modal fusion features, the point cloud data is segmented to obtain the instance point cloud of the target drainage clamp;

[0037] S6: Extract boundary point pair features from the instance point cloud, match them with the preset template point cloud, and generate candidate poses through a voting mechanism;

[0038] S7: Iteratively optimize the candidate pose and output the six-degree-of-freedom pose information of the target drainage clamp.

[0039] Specifically, the candidate pose is used as the initial transformation to transform the template point cloud to the scene coordinate system;

[0040] Establish a point correspondence between the transformed template point cloud and the instance point cloud;

[0041] By minimizing the distance error between corresponding points, the rotation matrix and translation vector are iteratively updated until the error converges or the preset number of iterations is reached.

[0042] It should be noted that convergence refers to the process by which the ICP algorithm, during its iterations, continuously adjusts the rotation matrix and translation vector, gradually reducing the distance error between the model point cloud and the scene point cloud, eventually reaching a stable state. This state indicates that the algorithm has found a local optimum or reached a preset stopping condition. In the ICP algorithm, the preset number of iterations refers to the maximum number of iterations set manually to control the algorithm's running time and prevent it from looping indefinitely. Since ICP is an iterative optimization algorithm, it reduces the matching error between point clouds by continuously updating rotation and translation parameters. However, in some cases, it may be difficult to converge quickly due to poor initial pose, high point cloud noise, or local minima. To ensure the efficiency and stability of the algorithm, an upper limit is usually set (such as 50 or 100 iterations). When this number is reached, regardless of whether it has fully converged, the algorithm stops and outputs the current optimal pose estimation result. This approach ensures real-time computation while avoiding excessive resource consumption.

[0043] It should be explained that the multi-modal drainage wire clip pose estimation algorithm proposed by the application is composed of data acquisition, feature extraction, cross-modal contrastive learning, instance segmentation, geometric modeling and pose estimation, and forms an end-to-end, high-robustness, high-precision perception scheme.

[0044] RGB images and three-dimensional point cloud data are synchronously collected on the live working platform D435i. Image data is extracted by a YOLO11 backbone network (YOLO11 is the 11th generation of the You Only Look Once series target detection algorithm, which is a high-efficiency real-time target detection network architecture. In the application, YOLO11 is used as a backbone network to extract semantic features from RGB images, which can quickly identify target objects in the image and output their category and position information) to extract semantic features, while point cloud data is input into a CMCL-PENet network (CMCL-PENet is a deep neural network structure specially designed for point cloud data, and the full name is Cross-Modal Contrastive Learning-based Point Embedding Network, i.e. point cloud embedding network based on cross-modal contrastive learning). The network is used to extract local and global geometric features from three-dimensional point cloud data, which supports subsequent alignment and fusion of image features in a unified embedding space. The local and global geometric features are extracted. Subsequently, the two types of modal features are mapped to a unified embedding feature space through modal projection, ensuring that the image and the point cloud have comparability, and building a semantically consistent representation basis for subsequent contrastive learning.

[0045] After the feature space is unified, the point cloud is rotated to learn the spatial invariance, and then a modal contrastive learning mechanism is introduced to strengthen the consistency learning of the representation from the image-point cloud pair and the point cloud-point cloud pair, thereby constructing a robust cross-modal embedding feature representation. The representation not only retains the effective information in each modality, but also maximizes the alignment of the distribution difference between the two in structure and semantics.

[0046] Based on this embedding space, instance-level segmentation is further performed in the point cloud. By combining semantic features, instance features, template point clouds, and using a kNN clustering strategy, effective instance recognition of the target region can be achieved. Subsequently, a discriminative feature learning mechanism is applied to extract clear instance boundaries and separate the structure point cloud corresponding to the target drainage wire clip.

[0047] After obtaining the target point cloud, the boundary key points thereof are detected, and geometric point pair features are calculated based on PPF (Point Pair Feature). These features are used for matching with the modeling point clouds in a standard template library, and a fast corresponding search is performed through a hash index table. After the matching is completed, candidate pose transformations are generated under a local coordinate system, and a voting mechanism is adopted to screen out optimal pose candidates.

[0048] To further improve the pose estimation accuracy, an ICP (Iterative Closest Point) algorithm is adopted to refine and optimize the preliminary results, and finally stable and accurate six-dimensional pose information is output. The pose information is transmitted to a mechanical arm execution module to realize accurate grasping, alignment and assembly operations of the drainage wire clamp, and ensure the safety and efficiency of live working.

[0049] The entire algorithm process fully combines cross-modal deep feature learning and point cloud structure modeling technology, not only realizes high robustness and high generalization ability in the perception stage, but also ensures the operability of the output through geometric registration and pose optimization, and is suitable for complex and variable high-voltage substation working scenes.

[0050] In summary, the present application realizes the alignment and semantic consistency modeling between modalities by constructing a unified cross-modal embedding space and deeply integrating image semantics and point cloud geometric features through a contrast learning mechanism. Combined with discriminative instance segmentation and boundary geometric matching, high-precision 6D poses are output, which have weak supervision training, high robustness and deployability, and are suitable for intelligent perception and control in the power working environment.

[0051] Embodiment 2, refer to Figure 1 The present application provides a multi-modal drainage wire clamp pose estimation method based on cross-modal contrast learning.

[0052] S1: Obtain the RGB image and point cloud data of the target drainage wire clamp, and pre-process the RGB image and point cloud data to generate a space-aligned image-point cloud data pair.

[0053] This step mainly installs a D435i depth camera on the live working platform to obtain an input data set: point cloud, RGB image, grayscale image and rendered image.

[0054] Specifically, a D435i depth camera is installed on the live working platform to obtain an input data set:

[0055] Point cloud: ;

[0056] RGB image: ;

[0057] Grayscale image: ;

[0058] rendered image: , represents the rendered image of the i-th frame, is a structure rendered image generated from the point cloud, and the superscript r indicates that it is a rendered image, which is usually used for visualization or further processing;

[0059] It should be noted that, represents the point cloud data set collected by the i-th frame, and the point cloud is a set of discrete points in three-dimensional space, each point representing a three-dimensional coordinate. Among them, is a single point in the point cloud (the subscript j represents the index of the point, ranging from 1 to ), represents a three-dimensional coordinate point in space, and , , is a three-dimensional real space, specifically (x, y, z), where x, y, and z are the coordinates of the point in the three-dimensional real space; is the number of points in the point cloud set ; represents the RGB color image synchronously acquired with the frame point cloud, wherein represents the RGB color image of the i-th frame, and the superscript rgb indicates that it is an RGB color image containing three color channels; and are the height (number of pixels in the vertical direction) and width (number of pixels in the horizontal direction) of the image, respectively, and 3 indicates that the image has three color channels (red, green, and blue), so it is a three-dimensional array; represents that the data type of the RGB image is real, with a shape of height H, width W, and three color channels; represents the corresponding grayscale image, which uses a single channel to represent image brightness information and is commonly used as auxiliary input in the process of enhancing structural feature modeling and multi-modal alignment. 1 indicates that the image has only one channel (brightness channel), so it is a two-dimensional array; represents the grayscale image of the i-th frame, where the superscript gray indicates that it is a grayscale image containing only brightness information without color channels; represents that the data type of the grayscale image is real, with a shape of height H, width W, and one brightness channel.

[0060] The image and the point cloud correspond to the same time, the same spatial position, so the positive and negative samples for subsequent contrastive learning are accurate.

[0061] Exemplarily, on the live-line working robot platform of a certain 220 kV high-voltage substation, an Intel RealSense D435i depth camera is installed as a multi-modal data acquisition device. The depth camera integrates an RGB color camera module and an infrared depth sensor, and can synchronously acquire color image information and depth information of a target scene.

[0062] After starting, the D435i depth camera synchronously acquires data in the target scene of the drainage wire clamp at a collection frequency of 30 Hz. Specifically, the RGB camera module captures an RGB image, and the infrared depth sensor acquires a depth map of the same resolution. Since there is an offset between the two sensor modules in the physical position and different intrinsic characteristics, preprocessing is needed to ensure the spatial consistency of the data.

[0063] Firstly, the intrinsic calibration of the RGB image is performed. The intrinsic matrix of the camera, including the focal length, principal point coordinates and distortion coefficients, is used to perform distortion correction on the original RGB image, eliminate the image distortion caused by lens distortion, and ensure the accuracy of the geometric relationship in the image.

[0064] Secondly, the depth map is converted into three-dimensional point cloud data. According to the intrinsic parameters of the depth camera and the depth value of each pixel point, the two-dimensional depth image pixel is converted into a three-dimensional space coordinate through the camera projection model to generate the original point cloud data. Each point contains its three-dimensional position coordinates (x, y, z) in the camera coordinate system.

[0065] Subsequently, voxel filter processing is performed on the generated point cloud data. The voxel size is set to 0.005 meters, and the three-dimensional space is divided into a regular cubic grid, and all points in each voxel are replaced by the centroid point, thereby effectively reducing the point cloud density, reducing the calculation amount of subsequent processing, and filtering out part of the noise points while preserving the overall structural features of the point cloud.

[0066] Finally, the spatial registration of the RGB image and the point cloud is performed. The extrinsic calibration information provided by the D435i camera, including the rotation matrix and the translation vector between the RGB camera and the depth sensor, is used to transform the point cloud coordinates from the depth camera coordinate system to the RGB camera coordinate system. Through this registration operation, it is ensured that each three-dimensional point in the point cloud can be accurately corresponding to the corresponding pixel point in the RGB image, forming a spatially aligned image-point cloud data pair.

[0067] After the above preprocessing steps, a time-synchronized and spatially consistent RGB image and point cloud data pair can be obtained, laying a foundation for subsequent multi-modal feature extraction and fusion. Through this data alignment, the accuracy of the positive and negative sample pairs constructed in the subsequent contrastive learning can be ensured, which is a key prerequisite for realizing high-precision cross-modal pose estimation.

[0068] S2: input the RGB image into an image feature extraction network to obtain image semantic features, and input the point cloud data into a point cloud feature extraction network to obtain point cloud geometric features and point cloud instance features.

[0069] This step mainly inputs the RGB image into a YOLO11 network to extract image semantic features, and inputs the point cloud into a CMCL-PENet network to extract instance and geometric features.

[0070] Specifically, in the present application, the feature extraction network structure is to use two deep neural network paths to extract modal features respectively.

[0071] Point cloud feature extractor : using the CMCL-PENet network given in the present application to extract features , specifically instance features (wherein, represents a feature vector extracted from point cloud data, the subscript 3D indicates that it is extracted from three-dimensional point cloud data, and the superscript i indicates that it is an instance feature of the point cloud, which is usually used to identify and distinguish different object instances) and point cloud semantic features ; wherein, represents a function or network model for extracting point cloud features, wherein the subscript 3D indicates that the model is specially designed for three-dimensional point cloud data; the superscript θ represents the parameter set of the function or network. In deep learning, θ usually represents the trainable parameters (such as weights and biases) of the network. represents the point cloud data of the i-th frame input into the point cloud feature extraction network , and the features of the point cloud are extracted.

[0072] Image feature extractor : YOLO11 backbone network, extracting image semantic features . Wherein, represents a function or network model for extracting two-dimensional image features, and the subscript 2D indicates that it is specially designed for two-dimensional image data; the superscript represents the parameter set of the function or network, similar to θ in the point cloud feature extractor; represents the i-th frame of the rendered image, which is a structured image generated from the point cloud and used to assist feature extraction.

[0073] It should be noted that the CMCL-PENet network is the core network structure in the present application for processing point cloud modal features, and the overall architecture thereof includes two parts of a point cloud branch and an RGB image branch, and the two modalities are cooperatively modeled through deep fusion. In the point cloud branch, the input is boundary structure information of local point clouds, and geometric features of multiple local regions are extracted respectively. The branch is internally composed of multiple KAN (Kolmogorov-Arnold Network) modules connected in series, which are used to replace the traditional MLP (linear projection layer) to achieve efficient nonlinear modeling and feature expression. Different geometric channels extract structural features through multiple stacked KAN layers, and are respectively converged into a semantic branch and an instance branch, and finally high-level point cloud features of a unified dimension are output.

[0074] The semantic branch is responsible for generating the class representation of each point, such as color, contour, class label and other semantic information, while the instance branch focuses on object shape and local structure modeling to support instance-level segmentation and boundary perception. The point cloud branch as a whole emphasizes geometric level understanding, and is fused with the image branch through a cross-modality learning module in the subsequent stage to provide structural rich geometric support for the final pose estimation.

[0075] The image branch is constructed based on the YOLO11 backbone network, and the texture, color and edge semantic features in the RGB image are extracted layer by layer through multiple convolutional modules. This branch not only retains the appearance information of the target in the two-dimensional image, but also maps to a unified embedding space together with the point cloud branch to realize the deep alignment between the image and the point cloud. After feature extraction, the features of the two modalities respectively enter the intra-modality contrast learning and cross-modality contrast learning modules to form a unified embedding representation, which is used to support the subsequent pose regression and instance recognition tasks. The overall structure embodies the design concept of modality cooperation and structure perception, and is the key support for realizing high-precision cross-modality perception.

[0076] S3: mapping the image semantic features and the point cloud geometric features to a unified embedding space through a mapping network to generate image embedding features and point cloud embedding features.

[0077] This step mainly uses linear mapping to map the image and point cloud features to a unified embedding space, and constructs positive and negative sample pairs to provide embedding input for subsequent contrast learning.

[0078] Specifically, in step S2, the image semantic features are extracted , and then a linear projection layer (i.e. MLP) is used to map the two features to a unified space:

[0079] wherein, is a cross-modality feature representation, represents a real-valued vector space of dimension d; the notation d represents a real-valued vector space of dimension d; the notation and represents two different modalities of feature embedding vectors, represents the embedding feature vector of the point cloud modality after encoding and mapping, where the superscript is the parameter of the point cloud feature extraction network or the mapping function; represents the embedding vector of the image modality (RGB image) after feature extraction and mapping, where the superscript represents the network parameters or channels corresponding to the image branch; both are used to construct cross-modal feature pairs for subsequent contrastive learning loss function calculation, to realize the alignment of point cloud and image semantic representation in the same embedding space.

[0080] S4: Based on the image embedding features and point cloud embedding features, the intra-modal and inter-modal feature alignment is performed through the contrastive learning mechanism to generate cross-modal fusion features.

[0081] This step mainly performs intra-modal enhancement (such as image rotation / fuzziness, point cloud jitter), constructs positive and negative sample sets, and uses InfoNCE improved loss function to jointly optimize intra-modal loss and inter-modal loss. InfoNCE is the abbreviation of Information Noise Contrastive Estimation, which is a widely used loss function in self-supervised learning and contrastive learning. In this invention, InfoNCE is used as an improved loss function to jointly optimize intra-modal loss (such as feature consistency of image or point cloud itself) and inter-modal loss (such as alignment between image and point cloud features), so as to realize intra-modal and inter-modal feature alignment through contrastive learning mechanism, and improve the cross-modal fusion effect.

[0082] It should be noted that in steps S1 to S3, the data is processed in frames, and the subscript i represents the data or features of the i-th frame. After step S3, the point cloud embedding features and image embedding features are generated for each frame of data. In step S4, in order to further optimize feature alignment, each frame of data is treated as an independent sample, and a sample number i is introduced. By performing intra-modal enhancement on each sample i, two groups of enhanced views are generated, and features are extracted respectively. Positive and negative sample pairs are constructed for the contrastive learning mechanism to realize intra-modal consistency and cross-modal alignment.

[0083] Specifically, the specific content of modal enhancement and positive and negative sample construction is as follows:

[0084] For each sample Image enhancement (rotation, cropping, blurring) and point cloud enhancement (random rotation, translation, jitter) are performed to generate two sets of enhanced views, namely, point cloud views. Image view pairs

[0085] After extracting the representations from the network, four vectors are obtained: ;

[0086] Among them, the positive samples are , For image modality augmentation pairs, two views are formed by different data augmentations of the same image sample, used to learn intramodal consistency. A matching pair between an image and its corresponding point cloud represents observation data of the same object in different modalities, used to learn cross-modal alignment. Let represent the point cloud embedding features of the two augmented views of the i-th sample, respectively. These two point cloud embedding features come from the same sample but are generated through different augmentation operations.

[0087] Negative samples are , representing the feature pair between the image and the non-matching image, i.e., the first The image and the first The representation obtained after image enhancement This means they are not the same sample. `j` is used to identify other samples different from the current sample `i`, primarily for constructing negative sample pairs. These combinations constitute negative samples, used in contrastive learning to widen the feature distance between non-matching samples.

[0088] Furthermore, in this invention, the contrastive learning objective function is designed to use an improved loss function for the contrastive loss function, in the following form:

[0089] Intramodal loss (point cloud to point cloud): ;

[0090] in, It is an intramodal loss (point cloud to point cloud), which is used to encourage the consistency of feature representations of the same modality (such as point cloud) under different views, thereby enhancing the network's robust modeling ability of point cloud modalities.

[0091] It should be noted that the purpose of this loss function is to encourage two enhanced versions (views) of the same image sample. Figure 1 View 2) is closer to each other in feature space; and It is symmetrical and represents intramodal loss. The contrast loss term in the model is used to enhance the consistency of two-way information; is the number of sample pairs in a batch (i.e., the total number of samples in the current batch); is a normalization coefficient to ensure the total loss scale stable. Among them, the intra-modal loss The expression form of the contrast loss item in the inter-modal loss : ;

[0092] In the formula, is the feature of the current sample view Figure 1 ; is its positive sample (the feature of view 2 of the same sample); is view 2 of all other samples (negative samples); sim is a similarity function, and the cosine similarity is commonly used ; is a temperature coefficient used to control the smoothness of the distribution (adjust the importance of negative samples); the ultimate goal is to maximize the similarity ratio of positive samples (maximize the numerator and the most significant in the denominator).

[0093] is expressed as: ;

[0094] is expressed as: ;

[0095] Inter-modal loss (RGB image-point cloud): ;

[0096] Among them, represents the inter-modal loss, specifically the contrast loss item between the RGB image and the point cloud, which is used for cross-modal feature alignment learning. and are the contrast loss items in the inter-modal loss , which respectively calculate the similarity between the point cloud feature and the image feature, and optimize the consistency of the features through the contrast learning mechanism; is the number of sample pairs in a batch, i.e., the total number of samples in the current batch; is a normalization coefficient to ensure the total loss scale stable; the symmetric structure enhances the bidirectional consistency of the modal. Among them, the expression form of the contrast loss item in the inter-modal loss : ;

[0097] The expression of is: ​​ ;

[0098] The expression is: ;

[0099] The joint objective function is: ;

[0100] In the formula, represents the joint objective function, and the total loss term of jointly optimizing the intra-modal loss (L intra , point cloud-point cloud) and the inter-modal loss (L inter , RGB image-point cloud).

[0101] S5: According to the point cloud instance feature and the cross-modal fusion feature, the point cloud data is instance segmented to obtain the instance point cloud of the target drainage wire clamp.

[0102] This step mainly constructs an instance embedding space on the basis of the point cloud semantic enhanced feature, and performs instance level segmentation through kNN clustering (k nearest neighbor clustering), channel maximum pooling and discriminative loss.

[0103] Specifically, in the semantic branch, the application adopts a full connection layer to generate a feature buffer area, aiming to fuse local appearance information and global geometric structure information, so as to enhance the recognition ability of the network to object instances. On this basis, the fused semantic features are further processed through a full connection layer and KAN, and the network assigns semantic labels to each point after learning the features of each point, thereby realizing semantic classification of the point cloud.

[0104] The application fuses semantic features (class level features) into instance features (features of specific shape and position of the target). The fused semantic features , , the semantic enhanced instance features are represented as , and the expression form is as follows: ;

[0105] Among them, represents the fused semantic enhanced instance features, that is, the semantic information is added to the original instance features; represents the original instance features, which describe the specific shape and position and other geometric information of the target; represents the semantic features, which describe the category information (such as “drainage wire clamp”, “insulator” and the like) of the target; The weight of the class to the instance feature is represented, and different points of the same class are further distinguished, while different instances of the same class have less influence. Through this deep integration, the instance feature can fully consider the influence of semantic information when representing the object pose and class, thereby improving the detection accuracy.

[0106] To further optimize the expression ability of the instance feature, a learning mechanism based on similarity measurement is introduced in the invention. The core idea is that points within the same instance should have similar features, while points between different instances should exhibit differences. By learning the similarity measurement, the network can encourage points within the same object to be more closely gathered in the feature space, while points between different classes maintain a larger feature distance. This strategy effectively avoids class confusion, thereby realizing semantic-aware instance segmentation.

[0107] To ensure that the instance features of different classes of objects have obvious differences, the invention performs k-nearest neighbor search (kNN search) on each point in the instance embedding space, and combines edge filtering technology to remove noise points, ensuring that the selected points indeed belong to the same instance. This method makes the embedded points closer to the instance centroid mean, thereby improving the stability of the instance feature and reducing noise interference.

[0108] In the point cloud instance embedding space, The semantic feature of each point after searching is organized into a three-dimensional tensor , where, is the number of points in the instance, is the number of neighbors, is the feature dimension of each point.

[0109] The invention adopts a channel max-pooling method to select the maximum value on each channel from the points in each neighbor group, forming a compressed representative feature vector. For each instance centroid point , the fused global semantic feature can be represented as: ;

[0110] where, represents the semantic feature of the nearest neighbors of point .

[0111] The purpose of introducing channel max-pooling is to extract the most representative local structure information from each neighborhood group. Compared with methods such as average pooling or weighted summation, max-pooling has natural robustness, can effectively suppress the interference of noise points or invalid features, and retain the strongest feature value on each channel, thereby highlighting the information of key areas such as structure edges and geometric mutations. In addition, this method does not require the introduction of additional parameters, has small computational overhead, and is suitable for deployment on edge computing platforms. Especially in environments where point cloud density is uneven or there is occlusion, channel max-pooling can stably output a compact feature vector with discriminability, providing more reliable input support for subsequent instance recognition and pose estimation.

[0112] The semantic aggregation output forms a feature matrix with dimensions The number of points in the instance is The feature dimension of each point is That is, the semantic enhanced features of each instance, the next step is to complete the point classification task between different instances through a semantic classifier, thereby realizing instance segmentation of the point cloud.

[0113] In order to realize high-quality point cloud instance segmentation, the present application adopts a discriminative loss function for constraining the spatial distribution of instance features, so that the embedded features of points within the same instance are close to each other, and the embedded features between different instances are kept apart. The loss L is composed of three terms:

[0114] Among them, is the intra-class compactness loss (Variance Term), is the inter-class separation loss (Distance Term), is the regularization term (Regularization Term).

[0115] The embedded value of each point is pulled closer to the average embedded center of the instance it belongs to , improving the intra-class compactness; ;

[0116] The average embedded centers between different instances are pushed away, improving the inter-class separability; ;

[0117] The embedded centers of all instances are subjected to L1 regularization, constraining the range of embedded values; ;

[0118] In the formula, ​denotes the number of instances in the current batch, the point number of the i-th instance of the j-th class, the point number of the i-th instance of the j-th class, the hyperparameter control class content difference and the minimum distance between classes, denotes the embedding vector of the i-th point of the j-th class. It should be noted that the subscript i here represents the instance number, j represents the point number within the instance i in , and represents the number of another instance in .

[0119] S6: Extract the boundary point pair feature from the instance point cloud, match it with the preset template point cloud, and generate the candidate pose through the voting mechanism.

[0120] Specifically, after completing instance segmentation, the instance point cloud of the target object is obtained. By using the target geometric information (such as depth gradient and point normal vector), edge point pairs are selected and their point pair features are calculated to describe the local geometric relationship of the target in the scene point cloud (three-dimensional point cloud data collected from the real work site) and the model point cloud (standard reference point cloud generated based on three-dimensional modeling sampling). When a pair of scene points is given, its point pair descriptor can be calculated, and matched with the pre-calculated point pair feature in the model, so as to establish a corresponding relationship through a hash table, and estimate the target pose from the point pair set.

[0121] To improve the efficiency of pose estimation, the B2B-TL (Boundary-to-Boundary Tangent Line) descriptor is introduced to establish accurate point pair mapping between the model and the scene: ;

[0122] wherein, is the point pair feature (PPF) used to describe the spatial geometric relationship between two points, commonly used in point cloud registration and pose estimation; T represents the transpose operation, represents four feature values composed into a column vector.

[0123] is the Euclidean distance between the point and the reference point , representing the spatial interval of the two points.

[0124] is the angle between the normal vector of the point and the point pair vector , reflecting the directional relationship between the reference point normal and the connecting vector.​​

[0125] is the normal vector of point is the angle between the normal vector of point and the direction of vector

[0126] is the angle between the normal vectors of two points, reflecting the relative orientation between local surfaces.

[0127] is the Euclidean vector between two points; and are the normal vectors of reference point and target point respectively. The above features can express the spatial structure relationship between boundary points, independent of texture and color information, and have strong geometric robustness.

[0128] The tangent direction (rather than the normal) is used for alignment on the boundary points; by constructing angle features based on the tangent direction, false matches caused by inconsistent normals are avoided; the tangent direction ensures the consistency of the alignment direction, which helps to reduce the pose matching search space.

[0129] After establishing the local coordinate system , the pose transformation can be expressed by the following transformation formula: ;

[0130] where represents the point corresponding to the model point after transformation, and the coordinates in the local coordinate system where the current matching point pair is located. is the transformation from the model point cloud to the global coordinate; is the inverse transformation of the scene point cloud back to the local coordinate; is the rotation angle around the X-axis .

[0131] Based on the PPF feature, a hash index table is constructed, and the most similar template feature to the instance point pair is found through a fast matching mechanism. After successful matching, a series of candidate 6D pose transformations are generated, and the transformation process includes the following three steps:

[0132] 1. Template point cloud coordinate transformation.

[0133] Transform the local segment related to the matching point pair in the template point cloud to the local coordinate system where the matching point is located, expressed as: ;

[0134] where ​This represents the transformation matrix that converts the local coordinate system of the template point cloud to the global coordinate system. This transformation is typically used to unify the coordinate systems of different point clouds to facilitate subsequent matching and calculations.

[0135] 2. Angle compensation transformation.

[0136] To improve attitude matching accuracy, a bypass is introduced. - Axis rotation compensation: ;

[0137] 3. Align the target point cloud with the global coordinate system.

[0138] Transform the target point cloud of point-pair matching back into the global coordinate system of the workspace:

[0139] ;

[0140] Here, "inverse of local-to-global transform" represents the inverse transformation of the target point cloud from its local to its global coordinate system. This inverse transformation allows the target point cloud to be transformed back from its local coordinate system to its global coordinate system for comparison with the template point cloud.

[0141] Combining the above three parts yields the final candidate pose estimation transformation: ;

[0142] in, This indicates that the points, after angle compensation, are transformed from the global coordinate system back to the local coordinate system of the target point cloud.

[0143] By repeating the above calculation process on multiple point pairs, a large number of candidate poses can be generated. Then, a voting mechanism is introduced to statistically score all candidate poses, and the candidates with the most cumulative votes are selected as the initial estimation results.

[0144] S7: Iteratively optimize the candidate pose and output the six-degree-of-freedom pose information of the target drainage clamp.

[0145] To further improve the alignment accuracy of pose estimation, the candidate pose with the most votes is input into the refinement module based on the Iterative Closest Point (ICP) algorithm. ICP iteratively optimizes and minimizes the error function between the instance point cloud and the transformed model point cloud. ;

[0146] in, It is the first in the model point cloud One point; These are the points in the scene point cloud and the model points; are matched corresponding points; is a rotation matrix (orthogonal constraint must be met , I represents the identity matrix in the ICP algorithm); is a translation vector. The goal of this algorithm is to find the optimal rotation matrix and translation vector , so that the distance between the points in the model point cloud after a rigid transformation and their corresponding points in the scene point cloud is minimized.

[0147] After the iteration is completed, the pose information is output in the form of a homogeneous transformation matrix. The transformation matrix is composed of a three-dimensional rotation matrix and a three-dimensional translation vector, which is used to describe the spatial displacement and orientation change of the model point cloud in the task space coordinate system. The output result is a four-dimensional homogeneous matrix : ;

[0148] wherein, represents the rotation matrix, represents the translation vector. This transformation can be directly applied to the model point cloud to achieve the pose transformation from the local coordinate system to the scene global coordinate system.

[0149] The final output of the 6D pose result is not only used to guide the robot to perform precise grasping, alignment and assembly operations, but also used for system precision evaluation, virtual simulation comparison and other subsequent modules. If the system control platform requires a specific format (such as Euler angles or quaternions), the homogeneous matrix can also be converted into the required parameter form through mathematical decomposition, ensuring seamless connection with the control and perception subsystems.

[0150] In summary, the present application effectively breaks through the performance bottleneck of traditional single-modal perception methods and shallow multi-modal fusion technology, and has advantages in pose perception accuracy, structure understanding ability, system deployment efficiency and engineering adaptability, etc. Specifically:

[0151] Firstly, the application realizes high alignment of two modalities at the feature level by constructing a unified cross-modal embedding space, deeply fusing texture semantic information in the image modality and spatial geometric structure features in the point cloud modality. This feature fusion strategy can enhance the perception robustness of the system in the substation scene, so that it can still output high-precision six-degree-of-freedom pose estimation results to a certain extent under adverse conditions such as severe target occlusion, cluttered background, incomplete structure, and even strong light leading to image quality degradation. Especially in the case of high reflection of metal structure, the application compensates for the influence of unstable image information through point cloud and image collaborative modeling, reduces the positioning error of the robot in the grabbing and alignment process, and improves the overall operation success rate and stability.

[0152] Secondly, in view of the problem that the structure of the current-carrying clamp in the power field is heterogeneous and the target sample is difficult to label, the application introduces a self-supervised and weakly supervised learning mechanism, effectively reduces the dependence on a large amount of accurate pose labeled data, and greatly improves the trainability, adaptability and migration ability of the model. This method shows good generalization performance and environmental robustness in typical high-voltage environments such as strong light, electromagnetic interference, wind disturbance and structural occlusion. At the system deployment level, the application focuses on the lightweight design of the model structure and the optimization of the inference path to ensure that it has good real-time running ability and can be efficiently embedded in a work robot or an edge computing platform to meet the stringent requirements for system response speed and safety.

[0153] Thirdly, compared with the traditional point cloud matching method which is easily limited by initial pose error, structural occlusion and feature loss, the application further introduces an image-guided mechanism to optimize point cloud representation, enhancing the understanding ability of complex structure boundaries and geometric topological relationships. In the case of insufficient point cloud density or incomplete target information, stable output of pose estimation can still be achieved, improving the reliability and accuracy of the overall perception link.

[0154] In summary, the application has realized key breakthroughs in perception mechanism design, training strategy optimization, deployment capability enhancement and system adaptability, to a certain extent, overcoming the shortcomings of existing technologies in terms of precision, robustness, efficiency and adaptability, providing solid technical support for intelligent and high-precision disassembly and assembly of current-carrying clamps in high-voltage substations, and having outstanding technical advancement and practical application value.

[0155] Embodiment 3 is a third embodiment of the application, which provides a multi-modal current-carrying clamp pose estimation system based on cross-modal contrast learning, including a data acquisition module, a feature extraction module, a feature mapping module, a contrast learning module, an instance segmentation module, a pose generation module and a pose optimization module.

[0156] The data acquisition module is configured to acquire an RGB image and point cloud data of the target drainage line clamp, pre-process the RGB image and the point cloud data, and generate a spatially aligned image-point cloud data pair.

[0157] The feature extraction module is configured to input the RGB image into an image feature extraction network to obtain image semantic features, and input the point cloud data into a point cloud feature extraction network to obtain point cloud geometric features and point cloud instance features.

[0158] The feature mapping module is configured to map the image semantic features and the point cloud geometric features to a unified embedding space respectively through a mapping network to generate image embedding features and point cloud embedding features.

[0159] The contrast learning module is configured to perform intra-modal and inter-modal feature alignment through a contrast learning mechanism based on the image embedding features and the point cloud embedding features to generate cross-modal fusion features.

[0160] The instance segmentation module is configured to perform instance segmentation on the point cloud data according to the point cloud instance features and the cross-modal fusion features to obtain instance point clouds of the target drainage line clamp.

[0161] The pose generation module is configured to extract boundary point pair features from the instance point clouds, match the boundary point pair features with a preset template point cloud, and generate candidate poses through a voting mechanism.

[0162] The pose optimization module is configured to perform iterative optimization on the candidate poses to output six-degree-of-freedom pose information of the target drainage line clamp.

[0163] Embodiment 4 is the fourth embodiment of the present application, which is different from the first three embodiments in that the function, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application or the part that contributes to the prior art or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various storage medium that can store program codes.

[0164] The logic and / or steps represented in the flow diagrams or otherwise described herein, for example, can be considered as a sequence of executable instructions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or a combination of both. In the context of this specification, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0165] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via an optical scanner, then compiled, interpreted, or otherwise processed, and stored in a computer memory in a form that can be later executed by a computer.

[0166] It should be understood that aspects of the application can be implemented in hardware, software, firmware or a combination of both. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, or combination thereof, can be used: a discrete logic circuit having logic gates for implementing logic functions upon data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array (PGA), a field programmable gate array (FPGA), and so forth.

[0167] Embodiment 5, which is a fifth embodiment of the present application, provides a multi-modal drainage clip pose estimation method based on cross-modal contrastive learning. In order to verify the beneficial effects of the present application, scientific demonstration is carried out through experiments.

[0168] In a certain 220 kV high-voltage substation, the work robot needs to accurately grasp and align the current-carrying line clamp in the environment of strong light, structural obstruction, and serious target reflection. Before work, the robot installs an Intel RealSense D435i depth camera and starts the data collection process. When collecting a frame, the system synchronously acquires an RGB image (1280x720 pixels) and a depth map, converts them into a dense point cloud, and generates a spatially consistent image-point cloud pair after Voxel filtering (voxel size is set to 0.005 m) and depth map registration.

[0169] Subsequently, the RGB image is sent to the YOLO11 network to extract texture semantic features, and the point cloud is sent to the CMCL-PENet network to extract geometric and instance features. The KAN module inside the CMCL-PENet performs high-order nonlinear modeling on the point cloud, outputting a 128-dimensional geometric embedding vector for each point. After independent linear projection (MLP, output dimension is 64), the two modal features are mapped to a unified embedding space.

[0170] In the contrast learning stage, the system performs blur enhancement on the image and random rotation and jitter on the point cloud. Four enhanced views are generated for each sample, and 16 positive samples and 128 negative samples are constructed (based on a batch size of 32). In the loss function, the temperature parameter τ is set to 0.07 to optimize the intra-modal and inter-modal consistency. The training takes a total of 4 hours, and the top-1 alignment accuracy converges to 92.4% on the NVIDIA RTX 3090 platform.

[0171] In the model inference stage, after obtaining the features, the system performs instance segmentation in the point cloud. Through kNN (k=16) clustering + channel max-pooling operation, each current-carrying line clamp is accurately segmented, and its semantic-enhanced structural point cloud (average point number is 430) is output. The system further extracts boundary points, calculates point pair descriptors (B2B-TL), and matches them with 20 different models of current-carrying line clamp templates in the standard library. After hash table search, 50 candidate poses are generated, the system votes for the top 3 groups with the most votes, and finally inputs the candidate with a vote rate of 82% into the ICP module for fine alignment.

[0172] The upper limit of ICP iteration is set to 30 times, and the termination threshold is 1e-6. The final output 6D pose error is: translation error 2.4 mm, rotation error 1.2°. The pose information is sent to the robot controller in real time, and the robot performs the grasping action and successfully completes the live installation operation.

[0173] Through the present embodiment, the system can still stably output high-precision 6D pose information in power scenarios such as obstruction, reflection, and incomplete structure.

[0174] In addition, the present application has low dependence on labeled data, and only a small amount of geometric modeling templates and registration data of the drainage clips are required to complete the training, thereby having good scalability and engineering deployment value.

[0175] The overall delay of the reasoning stage is low, the edge computing unit is adapted, and the requirements of high frequency response and accurate control of the robot on the site are met.

[0176] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the present application, and they should be covered in the scope of the claims of the present application.

Claims

1. A multimodal drainage line clamp pose estimation method based on cross-modal contrastive learning, characterized in that: include, Acquire the RGB image and point cloud data of the target drainage clamp, preprocess the RGB image and the point cloud data to generate spatially aligned image-point cloud data pairs; The RGB image is input into an image feature extraction network to obtain image semantic features, and the point cloud data is input into a point cloud feature extraction network to obtain point cloud geometric features and point cloud instance features. The image semantic features and the point cloud geometric features are mapped to a unified embedding space through a mapping network, generating image embedding features and point cloud embedding features respectively. Based on the image embedding features and the point cloud embedding features, feature alignment within and between modalities is performed through a contrastive learning mechanism to generate cross-modal fusion features; Based on the point cloud instance features and the cross-modal fusion features, the point cloud data is segmented to obtain the instance point cloud of the target drainage clamp; Boundary point pair features are extracted from the instance point cloud and matched with a preset template point cloud to generate candidate poses through a voting mechanism; The candidate poses are iteratively optimized to output the six-degree-of-freedom pose information of the target drainage clamp.

2. The multimodal drainage line clamp pose estimation method based on cross-modal contrastive learning as described in claim 1, characterized in that: The point cloud feature extraction network includes multiple cascaded feature extraction modules, each of which is used to extract geometric features at different levels. The features output by the point cloud feature extraction network are respectively input into the semantic branch and the instance branch. The semantic branch generates point cloud semantic features, and the instance branch generates point cloud instance features.

3. The multimodal drainage line clamp pose estimation method based on cross-modal contrastive learning as described in claim 2, characterized in that: The step of segmenting the point cloud data into instances based on the point cloud instance features and the cross-modal fusion features includes: The semantic features of the point cloud are fused into the instance features of the point cloud to generate semantically enhanced instance features; For each point, perform a k-nearest neighbor search in the semantically enhanced instance feature space to obtain the nearest neighbor feature set for each point; Channel max pooling is performed on the nearest neighbor feature set to generate aggregated features for each instance; Based on the aggregated features, the spatial distribution of the semantically enhanced instance features is constrained by a discriminative loss function to achieve instance-level point cloud segmentation.

4. The multimodal drainage line clamp pose estimation method based on cross-modal contrastive learning as described in claim 3, characterized in that: The feature alignment within and between modalities via a contrastive learning mechanism includes: Data augmentation is performed on the RGB image and the point cloud data respectively to generate multiple enhanced views; Construct positive sample pairs and negative sample pairs; wherein, the positive sample pairs include different augmented views of the same data and matching image-point cloud pairs, and the negative sample pairs include non-matching image-point cloud pairs; By optimizing the image embedding features and the point cloud embedding features using a comparative loss function, the distance between the positive sample pairs in the embedding space is reduced, and the distance between the negative sample pairs in the embedding space is increased.

5. The multimodal drainage line clamp pose estimation method based on cross-modal contrastive learning as described in claim 4, characterized in that: Extracting boundary point pair features from the instance point cloud includes: Identify boundary points from the instance point cloud; For the boundary points, point pairs are constructed, and a geometric descriptor for each point pair is calculated; wherein, the geometric descriptor includes the relative positional relationship and tangential direction information between the point pairs; A hash index table is constructed based on the geometric descriptor to quickly match the scene point cloud with the template point cloud.

6. The multimodal drainage line clamp pose estimation method based on cross-modal contrastive learning as described in claim 5, characterized in that: The generation of candidate poses through a voting mechanism includes: Match the geometric descriptor of the scene point pair with the geometric descriptor of the template point cloud; For each matched pair of points, calculate the transformation matrix from the template coordinate system to the scene coordinate system; Clustering and voting are performed on all the transformation matrices, and the transformation matrix with the most votes is selected as the candidate pose.

7. The multimodal drainage line clamp pose estimation method based on cross-modal contrastive learning as described in claim 6, characterized in that: The iterative optimization of the candidate pose includes: The candidate pose is used as the initial transformation to transform the template point cloud to the scene coordinate system; Establish a point correspondence between the transformed template point cloud and the instance point cloud; By minimizing the distance error between corresponding points, the rotation matrix and translation vector are iteratively updated until the error converges or the preset number of iterations is reached.

8. A multimodal drainage wire clamp pose estimation system based on cross-modal contrastive learning, employing the multimodal drainage wire clamp pose estimation method based on cross-modal contrastive learning as described in any one of claims 1 to 7, characterized in that, include: The module comprises a data acquisition module, a feature extraction module, a feature mapping module, a contrastive learning module, an instance segmentation module, a pose generation module, and a pose optimization module, among which: The data acquisition module is used to acquire the RGB image and point cloud data of the target drainage clamp, preprocess the RGB image and the point cloud data, and generate spatially aligned image-point cloud data pairs. The feature extraction module is used to input the RGB image into the image feature extraction network to obtain image semantic features, and to input the point cloud data into the point cloud feature extraction network to obtain point cloud geometric features and point cloud instance features; The feature mapping module is used to map the image semantic features and the point cloud geometric features to a unified embedding space through a mapping network, thereby generating image embedding features and point cloud embedding features. The contrastive learning module is used to perform intra-modal and inter-modal feature alignment based on the image embedding features and the point cloud embedding features through a contrastive learning mechanism to generate cross-modal fusion features; The instance segmentation module is used to segment the point cloud data into instances based on the point cloud instance features and the cross-modal fusion features, and obtain the instance point cloud of the target drainage clamp. The pose generation module is used to extract boundary point pair features from the instance point cloud, match them with a preset template point cloud, and generate candidate poses through a voting mechanism. The pose optimization module is used to iteratively optimize the candidate pose and output the six-degree-of-freedom pose information of the target drainage clamp.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the multimodal drainage line clamp pose estimation method based on cross-modal contrastive learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal drainage line clamp pose estimation method based on cross-modal contrastive learning as described in any one of claims 1 to 7.

Images