GIL unit flange plate and bolt hole pose recognition method and system based on multi-modal fusion, and medium

Abstract

This invention relates to a method, system, and medium for recognizing the pose of a GIL unit flange and bolt holes based on multimodal fusion. The method includes: acquiring a 3D point cloud and a synchronized RGB texture image of the GIL unit's mating end; performing multimodal fusion and alignment processing to obtain a multimodal fused point cloud; performing preprocessing followed by semantic segmentation using a point cloud semantic segmentation model to output the flange end face point cloud, bolt hole region point cloud, and background noise; fitting spatial geometric elements based on the flange end face point cloud to extract geometric features reflecting the flange's orientation; and calculating the geometric center coordinates of each bolt hole in space based on the bolt hole region point cloud, using these coordinates as the bolt hole pose; and combining the relative spatial relationship between the geometric features and the geometric center coordinates to calculate the multi-degree-of-freedom pose parameters of the flange relative to the sensor coordinate system. Compared with existing technologies, this invention has advantages such as improved robustness and accuracy of pose recognition under complex working conditions.

Description

Technical Field

[0001] This invention relates to the field of automated installation of power equipment, and in particular to a method, system and medium for recognizing the position and orientation of GIL unit flanges and bolt holes based on multimodal fusion. Background Technology

[0002] With the rapid development of ultra-high voltage power transmission projects, gas-insulated transmission lines (GILs) have been widely used due to their advantages such as large transmission capacity and small footprint. During GIL field installation, precise flange alignment and bolt hole positioning are key aspects of achieving automated construction.

[0003] Traditional GIL installation primarily relies on manual visual guidance or single sensor identification. However, GIL unit flanges exhibit typical highly reflective metal characteristics, and the lighting conditions at construction sites are complex. Single 2D image recognition methods are easily affected by light and shadow interference, making it difficult to obtain reliable 3D spatial pose. On the other hand, recognition methods that rely solely on 3D point clouds suffer from missing point clouds or severe outlier noise due to specular reflections from the metal surface. Furthermore, in the absence of color information, the semantic extraction accuracy for subtle geometric features such as bolt holes is poor, leading to significant errors in the final motion control input. This affects docking efficiency and may even pose a risk of equipment collision.

[0004] Therefore, existing recognition methods are insufficient to simultaneously meet the requirements of robustness in recognition and accuracy in pose calculation in environments with high dynamics, high reflectivity, and high precision, such as GIL construction. There is an urgent need for a recognition method that can integrate multimodal information and has efficient semantic parsing capabilities. Summary of the Invention

[0005] The purpose of this invention is to provide a method, system, and medium for recognizing the pose of GIL unit flanges and bolt holes based on multimodal fusion, achieving robust recognition and accurate pose calculation.

[0006] The objective of this invention can be achieved through the following technical solutions: A method for recognizing the pose of a GIL unit flange and bolt holes based on multimodal fusion includes the following steps: Obtain the 3D point cloud of the GIL unit docking end P raw and synchronized RGB texture images I Multimodal fusion and alignment processing is performed to obtain a multimodal fused point cloud. P fusion ; For the multimodal fused point cloud P fusion First, preprocessing is performed, then semantic segmentation is performed using a point cloud semantic segmentation model to output the point cloud of the flange end face. Pface Point cloud of bolt hole area P hole and background noise; Based on the point cloud of the flange end face P face Spatial geometric element fitting is performed to extract geometric features reflecting the flange orientation, and the point cloud of the bolt hole area is used as the basis for this extraction. P hole Calculate the geometric center coordinates of each bolt hole in space, which are used as the pose of the bolt holes; Based on the relative spatial relationship between the geometric features and the coordinates of the geometric center, the multi-degree-of-freedom pose parameters of the flange relative to the sensor coordinate system are calculated.

[0007] Furthermore, the obtained multimodal fused point cloud P fusion The steps include: Obtain the set calibration information, including the intrinsic parameter matrix of the texture camera. K The extrinsic transformation matrix between the texture camera and the point cloud coordinate system. R | t ]; For the three-dimensional point cloud P raw Each three-dimensional point in P i ( X i , Y i , Z i ), using perspective projection model to calculate P i In the RGB texture image I The corresponding pixel coordinates ( u i , v i The perspective projection model is represented as follows: , In the formula, As a scale factor, For the first Points in RGB texture image I homogeneous pixel coordinates on The extrinsic transformation matrix, For the first The three-dimensional coordinates of a point in the world coordinate system It is a translation vector; Based on the pixel coordinates ( u i , vi From the RGB texture image I Extract the corresponding color information ( R i , G i , B i The color information is then assigned to the corresponding three-dimensional point. P i This forms a set of six-dimensional attributes. X i , Y i , Z i , R i , G i , B i ] three-dimensional points P i Finally, a multimodal fused point cloud was obtained. P fusion .

[0008] Furthermore, the preprocessing process includes: For the multimodal fused point cloud P fusion For each point in the array, calculate the average distance to its k nearest neighbors; calculate P fusion The mean μ and standard deviation σ of the average distance of all points are calculated, and a dynamic threshold interval is set based on the mean μ and standard deviation σ. Multimodal fusion of point clouds P fusion Points falling outside the dynamic threshold range are removed to obtain a multimodal fused point cloud after noise removal; For the multimodal fused point cloud after noise removal, its three-dimensional space is divided into multiple voxels, and all points located in the same voxel are grouped together. For all points in each group, calculate the geometric center of its three-dimensional coordinates, and use the geometric center point to represent all points within its voxel, finally obtaining the preprocessed multimodal fused point cloud.

[0009] Furthermore, the point cloud semantic segmentation model is constructed using a PointNeXt network, which includes multiple stacked SA modules, an inverse residual multilayer perceptron following each SA module, and a decoding output module at the tail end. The SA modules are used to downsample and initially aggregate the preprocessed multimodal fusion point cloud. The inverse residual multilayer perceptron is used to enhance the feature representation ability based on the SA modules. The decoding output module is used to output the flange end face point cloud through feature decoding and mapping. P face Point cloud of bolt hole area P hole And background noise.

[0010] Furthermore, each SA module includes a fusion layer and texture branches, geometry branches, and downsampling layers respectively connected to it. The execution steps of each SA module include: The output of the previous SA module is used as the input of the current SA module. The farthest point is sampled using the downsampling layer to obtain the set of center points. Then, a neighborhood query is performed using the set sphere query radius to obtain the set of neighborhood points. In the case that the current SA module is the first SA module, its input is the preprocessed multimodal fusion point cloud. The neighborhood point set is used as input for the texture branch and the geometric branch to extract pixel-level texture features and three-dimensional point cloud features, respectively. The texture features are color texture and edge prior information, and the point cloud features are normal vectors and local curvature features. The texture features are applied as weights to the point cloud features, and an attention mechanism is used for fusion to obtain fused features. The specific steps include: The texture features extracted from the texture branch Point cloud features extracted by geometric branching Mapped to the same feature dimension D : ,in, This represents the number of points after downsampling by the current SA module. It is the set of real numbers; Spatial weights are generated using the texture features. and utilize the spatial weights The point cloud features are weighted to obtain the weighted point cloud features. , is represented as: , , In the formula, For gated multilayer sensing, Use the Sigmoid activation function; Based on the texture features and weighted point cloud features An attention mechanism is used to capture the non-linear correlation between features of two modalities: , , , In the formula, For query volume, As key-value pair For numerical values, To query the weight matrix, This is the key-value weight matrix. This is a numerical weight matrix; Based on weighted point cloud features Combined with the query volume Key value and numerical values By performing fusion, fusion characteristics are obtained. , is represented as: , , In the formula, To output the projection layer, For splicing operations, For attention mechanisms, Scaling factor The dimension of the feature vector. This is the normalization function.

[0011] Furthermore, the geometric feature extraction step includes: The point cloud of the flange end face is obtained by using a random sampling consensus algorithm. P face By fitting spatial geometric elements, a spatial plane is obtained, represented as: , In the formula, , , , It is a constant. , , Not both equal to 0, The coordinates of the point; Calculate the unit normal vector of the space plane. n =( A , B , C The unit normal vector n For the target coordinate system of the flange { TIn the sensor coordinate system { S} X The axial vector direction is used as a geometric feature reflecting the flange orientation, wherein the flange target coordinate system { T The origin is taken as the geometric center point of the flange. O t ,That X t The normal vector of the shaft coincides with that of the flange end face. Y t The axis points to the preset starting bolt hole, and the sensor coordinate system { S} Taking the optical center of the sensor as the origin O s ,That X s The axis points in the negative direction of the optical axis. Y s The axis points to the right side of the sensor plane, which is the horizontal rightward direction of the RGB texture image.

[0012] Furthermore, the calculation steps for the geometric center coordinates include: Point cloud of bolt hole area P hole The points are projected onto the spatial plane, and then Euclidean clustering algorithm is used to generate the point cloud of the bolt hole region. P hole Divide into N independent bolt hole clusters, where N corresponds to the number of bolt holes on the flange; For each bolt hole cluster, the least squares method is used to perform circle fitting to obtain the geometric center coordinates of each bolt hole in space. C i ( x i , y i , z i ).

[0013] Furthermore, the calculation steps for the multi-degree-of-freedom pose parameters include: Based on the aforementioned geometric center coordinates, the average geometric center point of all bolt holes is calculated and used as the geometric center point of the flange. O t , is represented as: , In the formula, N is the number of bolt holes on the flange. For the first The geometric center coordinates of each bolt hole in space; The geometric center point O t ( x0 , y 0 As the initial solution, combined with the radius dimension of the flange. R The optimized geometric center point is obtained by using the least squares circle fitting algorithm in reverse iteration. O t The origin is called the origin and is represented as: , In the formula, For the first The two-dimensional coordinates of the extracted bolt hole center point in the fitting plane, subscript This is the index of the observation point, with a value ranging from 1 to... ;; Based on the aforementioned geometric features and the optimized geometric center point O t The pose matrix of the flange relative to the sensor coordinate system {S} is constructed. T ST The pose matrix T ST Includes the rotating portion determined by the aforementioned geometric features and the bolt hole phase vector. R ST The optimized geometric center point O t The coordinates in the sensor coordinate system {S} are used as the translation component. t ST ; The pose matrix T ST Combined with the set calibration matrix T BS Transform to reference coordinate system { B The multi-degree-of-freedom pose parameters of the flange in the robot's workspace are obtained below. T BT The reference coordinate system { B} refers to the robot base coordinate system or the global coordinate system of the construction site, and the multi-degree-of-freedom pose parameters T BT Represented as: .

[0014] This invention also provides a multimodal fusion-based system for recognizing the pose of GIL unit flanges and bolt holes, including a vision sensor and a control and computing unit connected thereto. The vision sensor is used to scan the GIL unit flange and acquire a three-dimensional point cloud of the GIL unit mating end. P raw and synchronized RGB texture images IThe control computing unit is used to execute a multimodal fusion-based GIL unit flange and bolt hole pose recognition method as described above.

[0015] The present invention also provides a storage medium storing a program thereon, which, when executed, implements the above-described method for recognizing the pose of a GIL unit flange and bolt holes based on multimodal fusion.

[0016] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention considers the three-dimensional spatial structure of point cloud, color and texture information of RGB texture image. By merging and aligning the three-dimensional point cloud with the synchronized RGB texture image, the original three-dimensional point cloud is given texture characteristics. Thus, in semantic segmentation, the point cloud of flange end face and the point cloud of bolt hole area can be accurately and robustly segmented by combining point cloud features and image texture features. Then, the corresponding pose is calculated by coordinate solution, realizing the robustness of recognition and the accuracy of pose calculation.

[0017] (2) In view of the lack of texture variation on the flange end face, which makes it difficult to distinguish, the present invention adopts the method of combining GRB texture image and point cloud to achieve complementary enhancement of geometry and texture. By attaching the pixels in the GRB texture image to each point, it is equivalent to giving each three-dimensional point a visual appearance. This enables the point cloud semantic segmentation model to accurately identify the flange end face and bolt holes through shape and color, making up for the lack of geometric information.

[0018] (3) In the point cloud semantic segmentation model, the present invention uses texture features as weights and applies them to the point cloud feature vectors of the geometric branch through the attention mechanism, so that even when the local point cloud is sparse due to high reflectivity, the model can still accurately identify the semantic boundary of the bolt hole by relying on image features.

[0019] (4) Considering the high reflectivity of GIL flanges and the complex construction background, this invention employs noise removal and data simplification in the preprocessing process. This not only ensures the quality of the input data and eliminates outliers caused by mirror reflection on the metal surface, but also avoids the interference of noise on the calculation of the centroid of the downsampling. Furthermore, it can obtain a computationally friendly and uniformly distributed point cloud dataset while ensuring geometric features, providing a high-quality data foundation for subsequent point cloud segmentation and pose calculation.

[0020] (5) This invention uses the semantic segmentation results for subsequent pose calculation. By fitting the point cloud of the flange end face to the plane, the interference of the background and bolt holes is eliminated, ensuring the high accuracy of the plane normal vector. When calculating the bolt hole position, the Euclidean clustering algorithm is used to fit the circle, avoiding the pulling effect of the point-to-circle fitting of other parts of the flange, thereby achieving high-precision pose calculation.

[0021] (6) This invention effectively suppresses the high reflectivity interference of the GIL metal surface through multimodal fusion technology. By utilizing the deep coupling of deep learning semantic segmentation model and precise geometric fitting algorithm, the pose recognition accuracy and robustness of flange and bolt holes in complex construction environments are significantly improved. At the same time, it can accurately analyze and generate high-precision multi-degree-of-freedom pose parameters, providing key motion control inputs for automated robot docking tasks, significantly reducing the dependence on human experience in the construction process, and laying the core technical foundation for realizing intelligent and precise docking and installation of GIL units. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the overall system structure of the present invention; Figure 3 The following are comparison diagrams before and after preprocessing in this invention, wherein (a) is a schematic diagram of the point cloud before preprocessing, and (b) is a schematic diagram of the point cloud after preprocessing. Figure 4 This is a schematic diagram showing the geometric center coordinates of each bolt hole in this invention; Figure 5 These are the three coordinate systems defined in this invention. Detailed Implementation

[0023] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0024] Example 1 This embodiment provides a method for recognizing the pose of a GIL unit flange and bolt holes based on multimodal fusion, aiming to improve the robustness and accuracy of pose recognition under complex working conditions. Specifically, as shown... Figure 1 As shown, the method includes the following steps: Step 1: Acquisition, fusion, and alignment of multimodal raw data: This step primarily utilizes a 3D vision sensor to acquire the raw 3D point cloud data of the GIL unit docking end. P raw and synchronized RGB texture images I Based on the sensor's intrinsic and extrinsic parameter calibration matrix, the pixel features of the RGB texture image are mapped and aligned with the spatial coordinates of the point cloud to construct a multimodal fusion dataset.

[0025] like Figure 2As shown, in this embodiment, a high-precision 3D camera based on the structured light imaging principle is used as the 3D vision sensor 1. This 3D vision sensor 1 integrates a texture camera module and a depth sensing module, enabling it to simultaneously acquire two-dimensional RGB texture images and three-dimensional point cloud data of the target area.

[0026] In this embodiment, the 3D vision sensor 1 is fixed to the end effector of an industrial robot (not shown) via a dedicated mounting bracket, forming an "eye-in-hand" configuration mode; or it is fixed to a pre-set bracket at the construction site to ensure that its field of view (FOV) completely covers the end face of the GIL unit flange 2 to be docked.

[0027] The GIL unit flange 2 is made of a typical metal material, and its surface has multiple bolt holes arranged in a circle. Due to the specular reflection characteristics of the metal surface, it is prone to produce local bright areas (i.e., high reflective noise) under strong light or supplementary light conditions.

[0028] In this embodiment, the acquisition and alignment of multimodal data are prerequisites for achieving accurate identification. This step specifically includes: First, the GIL unit flange 2 is scanned using 3D vision sensor 1. The texture camera module inside 3D vision sensor 1 acquires a two-dimensional RGB texture image. I Meanwhile, the depth perception module generates the corresponding raw point cloud data. P raw To ensure data consistency, a synchronization trigger signal is sent to the 3D vision sensor via an external server. I and P raw Generated at the same timestamp, thus eliminating deviations caused by dynamic changes in the environment.

[0029] Obtain raw point cloud data P raw Then, multimodal alignment is performed using the calibration information pre-set by the 3D vision sensor 1. This calibration information includes the intrinsic parameter matrix of the texture camera. K And the extrinsic transformation matrix between the texture camera and the point cloud coordinate system[ R | t ].

[0030] For the original point cloud P raw Each three-dimensional point in P i ( X i , Y i , Z i The following perspective projection model is used to calculate its representation in the RGB texture image.I The corresponding pixel coordinates ( u i , v i ): , In the formula, As a scale factor, For the first Points in RGB texture image I homogeneous pixel coordinates on The extrinsic transformation matrix, For the first The three-dimensional coordinates of a point in the world coordinate system It is a translation vector; Based on the calculated pixel coordinates ( u i , v i ), to extract from RGB texture images I Extract the corresponding color information ( R i , G i , B i ), and assign it to the corresponding 3D point. P i This forms a set of six-dimensional attributes. X i , Y i , Z i , R i , G i , B i ] three-dimensional points P i .

[0031] After the above mapping process, the original point cloud P raw Transforming simple geometric coordinate data into data containing six-dimensional attributes. X , Y , Z , R , G , B Multimodal fusion point cloud P fusion .

[0032] The algorithm in this embodiment runs on Linux or Windows operating systems and processes the raw data through a core algorithm module.

[0033] The algorithm framework uses Python as the development language and integrates Open3D and OpenCV libraries for basic point cloud and image processing. The deep learning part is built on the PyTorch or TensorFlow framework and utilizes GPUs (Graphics Processing Units) for real-time semantic segmentation and feature extraction on multimodal fusion datasets.

[0034] Step 2, adaptive preprocessing of point clouds: This step primarily targets the reflection interference on the metal surface of the GIL flange. It employs a strategy that combines noise removal based on spatial distribution characteristics with data simplification to suppress artifact data generated by reflection and optimize point cloud density in order to obtain a target point cloud with geometric consistency.

[0035] This step involves acquiring a multimodal fused point cloud. P fusion Then, adaptive preprocessing (denoising and downsampling) is performed first, and the comparison of the effects before and after processing is as follows: Figure 3 As shown, Figure 3 Figure (a) shows the original point cloud. P original , Figure 3 Figure (b) shows the preprocessed point cloud. P after .

[0036] For outliers caused by specular reflection on metal surfaces, a statistical filter (SOR) is preferred for point removal: For multimodal fusion point clouds P fusion For each point in the array, calculate the average distance to its k nearest neighbors; calculate P fusion The mean μ and standard deviation σ of the average distance between all points; The dynamic threshold range is set to [μ-σ*1.0, μ+σ*1.0]. If the average distance of a point falls outside this range, it is identified as an outlier and removed. The result is a multimodal fused point cloud after noise removal.

[0037] Preferred parameters in this embodiment: Number of neighborhood analysis points k The value is 50, which is the threshold for multiples of standard deviation. σ The value is 1.0.

[0038] To balance recognition accuracy and computational efficiency, a voxel grid downsampling strategy is used for data simplification: For the multimodal fused point cloud after noise removal, its three-dimensional space is divided into multiple voxels; Group all points that fall within the same voxel into one group; For all points in each group, calculate the geometric center (centroid) of its three-dimensional coordinates, and use this centroid to uniquely represent all points within that voxel, ultimately obtaining the preprocessed multimodal fused point cloud.

[0039] Preferred parameters for this embodiment: The leaf node size of the voxel grid is 1.0 mm.

[0040] This step, through denoising and a streamlined preprocessing workflow, not only ensures the quality of the input data and avoids noise interference in the calculation of the downsampling centroid, but also obtains a computationally friendly and uniformly distributed point cloud dataset while preserving geometric features, providing a high-quality data foundation for subsequent point cloud segmentation and pose calculation.

[0041] Step 3: Semantic segmentation based on multimodal feature fusion: This step mainly involves constructing a point cloud semantic segmentation model, using the texture edge features of the RGB texture image as prior information input to the model; and achieving deep fusion of image texture and point cloud geometric features through feature fusion processing, accurately segmenting the flange end face point cloud, the point cloud of each bolt hole area, and background noise from the original data.

[0042] The core of this step lies in overcoming the limitations of single-modal recognition through deep fusion of 2D / 3D features. The specific model architecture is constructed using the PointNeXt network, which includes multiple stacked SA modules, an inverted residual MLP following each SA module, and a decoding output module at the end. The SA modules are used to downsample and initially aggregate the preprocessed multimodal fusion point cloud. The inverted residual MLP enhances the expressive power of the features based on the SA modules. The decoding output module outputs the flange end face point cloud through feature decoding and mapping. P face Point cloud of bolt hole area P hole And background noise. This embodiment introduces a reciprocal residual multilayer perceptron, which enhances the network's ability to represent subtle geometric features of the GIL flange (such as bolt hole edges) by first increasing and then decreasing the dimensionality. Given the prior characteristic that the GIL unit flange 2 has a fixed size, the model is able to simultaneously capture the global contour features of the GIL unit flange 2 and the local void features of the bolt holes by adjusting the ball query radius of the SA module.

[0043] Each SA module includes a fusion layer and texture branches, geometry branches, and downsampling layers connected to it. The execution steps of each SA module include: The output of the previous SA module is used as the input of the current SA module. The farthest point is sampled using the downsampling layer to obtain the set of center points. Then, a neighborhood query is performed using the set sphere query radius to obtain the set of neighborhood points. In the case that the current SA module is the first SA module, its input is the preprocessed multimodal fusion point cloud. The neighborhood point set is used as input to the texture branch, which uses a convolutional neural network to extract color texture and edge prior information from the RGB texture image as texture features. Since bolt holes appear as obvious dark shadow areas in two-dimensional images, this branch can effectively provide initial positioning information for the holes. Simultaneously, the neighborhood point set is used as input to the geometry branch, which uses a point cloud encoder to extract normal vectors and local curvature features in three-dimensional space as point cloud features. Texture features are applied as spatial weights to point cloud features. Even when high reflectivity causes local point cloud sparseness, the model can still accurately identify the semantic boundaries of bolt holes guided by image features. An attention mechanism is then used for fusion to obtain fused features. The specific steps include: First, the image features extracted from the texture branches Point cloud features extracted with geometric branching Mapped to the same feature dimension .

[0044] , In the formula, It is the set of real numbers; This represents the number of points after downsampling by the current SA module.

[0045] Generate spatial weight vector using texture features It is used to suppress geometric noise in highly reflective areas and enhance the response in edge areas.

[0046] , , In the formula, τ Use the Sigmoid activation function; This represents the Gating Multi-Layer Perceptron, a small neural network consisting of fully connected layers and activation functions. This represents the weighted point cloud features.

[0047] Based on weighted features, an attention mechanism is used to capture the nonlinear correlation between two modalities. Definition (Query volume) (Key, value) and (Value, numerical value): Weighted point cloud features , Original texture features , Original texture features , In the formula, To query the weight matrix; This is the key-value weight matrix; This is a numerical weight matrix.

[0048] Fusion features The calculation formula is: V , , In the formula, This is the normalization function; For matrix transpose; This indicates the calculation of the similarity score between "point cloud query" and "texture key value". Scaling factor It is the dimension of the feature vector. This represents a stitching operation that "combines" the original (weighted) point cloud features with the newly extracted attention features along the channel dimension. The output projection layer compresses the stitched high-dimensional features back to the preset dimension and performs non-linear feature extraction.

[0049] During the training phase, random jittering and rotation enhancements were introduced before inputting PointNeXt to address noise caused by the high reflectivity of metal, thereby improving the robustness of the model in complex field environments.

[0050] After completing the semantic segmentation of the point cloud, the point cloud set belonging to the flange end face is obtained. P face Point clusters belonging to the bolt hole area P hole And background noise.

[0051] Step 4: Multi-stage geometric feature extraction and pose calculation: This step mainly utilizes a pose calculation algorithm to fit spatial geometric elements based on the extracted point cloud of the flange end face, extracting geometric features that reflect the orientation of the GIL unit flange 2; and identifies the structural features of bolt holes within the target area, determining the spatial center position of each bolt hole through coordinate calculation; and calculates the multi-degree-of-freedom pose parameters of the GIL unit flange 2 relative to the sensor coordinate system by combining the relative spatial relationship between the geometric features and the center position.

[0052] To achieve the logical mapping from "visual perception" to "action execution", this embodiment defines as follows: Figure 5 The three core coordinate systems shown are: Sensor coordinate system { S}: Taking the optical center of 3D vision sensor 1 as the origin O s It is used to define the metric space of the original multimodal data. X s The axis points in the negative direction of the optical axis. Y s The axis points to the right side of the sensor plane, which is the horizontal rightward direction of the RGB texture image.

[0053] Flange target coordinate system { T}: The origin is the geometric center of GIL unit flange 2. O t ,That X t The axis coincides with the normal vector of the end face. Y t The axis points to the preset starting bolt hole, used to describe the instantaneous pose of the target.

[0054] Reference coordinate system { B In this embodiment, the preferred coordinate system is the robot base coordinate system or the global coordinate system of the construction site. This coordinate system serves as the absolute reference for motion control, and it is related to the sensor coordinate system. S The transformation relationship between them is obtained through pre-performed hand-eye calibration and is denoted as the transformation matrix. T BS .

[0055] like Figure 4 As shown, the points on the flange end face are first clustered. P face Planar feature extraction is performed using the Random Sample Consensus (RANSAC) algorithm. P face By fitting spatial geometric elements, the equation of the spatial plane is obtained: , In the formula, , , , It is a constant. , , Not both equal to 0, The coordinates of the point; Calculate the unit normal vector of the space plane. n =( A , B , C This vector is the target coordinate system { T In the sensor coordinate system { S} X Axis vector direction.

[0056] After determining the reference plane (space plane), the bolt hole points are clustered. P hole Perform analysis. P hole Projecting 3D points to Figure 4 Fitting plane in Above, using the Euclidean clustering algorithm to... P hole Divided into N A separate cluster of bolt holes ( N (The number of bolt holes is preset for GIL unit flange 2). For each point cluster, a circle fit is performed using the least squares method to obtain the geometric center coordinates of each bolt hole in space. C i ( x i , y i , z i ), as the spatial pose of the bolt hole, such as Figure 4 As shown.

[0057] Calculate the geometric mean of the center points of all bolt holes, and use it as the target center point of the GIL unit flange 2. O t : , Furthermore, to improve robustness under partial occlusion conditions, this embodiment determines the origin... O t ( x 0 , y 0 When ), it is not simply an arithmetic mean, but rather the extracted... m The center point of each bolt hole is used as the observation value, combined with the preset radius dimension of the GIL unit flange 2. R The optimized geometric center point is obtained by using the least squares circle fitting algorithm in reverse iteration.O t That is, the coordinates of the center of the circle (origin): , In the formula, ( x 0 , y 0 ) is the initial solution.

[0058] Based on the above geometric features, the pose matrix is ​​constructed and transformed step by step. First, the pose matrix of the GIL unit flange 2 relative to the sensor coordinate system {S} is constructed. T ST Among them, the rotating part R ST From the normal vector n Determined together with the bolt hole phase vector, the translation part t ST Origin O t exist{ S The coordinates are determined using a pre-defined calibration matrix. T BS The recognition results ( T ST Transform to the reference coordinate system { B The calculation formula is as follows: , like Figure 5 As shown, the final result is T BT This refers to the 6D pose parameters of the GIL unit flange 2 in the robot's workspace, including three translational components. x , y , z ) and three rotational components ( α , β , γ ).

[0059] In summary, this embodiment effectively suppresses the high reflectivity interference of the GIL metal surface through multimodal fusion technology. By deeply coupling the deep learning semantic segmentation model with the precise geometric fitting algorithm, it significantly improves the pose recognition accuracy and robustness of the flange and bolt holes in complex construction environments. At the same time, this embodiment can accurately analyze and generate high-precision multi-degree-of-freedom pose parameters, providing key motion control inputs for automated robot docking tasks. This significantly reduces the reliance on human experience in the construction process and lays the core technical foundation for realizing the intelligent and precise docking and installation of GIL units.

[0060] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0061] Example 2 This embodiment provides a multimodal fusion-based system for recognizing the position and orientation of GIL unit flanges and bolt holes, such as... Figure 2 As shown, it includes the 3D vision sensor 1 shown in Embodiment 1 above and the control computing unit 3 connected thereto.

[0062] 3D vision sensor 1 will collect 3D point clouds P raw and synchronized RGB texture images I The data is sent to the control and computing unit 3, where, to ensure data consistency, the control and computing unit 3 sends a synchronization trigger signal to the 3D vision sensor 1 to ensure... I and P raw Generated at the same timestamp, thus eliminating deviations caused by dynamic changes in the environment.

[0063] The control computing unit 3 uses an industrial control computer with high-performance graphics processing capabilities. It is equipped with the point cloud semantic segmentation model and pose calculation algorithm shown in Embodiment 1 above, and communicates with the 3D vision sensor 1 at high speed through an Ethernet or USB 3.0 interface.

[0064] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0065] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0066] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0067] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0068] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0069] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A GIL unit flange plate and bolt hole position recognition method based on multi-modal fusion, characterized by, Includes the following steps: Acquiring a three-dimensional point cloud of a GIL unit mating end P raw and synchronized RGB texture images I , performing multi-modal fusion alignment processing to obtain a multi-modal fusion point cloud P fusion ; For the multimodal fused point cloud P fusion First, preprocessing is performed, then semantic segmentation is performed using a point cloud semantic segmentation model to output the point cloud of the flange end face. P face Point cloud of bolt hole area P hole and background noise; based on P face Spatial geometric element fitting is performed to extract geometric features reflecting the flange orientation, and based on... P hole Calculate the geometric center coordinates of each bolt hole in space, which are used as the pose of the bolt holes; Based on the relative spatial relationship between the geometric features and the coordinates of the geometric center, the multi-degree-of-freedom pose parameters of the flange relative to the sensor coordinate system are calculated.

2. The GIL unit flange plate and bolt hole position recognition method based on multi-modal fusion according to claim 1, characterized in that, The obtained multi-modal fusion point cloud P fusion The steps include: Obtain the set calibration information, including the intrinsic parameter matrix of the texture camera. K The extrinsic transformation matrix between the texture camera and the point cloud coordinate system. R | t ]; For the three-dimensional point cloud P raw Each three-dimensional point in P i ( X i , Y i , Z i ), using perspective projection model to calculate P i In the RGB texture image I The corresponding pixel coordinates ( u i , v i The perspective projection model is represented as follows: ， In the formula, As a scale factor, For the first Points in RGB texture image I homogeneous pixel coordinates on The extrinsic transformation matrix, For the first The three-dimensional coordinates of a point in the world coordinate system It is a translation vector; Based on the pixel coordinates ( u i , v i From the RGB texture image I Extract the corresponding color information ( R i , G i , B i The color information is then assigned to the corresponding three-dimensional point. P i This forms a set of six-dimensional attributes. X i , Y i , Z i , R i , G i , B i ] three-dimensional points P i Finally, a multimodal fused point cloud was obtained. P fusion .

3. The GIL unit flange plate and bolt hole pose recognition method based on multi-modal fusion according to claim 1, characterized in that, The preprocessing process includes: For each point in the multi-modal fused point cloud P fusion compute its average distance to the nearest k neighborhood points; Computing P fusion average the mean μ and the standard deviation σ of the distances of all points, and set a dynamic threshold interval based on the mean μ and the standard deviation σ; Fusing multi-modal point clouds P fusion The points falling outside the dynamic threshold interval are rejected to obtain the multi-modal fused point cloud after noise point rejection. For the multimodal fused point cloud after noise removal, its three-dimensional space is divided into multiple voxels, and all points located in the same voxel are grouped together. For all points in each group, calculate the geometric center of its three-dimensional coordinates, and use the geometric center point to represent all points within its voxel, finally obtaining the preprocessed multimodal fused point cloud.

4. The GIL unit flange plate and bolt hole position recognition method based on multi-modal fusion according to claim 1, characterized in that, The point cloud semantic segmentation model is constructed using the PointNeXt network, which includes multiple stacked SA modules, an inverse residual multilayer perceptron following each SA module, and a decoding output module at the end. The SA modules are used to downsample and initially aggregate the preprocessed multimodal fusion point cloud. The inverse residual multilayer perceptron is used to enhance the feature representation ability based on the SA modules. The decoding output module is used to output the flange end face point cloud through feature decoding and mapping. P face Point cloud of bolt hole area P hole And background noise.

5. The GIL unit flange plate and bolt hole position recognition method based on multi-modal fusion according to claim 4, characterized in that, Each SA module includes a fusion layer and texture branches, geometry branches, and downsampling layers connected to it respectively. The execution steps of each SA module include: The output of the previous SA module is used as the input of the current SA module. The farthest point is sampled using the downsampling layer to obtain the set of center points. Then, a neighborhood query is performed using the set sphere query radius to obtain the set of neighborhood points. In the case that the current SA module is the first SA module, its input is the preprocessed multimodal fusion point cloud. The neighborhood point set is used as input for the texture branch and the geometric branch to extract pixel-level texture features and three-dimensional point cloud features, respectively. The texture features are color texture and edge prior information, and the point cloud features are normal vectors and local curvature features. The texture features are applied as spatial weights to the point cloud features, and an attention mechanism is used for fusion to obtain fused features. The specific steps include: texture features extracted by the texture branch point cloud features extracted by the geometry branch mapped to the same feature latitude D : wherein, is the number of points after down-sampling by the current SA module, is a set of real numbers; generate a spatial weight using the texture feature and generate the spatial weight using the spatial weight weight the point cloud feature to obtain a weighted point cloud feature , denoted as: ， ， In the formula, For gated multilayer sensing, Use the Sigmoid activation function; based on the texture features and weighted point cloud features with attention mechanism to capture the nonlinear correlation between the two modal features: ， ， ， In the formula, For query volume, As key-value pair For numerical values, To query the weight matrix, This is the key-value weight matrix. This is a numerical weight matrix; Based on weighted point cloud features Combined with the query volume Key value and numerical values By performing fusion, fusion characteristics are obtained. , is represented as: ， ， In the formula, To output the projection layer, For splicing operations, For attention mechanisms, Scaling factor The dimension of the feature vector. This is the normalization function.

6. The GIL unit flange plate and bolt hole pose recognition method based on multi-modal fusion according to claim 1, characterized in that, The steps for extracting the geometric features include: Adopt the random sampling consistency algorithm to the flange end face point cloud P face Carrying out space geometric element fitting, obtaining a space plane, and being expressed as: ， In the formula, , , , It is a constant. , , Not both equal to 0, The coordinates of the point; Calculate the unit normal vector of the space plane. n =( A , B , C The unit normal vector n For the target coordinate system of the flange { T In the sensor coordinate system { S } X The axial vector direction is used as a geometric feature reflecting the flange orientation, wherein the flange target coordinate system { T The origin is taken as the geometric center point of the flange. O t ,That X t The normal vector of the shaft coincides with that of the flange end face. Y t The axis points to the preset starting bolt hole, and the sensor coordinate system { S } Taking the optical center of the sensor as the origin O s ,That X s The axis points in the negative direction of the optical axis. Y s The axis points to the right side of the sensor plane, which is the horizontal rightward direction of the RGB texture image.

7. The GIL unit flange plate and bolt hole pose recognition method based on multi-modal fusion according to claim 6, characterized in that, The steps for calculating the coordinates of the geometric center include: Point cloud of bolt hole area P hole The points are projected onto the spatial plane, and then Euclidean clustering algorithm is used to generate the point cloud of the bolt hole region. P hole Divide into N independent bolt hole clusters, where N corresponds to the number of bolt holes on the flange; For each bolt hole cluster, the least squares method is used to perform circle fitting to obtain the geometric center coordinates of each bolt hole in space. C i ( x i , y i , z i ).

8. The GIL unit flange plate and bolt hole pose recognition method based on multi-modal fusion according to claim 1, characterized in that, The calculation steps for the multi-degree-of-freedom pose parameters include: Based on the geometric center coordinates, the geometric center point mean value of all bolt holes is calculated as the geometric center point of the flange plate O t is expressed as: ， In the formula, N is the number of bolt holes on the flange plate, is the first geometric center coordinate of the first bolt hole in space; The geometric center point O t ( x 0 , y 0 As the initial solution, combined with the radius dimension of the flange. R The optimized geometric center point is obtained by using the least squares circle fitting algorithm in reverse iteration. O t The origin is called the origin and is represented as: ， In the formula, For the first The two-dimensional coordinates of the extracted bolt hole center point in the fitting plane, subscript This is the index of the observation point, with a value ranging from 1 to... ; Based on the aforementioned geometric features and the optimized geometric center point O t The pose matrix of the flange relative to the sensor coordinate system {S} is constructed. T ST The pose matrix T ST Includes the rotating portion determined by the aforementioned geometric features and the bolt hole phase vector. R ST The optimized geometric center point O t The coordinates in the sensor coordinate system {S} are used as the translation component. t ST ; The pose matrix T ST Combined with the set calibration matrix T BS Transform to reference coordinate system { B The multi-degree-of-freedom pose parameters of the flange in the robot's workspace are obtained below. T BT The reference coordinate system { B } refers to the robot base coordinate system or the global coordinate system of the construction site, and the multi-degree-of-freedom pose parameters T BT Represented as: 。 9. A GIL unit flange and bolt hole pose recognition system based on multimodal fusion, characterized in that, It includes a vision sensor and a connected control computing unit. The vision sensor is used to scan the flange of the GIL unit and acquire a three-dimensional point cloud of the GIL unit docking end. P raw and synchronized RGB texture images I The control and computing unit is used to process the multimodal fusion-based GIL unit flange and bolt hole pose recognition method described in any one of claims 1-8.

10. A storage medium having stored thereon a program, characterized by When the program is executed, it implements a multimodal fusion-based method for recognizing the position of a GIL unit flange and bolt holes as described in any one of claims 1-8.

Images