An anode copper plate clamp high-precision positioning method based on multi-modal data fusion

Abstract

The application discloses a kind of based on multimodal data fusion's anode copper plate clamp high-precision positioning method, belong to computer vision technical field.It includes: first, the situation of anode copper plate in image is obtained by processing RGB image through convolutional neural network, ensure the accurate identification of the position and state of anode copper plate in image;Second, the feature of point cloud graph is extracted by multilayer perception, after voxel-based processing method, the graph after voxelization is projected to two-dimensional plane, obtain the feature after processing, retain spatial detail information;RGB image data features and the features obtained after processing of point cloud graph are spliced and fused, make full use of the complementarity of two kinds of modal data;Finally, a new type of deep learning network is constructed, based on the method, after model training is completed, the coordinates of anode copper plate center point, pitch angle alpha and yaw angle beta are output by inputting RGB image and point cloud graph, and high-precision positioning of anode copper center point coordinates is realized by continuous optimization.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a high-precision positioning method for anodized copper plate fixtures based on multimodal data fusion in complex environments. Background Technology

[0002] Currently, the application of multimodal fusion technology in the field of computer vision has opened up new avenues for improving the accuracy and reliability of positioning. Considering the low accuracy of traditional positioning methods under complex working conditions, the positioning operation scenario for anode copper is evolving into a hybrid mode where single-modal positioning and multimodal fusion positioning coexist.

[0003] Current research on anode copper positioning methods mainly focuses on single-mode positioning frameworks, which rely on data collected by a specific sensor and utilize its corresponding positioning algorithm to determine the location. However, this type of method focuses on optimizing the positioning effect of a single information source, resulting in poor accuracy. Furthermore, it struggles to achieve good adaptability in complex scenarios and cannot significantly improve the accuracy of target object positioning across a wide range of viewing angles. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a high-precision positioning method for anode copper plate fixtures based on multimodal data fusion, thus solving the aforementioned technical problems.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a high-precision positioning method for an anode copper plate fixture based on multimodal data fusion, comprising the following steps:

[0006] S1. The dataset consists of 2000 sets of copper plate images (the working environment of the anode copper includes the sampling site, where various large equipment are working together, and there are risks such as mechanical, noise, and chemical substances). Each set contains one RGB image and one point cloud image of an anode copper plate.

[0007] S2. RGB Image Acquisition: Use an RGB camera to acquire an image containing the anode copper plate. Perform convolution, pooling, and downsampling operations on the RGB channels of the acquired image to accurately capture the key information of the anode copper plate in the image. (Compared with the "threshold segmentation method", this method captures the details of the anode copper feature more accurately and is more adaptable to complex backgrounds and lighting changes.) This yields accurate RGB image features, denoted as [B,h,w,c'], where B = 64, h = 1080, w = 1920, and c' = 256.

[0008] S3. Point cloud image acquisition and preprocessing: The point cloud image features are extracted by a three-layer perceptron. The three-dimensional space is divided into voxels by a voxel-based processing method. The points in the point cloud image are assigned to the corresponding voxels. The voxelized data is projected onto a two-dimensional plane to obtain the processed features, denoted as [B,h,w,d], where d ranges from [0,10].

[0009] S4. Multimodal Data Fusion: The preprocessed RGB image features from S2 and the transformed two-dimensional data from the point cloud map in S3 are concatenated along the channel dimension to obtain the fused features, denoted as [B,h,w,c'+d]. This concatenation method organically combines the data features of the two modalities, providing accurate information for subsequent processing. (The "single-modal RGB image direct positioning method" only locates the anode copper based on the RGB image, which is easily affected by illumination and occlusion.) However, fusing RGB image and point cloud map features can comprehensively utilize the advantages of both, enabling more accurate positioning of the anode copper in complex environments.

[0010] S5. Semantic Segmentation: The fused features from S4 are used for semantic segmentation, employing an encoder-decoder structure. The encoding process extracts more abstract features step by step through convolution and pooling operations, while the decoding process restores the features to a size similar to the original image through upsampling and other operations. At the same time, features from different levels are fused to obtain the final semantic segmentation result, denoted as [B,h,w,2], where 2 indicates segmentation into two categories: anode copper plate and background.

[0011] S6. Model Establishment: The semantic segmentation results obtained in S5 are processed through a convolutional neural network and residual coding is applied to fuse and encode features at different levels. (The "convolutional stacking localization method" only relies on stacking convolutional layers to process semantic segmentation results, which is difficult to effectively fuse multi-scale features. By using residual coding, features at different levels can be better fused, accurately capturing the complex features of anode copper and achieving higher localization accuracy. This results in a compact and accurate feature vector sequence [B,C,H,W], where C is the encoded feature dimension, and H and W are the height and width of the encoded feature map. The feature vector sequence [B,C,H,W] is then processed through a generative adversarial network to gradually expand the low-dimensional feature map into a voxel grid with a certain spatial resolution. During training, the network continuously optimizes itself through self-play. After this stage of adversarial training, the voxel model output by the generator is denoted as [B,V]. x V y V z ], V x V y V z These represent the number of voxels in the x, y, and z directions, respectively; the voxel model [B, V] x V y V zThe point cloud is converted to a size of [B, N, 3+d], where N is the number of points in the converted point cloud, ranging from [1000, 10000], and 3 represents the spatial coordinates x, y, z. Through multiple ensemble abstraction layers, local regions are divided into the point cloud, and features are extracted. A fully connected layer is then used to process the point cloud image after local feature extraction, ultimately outputting the center point coordinates, i.e., [B, x, y, z, α, β].

[0012] As a preferred embodiment of the present invention, step S2 includes the following steps:

[0013] S2.1. Local features are extracted through convolution operations, with the convolution kernel set to 3*3, the number of output channels set to 128, the stride set to 2, and the padding set to 1. Resolution is reduced through pooling operations, with a pooling window size of 2*2 and a stride set to 2. Then, RGB image features are acquired through downsampling operations. Specifically, downsampling is performed twice: the first downsampling uses a 3*3 convolution kernel with 128 output channels, and the second downsampling uses a 3*3 convolution kernel with 256 output channels.

[0014] As a preferred embodiment of the present invention, step S3 includes the following steps:

[0015] S3.1. Normalize the point cloud data, where the point cloud is [B, N, f], where N represents the number of points, and f represents the coordinates and depth of each point, i.e., [x, y, z, d]. Input the processed point cloud features into a network structure containing a three-layer perceptron.

[0016] First, the perceptron is configured with 64 neurons in the first hidden layer, 128 neurons in the second hidden layer, and 256 neurons in the output layer. For each point's 4-dimensional input feature vector, a connection is made with a neuron in the first hidden layer. The connection weight is denoted as W1 (dimension 256), and the bias term is denoted as b1 (dimension 64). After a linear transformation and the ReLU activation function, the output of the first hidden layer is obtained. The calculation formula is as follows (expressed in vector form, where f1 represents the input feature vector [x, y, z, d]):

[0017] h1 = ReLU(f1W1 + b1)

[0018] Next, the output h1 of the first hidden layer is used as the input of the second hidden layer. The connection weights between the neurons in the second hidden layer are W2 (dimension 8192), and the bias term is b2 (dimension 128). Similarly, after linear transformation and activation function, the output of the second hidden layer is obtained.

[0019] h2 = ReLU(f2W2 + b2)

[0020] Finally, the output h2 of the second hidden layer is fed into the output layer, with connection weights W3 (32768 dimensions) and a bias term b3 (256 dimensions), resulting in a 256-dimensional feature vector for each point:

[0021] y = h²W³ + b³

[0022] The model output, i.e., feature data y = [B, N, 256], is calculated through the above process. The connection weights (W1, W2, W3) and bias terms (b1, b2, b3) of each layer are optimized and adjusted during training using the backpropagation algorithm combined with the loss function.

[0023] Initialize weights: W1 ~ N(0, 0.5), Initialize biases: b1, b2, b3, initialized to 0; weight update of layer l: Bias term update: Where L total This represents the loss in the final output.

[0024] S3.2. Given a cube with side length s = 0.01m, calculate the voxel quantity that can be divided in each direction, using V = V x *V y *V z The total number of voxels, V, is obtained. Each point in the point cloud is traversed, and the voxel index to which each point belongs is determined according to its coordinates. For each voxel, the 256-dimensional features of the points falling into the same voxel are counted from the feature extraction steps. After voxelization, the data format is changed from the original point-based representation [B,N,256] to the voxel-based representation [B,V,256], where V represents the number of voxels, ranging from [1000,10000].

[0025] S3.3. The voxelized data [B, V, 256] undergoes a projection transformation. The coordinate information of each voxel is projected onto a two-dimensional plane according to rules. When projecting along the x-axis onto the yz-plane, only the y and z coordinates are retained as the projected coordinates. The feature information corresponding to each projection position and the distance d from the laser to the corresponding point are recorded. The pixel position of each projection point in the target image is calculated through linear mapping. The coordinate range in the y-direction is [y...]. min ,y max The coordinate range in the z-direction is [z min ,z max ], voxel coordinates are (x v ,y v ,z v ), where x v y v The range is [-100, 100], zv The coordinates obtained after projection are in the range [0, 200].

[0026]

[0027] After this projection step, the data format changes from the original voxelized data [B,V,g] to [B,h,w,d], where h is the set image height, w is the image width, and d represents the distance from the laser radar to the point.

[0028] As a preferred embodiment of the present invention, step S6 includes the following steps:

[0029] S6.1. Input the semantically segmented image data [B,h,w,2] into a convolutional neural network architecture based on an attention mechanism. First, use three convolutional layers to extract features at different levels of the image. Set the kernel of the first convolutional layer to 3*3, the output channels to 16, the stride to 1, and the padding to 1. The first convolution yields F. out1 =ReLU(W*F in1 +b), where F in1 For [B,h,w,2], F out1 The region is [B, h, w, 16], and ReLU is the activation function. Each convolutional layer is followed by a pooling layer, using 2×2 max pooling with a stride of 2. The first pooling yields the feature data F. out11 = [B, h / 2, w / 2, 16]. The max pooling formula is:

[0030]

[0031] Similarly, setting the kernel of the second convolutional layer to 3*3, the output channels to 32, the stride to 1, and the padding to 1, the second convolution yields F. out2 =ReLU(W*F out11 +b). Where F out2 The feature data is defined as [B, h / 2, w / 2, 32], with ReLU as the activation function. A 2×2 max pooling process with a stride of 2 is used, and the second pooling operation yields the feature data F. out22 = [B, h / 4, w / 4, 32]. The max pooling formula is:

[0032]

[0033] The third convolutional layer has a 3x3 kernel, 64 output channels, a stride of 1, and padding of 1. The third convolution yields F. out3 =ReLU(W*F out22 +b). Where F out3The feature data is defined as [B, h / 4, w / 4, 64], with ReLU as the activation function, ReLU(x) = max(0, x). A 2×2 max pooling process with a stride of 2 is used, and the third pooling operation yields the feature data F. out33 = [B, h / 8, w / 8, 64]. The max pooling formula is:

[0034]

[0035] Introducing a spatial attention module to process feature map F out33 An attention weight map A is generated through a convolution operation. The convolution kernel size is 7x7, the number of output channels is 1, and the activation function is σ: A = σ(W4*F). out33 +b). Multiply the attention weight map A element-wise with the input feature map F to obtain the weighted feature map F′=[B,h / 8,w / 8,64]; then, use residual coding to fuse and encode the feature data obtained above according to different levels to obtain a compact and information-rich feature vector sequence, which can be denoted as [B,C,H,W], where C is the dimension of the encoded feature, and H and W are the height and width of the encoded feature map.

[0036] S6.2 Construct a generative adversarial network architecture, where the generator takes the feature vector sequence [B,C,H,W] obtained in S5 as input. The generator consists of three transposed convolutional layers with a kernel size of 4*4, a stride of 2, and padding of 1. These layers progressively upsample the feature maps, attempting to reconstruct the voxel representation [B,V] in three-dimensional space. x V y V z ], where V x V y V z These represent the number of voxels in the x, y, and z directions, respectively. The discriminator is built on a convolutional neural network and performs multi-scale feature analysis on the input 3D voxel model, with a 4x4 kernel, a stride of 2, and padding of 1. The generator aims to maximize the probability of the discriminator misclassifying its generated samples; the generator loss function L... G : Where G(z) represents the sample generated by the generator, and D(G(z)) represents the discriminator's judgment result on the generated sample. The discriminator aims to correctly distinguish between real samples and generated samples, and the discriminator loss function L... D : Where D(x) is the discriminator's judgment result on the real sample. Through adversarial training between the generator and the discriminator, the generator can continuously optimize its generated voxel model, making it closer to the real three-dimensional structure of the anode copper plate. After this stage of adversarial training, the voxel model [B, V] output by the generator becomes... x V yV z ].

[0037] S6.3, Generate the voxel model [B, V x V y V z The data is converted to point cloud format, and the center coordinates of each voxel are extracted as points in the point cloud, resulting in a point cloud image [B, N, 3+w], where N is the number of points in the converted point cloud, 3 represents the spatial coordinates x, y, z, and w is the auxiliary information dimension. The point cloud image [B, N, 3+w] is then input into the PointNet++ network. First, an abstraction layer is used to divide the point cloud into local regions, and features are extracted within each local region. Fully connected layers are then used to process the point cloud image after local feature extraction, ultimately outputting [B, x, y, z, α, β]. Here, x, y, and z are the precise 3D coordinates of the center point of the anode copper plate; α and β are angles calculated based on the overall pose analysis of the point cloud and comparison with a preset coordinate system.

[0038] The following is the calculation process for this step:

[0039] By using an abstraction layer, the point cloud is divided into local regions, and the centroid of points within each local region is calculated. Each local region contains M points, and the coordinates of the i-th point within that region are (x...). i ,y i ,z i If the centroid of the local region (x) is... c ,y c ,z c The calculation formula is: Calculate the relative coordinates (Δx) of each point with respect to the centroid. i ,Δy i ,Δz i ): Δx i =x i -x c Δy i =y i -y c Δz i =z i -z c These relative coordinates are processed using a perceptron. The perceptron has weights of W4, a bias of b4, and output features of e. i :e i =ReLU(W4·[Δx) i ,Δy i ,Δz i ]+b4) where ReLU is the activation function, defined as: ReLU(x)=max(0,x).

[0040] Features are extracted within each local region, and the point cloud image after local feature extraction is processed using a fully connected layer. Assume the weights of the fully connected layer are W. fc The bias is b fc Then the output feature is: F = W fc ·k i +b fc The final output includes the precise three-dimensional coordinates (x, y, z) and angle (α, β) of the center point of the anode copper plate.

[0041] The center point coordinate loss function and the attitude angle loss function are as follows:

[0042]

[0043] Where and are the coordinates and angle of the center point in the actual label, respectively. The total loss is: L total =λ1L center +λ2L angle Initialize weights: λ1 = λ2 = 0.5; Weights: The Adam optimizer is used for updates, with a learning rate η = 0.001, β1 = 0.9, and β2 = 0.999. Through continuous learning and optimization, the loss is reduced, and the accuracy of localization is improved.

[0044] This invention provides a high-precision positioning method for anode copper plate fixtures based on multimodal data fusion, which has the following advantages:

[0045] This invention first performs convolution, pooling, and downsampling operations on the RGB channels of the acquired image to progressively extract effective local features. This refined processing helps to accurately identify the position and state of the anode copper plate in the image, improving the accuracy of positioning-related information acquisition. Secondly, features are extracted using a three-layer perceptron, and through complex linear transformations and activation function calculations, the point cloud features are fully explored. Then, based on voxel processing, the image is precisely projected onto a two-dimensional plane, and detailed coordinate and distance information is recorded. These operations ensure high precision in point cloud feature extraction and transformation, which is beneficial for accurately reflecting the spatial features of the anode copper plate to aid in positioning. Next, an encoder-decoder structure is used to accurately segment the anode copper plate from the background, providing accurate target area information for subsequent positioning, reducing interference, and improving positioning accuracy. Finally, an attention mechanism focuses on key image regions, a generative adversarial network optimizes the voxel model to approximate the real structure, and a PointNet++ network refines the transformed point cloud, outputting high-precision center point coordinates and angles. The entire process continuously improves positioning accuracy through a complex network structure and rigorous training.

[0046] Compared to existing technologies, such as the PointPainting method, which enhances point cloud features by projecting segmentation information from RGB images onto the point cloud, its performance is highly dependent on the accuracy of image segmentation. Inaccurate image segmentation leads to poor point cloud feature enhancement, thus affecting localization accuracy. In contrast, the attention mechanism introduced in this invention dynamically adjusts weights, reducing reliance on single-modality data. Even with inaccurate image segmentation, it maintains high feature extraction capabilities, significantly improving system robustness. Furthermore, the 3D model generated by the generative adversarial network (GAN) through adversarial training does not depend on the accuracy of a single modality, enabling better handling of complex scenes and further improving localization accuracy. While the Transformer-based multimodal fusion model can automatically learn the feature relationships between point clouds and RGB images and improve localization accuracy through global feature learning, its computational complexity is high, and it is less accurate than the voxel models generated by GANs when handling complex 3D structures. The advantages of this invention lie in the attention mechanism's ability to focus on key regions and enhance feature extraction capabilities. The GAN can generate high-precision 3D models and optimize center point coordinates. The PointNet++ network can extract local features and capture geometric details. High-precision 3D models are generated by using convolutional neural networks with attention mechanisms and generative adversarial networks. The PointNet++ network generates accurate coordinates of the center point of the anode copper and the included angle, which can better handle complex scenes and locate the anode copper plate. The positioning accuracy is higher than that of the methods mentioned above. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of a camera taking a picture of an anode copper electrode.

[0048] Figure 2 This is a flowchart illustrating the high-precision positioning method for anode copper plate fixtures based on multimodal data fusion in this invention.

[0049] Figure 3 This is a schematic diagram of the projection transformation of a point cloud image;

[0050] Figure 4 This is a schematic diagram of the deep learning structure of the present invention; Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Please see Figures 1-4A high-precision positioning method for an anode copper plate fixture based on multimodal data fusion, characterized by the following steps:

[0053] S1, such as Figure 1 As shown, the model of this invention is trained on a dataset consisting of 2000 sets of manually collected copper plate images. Each set contains an RGB image and a point cloud image of a copper plate. The goal is to obtain the coordinates of the center point of the anode copper plate and the pitch angle α and yaw angle β.

[0054] S2, such as Figure 2 As shown, RGB image acquisition and preprocessing: An RGB camera is used to acquire an image containing the anode copper plate. Convolution, pooling, and downsampling operations are performed on the RGB channels of the acquired image to obtain the processed RGB image features.

[0055] S3, such as Figure 3 As shown, the point cloud image data is normalized, and the features of the point cloud image data are extracted by a three-layer perceptron. The three-dimensional space is divided into voxels by a voxel-based processing method. The points in the point cloud image data are assigned to the corresponding voxels. The voxelized data is projected onto a two-dimensional plane by projection. Image enhancement technology is used to improve the visual effect of the image. The processed features are obtained and denoted as [B,h,w,d].

[0056] Step S3 includes the following steps:

[0057] S3.1. The point cloud image is normalized, and the processed point cloud features are input into a network structure containing a three-layer perceptron (MLP). The first hidden layer has 64 neurons, the second hidden layer has 128 neurons, and the output layer has 256 neurons. For the 4-dimensional input feature vector of each point, it is connected to the neurons of the first hidden layer, and after linear transformation and ReLU activation function, the output of the first hidden layer is obtained. Then, the output h1 of the first hidden layer is used as the input of the second hidden layer, and after linear transformation and activation function, the output of the second hidden layer is obtained. Finally, the output h2 of the second hidden layer is fed into the output layer, resulting in the final 256-dimensional feature vector of each point. The error between the model output and the true label is measured by a loss function, and the connection weights (W1, W2, W3) and bias terms (b1, b2, b3) of each layer are optimized and adjusted during training using the backpropagation algorithm combined with the loss function.

[0058] S3.2. Set a cube with a side length of s = 0.01m, calculate the number of voxels that can be divided in each direction, and obtain the total number of voxels V. Count the 256-dimensional features of the points that fall into the same voxel, obtained in the previous feature extraction step. After voxelization, the data format is changed from the original point-based representation [B,N,256] to the voxel-based representation [B,V,256].

[0059] S3.3. The voxelized data [B,V,256] is subjected to projection transformation. The projection range and resolution are planned based on the preset image height h and image width w. The pixel position of each projection point in the target image is calculated through linear mapping.

[0060] S4. Multimodal data fusion: The preprocessed RGB image features in S2 and the transformed two-dimensional data of the point cloud in S3 are spliced ​​together in the channel dimension to obtain the fused features.

[0061] S5. Semantic Segmentation: The fused features from S4 are used for semantic segmentation, employing an encoder-decoder structure. The encoding process extracts more abstract features step by step through convolution and pooling operations, while the decoding process restores the features to a size similar to the original image through upsampling and other operations. At the same time, features from different levels are fused to finally obtain the semantic segmentation result, denoted as [B,h,w,2].

[0062] S6, such as Figure 4 As shown, the semantic segmentation result obtained in S5 is processed by a convolutional neural network and residual coding is applied to fuse and encode features at different levels, resulting in a compact and information-rich feature vector sequence [B, C, H, W], where C is the dimension of the encoded feature, and H and W are the height and width of the encoded feature map. The feature vector sequence obtained in S5.1 is then processed by a generative adversarial network to gradually expand the low-dimensional feature map into a voxel grid with a certain spatial resolution. During training, the network continuously optimizes itself through self-play. After this stage of adversarial training, the voxel model output by the generator is denoted as [B, V]. x V y V z ], V x V y V z These represent the number of voxels in the x, y, and z directions, respectively; the voxel model [B, V] x V y V z The data is converted to a point cloud format [B, N, 3+d]. A set abstraction layer is used to divide the point cloud into local regions and extract features. A fully connected layer is then used to process the point cloud image after local feature extraction, ultimately outputting the desired center point coordinates, i.e., [B, x, y, z, a, b].

[0063] Step S6 includes the following steps:

[0064] S6.1. Input the semantically segmented image data [B,h,w,2] into a convolutional neural network architecture based on an attention mechanism. First, use three convolutional layers to extract features at different levels of the image. Set the kernel of the first convolutional layer to 3*3, the output channels to 16, the stride to 1, and the padding to 1. The first pooling operation yields the feature data F. out11 =[B,h / 2,w / 2,16]. The second convolutional layer has a 3x3 kernel, 32 output channels, a stride of 1, and padding of 1. The second convolution yields F. out2 = [B, h / 2, w / 2, 32]. The second pooling yields the feature data F. out22 =[B,h / 4,w / 4,32]. The third convolutional layer has a 3x3 kernel, 64 output channels, a stride of 1, and padding of 1. The third convolution yields F. out3 = [B, h / 4, w / 4, 64], the third pooling operation yields the feature data F. out33 =[B,h / 8,w / 8,64]. A spatial attention module is introduced to process the feature map F. out33 An attention weight map A is generated through convolution operations. The convolution kernel size is 7*7, the number of output channels is 1, and the activation function is σ. The attention weight map A is then multiplied element-wise with the input feature map F to obtain a weighted feature map F′=[B,h / 8,w / 8,64]. Next, residual coding is used to fuse and encode the feature data obtained above according to different levels, resulting in a compact and information-rich feature vector sequence [B,C,H,W].

[0065] S6.2 Construct a generative adversarial network architecture, where the generator takes the feature vector sequence [B,C,H,W] obtained in S6.1 as input. The generator consists of three transposed convolutional layers with a kernel size of 4*4, a stride of 2, and padding of 1. These layers progressively upsample the feature maps, attempting to reconstruct the voxel representation [B,V] in three-dimensional space. x V y V z ], where V x V y V z These represent the number of voxels in the x, y, and z directions, respectively. The discriminator, built on a convolutional neural network, performs multi-scale feature analysis on the input 3D voxel model, with a 4*4 kernel, a stride of 2, and padding of 1. Through adversarial training between the generator and the discriminator, the generator can continuously optimize its generated voxel model, making it closer to the real 3D structure of an anode copper plate. After this stage of adversarial training, the voxel model [B, V] output by the generator...x V y V z ].

[0066] S6.3, Generate the voxel model [B, V x V y V z The image is converted to a point cloud format, and the center coordinates of each voxel are extracted as points in the point cloud, resulting in a point cloud image [B, N, 3+w], where N is the number of points in the converted point cloud, 3 represents the spatial coordinates x, y, z, and w is the auxiliary information dimension. The point cloud image [B, N, 3+w] is then input into the PointNet++ network. First, an ensemble abstraction layer is used to divide the point cloud into local regions, and features are extracted within each local region. Fully connected layers then process the point cloud image after local feature extraction, ultimately outputting [B, x, y, z, α, β]. The model parameters are updated and optimized based on the loss function, continuously improving the model's ability to generate accurate coordinate and angle information from semantically segmented images.

[0067] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended specification and its equivalents.

Claims

1. A high-precision positioning method for an anode copper plate fixture based on multimodal data fusion, characterized in that, include: Obtain the RGB image and point cloud map of the anode copper plate; For the RGB image, RGB image features are obtained through convolution, pooling, and downsampling operations; For the point cloud image, features are extracted using a three-layer perceptron, and after voxelization, they are projected onto a two-dimensional plane to obtain the point cloud image features. The RGB image features and point cloud features are concatenated along the channel dimension to obtain the fused features; The fused features are semantically segmented using an encoder-decoder structure to obtain the segmentation result; The segmentation results are processed through a convolutional neural network and residual coding is applied to obtain a sequence of feature vectors; The process of converting the segmentation result into a sequence of feature vectors includes: Multi-scale features are extracted using a convolutional neural network with an attention mechanism. Three convolutional layers are set up, each with a kernel size of 3*3, and the number of output channels are 16, 32, and 64, respectively. The pooling window size is 2*2, and the stride is 2. A spatial attention module is introduced, which generates a weight map using a 7*7 convolutional kernel and multiplies it element-wise with the feature map to enhance the features of key regions. Residual coding is used to fuse features at different levels to obtain a compact feature vector sequence [B,C,H,W]. The feature vector sequence is expanded into a voxel grid through a generative adversarial network and then converted into a point cloud format; The point cloud map is processed by the PointNet++ network, and the center point coordinates and angle information of the anode copper plate are finally output. The process of converting the voxel mesh into a point cloud and processing it through a PointNet++ network includes: Extract the center coordinates of each voxel as points in the point cloud to obtain the point cloud map [B,N,3+d]. The PointNet++ network is used to segment local regions and extract features through an ensemble abstraction layer. Then, a fully connected layer processes the extracted local features, and a loss function is used to optimize the model parameters. Finally, the precise center point coordinates [B, x, y, z] and angle information of the anode copper plate are output. , ].

2. The method as described in claim 1, characterized in that, For RGB images, convolution, pooling, and downsampling operations are used, including: Local features are extracted through multi-layer convolution, with a kernel size of 3*3, an output channel count of 128, a stride of 2, and padding of 1. The resolution is reduced by pooling operations, with a pooling window size of 2*2 and a stride of 2. Then, the parameters are downsampled twice. The first downsampling parameter is a 3*3 convolution kernel with 128 output channels, and the second downsampling parameter is a 3*3 convolution kernel with 256 output channels. The final RGB image features of size [B,h,w,c'] are obtained.

3. The method as described in claim 1, characterized in that, Processing of point cloud images includes: Normalize the point cloud image to obtain data in the format [B,N,x,y,z,d]. Features are extracted using a three-layer perceptron. The first, second, and third layers have 64, 128, and 256 neurons, respectively. The features are calculated using linear transformation and ReLU activation function, and the weights and bias terms are optimized and adjusted to output 256-dimensional features. The three-dimensional space is divided into voxels, and the point cloud features within each voxel are statistically analyzed. The data format is converted from [B,N,256] to [B,V,256], where V is the number of voxels. The voxelized data is projected onto a two-dimensional plane along the x-axis and then mapped to coordinates to obtain a point cloud feature in the format [B,h,w,d].

4. The method as described in claim 1, characterized in that, The semantic segmentation adopts an encoder-decoder structure. The encoding process extracts abstract features step by step through convolution and pooling. The decoding process restores the features to their original size through upsampling and fuses features at different levels. Finally, the anode copper plate and the background are segmented into two classes. The segmentation result is in the format [B,h,w,2].

5. The method as described in claim 1, characterized in that, The generative adversarial network includes a generator and a discriminator. The generator consists of three transposed convolutional layers with a kernel size of 4*4 and a stride of 2, progressively upsampling the feature vector sequence into a three-dimensional voxel grid [B, , , ]; The discriminator uses a convolutional neural network to perform multi-scale feature extraction and real / false discrimination on the voxel model. The convolutional kernel size is 4*4 and the stride is 2. By training the generator and discriminator adversarially, the generated voxel model is optimized to approximate the real structure.

Images