Anode copper plate clamp high-precision positioning method based on multi-modal data fusion

Abstract

The invention discloses an anode copper plate clamp high-precision positioning method based on multi-modal data fusion, and belongs to the technical field of computer vision. Comprising the steps that firstly, an RGB image is processed through a convolutional neural network to obtain the condition of an anode copper plate in the image, and it is ensured that the position and state of the anode copper plate in the image are accurately recognized; secondly, extracting point cloud picture features through a multi-layer perceptron, projecting a voxelized picture onto a two-dimensional plane through a voxel-based processing method to obtain processed features, and reserving space detail information; rGB image data features and features obtained after point cloud image processing are spliced and fused, and complementarity of two kinds of modal data is fully utilized; finally, a novel deep learning network is constructed, after model training is completed based on the method, coordinates, a pitch angle alpha and a yaw angle beta of the center point of the anode copper plate are output by inputting an RGB image and a point cloud picture, and high-precision positioning of the coordinates of the center point of the anode copper plate is achieved through continuous optimization.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a high-precision positioning method for an anode copper plate fixture based on multimodal data fusion in a complex environment. Background Art

[0002] The application of multimodal fusion technology in computer vision has opened up new avenues for improving positioning accuracy and reliability. Given the low accuracy of traditional positioning methods under complex working conditions, anode copper positioning operations are evolving into a hybrid model combining single-modal positioning with multimodal fusion positioning.

[0003] Current research on anode copper positioning methods primarily focuses on single-modal positioning frameworks, which use data collected by a specific sensor and its corresponding positioning algorithm to determine the position. However, these methods focus on optimizing the positioning effect of a single information source, resulting in poor accuracy. Furthermore, they struggle to adapt to complex scenarios and are unable to effectively improve the accuracy of target object positioning over a wide range of viewing angles. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the present invention provides a high-precision positioning method for an anode copper plate fixture based on multimodal data fusion, which solves the above technical problems.

[0005] To achieve the above object, the present invention provides the following technical solution: a high-precision positioning method for an anode copper plate fixture based on multimodal data fusion, comprising the following steps:

[0006] S1. The dataset consists of 2,000 sets of copper plate images (the working environment of anode copper includes the sampling site, where various large-scale equipment work together, and there are risk factors such as machinery, noise, and chemicals). Each set contains an RGB image and a point cloud map of an anode copper plate.

[0007] S2. RGB image acquisition: Use an RGB camera to acquire an image containing the anode copper plate. Perform convolution, pooling, and downsampling operations on the RGB channels of the acquired image to accurately capture the key information of the anode copper plate in the image. (Compared with the "threshold segmentation method", this method captures the details of the anode copper features more accurately and is more adaptable to complex backgrounds and lighting changes.) The resulting precise RGB image features are denoted as [B, h, w, c'], where B = 64, h = 1080, w = 1920, and c' = 256.

[0008] S3. Point cloud acquisition and preprocessing: Extract point cloud features using a three-layer perceptron. Using a voxel-based processing method, divide the three-dimensional space into voxels. Assign points in the point cloud to corresponding voxels. Project the voxelized data onto a two-dimensional plane to obtain processed features, denoted as [B, h, w, d], where d is in the range [0, 10].

[0009] S4, multimodal data fusion: The pre-processed RGB image features in S2 and the transformed two-dimensional data of the point cloud in S3 are spliced in the channel dimension to obtain the fused features, recorded as [B, h, w, c'+d]. This splicing method can organically combine the data features of the two modalities, providing accurate information for subsequent processing. (The "single-modal RGB image direct positioning method" locates anode copper based solely on RGB images and is easily affected by lighting and occlusion.) Fusion of RGB image and point cloud features can comprehensively utilize the advantages of both and more accurately locate anode copper in complex environments.

[0010] S5, semantic segmentation: The features fused in S4 are subjected to semantic segmentation using an encoder-decoder architecture. The encoding process gradually extracts more abstract features through convolution and pooling operations, while the decoding process restores the features to a size close to the original image through operations such as upsampling. Features at different levels are also integrated to obtain the semantic segmentation result, which is denoted as [B, h, w, 2], where 2 indicates the segmentation into two categories: the anode copper plate and the background.

[0011] S6. Model establishment: The result of semantic segmentation in S5 is passed through a convolutional neural network and residual coding is used to fuse and encode features at different levels. (The "convolution stacking positioning method" only relies on stacking convolution layers to process semantic segmentation results, which is difficult to effectively fuse multi-scale features. In this way, the use of residual coding can better fuse features at different levels, accurately capture the complex features of anode copper, and achieve higher positioning accuracy. In this way, a compact and accurate feature vector sequence [B, C, H, W] is obtained, where C is the feature dimension after encoding, and H and W are the height and width of the feature map after encoding; the feature vector sequence [B, C, H, W] is passed through a generative adversarial network to gradually expand the low-dimensional feature map into a voxel grid with a certain spatial resolution, and continuously self-game optimization during the training process. After this stage of adversarial training, the voxel model output by the generator is recorded as [B, V x ,V y ,V z ],V x 、V y 、V z Represents the number of voxels in the x, y, and z directions respectively; the voxel model [B, V x ,V y ,V z] is converted to a point cloud of size [B, N, 3 + d], where N is the number of points in the converted point cloud, ranging from [1000 to 10000], and 3 represents the spatial coordinates x, y, and z. Through multiple set abstraction layers, the point cloud is divided into local regions and features are extracted. The fully connected layer is used to process the point cloud after local feature extraction, and the final output is the center point coordinates [B, x, y, z, α, β].

[0012] As a preferred technical solution of the present invention, step S2 includes the following steps:

[0013] S2.1. Extract local features through convolution, setting the convolution kernel to 3*3, the number of output channels to 128, the stride to 2, and the padding to 1. Reduce the resolution through pooling, with a pooling window size of 2*2 and a stride of 2. Then, perform downsampling to complete the RGB image feature acquisition. The first downsampling is performed with a convolution kernel of 3*3 and 128 output channels, and the second downsampling is performed with a convolution kernel of 3*3 and 256 output channels.

[0014] As a preferred technical solution of the present invention, step S3 includes the following steps:

[0015] S3.1. Normalize the point cloud data to [B, N, f], where N represents the number of points and f represents the coordinates and depth of each point, i.e., [x, y, z, d]. Input the processed point cloud features into a network structure consisting of a three-layer perceptron.

[0016] First, the perceptron is configured with 64 neurons in the first hidden layer, 128 neurons in the second hidden layer, and 256 neurons in the output layer. For each point's 4-dimensional input feature vector, it is connected to the neurons in the first hidden layer, with the connection weight denoted by W1 (dimension 256) and the bias term denoted by b1 (dimension 64). After a linear transformation and the activation function ReLU, the output of the first hidden layer is obtained. The calculation formula is as follows (expressed in vector form, with f1 representing the input feature vector [x, y, z, d]):

[0017] h1=ReLU(f1W1+b1)

[0018] Next, the output h1 of the first hidden layer is used as the input of the second hidden layer. The weight of the connection with the neurons in the second hidden layer is W2 (dimension is 8192), and the bias term is b2 (dimension is 128). The output of the second hidden layer is obtained by the same linear transformation and activation function:

[0019] h2=ReLU(f2W2+b2)

[0020] Finally, the output h2 of the second hidden layer is passed to the output layer, with the connection weight W3 (dimension is 32768) and the bias term b3 (dimension is 256), and the final 256-dimensional feature vector of each point is obtained:

[0021] y=h2W3+b3

[0022] The model output, i.e., the feature data y = [B, N, 256], is calculated through the above process. The connection weights (W1, W2, W3) and bias terms (b1, b2, b3) of each layer are optimized and adjusted during training using the backpropagation algorithm combined with the loss function.

[0023] Initialization weight: W1~N(0,0.5), Initialization bias: b1, b2, b3, initialized to 0; weight update of the lth layer: Bias update: Among them L total is the loss of the final output result.

[0024] S3.2, set the side length of the cube to be s = 0.01m, calculate the number of voxels that can be divided in each direction, and use V = V x *V y *V z , the total number of voxels V is obtained, and each point in the point cloud is traversed. The voxel index to which it belongs is determined according to the coordinates of each point. For each voxel, the 256-dimensional features of the points falling into the same voxel are counted in the previous feature extraction step. After voxelization, the data format is converted from the original point-based representation [B, N, 256] to the voxel-based representation [B, V, 256], where V represents the number of voxels in the range [1000, 10000].

[0025] S3.3. Perform projection transformation on the voxelized data [B, V, 256]. Project the coordinate information of each voxel onto a two-dimensional plane according to the rules. When projecting along the x-axis onto the yz plane, only retain the y and z coordinates as the projected coordinates. Record the feature information corresponding to each projection position and the distance d from the laser radar to the corresponding point. Calculate the pixel position corresponding to each projection point in the target image through linear mapping. The coordinate range of the y direction is [y min ,y max ], the coordinate range of the z direction is [z min ,z max ], the voxel coordinates are (x v ,y v ,z v ), where x v ,y v The range is [-100,100], zv The range is [0,200], and the coordinates obtained after projection are

[0026]

[0027] After this projection step, the data format changes from the original voxelized data [B, V, g] to [B, h, w, d], where h is the set image height, w is the image width, and d represents the distance from the lidar to the point.

[0028] As a preferred technical solution of the present invention, step S6 includes the following steps:

[0029] S6.1. Input the semantically segmented image data [B, h, w, 2] into a convolutional neural network architecture based on the attention mechanism. First, use three convolutional layers to extract features at different levels of the image. Set the convolution kernel of the first convolutional layer to 3*3, the output channel to 16, the stride to 1, and the padding to 1. The first convolution obtains F out1 =ReLU(W*F in1 +b), where F in1 is [B,h,w,2], F out1 is [B, h, w, 16], and ReLU is the activation function. Each convolution layer is followed by a pooling layer, using 2×2 maximum pooling with a stride of 2. The first pooling obtains the feature data F out11 =[B,h / 2,w / 2,16]. The maximum pooling formula is:

[0030]

[0031] Similarly, set the convolution kernel of the second convolution layer to 3*3, the output channel to 32, the stride to 1, and the padding to 1. The second convolution obtains F out2 =ReLU(W*F out11 +b). where F out2 is [B,h / 2,w / 2,32], and ReLU is the activation function. 2×2 maximum pooling is used with a stride of 2. The second pooling obtains the feature data F out22 =[B,h / 4,w / 4,32]. The maximum pooling formula is:

[0032]

[0033] Set the convolution kernel of the third convolution layer to 3*3, the output channel to 64, the stride to 1, and the padding to 1. The third convolution obtains F out3 =ReLU(W*F out22 +b). where F out3is [B,h / 4,w / 4,64], ReLU is the activation function, ReLU(x)=max(0,x). 2×2 maximum pooling is used with a stride of 2, and the third pooling is used to obtain the feature data F out33 =[B,h / 8,w / 8,64]. The maximum pooling formula is:

[0034]

[0035] Introducing the spatial attention module, the feature map F out33 The attention weight map A is generated by convolution operation. The convolution kernel size is 7*7, the number of output channels is 1, and the activation function is σ: A=σ(W4*F out33 +b). The attention weight map A is element-wise multiplied by the input feature map F to obtain the weighted feature map F′ = [B, h / 8, w / 8, 64]. Next, residual coding is used to fuse and encode the feature data obtained above at different levels to obtain a compact and information-rich feature vector sequence in the format of [B, C, H, W], where C is the encoded feature dimension, and H and W are the height and width of the encoded feature map.

[0036] S6.2. Build a generative adversarial network architecture, where the generator takes the feature vector sequence [B, C, H, W] obtained in S5 as input. The generator consists of three transposed convolutional layers, with a kernel size of 4*4, a stride of 2, and a padding of 1. These layers gradually upsample the feature map and attempt to reconstruct a voxel representation of the 3D space [B, V x ,V y ,V z ], where V x 、V y 、V z Represents the number of voxels in the x, y, and z directions respectively. The discriminator is built based on a convolutional neural network and performs multi-scale feature analysis on the input 3D voxel model. The convolution kernel is set to 4*4, the stride is 2, and the padding is 1. The goal of the generator is to maximize the probability of the discriminator misjudging its generated samples. The generator loss function L G : Among them, G(z) is the sample generated by the generator, and D(G(z)) is the discriminant result of the generated sample. The goal of the discriminator is to correctly distinguish between real samples and generated samples. The discriminator loss function L D : Among them, D(x) is the discriminator’s judgment result on the real sample. Through the adversarial training between the generator and the discriminator, the generator can continuously optimize the voxel model it generates, making it closer to the real three-dimensional structure of the anode copper plate. After this stage of adversarial training, the voxel model [B, V x ,V y,V z ].

[0037] S6.3, the generated voxel model [B,V x ,V y ,V z ] is converted into a point cloud format, and the center coordinates of each voxel are extracted as the points of the point cloud to obtain the point cloud image [B, N, 3 + w], where N is the number of points in the converted point cloud, 3 represents the spatial coordinates x, y, z, and w is the auxiliary information dimension. The point cloud image [B, N, 3 + w] is input into the PointNet++ network. First, the point cloud is divided into local areas through the set abstraction layer, and features are extracted in each local area. The point cloud image after local feature extraction is processed using the fully connected layer, and the final output is [B, x, y, z, α, β]. Among them, x, y, z are the precise three-dimensional coordinates of the center point of the anode copper plate; α and β are the angles calculated based on the overall posture analysis of the point cloud and the comparison with the preset coordinate system.

[0038] The calculation process for this step is as follows:

[0039] Through the set abstraction layer, the point cloud is divided into local areas and the center of gravity of the points in the local area is calculated. There are M points in the local area, and the coordinates of the i-th point in the local area are (x i ,y i ,z i ), then the centroid of the local area (x c ,y c ,z c ) is calculated as: Calculate the relative coordinates (Δx i ,Δy i ,Δz i ):Δx i =x i -x c , Δy i =y i -y c , Δz i =z i -z c ; These relative coordinates are processed by the perceptron. The weight of the perceptron is W4, the bias is b4, and the output feature is e i :e i =ReLU(W4·[Δx i ,Δy i ,Δz i ]+b4) Where ReLU is the activation function, defined as: ReLU(x)=max(0,x).

[0040] Extract features in each local area and use the fully connected layer to process the point cloud after local feature extraction. Assume that the weight of the fully connected layer is W fc , bias is b fc , then the output feature is: F = W fc ·k i +b fc ; Finally, the precise three-dimensional coordinates (x, y, z) and angles (α, β) of the center point of the anode copper plate are output:

[0041] The center point coordinate loss function and the posture angle loss function are:

[0042]

[0043] Among them, are the center point coordinates and angles in the true label respectively. The total loss is: L total =λ1L center +λ2L angle Initialization weight: λ1 = λ2 = 0.5; weight: The Adam optimizer is used for updating, with a learning rate of η = 0.001, β1 = 0.9, and β2 = 0.999. Through continuous learning and optimization, the loss is reduced and the positioning accuracy is improved.

[0044] The present invention provides a high-precision positioning method for an anode copper plate fixture based on multimodal data fusion, which has the following beneficial effects:

[0045] The present invention first performs convolution, pooling and downsampling operations on the RGB channels of the acquired image, and gradually extracts effective local features. This fine processing helps to accurately identify the position and state of the anode copper plate in the image, and improves the accuracy of obtaining relevant information on positioning accuracy. Secondly, the three-layer perceptron extracts features, and through complex linear transformations and activation function calculations, fully exploits the features of the point cloud map; then based on voxel processing, it is accurately projected onto a two-dimensional plane and records detailed coordinates and distance information. These operations ensure high precision in feature extraction and conversion of the point cloud map, which is conducive to accurately reflecting the spatial features of the anode copper plate to assist in positioning. Then, an encoding-decoding structure is used to accurately segment the anode copper plate and the background, providing accurate target area information for subsequent positioning, reducing interference, and improving positioning accuracy. Finally, the attention mechanism focuses on key image areas, the generative adversarial network optimizes the voxel model to approximate the real structure, and the PointNet++ network finely processes the converted point cloud map to output high-precision center point coordinates and angles. The entire process continuously improves positioning accuracy through complex network structure and rigorous training.

[0046] Compared to existing technologies, such as the PointPainting method, which enhances point cloud features by projecting segmentation information from RGB images onto a point cloud, its performance is highly dependent on image segmentation accuracy. Inaccurate image segmentation results in poor point cloud feature enhancement, which in turn affects localization accuracy. In contrast, the attention mechanism introduced in this invention dynamically adjusts weights, reducing reliance on single-modality data. Even with inaccurate image segmentation, it maintains high feature extraction capabilities, significantly improving system robustness. Furthermore, the 3D model generated by the generative adversarial network through adversarial training is independent of the accuracy of a single modality, enabling better handling of complex scenes and further improving localization accuracy. While the Transformer-based multimodal fusion model can automatically learn feature associations between point clouds and RGB images and improve localization accuracy through global feature learning, it has high computational complexity and is not as accurate as the voxel-based model generated by the generative adversarial network when processing complex 3D structures. The advantage of this invention lies in its ability to focus on key areas, enhancing feature extraction capabilities. The generative adversarial network can generate high-precision 3D models and optimize center point coordinates. The PointNet++ network can extract local features and capture geometric details. A high-precision three-dimensional model is generated through the convolutional neural network and generative adversarial network with an attention mechanism, and the precise coordinates and angles of the anode copper center point are generated through the PointNet++ network. This can better handle complex scenes and locate the anode copper plate, and the positioning accuracy is higher than the above-mentioned methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a schematic diagram of the camera shooting anode copper;

[0048] Figure 2 Schematic diagram of the process of the high-precision positioning method of the anode copper plate fixture based on multimodal data fusion in the present invention;

[0049] Figure 3 This is a schematic diagram of point cloud projection transformation;

[0050] Figure 4 This is a schematic diagram of the deep learning structure of the present invention; DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0052] See also Figures 1-4A high-precision positioning method for an anode copper plate fixture based on multimodal data fusion is characterized by comprising the following steps:

[0053] S1, such as Figure 1 As shown, the model of the present invention is trained on a data set, which is 2000 groups of copper plate images collected manually. Each group contains an RGB image and a point cloud map of a copper plate. The goal is to obtain the coordinates of the center point of the anode copper plate and the pitch angle α and yaw angle β.

[0054] S2, such as Figure 2 As shown in the figure, RGB image acquisition and preprocessing: Use an RGB camera to acquire an image containing the anode copper plate. Perform convolution, pooling, and downsampling operations on the RGB channels of the acquired image to obtain the processed RGB image features;

[0055] S3, such as Figure 3 As shown in the figure, the point cloud data is normalized, and the features of the point cloud data are extracted through a three-layer perceptron. The three-dimensional space is divided into voxels through a voxel-based processing method, and the points in the point cloud data are assigned to the corresponding voxels. The voxelized data is projected onto a two-dimensional plane through projection, and image enhancement technology is used to improve the visual effect of the image. The processed features are obtained and recorded as [B, h, w, d].

[0056] The step S3 comprises the following steps:

[0057] S3.1. Normalize the point cloud image and input the processed point cloud image features into a network structure consisting of a three-layer perceptron (MLP). The first hidden layer has 64 neurons, the second hidden layer has 128 neurons, and the output layer has 256 neurons. For each point's 4-dimensional input feature vector, it is connected to the neurons in the first hidden layer, and the output of the first hidden layer is obtained through a linear transformation and the activation function ReLU. Then, the output h1 of the first hidden layer serves as the input of the second hidden layer, and the output of the second hidden layer is obtained through the same linear transformation and activation function. Finally, the output h2 of the second hidden layer is passed to the output layer, resulting in a final 256-dimensional feature vector for each point. The error between the model output and the true label is measured using a loss function, and the connection weights (W1, W2, W3) and bias terms (b1, b2, b3) of each layer are optimized and adjusted during training using the backpropagation algorithm combined with the loss function.

[0058] S3.2. Set the side length of a cube to s = 0.01m, calculate the number of voxels that can be divided in each direction, and obtain the total number of voxels V. Count the 256-dimensional features of the points falling into the same voxel obtained in the previous feature extraction step. After voxelization, the data format is converted from the original point-based representation [B, N, 256] to the voxel-based representation [B, V, 256].

[0059] S3.3. Perform a projection transformation on the voxelized data [B, V, 256]. The projection range and resolution are planned based on the preset image height h and image width w. The corresponding pixel position of each projection point in the target image is calculated using a linear mapping method.

[0060] S4, multimodal data fusion: The preprocessed RGB image features in S2 and the transformed two-dimensional data of the point cloud image in S3 are spliced in the channel dimension to obtain the fused features.

[0061] S5, semantic segmentation: The features fused in S4 are subjected to semantic segmentation using an encoding-decoding structure. The encoding process gradually extracts more abstract features through convolution and pooling operations, while the decoding process restores the features to a size close to the original image through operations such as upsampling. At the same time, features at different levels are integrated to obtain the semantic segmentation result, which is recorded as [B, h, w, 2];

[0062] S6, such as Figure 4 As shown in the figure, the result of semantic segmentation in S5 is passed through a convolutional neural network and residual coding is used to fuse and encode features at different levels to obtain a compact and information-rich feature vector sequence [B, C, H, W], where C is the feature dimension after encoding, and H and W are the height and width of the feature map after encoding; the feature vector sequence obtained in S5.1 is passed through a generative adversarial network to gradually expand the low-dimensional feature map into a voxel grid with a certain spatial resolution, and continuously self-game optimization is carried out during the training process. After this stage of adversarial training, the voxel model output by the generator is recorded as [B, V x ,V y ,V z ],V x 、V y 、V z Represents the number of voxels in the x, y, and z directions respectively; the voxel model [B, V x ,V y ,V z ] to convert it into the point cloud format [B, N, 3+d]. Through the set abstraction layer, the point cloud is divided into local areas and features are extracted. The fully connected layer is used to process the point cloud image after local feature extraction, and finally the desired center point coordinates are output, namely [B, x, y, z, a, b].

[0063] The step S6 comprises the following steps:

[0064] S6.1. Input the semantically segmented image data [B, h, w, 2] into a convolutional neural network architecture based on the attention mechanism. First, use three convolutional layers to extract features at different levels of the image. Set the convolution kernel of the first convolutional layer to 3*3, the output channel to 16, the stride to 1, and the padding to 1. The first pooling obtains the feature data F out11 =[B,h / 2,w / 2,16]. Set the convolution kernel of the second convolution layer to 3*3, the output channel to 32, the stride to 1, and the padding to 1. The second convolution obtains F out2 =[B,h / 2,w / 2,32]. The second pooling obtains the feature data F out22 =[B,h / 4,w / 4,32]. Set the convolution kernel of the third convolution layer to 3*3, the output channel to 64, the stride to 1, and the padding to 1. The third convolution obtains F out3 =[B,h / 4,w / 4,64], the third pooling obtains the feature data F out33 =[B,h / 8,w / 8,64]. Introduce the spatial attention module and transform the feature map F out33 The attention weight map A is generated through a convolution operation. The convolution kernel size is 7*7, the number of output channels is 1, and the activation function is σ. The attention weight map A is element-wise multiplied with the input feature map F to obtain the weighted feature map F′ = [B, h / 8, w / 8, 64]. Next, residual coding is used to fuse and encode the feature data obtained above at different levels, resulting in a compact and information-rich feature vector sequence [B, C, H, W].

[0065] S6.2. Build a generative adversarial network architecture, where the generator takes the feature vector sequence [B, C, H, W] obtained in S6.1 as input. The generator consists of three transposed convolutional layers, with a kernel size of 4*4, a stride of 2, and a padding of 1. These layers gradually upsample the feature map and attempt to reconstruct a voxel representation of the 3D space [B, V x ,V y ,V z ], where V x 、V y 、V z Represents the number of voxels in the x, y, and z directions respectively. The discriminator is built based on a convolutional neural network and performs multi-scale feature analysis on the input three-dimensional voxel model. The convolution kernel is set to 4*4, the stride is 2, and the padding is 1. Through the adversarial training between the generator and the discriminator, the generator can continuously optimize the voxel model it generates, making it closer to the real three-dimensional structure of the anode copper plate. After this stage of adversarial training, the voxel model [B, Vx ,V y ,V z ].

[0066] S6.3, the generated voxel model [B,V x ,V y ,V z ] is converted to a point cloud format, and the center coordinates of each voxel are extracted as the point cloud points, resulting in a point cloud image [B, N, 3 + w], where N is the number of points in the converted point cloud, 3 represents the spatial coordinates x, y, and z, and w is the auxiliary information dimension. The point cloud image [B, N, 3 + w] is input into the PointNet++ network. First, the point cloud is divided into local regions through the set abstraction layer, and features are extracted within each local region. The point cloud image after local feature extraction is processed using a fully connected layer, and the final output is [B, x, y, z, α, β]. The model parameters are updated and optimized based on the loss function, continuously improving the model's ability to generate accurate coordinate and angle information from the semantically segmented image.

[0067] Although the embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the invention, and the scope of the invention is limited by the accompanying description and its equivalents.

Claims

1. A high-precision positioning method for anode copper plate fixture based on multimodal data fusion, characterized in that: include: Get the RGB image and point cloud image of the anode copper plate; For the RGB image, obtain RGB image features through convolution, pooling and downsampling operations; For the point cloud image, features are extracted by a three-layer perceptron, and after voxel processing, they are projected onto a two-dimensional plane to obtain point cloud features; The RGB image features and the point cloud features are spliced in the channel dimension to obtain fusion features; Performing semantic segmentation on the fused features using an encoding-decoding structure to obtain a segmentation result; Passing the segmentation results through a convolutional neural network and applying residual coding to obtain a feature vector sequence; Expanding the feature vector sequence into a voxel grid through a generative adversarial network and then converting it into a point cloud format; The point cloud image is processed by the PointNet++ network, and the center point coordinates and angle information of the anode copper plate are finally output.

2. The method according to claim 1, wherein For RGB images, the convolution, pooling and downsampling operations are as follows: Local features are extracted through multi-layer convolution, with a convolution kernel size of 33, an output channel number of 128, a stride of 2, and a padding of 1; Reduce the resolution by pooling operation, with a pooling window size of 22 and a stride of 2; Then downsample twice. The first downsampling parameter is convolution kernel 33, output channel 128, and the second downsampling parameter is convolution kernel 33, output channel 256. Finally, the RGB image features of size [B,h,w,c'] are obtained.

3. The method according to claim 1, wherein The processing of the point cloud image includes: Normalize the point cloud image to obtain data in the format of [B, N, x, y, z, d]; A three-layer perceptron is used to extract features. The first, second, and third layers have 64, 128, and 256 neurons, respectively. The output is 256-dimensional features through linear transformation and ReLU activation function calculations, and weights and bias terms are optimized and adjusted. Divide the three-dimensional space into voxels, count the point cloud features in each voxel, and convert the data format from [B,N,256] is converted to [B,V,256], where V is the number of voxels; The voxelized data is projected onto a two-dimensional plane along the x-axis direction and coordinate mapping is performed to finally obtain a point cloud feature in the format of [B, h, w, d].

4. The method according to claim 1, wherein The semantic segmentation adopts an encoding-decoding structure. The encoding process gradually extracts abstract features through convolution and pooling. The decoding process restores the features to their original size through upsampling and fuses features at different levels. Finally, the anode copper plate and the background are segmented into two categories. The format of the segmentation result is [B, h, w, 2].

5. The method according to claim 1, wherein The process of converting the segmentation result into a feature vector sequence includes: The multi-scale features are extracted through the convolutional neural network with attention mechanism. Three layers of convolution are set, the convolution kernel size is 33, the number of output channels is 16, 32, and 64 respectively, the pooling window size is 22, and the stride is 2; A spatial attention module is introduced to generate a weight map using 77 convolution kernels, which is then multiplied element-wise with the feature map to enhance the features of key areas. Residual coding is used to fuse features at different levels to obtain a compact feature vector sequence [B, C, H, W].

6. The method according to claim 1, wherein The generative adversarial network includes a generator and a discriminator. The generator consists of three transposed convolutional layers with a convolution kernel size of 44 and a stride of 2. The feature vector sequence is gradually upsampled into a three-dimensional voxel grid [B, V x ,V y ,V z ]; The discriminator uses a convolutional neural network to extract multi-scale features and distinguish true from false on the voxel model, with a convolution kernel size of 44 and a stride of 2; Through adversarial training of the generator and the discriminator, the generated voxel model is optimized to approximate the real structure.

7. The method according to claim 1, wherein The process of converting a voxel grid into a point cloud and processing it through the PointNet++ network includes: Extract the center coordinates of each voxel as the point of the point cloud and obtain the point cloud map [B, N, 3+d]; The PointNet++ network’s set abstraction layer is used to divide local areas and extract features. The fully connected layer is then used to process the extracted local features. The model parameters are optimized through the loss function, and the precise center point coordinates [B, x, y, z] and angle information [α, β] of the anode copper plate are finally output.

Images