Object point cloud reconstruction method based on guided cross-modal robot

Abstract

The invention relates to an object point cloud reconstruction method based on a guided cross-modal robot, and the method comprises the steps: obtaining a depth image and a corresponding RGB image of an object under a single visual angle of the robot, carrying out the preprocessing, and obtaining the incomplete data of the point cloud of the object; a point cloud classification encoder is adopted to encode the point cloud classification task, a point cloud classification decoder is adopted to decode the point cloud classification task, and a classification result of the object is obtained; based on the object point cloud incomplete data, a point cloud data encoder is adopted to carry out position embedded coding, and point cloud features containing position codes are obtained; acquiring tactile data of an object at a single-view blind area position of the robot, and encoding the tactile data by a tactile space encoder to obtain tactile features; a visual touch cross-modal fusion module is adopted to fuse the point cloud features and the touch features, and a classification result is utilized to guide in the fusion process; and decoding the fused features by adopting a pyramid decoder to obtain a complete point cloud of the object. Compared with the prior art, the method has the advantages of high precision and the like.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to an object point cloud reconstruction method based on a guided cross-modal robot. Background Art

[0002] In such a complex operating environment, the embodied robot can build an understanding of the current operating environment through a variety of perception methods. Especially in complex unknown environments, accurately estimating and characterizing the three-dimensional shape of an unknown object or environment is one of the challenges in many engineering applications.

[0003] Robots have no knowledge of the external environment or objects, and usually interact with the environment through multimodal data. Currently, embodied robots basically use visual and tactile data to overcome the limitations of a single modality. Although visual sensors can provide rich color and texture information, their performance will be affected when objects are blocked or lighting conditions are not ideal. Although tactile sensors can provide accurate contact information, their perception range is limited and they cannot obtain comprehensive information about the entire object at one time. Therefore, robots need to make up for the shortcomings of a single modality through multiple explorations to obtain a more complete understanding of the environment.

[0004] Many manipulation tasks are achieved based on the correct establishment of the three-dimensional representation of the object to be manipulated. In a complex manipulation space, constructing a reasonable three-dimensional representation of the object to be manipulated can greatly improve the efficiency of embodied robot manipulation. The object information obtained by vision in an environment such as occlusion is limited by the receptive field of touch, which increases the number of explorations required by the robot. Therefore, a reconstruction method is urgently needed to overcome the defects of the existing technology. Summary of the invention

[0005] The purpose of the present invention is to provide an object point cloud reconstruction method based on a guided cross-modal robot to achieve more accurate point cloud reconstruction.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A method for object point cloud reconstruction based on a guided cross-modal robot comprises the following steps:

[0008] Obtain the depth image and corresponding RGB image of the object under the robot's single view, perform preprocessing, obtain the incomplete point cloud data of the object, and input it into the point cloud reconstruction network for reconstruction, and output the complete point cloud P of the object. c , where the point cloud reconstruction network includes a point cloud classification encoder, a point cloud data encoder, a tactile space encoder, a visual-tactile cross-modal fusion module, and a pyramid decoder;

[0009] The execution steps of the point cloud reconstruction network include:

[0010] Based on the incomplete point cloud data of the object, a point cloud classification encoder is used to encode the point cloud classification task to extract the point cloud classification feature F p_1 , and use the point cloud classification decoder to decode and obtain the classification result of the object;

[0011] Based on the incomplete point cloud data of the object, a point cloud data encoder is used to perform position embedding encoding to obtain a point cloud feature F containing position encoding. p_2 ;

[0012] The tactile data T of the object in the blind spot of the robot's single view is obtained, and the tactile space encoder encodes it to obtain the tactile feature F T ;

[0013] The visual-tactile cross-modal fusion module is used to transform the point cloud feature F containing the position code p_2 and tactile characteristics F T Fusion is performed to obtain the fusion feature F f , and use the classification results to fusion feature F in the fusion process f Conduct guidance;

[0014] The pyramid decoder is used to decode the fusion feature F f Decode and get the complete point cloud P of the object c .

[0015] Furthermore, the steps of the pre-processing operation include:

[0016] The depth image pixel coordinate system is converted into a world coordinate system using camera internal and external parameters to obtain initial single-view three-dimensional point cloud data in the real world;

[0017] Based on the initial single-view three-dimensional point cloud data, a point cloud quantity conversion is performed through a downsampling operation to meet network input requirements, thereby obtaining final single-view three-dimensional point cloud data;

[0018] The final single-view 3D point cloud data is segmented using a pre-trained segment anything model to obtain an initial segmentation result as initial object point cloud incomplete data;

[0019] Based on the RGB image, the initial object point cloud incomplete data and the corresponding point cloud position are matched to obtain final object point cloud incomplete data.

[0020] Furthermore, the point cloud classification encoder includes a mamba model and a multi-head self-attention layer, and the execution steps of the point cloud classification encoder include:

[0021] Based on the incomplete point cloud data of the object, the mamba model uses multiple convolution layers to perform convolution processing respectively, and uses the maximum pooling layer to splice to obtain the initial point cloud classification features;

[0022] Based on the initial point cloud classification features, a multi-head self-attention layer is used for further processing to obtain the final point cloud classification features F p_1 , where the formula for the multi-head self-attention layer is:

[0023]

[0024] Where Attention(Q,K,V) is the self-attention value, Q, K, V are query, key, and value respectively, softmax is the normalized exponential function, and d k The scaling factor makes the results of Q and K satisfy the distribution with mean 0 and variance 1.

[0025] Furthermore, the execution steps of the point cloud classification decoder include:

[0026] Based on the point cloud classification feature F p_1 , the point cloud classification decoder is used for decoding, and the classification result of the object is output. During the decoding process, the point cloud classification decoder uses the Softmax cross entropy loss as the category loss, and the Softmax cross entropy loss is:

[0027]

[0028] In the formula, Loss class is the category loss for point cloud classification, y i is the probability distribution of the true sample label, is the probability distribution of the normalized prediction output.

[0029] Furthermore, the point cloud data encoder includes a position embedding module, a multi-layer convolution layer and a maximum pooling layer, and the execution steps of the point cloud data encoder include:

[0030] The position embedding module groups and sorts the incomplete point cloud data of the object through the farthest point sampling algorithm and the K nearest neighbor algorithm, and constructs the position encoding according to the sorting result to achieve position embedding of each group;

[0031] According to the position embedding results of each group, multiple convolution layers are used to perform convolution processing respectively, and then spliced ​​after maximum pooling to obtain the point cloud feature F containing the position code p_2 .

[0032] Furthermore, the tactile space encoder is constructed by a graph convolutional neural network model, and the tactile space encoder is used to encode the spatial position of the sensing unit and match the tactile data T, so as to embed the encoded spatial position into the tactile data T, and then perform tactile feature F according to the matching result. T Extraction, where the spatial position encoding formula is:

[0033]

[0034] In the formula, H(l+1) represents the node feature matrix of the l+1th layer, that is, the node features updated after the graph convolution operation, and H (l) is the node feature matrix of the lth layer, where each row represents the feature vector of a node, σ(·) is the nonlinear activation function, is the adjacency matrix A of the graph plus the identity matrix In, n is the number of nodes, yes The degree matrix of A is a diagonal matrix whose diagonal elements are the sum of each row of A. (l) is the weight matrix of the lth layer, which is used to perform linear transformation on node features.

[0035] Furthermore, the visual-tactile cross-modal fusion module includes a splicing layer, a mamba model, a multi-layer convolutional layer and a linear layer, and the execution steps of the visual-tactile cross-modal fusion module include:

[0036] The splicing layer is used to convert the F containing the position code p_2 and tactile characteristics F T Process them into the same format and concatenate them by column through concatenation operation to obtain concatenated features;

[0037] Based on the splicing features, a multi-layer convolutional layer is used to extract local and global patterns, and the mamba model is used for fusion. At the same time, the classification results are used to guide the fusion process to obtain preliminary fusion features, wherein the expression of the mamba model is:

[0038] h(t)=Ah(t-1)+Bx(t)

[0039] y(t)=Ch(t)+Dx(t)

[0040] In the formula, h(t) represents the state quantity at the current time t, which is a hidden state inside the model, A is the state transfer matrix, which describes how the state quantity changes over time, x(t) represents the input control quantity at the current time t, B represents the influence of the control quantity x(t) on the state quantity h(t), y(t) represents the system output at the current time t, C represents the influence of the current state quantity h(t) on the output y(t), and D represents the direct influence of the current control quantity x(t) on the output y(t);

[0041] Based on the preliminary fusion features, a linear layer is used for mapping to obtain the final fusion feature F f .

[0042] Furthermore, the pyramid decoder includes a fully connected network layer and a convolutional network layer, and the steps of decoding by the pyramid decoder include:

[0043] Based on the fusion feature F f Multiple fully connected network layers are used to extract channel features and obtain channel decoding input features F c_1 and F c_2 ;

[0044] The fusion feature F f , channel decoding input feature F c_1 and F c_2 Multiple convolutional network layers are used for multi-channel decoding to obtain point cloud results of different scales P c_1 , P c_2 and P c_3 , where P c_2 By P c_1 With F c_1 The combined calculation yields P c_3 By P c_2 With F c_2 The combined calculation results are:

[0045] The P c_3 Input the head layer for processing to obtain the reconstructed complete point cloud Pc.

[0046] Furthermore, the loss function adopted by the pyramid decoder is:

[0047] Loss gen =d CD (Y1,Y GT_1 )+d CD (Y2,Y GT_2 )+d CD (Y3,Y GT_3 )

[0048]

[0049] In the formula, Loss gen is the pyramid decoder loss, Y1, Y2 and Y3 are the predicted point clouds of three scales, Y GT_1 , Y GT_2 , Y GT_3 is the real point cloud at three scales, d CD is the chamfer distance, S1 and S2 represent the two point cloud sets for calculating the chamfer distance.

[0050] Furthermore, the loss function used by the point cloud reconstruction network is:

[0051]

[0052] In the formula, Loss class To reconstruct the total loss, Loss class is the category loss for point cloud classification, Loss gen is the loss function for training the pyramid decoder, are the weights of the two loss functions respectively.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] (1) In a complex environment, the visual information obtained by the robot from a single perspective is limited. The robot of the present invention obtains visual data from a single perspective, obtains tactile data of the occluded part by touching, and then fuses the visual and tactile features through a cross-modal fusion module. In the fusion process, the classification results are used to guide the optimization of the fusion features, which can more accurately represent the three-dimensional information of the object. Finally, through decoding, more accurate point cloud reconstruction is achieved.

[0055] (2) The present invention guides the fusion of features by using classification results to optimize feature representation, which can not only effectively reduce the number of times the robot explores the environment through accurate three-dimensional representation, but also improve the robot's perception and cognition.

[0056] (3) In complex or unknown environments, the present invention can better help robots understand and adapt to the environment and improve their decision-making ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0058] Figure 2 Schematic diagram of the point cloud reconstruction network of the present invention. DETAILED DESCRIPTION

[0059] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0060] This embodiment provides an object point cloud reconstruction method based on a guided cross-modal robot. Figure 2As shown in the figure, firstly, the depth camera is used to obtain the operating environment information, then the tactile sensor is used to touch the object to obtain the object surface data, and the encoder in the point cloud reconstruction network is used to extract the visual and tactile features. Then, the decoder is used to predict and complete the sparse point cloud to complete the 3D point cloud reconstruction of the operating object in the environment. The point cloud reconstruction network includes a point cloud classification encoder, a point cloud data encoder, a tactile space encoder, a visual and tactile cross-modal fusion module, and a pyramid decoder. Specifically, combined with Figure 1 and Figure 2 , the method comprises the following steps:

[0061] Step S1: The depth image and RGB color image of the operating environment within the robot's field of view are acquired through an RGB-D camera, and a single-view three-dimensional point cloud P is obtained through preprocessing.

[0062] In step S1, preprocessing is used to construct single-view 3D point cloud data based on the depth image acquired by the camera and the internal and external parameters of the camera; the depth image pixel coordinate system is converted to the world coordinate system using the internal and external parameters of the camera to obtain the true position of the point cloud in the real world, and the number of the acquired object point cloud is converted to the 2048 points required by the network input through downsampling operation: i =(x,y,z), where x, y, and z are the three-dimensional coordinates of space;

[0063] Step S2: Migrate the image segmentation tool to the single-view three-dimensional point cloud space, perform point cloud segmentation processing on the three-dimensional point cloud P, and obtain incomplete point cloud data of the object.

[0064] In step S2, the image segmentation tool is migrated to adapt the pre-trained segment anything model to the three-dimensional point cloud data; the grouping results after segmenting the point cloud data through the RGB image match the point cloud position, and the segmentation results are matched with the point cloud data.

[0065] Step S3: by touching the high point of curvature of the object to be operated in the space, tactile data under the occluded viewing angle is obtained, and tactile data T of the blind spot position of the object is obtained.

[0066] In step S3, the tactile data is obtained by touching the blind spot of the single-viewing angle with a tactile sensor to obtain the information of the blocked part of the object to be operated: i =(N,C,x,y,z), N is the perceived position, C is the perceived value, and x, y, and z are the three-dimensional coordinates of the space respectively.

[0067] Step S4: Use the point cloud classification encoder based on the fusion of Mamba and attention mechanism to classify the segmentation results and obtain the classification results.

[0068] Step S5: Use the farthest point sampling and grouping strategy to reconstruct the point cloud data and generate position coding. Use a multi-layer hybrid perception network and mamba to build a point cloud data encoder to achieve feature embedding and obtain the point cloud feature F containing the position coding. p .

[0069] Step S6: Use the graph structure to match the spatial position of tactile perception, construct a tactile spatial encoder, and obtain the tactile feature F T .

[0070] The encoder in steps S4, S5, and S6 is used to complete the feature processing operation of the data. It is composed of three encoders: a point cloud classification encoder, a point cloud data encoder, and a tactile space encoder. One encoder encodes the point cloud classification task, one encoder encodes the cross-modal fusion of the point cloud data, and one encoder encodes the tactile information to obtain the classification result and the point cloud feature F respectively. p and tactile characteristics F T ;

[0071] The point cloud classification encoder consists of a mamba model and a multi-head self-attention layer. The mamba model consists of multiple convolutional layers. First, the outputs of the last four convolutional layers of the multi-layer convolutional layer are respectively concatenated after passing through the maximum pooling layer to obtain the initial point cloud classification features; the self-attention layer is responsible for further extracting point cloud features to obtain the initial point cloud classification features. The formula of the self-attention layer is:

[0072]

[0073] Among them, Q, K, V are query, key and value respectively, and softmax is the normalized exponential function.

[0074] The point cloud classification decoder outputs the category result of the object; the category loss uses Softmax cross entropy loss:

[0075]

[0076] In the formula, y i is the probability distribution of the true sample label, is the probability distribution of the normalized prediction output;

[0077] The point cloud data encoder is composed of a position embedding module, a multi-layer convolutional layer and a maximum pooling layer. The position embedding module groups the point cloud through the farthest point sampling and K nearest neighbor algorithm, and completes the sorting. At the same time, the position encoding is constructed according to the sorting result to realize the position embedding of the encoder of each group; the outputs of the last four convolutional layers composed of multi-layer convolutional layers are respectively concatenated after passing through the maximum pooling layer to obtain the point cloud feature F containing the position encoding. p_2 ;

[0078] The tactile spatial encoder is constructed by a graph convolutional neural network model, which encodes and matches the tactile data T according to the spatial position of the sensing unit, so as to embed the encoded spatial position into the tactile data T, and then performs tactile feature F according to the matching result. T Extract tactile data. The characteristics of graph neural networks can be used to match sensor units with space, and the importance of different sensor units can be reflected by manually building maps; the position encoding formula is:

[0079]

[0080] Among them, H l H represents the node feature matrix of the lth layer, where each row represents the feature vector of a node. (l+1) Represents the node feature matrix of the l+1th layer, that is, the node features updated after the graph convolution operation. =A+In is the adjacency matrix A of the graph plus the identity matrix In (to account for self-connections), where N is the number of nodes. yes The degree matrix of is a diagonal matrix whose diagonal elements are The sum of each row. W l is the weight matrix of the lth layer, which is used to linearly transform the node features, and σ is the nonlinear activation function.

[0081] Step S7: Use the visual-tactile cross-modal fusion module to fuse the visual features and the tactile features to obtain the visual-tactile cross-modal fusion feature F f , and use the classification results to guide the fusion features.

[0082] In step S7, the visual-touch cross-modal fusion module is used to transform the point cloud feature F p_2 and tactile characteristics F T Fusion is performed. This module consists of a concatenation layer, a mamba model, a multi-layer convolution layer, and a linear layer, and finally outputs the result;

[0083] The visual-tactile cross-modal fusion module first transforms the point cloud feature F T and tactile characteristics F T The two types of features are processed into the same format, and the two types of features are spliced ​​by column through the concatenation operation to obtain a 1920×2 splicing feature; based on the splicing feature, a multi-layer convolution layer is used to extract local and global patterns, and the mamba model is used for fusion. At the same time, the classification results are used to guide the fusion process to obtain a preliminary fusion feature; a linear layer is used to linearly map the preliminary fusion feature to obtain a fusion feature F f The formula of the mamba model is:

[0084] h(t) = Ah(t-1) + Bx(t) (4)

[0085] y(t) = Ch(t) + Dx(t) (5)

[0086] In the formula, h(t) represents the state quantity at the current time t, which is a hidden state inside the model. A is the state transfer matrix, which describes how the state quantity changes over time. x(t) represents the input control quantity at the current time t. B represents the influence of the control quantity x(t) on the state quantity h(t). y(t) represents the system output at the current time t. C represents the influence of the current state quantity h(t) on the output y(t). D represents the direct influence of the current control quantity x(t) on the output y(t).

[0087] Step S8: Use the pyramid decoder to decode the fused features to obtain the complete point cloud P c .

[0088] In step S8, the pyramid decoder decodes the fused features and predicts the complete point cloud to obtain the complete point cloud P c The pyramid decoder consists of two fully connected networks and three convolutional networks. The two fully connected networks further extract the fusion feature F f , and get F c_1 and F c_2 ; The inputs of the three convolutional networks are F f 、F c_1 and F c_2 , the output is the reconstructed point cloud results of three different scales, namely W1, W2, and W3. For single-view scenes, they are set to W1=128, W2=512, and W3=2048;

[0089] The loss function for pyramid decoder training is:

[0090] Loss gen = d CD (Y1,Y GT_1 )+d CD (Y2,Y GT_2 )+d CD (Y3,Y GT_3 ) (6)

[0091]

[0092] Among them, d CD is the chamfer distance, Y1, Y2 and Y3 are the predicted point clouds of three scales, Y GT_1 is the real point cloud Y at three scales GT_2 Y GT_3are real point clouds at three scales, S1 and S2 represent the two point cloud sets for calculating the chamfer distance;

[0093] The loss function of the overall point cloud reconstruction network is:

[0094]

[0095] are the weights of the two loss functions.

[0096] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0097] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0098] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0101] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0102] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for object point cloud reconstruction based on a guided cross-modal robot, characterized in that: The following steps are involved: Obtain the depth image and corresponding RGB image of the object under the robot's single view, perform preprocessing, obtain the incomplete point cloud data of the object, and input it into the point cloud reconstruction network for reconstruction, and output the complete point cloud P of the object. c , where the point cloud reconstruction network includes a point cloud classification encoder, a point cloud data encoder, a tactile space encoder, a visual-tactile cross-modal fusion module, and a pyramid decoder; The execution steps of the point cloud reconstruction network include: Based on the incomplete point cloud data of the object, a point cloud classification encoder is used to encode the point cloud classification task to extract the point cloud classification feature F p_1 , and use the point cloud classification decoder to decode and obtain the classification result of the object; Based on the incomplete point cloud data of the object, a point cloud data encoder is used to perform position embedding encoding to obtain a point cloud feature F containing position encoding. p_2 ; The tactile data T of the object in the blind spot of the robot's single view is obtained, and the tactile space encoder encodes it to obtain the tactile feature F T ; The visual-tactile cross-modal fusion module is used to transform the point cloud feature F containing the position code p_2 and tactile characteristics F T Fusion is performed to obtain the fusion feature F f , and use the classification results to fusion feature F in the fusion process f Conduct guidance; The pyramid decoder is used to decode the fusion feature F f Decode and get the complete point cloud P of the object c .

2. The object point cloud reconstruction method based on a guided cross-modal robot according to claim 1, characterized in that: The steps of the pre-processing operation include: The depth image pixel coordinate system is converted into a world coordinate system using camera internal and external parameters to obtain initial single-view three-dimensional point cloud data in the real world; Based on the initial single-view three-dimensional point cloud data, a point cloud quantity conversion is performed through a downsampling operation to meet network input requirements, thereby obtaining final single-view three-dimensional point cloud data; The final single-view 3D point cloud data is segmented using a pre-trained segment anything model to obtain an initial segmentation result as initial object point cloud incomplete data; Based on the RGB image, the initial object point cloud incomplete data and the corresponding point cloud position are matched to obtain final object point cloud incomplete data.

3. The object point cloud reconstruction method based on a guided cross-modal robot according to claim 1, characterized in that: The point cloud classification encoder includes a mamba model and a multi-head self-attention layer, and the execution steps of the point cloud classification encoder include: Based on the incomplete point cloud data of the object, the mamba model uses multiple convolution layers to perform convolution processing respectively, and uses the maximum pooling layer to splice to obtain the initial point cloud classification features; Based on the initial point cloud classification features, a multi-head self-attention layer is used for further processing to obtain the final point cloud classification features F p_1 , where the formula for the multi-head self-attention layer is: Where Attention(Q,K,V) is the self-attention value, Q, K, V are query, key, and value respectively, softmax is the normalized exponential function, and d k The scaling factor makes the results of Q and K satisfy the distribution with mean 0 and variance 1.

4. The object point cloud reconstruction method based on a guided cross-modal robot according to claim 3, characterized in that: The execution steps of the point cloud classification decoder include: Based on the point cloud classification feature F p_1 , the point cloud classification decoder is used for decoding, and the classification result of the object is output. During the decoding process, the point cloud classification decoder uses the Softmax cross entropy loss as the category loss, and the Softmax cross entropy loss is: In the formula, Loss class is the category loss for point cloud classification, y i is the probability distribution of the true sample label, is the probability distribution of the normalized prediction output.

5. The object point cloud reconstruction method based on a guided cross-modal robot according to claim 1, characterized in that: The point cloud data encoder includes a position embedding module, a multi-layer convolution layer and a maximum pooling layer. The execution steps of the point cloud data encoder include: The position embedding module groups and sorts the incomplete data of the object point cloud by using the farthest point sampling algorithm and the K nearest neighbor algorithm, and constructs a position code according to the sorting result to achieve position embedding of each group; According to the position embedding results of each group, multiple convolution layers are used to perform convolution processing respectively, and then spliced ​​after maximum pooling to obtain the point cloud feature F containing the position code p_2 .

6. The object point cloud reconstruction method based on a guided cross-modal robot according to claim 1, characterized in that: The tactile space encoder is constructed by a graph convolutional neural network model, and the tactile space encoder is used to encode the spatial position of the sensing unit and match the tactile data T, so as to embed the encoded spatial position into the tactile data T, and then perform tactile feature F according to the matching result. T Extraction, where the spatial position encoding formula is: In the formula, H(l+1) represents the node feature matrix of the l+1th layer, that is, the node features updated after the graph convolution operation, and H (l) is the node feature matrix of the lth layer, where each row represents the feature vector of a node, σ(·) is the nonlinear activation function, is the adjacency matrix A of the graph plus the identity matrix In, n is the number of nodes, yes The degree matrix of A is a diagonal matrix whose diagonal elements are the sum of each row of A. (l) is the weight matrix of the lth layer, which is used to perform linear transformation on node features.

7. The object point cloud reconstruction method based on a guided cross-modal robot according to claim 1, characterized in that: The visual-tactile cross-modal fusion module includes a splicing layer, a mamba model, a multi-layer convolutional layer and a linear layer. The execution steps of the visual-tactile cross-modal fusion module include: The splicing layer is used to convert the F containing the position code p_2 and tactile characteristics F T Process them into the same format and concatenate them by column through concatenation operation to obtain concatenated features; Based on the splicing features, a multi-layer convolutional layer is used to extract local and global patterns, and the mamba model is used for fusion. At the same time, the classification results are used to guide the fusion process to obtain preliminary fusion features, wherein the expression of the mamba model is: h(t)=Ah(t-1)+Bx(t) y(t)=Ch(t)+Dx(t) In the formula, h(t) represents the state quantity at the current time t, which is a hidden state inside the model, A is the state transfer matrix, which describes how the state quantity changes over time, x(t) represents the input control quantity at the current time t, B represents the influence of the control quantity x(t) on the state quantity h(t), y(t) represents the system output at the current time t, C represents the influence of the current state quantity h(t) on the output y(t), and D represents the direct influence of the current control quantity x(t) on the output y(t); Based on the preliminary fusion features, a linear layer is used for mapping to obtain the final fusion feature F f .

8. The object point cloud reconstruction method based on a guided cross-modal robot according to claim 1, characterized in that: The pyramid decoder includes a fully connected network layer and a convolutional network layer. The steps of decoding by the pyramid decoder include: Based on the fusion feature F f Multiple fully connected network layers are used to extract channel features and obtain channel decoding input features F c_1 and F c_2 ; The fusion feature F f , channel decoding input feature F c_1 and F c_2 Multiple convolutional network layers are used for multi-channel decoding to obtain point cloud results of different scales P c_1 , P c_2 and P c_3 , where P c_2 By P c_1 With F c_1 The combined calculation yields P c_3 By P c_2 With F c_2 The combined calculation results are: The P c_3 Input the head layer for processing to obtain the reconstructed complete point cloud Pc.

9. The object point cloud reconstruction method based on a guided cross-modal robot according to claim 1, characterized in that: The loss function used by the pyramid decoder is: Loss gen =d CD (Y1,Y GT_1 )+d CD (Y2,Y GT_2 )+d CD (Y3,Y GT_3 ) In the formula, Loss gen is the pyramid decoder loss, Y1, Y2 and Y3 are the predicted point clouds of three scales, Y GT_1 , Y GT_2 , Y GT_3 is the real point cloud at three scales, d CD is the chamfer distance, S1 and S2 represent the two point cloud sets for calculating the chamfer distance.

10. The object point cloud reconstruction method based on a guided cross-modal robot according to claim 1, characterized in that: The loss function used by the point cloud reconstruction network is: In the formula, Loss class To reconstruct the total loss, Loss class is the category loss for point cloud classification, Loss gen is the loss function for training the pyramid decoder, are the weights of the two loss functions respectively.

Images