A method for object point cloud reconstruction based on guided cross-modal robot

Abstract

The present invention relates to a method for reconstructing object point clouds based on a guided cross-modal robot. The method comprises: obtaining a depth image and a corresponding RGB image of an object from a single perspective of the robot, performing preprocessing to obtain incomplete point cloud data for the object; encoding the point cloud classification task using a point cloud classification encoder, and decoding the result using a point cloud classification decoder to obtain the object classification result; performing position embedding encoding based on the incomplete point cloud data using a point cloud data encoder to obtain point cloud features containing position encoding; obtaining tactile data of the object in the blind spot of the robot's single perspective, encoding the data using a tactile space encoder to obtain tactile features; fusing the point cloud features and tactile features using a visual-tactile cross-modal fusion module, and using the classification results for guidance during the fusion process; and decoding the fused features using a pyramid decoder to obtain a complete point cloud for the object. Compared with existing technologies, the present invention has the advantages of higher accuracy.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a method for reconstructing object point clouds based on a guided cross-modal robot. Background Art

[0002] In such a complex operating environment, embodied robots can build an understanding of the current operating environment through various perception methods. In particular, accurately estimating and representing the 3D shape of an unknown object or environment in complex unknown environments is one of the challenges in many engineering applications.

[0003] Robots lack knowledge of the external environment or objects and typically interact with it through multimodal data. Currently, embodied robots primarily utilize a combination of visual and tactile data to overcome the limitations of a single modality. While visual sensors can provide rich color and texture information, their performance is affected when objects are obscured or lighting conditions are suboptimal. While tactile sensors can provide precise contact information, their sensing range is limited, preventing them from capturing comprehensive information about an entire object all at once. Therefore, robots need to conduct multiple explorations to compensate for the limitations of a single modality and achieve a more complete understanding of their environment.

[0004] Many manipulation tasks rely on a correct three-dimensional representation of the object being manipulated. Within complex operating spaces, constructing a reasonable three-dimensional representation of the object can significantly improve the efficiency of embodied robots. However, object information acquired by vision in environments with occlusion and other conditions is limited by the receptive field of tactile perception, requiring the robot to perform more exploration. Therefore, a reconstruction method is urgently needed to overcome the limitations of existing technologies. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for object point cloud reconstruction based on a guided cross-modal robot to achieve more accurate point cloud reconstruction.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A method for object point cloud reconstruction based on a guided cross-modal robot comprises the following steps:

[0008] Obtain the depth image and corresponding RGB image of the object under the robot's single view, perform preprocessing, obtain the incomplete point cloud data of the object, and input it into the point cloud reconstruction network for reconstruction, and output the complete point cloud P of the object. c , where the point cloud reconstruction network includes a point cloud classification encoder, a point cloud data encoder, a tactile space encoder, a visual-tactile cross-modal fusion module, and a pyramid decoder;

[0009] The execution steps of the point cloud reconstruction network include:

[0010] Based on the incomplete point cloud data of the object, a point cloud classification encoder is used to encode the point cloud classification task to extract the point cloud classification feature F p_1 , and use the point cloud classification decoder to decode and obtain the classification result of the object;

[0011] Based on the incomplete point cloud data of the object, the point cloud data encoder is used to perform position embedding encoding to obtain the point cloud feature F containing the position encoding p_2 ;

[0012] Obtain the tactile data T of the object in the robot's single-view blind spot, encode it with the tactile space encoder, and obtain the tactile feature F T ;

[0013] The visual-tactile cross-modal fusion module is used to transform the point cloud feature F containing position encoding into p_2 and tactile characteristics F T Fusion is performed to obtain the fusion feature F f , and use the classification results to fusion feature F in the fusion process f Provide guidance;

[0014] The pyramid decoder is used to decode the fusion feature F f Decode and get the complete point cloud P of the object c .

[0015] Furthermore, the steps of the pre-processing operation include:

[0016] The depth image pixel coordinate system is converted to the world coordinate system using the camera's internal and external parameters to obtain initial single-view 3D point cloud data in the real world;

[0017] Based on the initial single-view three-dimensional point cloud data, converting the number of point clouds through a downsampling operation to meet network input requirements, thereby obtaining final single-view three-dimensional point cloud data;

[0018] The final single-view 3D point cloud data is segmented using a pre-trained segment anything model to obtain an initial segmentation result as the initial object point cloud incomplete data;

[0019] Based on the RGB image, the initial object point cloud incomplete data and the corresponding point cloud position are matched to obtain final object point cloud incomplete data.

[0020] Furthermore, the point cloud classification encoder includes a mamba model and a multi-head self-attention layer, and the execution steps of the point cloud classification encoder include:

[0021] Based on the incomplete point cloud data of the object, the mamba model uses multiple convolution layers to perform convolution processing respectively, and uses the maximum pooling layer to perform splicing to obtain the initial point cloud classification features;

[0022] Based on the initial point cloud classification features, a multi-head self-attention layer is used for further processing to obtain the final point cloud classification features F p_1 , where the formula for the multi-head self-attention layer is:

[0023]

[0024] Where Attention(Q,K,V) is the self-attention value, Q, K, V are query, key and value respectively, softmax is the normalized exponential function, d k The scaling factor makes the results of Q and K satisfy the distribution with mean 0 and variance 1.

[0025] Furthermore, the execution steps of the point cloud classification decoder include:

[0026] Based on the point cloud classification feature F p_1 , use the point cloud classification decoder to decode and output the classification result of the object, wherein the point cloud classification decoder uses Softmax cross entropy loss as the category loss during the decoding process, and the Softmax cross entropy loss is:

[0027]

[0028] Where, Loss class is the category loss for point cloud classification, y i is the probability distribution of the true sample label, is the probability distribution of the normalized prediction output.

[0029] Furthermore, the point cloud data encoder includes a position embedding module, a multi-layer convolution layer and a maximum pooling layer, and the execution steps of the point cloud data encoder include:

[0030] The position embedding module groups and sorts the incomplete point cloud data of the object using the farthest point sampling algorithm and the K-nearest neighbor algorithm, and constructs a position code based on the sorting results to achieve position embedding of each group;

[0031] According to the position embedding results of each group, multi-layer convolutional layers are used to perform convolution processing respectively, and then spliced ​​after maximum pooling to obtain the point cloud feature F containing position coding p_2 .

[0032] Furthermore, the tactile space encoder is constructed by a graph convolutional neural network model, and the tactile space encoder is used to encode the spatial position of the sensing unit and match the tactile data T, so as to embed the encoded spatial position into the tactile data T, and then perform tactile feature F according to the matching result. T Extraction, where the spatial position encoding formula is:

[0033]

[0034] Where H(l+1) represents the node feature matrix of the l+1th layer, that is, the node features updated after the graph convolution operation, and H (l) is the node feature matrix of the lth layer, where each row represents the feature vector of a node, σ(·) is the nonlinear activation function, It is the adjacency matrix A of the graph plus the identity matrix In, n is the number of nodes, yes The degree matrix is ​​a diagonal matrix whose diagonal elements are the sum of each row of A, W (l) is the weight matrix of the lth layer, which is used to perform linear transformation on node features.

[0035] Furthermore, the visual-tactile cross-modal fusion module includes a splicing layer, a mamba model, a multi-layer convolutional layer and a linear layer. The execution steps of the visual-tactile cross-modal fusion module include:

[0036] The splicing layer is used to convert the F containing position code into p_2 and tactile characteristics F T Process them into the same format and concatenate them by column through concatenation operation to obtain concatenated features;

[0037] Based on the splicing features, multi-layer convolutional layers are used to extract local and global patterns, and the Mamba model is used for fusion. At the same time, the classification results are used to guide the fusion process to obtain preliminary fusion features, where the expression of the Mamba model is:

[0038] h(t)=Ah(t-1)+Bx(t)

[0039] y(t)=Ch(t)+Dx(t)

[0040] Where h(t) represents the state at the current moment t, which is a hidden state within the model. A is the state transfer matrix, which describes how the state changes over time. x(t) represents the input control variable at the current moment t. B represents the influence of the control variable x(t) on the state variable h(t). y(t) represents the system output at the current moment t. C represents the influence of the current state variable h(t) on the output y(t). D represents the direct influence of the current control variable x(t) on the output y(t).

[0041] Based on the preliminary fusion features, a linear layer is used for mapping to obtain the final fusion feature F f .

[0042] Furthermore, the pyramid decoder includes a fully connected network layer and a convolutional network layer, and the steps of decoding by the pyramid decoder include:

[0043] Based on the fusion feature F f Multiple fully connected network layers are used to extract channel features and obtain channel decoding input features F c_1 and F c_2 ;

[0044] The fusion feature F f , channel decoding input feature F c_1 and F c_2 Multiple convolutional network layers are used for multi-channel decoding to obtain point cloud results P of different scales. c_1 、P c_2 and P c_3 , where P c_2 By P c_1 With F c_1 The joint calculation results show that P c_3 By P c_2 With F c_2 The joint calculation results are:

[0045] The P c_3 Input the head layer for processing to obtain the reconstructed complete point cloud Pc.

[0046] Furthermore, the loss function used by the pyramid decoder is:

[0047] Loss gen =d CD (Y1,Y GT_1 )+d CD (Y2,Y GT_2 )+d CD (Y3,Y GT_3 )

[0048]

[0049] Where, Loss gen is the pyramid decoder loss, Y1, Y2 and Y3 are the predicted point clouds of three scales, Y GT_1 、Y GT_2 、Y GT_3 is the real point cloud at three scales, d CD is the chamfer distance, S1 and S2 represent the two point cloud sets for calculating the chamfer distance.

[0050] Furthermore, the loss function used by the point cloud reconstruction network is:

[0051]

[0052] Where, Loss class To reconstruct the total loss, Loss class is the category loss for point cloud classification, Loss gen is the loss function for training the pyramid decoder, are the weights of the two loss functions respectively.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] (1) In complex environments, the visual information obtained by the robot from a single perspective is limited. The robot of the present invention obtains visual data from a single perspective, obtains tactile data of the occluded part by touching, and then fuses the visual and tactile features through a cross-modal fusion module. In the fusion process, the classification results are used to guide the optimization of the fusion features, which can more accurately represent the three-dimensional information of the object. Finally, through decoding, more accurate point cloud reconstruction is achieved.

[0055] (2) The present invention guides the fusion of features by using classification results to optimize feature representation, which can not only effectively reduce the number of times the robot explores the environment through accurate three-dimensional representation, but also improve the robot's perception and cognition.

[0056] (3) In complex or unknown environments, the present invention can better help robots understand and adapt to the environment and improve their decision-making ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Schematic diagram of the method flow of the present invention;

[0058] Figure 2 Schematic diagram of the point cloud reconstruction network of the present invention. DETAILED DESCRIPTION

[0059] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0060] This embodiment provides a method for object point cloud reconstruction based on a guided cross-modal robot. Figure 2As shown in the figure, firstly, the depth camera is used to obtain the operating environment information, then the tactile sensor is used to touch the object to obtain the object surface data, and the encoder in the point cloud reconstruction network is used to extract the visual and tactile features. Then, the decoder is used to predict and complete the sparse point cloud to complete the 3D point cloud reconstruction of the operating object in the environment. The point cloud reconstruction network includes a point cloud classification encoder, a point cloud data encoder, a tactile space encoder, a visual and tactile cross-modal fusion module and a pyramid decoder. Specifically, combined with Figure 1 and Figure 2 , the method comprises the following steps:

[0061] Step S1: The RGB-D camera is used to obtain the depth image and RGB color image of the operating environment within the robot's field of view, and a single-view 3D point cloud P is obtained through preprocessing.

[0062] In step S1, preprocessing is to construct single-view 3D point cloud data based on the depth image acquired by the camera and the internal and external parameters of the camera; the depth image pixel coordinate system is converted to the world coordinate system using the internal and external parameters of the camera to obtain the true position of the point cloud in the real world, and the number of the acquired object point cloud is converted to the 2048 points required by the network input through downsampling operation: i =(x,y,z), where x, y, and z are the three-dimensional coordinates in space;

[0063] Step S2: Migrate the image segmentation tool to the single-view 3D point cloud space, perform point cloud segmentation processing on the 3D point cloud P, and obtain incomplete object point cloud data.

[0064] In step S2, the image segmentation tool is migrated to adapt the pre-trained segment anything model to the three-dimensional point cloud data; the grouping results after segmenting the point cloud data through the RGB image match the point cloud position, and the segmentation results are matched with the point cloud data.

[0065] Step S3: obtaining tactile data under the occlusion viewing angle by touching the highest point of curvature of the object to be operated in the touch space, and obtaining tactile data T at the blind spot position of the object.

[0066] In step S3, the tactile data is obtained by touching the blind spot of the single view using a tactile sensor to obtain the information of the blocked part of the object to be operated: i =(N,C,x,y,z), where N is the perceived position, C is the perceived value, and x, y, and z are the three-dimensional coordinates in space.

[0067] Step S4: Use the point cloud classification encoder based on the fusion of Mamba and attention mechanism to classify the segmentation results and obtain the classification results.

[0068] Step S5: Use the farthest point sampling and grouping strategy to reconstruct the point cloud data and generate position coding. Use the multi-layer hybrid perception network and mamba to build a point cloud data encoder to achieve feature embedding and obtain the point cloud feature F containing position coding. p .

[0069] Step S6: Use the graph structure to match the spatial position of tactile perception, construct a tactile spatial encoder, and obtain the tactile feature F T .

[0070] The encoder in steps S4, S5, and S6 is used to complete the feature processing of the data. It consists of three encoders: a point cloud classification encoder, a point cloud data encoder, and a tactile space encoder. One encoder encodes the point cloud classification task, one encodes the cross-modal fusion of point cloud data, and one encodes the tactile information. The classification results and point cloud features F are obtained respectively. p and tactile characteristics F T ;

[0071] The point cloud classification encoder consists of a mamba model and a multi-head self-attention layer. The mamba model consists of multiple convolutional layers. First, the outputs of the last four convolutional layers of the multi-layer convolutional layer are respectively concatenated through the maximum pooling layer to obtain the initial point cloud classification features; the self-attention layer is responsible for further extracting point cloud features to obtain the initial point cloud classification features. The formula of the self-attention layer is:

[0072]

[0073] Among them, Q, K, V are query, key and value respectively, and softmax is the normalized exponential function.

[0074] The point cloud classification decoder outputs the object category result; the category loss uses Softmax cross entropy loss:

[0075]

[0076] Where y i is the probability distribution of the true sample label, is the probability distribution of the normalized prediction output;

[0077] The point cloud data encoder consists of a position embedding module, a multi-layer convolutional layer and a maximum pooling layer. The position embedding module groups the point cloud through the farthest point sampling and K nearest neighbor algorithm, and completes the sorting. At the same time, it constructs the position encoding according to the sorting result to realize the position embedding of the encoder of each group. The outputs of the last four convolutional layers composed of the multi-layer convolutional layers are respectively spliced ​​after the maximum pooling layer to obtain the point cloud feature F containing the position encoding. p_2 ;

[0078] The tactile spatial encoder is constructed by a graph convolutional neural network model, which encodes and matches the tactile data T according to the spatial position of the sensing unit, so as to embed the encoded spatial position into the tactile data T, and then performs tactile feature F according to the matching result. T Extracting tactile data. The characteristics of graph neural networks can be used to map sensor units to spaces, and the importance of different sensor units can be reflected through manual mapping. The position encoding formula is:

[0079]

[0080] Among them, H l H represents the node feature matrix of the lth layer, where each row represents the feature vector of a node. (l+1) Represents the node feature matrix of the l+1th layer, that is, the node features updated after the graph convolution operation. =A+In is the adjacency matrix A of the graph plus the identity matrix In (to account for self-connections), where N is the number of nodes. yes The degree matrix of is a diagonal matrix whose diagonal elements are The sum of each row. W l is the weight matrix of the lth layer, which is used to linearly transform the node features, and σ is the nonlinear activation function.

[0081] Step S7: Use the visual-tactile cross-modal fusion module to fuse the visual features and tactile features to obtain the visual-tactile cross-modal fusion feature F f , and use the classification results to guide the fusion features.

[0082] In step S7, the visual-tactile cross-modal fusion module is used to transform the point cloud feature F p_2 and tactile characteristics F T Fusion is performed. This module consists of a splicing layer, a mamba model, a multi-layer convolutional layer, and a linear layer, and finally outputs the result;

[0083] The visual-tactile cross-modal fusion module first transforms the point cloud feature F T and tactile characteristics F T Processed into the same format, the two types of features are spliced ​​by column through the concatenation operation to obtain a 1920×2 splicing feature; based on the splicing feature, a multi-layer convolution layer is used to extract local and global patterns, and the mamba model is used for fusion. At the same time, the classification results are used to guide the fusion process to obtain a preliminary fusion feature; the linear layer is used to linearly map the preliminary fusion feature to obtain the fusion feature F f The formula of the mamba model is:

[0084] h(t) = Ah(t-1) + Bx(t) (4)

[0085] y(t) = Ch(t) + Dx(t) (5)

[0086] Where h(t) represents the state at the current moment t, which is a hidden state inside the model. A is the state transfer matrix, which describes how the state changes over time. x(t) represents the input control variable at the current moment t. B represents the influence of the control variable x(t) on the state variable h(t). y(t) represents the system output at the current moment t. C represents the influence of the current state variable h(t) on the output y(t). D represents the direct influence of the current control variable x(t) on the output y(t).

[0087] Step S8: Use the pyramid decoder to decode the fused features to obtain the complete point cloud P c .

[0088] In step S8, the pyramid decoder decodes the fused features and predicts the complete point cloud to obtain the complete point cloud P c , the pyramid decoder consists of two fully connected networks and three convolutional networks. The two fully connected networks further extract the fusion feature F f , and get F c_1 and F c_2 ; The inputs of the three convolutional networks are F f 、F c_1 and F c_2 , the output is the reconstructed point cloud results of three different scales, namely W1, W2, and W3. For single-view scenes, they are set to W1=128, W2=512, and W3=2048;

[0089] The loss function for pyramid decoder training is:

[0090] Loss gen = d CD (Y1,Y GT_1 )+d CD (Y2,Y GT_2 )+d CD (Y3,Y GT_3 ) (6)

[0091]

[0092] Among them, d CD is the chamfer distance, Y1, Y2 and Y3 are the predicted point clouds of three scales, Y GT_1 is the real point cloud Y at three scales GT_2 Y GT_3These are real point clouds at three scales, where S1 and S2 represent the two point cloud sets for which the chamfer distance is to be calculated;

[0093] The loss function of the overall point cloud reconstruction network is:

[0094]

[0095] are the weights of the two loss functions.

[0096] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0097] It will be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention may be implemented in various computer languages, for example, the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0098] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0101] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0102] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for object point cloud reconstruction based on a guided cross-modal robot, characterized in that: The following steps are involved: Obtain the depth image and corresponding RGB image of the object under the robot's single view, perform preprocessing, obtain the incomplete point cloud data of the object, and input it into the point cloud reconstruction network for reconstruction, and output the complete point cloud P of the object. c , where the point cloud reconstruction network includes a point cloud classification encoder, a point cloud data encoder, a tactile space encoder, a visual-tactile cross-modal fusion module, and a pyramid decoder; The execution steps of the point cloud reconstruction network include: Based on the incomplete point cloud data of the object, a point cloud classification encoder is used to encode the point cloud classification task to extract the point cloud classification feature F p_1 , and use the point cloud classification decoder to decode and obtain the classification result of the object; Based on the incomplete point cloud data of the object, the point cloud data encoder is used to perform position embedding encoding to obtain the point cloud feature F containing the position encoding p_2 ; Obtain the tactile data T of the object in the robot's single-view blind spot, encode it with the tactile space encoder, and obtain the tactile feature F T ; The visual-tactile cross-modal fusion module is used to transform the point cloud feature F containing position encoding into p_2 and tactile features F T Fusion is performed to obtain the fusion feature F f , and use the classification results to fusion feature F in the fusion process f Provide guidance; The pyramid decoder is used to decode the fusion feature F f Decode and get the complete point cloud P of the object c .

2. The method for object point cloud reconstruction based on a guided cross-modal robot according to claim 1, characterized in that: The steps of the pre-processing operation include: The depth image pixel coordinate system is converted to the world coordinate system using the camera's internal and external parameters to obtain initial single-view 3D point cloud data in the real world; Based on the initial single-view three-dimensional point cloud data, converting the number of point clouds through a downsampling operation to meet network input requirements, thereby obtaining final single-view three-dimensional point cloud data; The final single-view 3D point cloud data is segmented using a pre-trained segment anything model to obtain an initial segmentation result as the initial object point cloud incomplete data; Based on the RGB image, the initial object point cloud incomplete data and the corresponding point cloud position are matched to obtain the final object point cloud incomplete data.

3. The method for object point cloud reconstruction based on a guided cross-modal robot according to claim 1, characterized in that: The point cloud classification encoder includes a mamba model and a multi-head self-attention layer. The execution steps of the point cloud classification encoder include: Based on the incomplete point cloud data of the object, the mamba model uses multiple convolution layers to perform convolution processing respectively, and uses the maximum pooling layer to perform splicing to obtain the initial point cloud classification features; Based on the initial point cloud classification features, a multi-head self-attention layer is used for further processing to obtain the final point cloud classification features F p_1 , where the formula for the multi-head self-attention layer is: Where Attention(Q,K,V) is the self-attention value, Q, K, V are query, key and value respectively, softmax is the normalized exponential function, d k The scaling factor makes the results of Q and K satisfy the distribution with mean 0 and variance 1.

4. The method for object point cloud reconstruction based on a guided cross-modal robot according to claim 3, characterized in that: The execution steps of the point cloud classification decoder include: Based on the point cloud classification feature F p_1 , use the point cloud classification decoder to decode and output the classification result of the object, wherein the point cloud classification decoder uses Softmax cross entropy loss as the category loss during the decoding process, and the Softmax cross entropy loss is: Where, Loss class is the category loss for point cloud classification, y i is the probability distribution of the true sample label, is the probability distribution of the normalized prediction output.

5. The method for object point cloud reconstruction based on a guided cross-modal robot according to claim 1, characterized in that: The point cloud data encoder includes a position embedding module, a multi-layer convolution layer and a maximum pooling layer. The execution steps of the point cloud data encoder include: The position embedding module groups and sorts the incomplete point cloud data of the object using the farthest point sampling algorithm and the K-nearest neighbor algorithm, and constructs a position code based on the sorting results to achieve position embedding of each group; According to the position embedding results of each group, multi-layer convolutional layers are used to perform convolution processing and then spliced ​​after maximum pooling to obtain the point cloud feature F containing position coding p_2 .

6. The method for object point cloud reconstruction based on a guided cross-modal robot according to claim 1, characterized in that: The tactile space encoder is constructed by a graph convolutional neural network model. The tactile space encoder is used to encode the spatial position of the sensing unit and match the tactile data T, so as to embed the encoded spatial position into the tactile data T, and then perform tactile feature F according to the matching result. T Extraction, where the spatial position encoding formula is: Where H (l+1) represents the node feature matrix of the l+1th layer, that is, the node features updated after the graph convolution operation, H (l) is the node feature matrix of the lth layer, where each row represents the feature vector of a node, σ(·) is the nonlinear activation function, It is the adjacency matrix A of the graph plus the identity matrix In, n is the number of nodes, yes The degree matrix is ​​a diagonal matrix whose diagonal elements are the sum of each row of A, W (l) is the weight matrix of the lth layer, which is used to perform linear transformation on node features.

7. The method for object point cloud reconstruction based on a guided cross-modal robot according to claim 1, characterized in that: The visual-tactile cross-modal fusion module includes a splicing layer, a mamba model, a multi-layer convolutional layer, and a linear layer. The execution steps of the visual-tactile cross-modal fusion module include: The splicing layer is used to convert the F containing position code into p_2 and tactile features F T Process them into the same format and concatenate them by column through concatenation operation to obtain concatenated features; Based on the splicing features, multi-layer convolutional layers are used to extract local and global patterns, and the Mamba model is used for fusion. At the same time, the classification results are used to guide the fusion process to obtain preliminary fusion features, where the expression of the Mamba model is: h(t)=Ah(t-1)+Bx(t) y(t)=Ch(t)+Dx(t) Where h(t) represents the state at the current moment t, which is a hidden state within the model. A is the state transfer matrix, which describes how the state changes over time. x(t) represents the input control variable at the current moment t. B represents the influence of the control variable x(t) on the state variable h(t). y(t) represents the system output at the current moment t. C represents the influence of the current state variable h(t) on the output y(t). D represents the direct influence of the current control variable x(t) on the output y(t). Based on the preliminary fusion features, a linear layer is used for mapping to obtain the final fusion feature F f .

8. The method for object point cloud reconstruction based on a guided cross-modal robot according to claim 1, characterized in that: The pyramid decoder includes a fully connected network layer and a convolutional network layer. The steps of decoding by the pyramid decoder include: Based on the fusion feature F f Multiple fully connected network layers are used to extract channel features and obtain channel decoding input features F c_1 and F c_2 ; The fusion feature F f , channel decoding input feature F c_1 and F c_2 Multiple convolutional network layers are used for multi-channel decoding to obtain point cloud results P of different scales. c_1 、P c_2 and P c_3 , where P c_2 By P c_1 With F c_1 The joint calculation results show that P c_3 By P c_2 With F c_2 The joint calculation results are: The P c_3 Input the head layer for processing to obtain the reconstructed complete point cloud Pc.

9. The method for object point cloud reconstruction based on a guided cross-modal robot according to claim 1, characterized in that: The loss function used by the pyramid decoder is: Loss gen =d CD (Y1,Y GT_1 )+d CD (Y2,Y GT_2 )+d CD (Y3,Y GT_3 ) Where, Loss gen is the pyramid decoder loss, Y1, Y2 and Y3 are the predicted point clouds of three scales, Y GT_1 、Y GT_2 、Y GT_3 is the real point cloud at three scales, d CD is the chamfer distance, S1 and S2 represent the two point cloud sets for calculating the chamfer distance.

10. The method for object point cloud reconstruction based on a guided cross-modal robot according to claim 1, characterized in that: The loss function used by the point cloud reconstruction network is: Where, Loss class To reconstruct the total loss, Loss class is the category loss for point cloud classification, Loss gen is the loss function for training the pyramid decoder, are the weights of the two loss functions respectively.

Images