Incomplete shape symmetry prediction method and system based on multi-modal feature fusion

Abstract

The application relates to a multi-modal feature fusion-based incomplete shape symmetry prediction method and system, which comprises the following steps: inputting an RGB image and a depth image of an incomplete shape into a channel perception network for fusion; cross-modal features extracted by the channel perception network are fused with color and depth modal features; the cross-modal features are input into a self-reconstruction network of a three-dimensional variational autoencoder for reconstruction to obtain a complete geometric shape, which provides global geometric features for a subsequent symmetry prediction network; and the cross-modal features and the global geometric features are input into the symmetry prediction network for symmetry prediction to obtain predicted symmetry of the incomplete shape. The method can reconstruct the incomplete shape into a complete geometric shape, overcome the defect that the incomplete shape lacks global geometric features, and realize symmetry prediction of the incomplete shape.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of symmetry prediction, in particular to a method and system for incomplete shape symmetry prediction based on multi-modal feature fusion. BACKGROUND

[0002] Detecting the symmetry of three-dimensional objects is a key and basic problem in many computer vision applications. In geometric processing, finding symmetry in geometric data (such as point clouds, polygonal meshes, and voxels) can help many applications use symmetry information to solve tasks or improve algorithms, such as shape matching, segmentation, completion, etc. Among all types of symmetry, the most common and important is the plane reflection symmetry. Traditional plane symmetry detection methods are usually based on the observation that after mirror transformation along the symmetry plane, all sampling points on the shape will coincide with the original shape. For example, a shape can be aligned with the principal axis, and then the planes formed by pairs of principal axes can be checked to see if they are symmetry planes. Recently, deep learning-based methods use neural networks to extract global features of shapes for capturing possible symmetries.

[0003] However, the above methods are all for symmetry detection of complete three-dimensional shapes. However, due to the limitations of sensor resolution, monocular and occlusion, the captured object shape is usually incomplete in practical applications, in which case the basic assumption of the above complete three-dimensional shape will be broken.

[0004] For symmetry detection of incomplete shapes, a common application scenario is to estimate the symmetry of three-dimensional shapes based on single-view RGB-D (Red-Green-Blue-Depth Map) images, but due to partial observation and occlusion of objects, it brings special challenges that cannot be solved by geometric detection, for example, the lack of global geometric information will cause a key difficulty in finding local symmetry correspondence supported by mirror transformation. SUMMARY

[0005] Therefore, it is necessary to provide a method and system for incomplete shape symmetry prediction based on multi-modal feature fusion in view of the above technical problems.

[0006] A method for incomplete shape symmetry prediction based on multi-modal feature fusion, the method comprising:

[0007] inputting the RGB image and the depth image of the incomplete shape into a channel perception network for fusion to obtain cross-modal features of the incomplete shape;

[0008] inputting the cross-modal features into a self-reconstruction network of a three-dimensional variational autoencoder for reconstruction to obtain a complete geometric shape, wherein the complete geometric shape provides global geometric features;

[0009] The cross-modal features and the global geometric features are input into the symmetry prediction network to predict the symmetry of the incomplete shape.

[0010] In one embodiment, the RGB image and the depth image of the incomplete shape are input into the channel perception network to be fused to obtain the cross-modal features of the incomplete shape, and the method further comprises:

[0011] The RGB image and the depth image of the incomplete shape are input, and two sub-networks are established to process the modal features of the RGB image and the modal features of the depth image respectively to obtain a color feature map and a depth feature map.

[0012] The color feature map and the depth feature map are normalized according to the batch normalization layer in the channel perception network to determine the importance of each channel in the two feature maps.

[0013] Whether the channel needs to be perceived is determined according to the importance of each channel in the two feature maps, if not, the channel is output, and if yes, the channel is replaced by a new channel obtained by fusing the information of the channel and the corresponding channel in the other feature map according to the channel perception network to obtain the cross-modal features fused with the information of the two feature maps.

[0014] In one embodiment, whether the channel needs to be replaced is determined according to the importance of each channel in the two feature maps, and the importance of each channel in the two feature maps is represented by a scale factor in the batch normalization layer, whether the channel needs to be perceived is determined according to the scale factor, if not, the channel is output, and if yes, a new channel is obtained by fusing the channel and the corresponding channel in the other feature map according to the attention perception network, the channel is replaced by the new channel, and the cross-modal features fused with the information of the two feature maps are obtained.

[0015] In one embodiment, the cross-modal features fused with the information of the two feature maps are obtained, and the method further comprises: inputting the cross-modal features into a nonlinear activation, and performing convolution on the activated cross-modal features in the self-reconstruction network of the three-dimensional variational autoencoder.

[0016] In one embodiment, the self-reconstruction network of the three-dimensional variational autoencoder further comprises:

[0017] The self-reconstruction network of the three-dimensional variational autoencoder is composed of an encoder network, a latent layer and a decoder network.

[0018] The encoder network is a three-dimensional down-sampling network composed of three three-dimensional convolution layers, and the down-sampling in the encoder network is realized by stride convolution.

[0019] The latent layer is a three-dimensional network to preserve geometric features.

[0020] The decoder network is a three-dimensional up-sampling network composed of three three-dimensional convolutional layers, and the up-sampling in the decoder network is realized by deconvolution.

[0021] In one embodiment, the cross-modal features are input into a self-reconstruction network of a three-dimensional variational autoencoder for reconstruction to obtain a complete geometric shape, wherein the complete geometric shape provides global geometric features.

[0022] The cross-modal features are input, and the cross-modal features are encoded into an implicit space according to an encoder network, and the implicit space is subject to a normal distribution.

[0023] The implicit space is sampled in the latent layer to obtain an input of a decoder network.

[0024] The input is reconstructed according to the decoder network to obtain a complete geometric shape, wherein the complete geometric shape provides global geometric features.

[0025] In one embodiment, the input is reconstructed according to the decoder network to obtain a complete geometric shape, further comprising defining a loss function of the three-dimensional variational autoencoder, training the reconstruction process of the decoder network according to the loss function, improving the accuracy of the reconstruction process, obtaining the complete geometric shape, and providing global geometric features, wherein the loss function represents the error between the complete geometric shape and the real complete shape of the incomplete shape.

[0026] In one embodiment, the cross-modal features and the global geometric features are input into a symmetry prediction network for symmetry prediction to obtain a predicted symmetry of the incomplete shape, further comprising:

[0027] The cross-modal features and the global geometric features are aggregated to obtain a final feature, and the final feature is input into the symmetry prediction network for symmetry prediction, wherein the symmetry prediction network is composed of six three-dimensional convolutional layers.

[0028] After convolution at each three-dimensional convolutional layer, the final feature after convolution is activated according to a max-pool and a rectified linear unit to obtain an activated final feature.

[0029] After convolution at each three-dimensional convolutional layer, the final feature after convolution is activated according to a max-pool and a rectified linear unit to obtain an activated final feature.

[0030] In one embodiment, the predicted symmetry of the incomplete shape is obtained, further comprising calculating a dense symmetry error according to the difference between the predicted symmetry of the incomplete shape and a true value symmetry to determine whether the predicted symmetry of the incomplete shape is correct.

[0031] An incomplete shape symmetry prediction system based on multi-modal feature fusion, the system comprising:

[0032] a multi-modal feature fusion module, configured to input an incomplete shape RGB image and a depth image into a channel perception network for fusion to obtain cross-modal features of the incomplete shape;

[0033] a reconstruction module, configured to input the cross-modal features into a self-reconstruction network of a three-dimensional variational autoencoder for reconstruction to obtain a complete geometric shape, wherein the complete geometric shape provides global geometric features;

[0034] a symmetry prediction module, configured to input the cross-modal features and the global geometric features into a symmetry prediction network for symmetry prediction to obtain predicted symmetry of the incomplete shape.

[0035] The multi-modal feature fusion-based incomplete shape symmetry prediction method and system reconstruct the incomplete shape to obtain a complete geometric shape, overcome the defect that the incomplete shape lacks global geometric features, and realize symmetry prediction of the incomplete shape. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 FIG. 1 is a flowchart of a multi-modal feature fusion-based incomplete shape symmetry prediction method according to an embodiment;

[0037] Figure 2 FIG. 2 is a structural block diagram of a multi-modal feature fusion-based incomplete shape symmetry prediction system according to an embodiment;

[0038] Figure 3 FIG. 3 is a structural block diagram of a multi-modal feature fusion module according to an embodiment;

[0039] Figure 4 FIG. 4 is a structural block diagram of a reconstruction module according to an embodiment;

[0040] Figure 5 FIG. 5 is a structural block diagram of a symmetry prediction module according to an embodiment;

[0041] Figure 6A comparison of the performance of a multimodal feature fusion-based incomplete shape symmetry prediction method in one embodiment with other methods for predicting reflection symmetry: (a) ShapeNet preserves the view

[0042] (b) ShapeNet preserved instance (c) YCB test (d) ScanNet preserved instance. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0044] In one embodiment, such as Figure 1 As shown, a method for predicting the symmetry of incomplete shapes based on multimodal feature fusion is provided, including the following steps:

[0045] Step 102: The incomplete shape RGB image and depth image are fused into the input channel perceptual network to obtain the cross-modal features of the incomplete shape.

[0046] It is understandable that incomplete shape refers to a shape that lacks global geometric features when captured due to limitations such as sensor resolution, single viewpoint, and occlusion. RGB image (three-channel color image) provides color features of incomplete shape, and depth image provides depth features of incomplete shape. By inputting these two modal features, color features and depth features, into the channel perception network (CTN) for fusion, the cross-modal features of incomplete shape are obtained.

[0047] Step 104: Input the cross-modal features into the self-reconstruction network of the 3D variational autoencoder for reconstruction to obtain the complete geometry, where the complete geometry provides global geometric features.

[0048] It is understandable that, since incomplete shapes lack global geometric features, it is difficult to find local symmetric correspondences supported by mirror transformation. Therefore, it is necessary to use the self-reconstruction network of a 3D variational autoencoder (3D-VAE) to reconstruct the incomplete shape to obtain the complete geometric shape, so as to provide global geometric features for the subsequent symmetry prediction network.

[0049] Step 106: Input the cross-modal features and global geometric features into the symmetry prediction network to perform symmetry prediction and obtain the predicted symmetry of the incomplete shape.

[0050] It is understandable that cross-modal features and global geometric features not only provide rich texture information and a large number of contour details, but also overcome the deficiency of incomplete shapes lacking global geometric features, and realize the prediction of the symmetry of incomplete shapes.

[0051] The aforementioned incomplete shape symmetry prediction method based on multimodal feature fusion fuses RGB and depth images of the incomplete shape into a channel perception network. The extracted cross-modal features, which integrate color and depth modal features, are then input into the self-reconstruction network of a 3D variational autoencoder to reconstruct the complete geometric shape. This provides global geometric features for the subsequent symmetry prediction network. The cross-modal features and global geometric features are then input into the symmetry prediction network for symmetry prediction, yielding the predicted symmetry of the incomplete shape. Compared to existing technologies, the incomplete shape symmetry prediction method and system proposed in this invention reconstruct the incomplete shape to obtain a complete geometric shape, overcoming the deficiency of incomplete shapes lacking global geometric features and achieving symmetry prediction for incomplete shapes.

[0052] In one embodiment, the i-th input data of the incomplete shape RGB image and depth image is: Where C represents the number of channels, H represents the height of the feature map, W represents the width of the feature map, and M represents the number of modal features (M=2);

[0053] Establish two 2D U-Net subnetworks f m (x) Process the modal features of the RGB image and the modal features of the depth image respectively to obtain RGB feature maps and depth feature maps. The sub-networks share all parameters except for the batch normalization layer, including the convolutional filter. The output of each sub-network is...

[0054]

[0055] The RGB and depth feature maps are normalized using the batch normalization layer (BN layer) in the channel-aware network, followed by an affine transformation to obtain the normalized RGB and depth feature maps.

[0056]

[0057] In the formula, x m,l,c Let x′ represent the feature map of the c-th channel of the l-th layer of the m-th subnetwork. m,l,c μ represents the feature map of the c-th channel of the l-th layer of the m-th subnetwork after normalization. m,l,c and σ m,l,c γ represents the mean and standard deviation of all activations at all pixel locations (H and W) in the c-th channel of the l-th layer of the m-th subnetwork. m,l,c and β m,l,c Let ε represent the trainable scaling factor and offset of the c-th channel in the l-th layer of the m-th subnetwork, where ε is a small constant to avoid division by zero.

[0058] For feature map x m,l,c Normalization is performed to obtain x′ m,l,c It is possible to determine x m,l,c The importance of the c-th channel in x m,l,c The importance of the c-th channel is determined to determine whether the c-th channel needs to be replaced. If not, the original channel c is output; if yes, the original channel c is fused with the corresponding channel in another feature map using an attention-aware network according to the channel-aware network to obtain a new channel. The original channel c is then replaced with the new channel to obtain a cross-modal feature that integrates the information from the two feature maps and a new feature map including the cross-modal feature is output.

[0059] In one embodiment, feature map x m,l,c The importance of the c-th channel is determined by the scaling factor γ in the batch normalization layer. m,l,c This means that if γ m,l,c →0 indicates that the gradient of loss will approach 0, x m,l,c The c-th channel in the batch normalization layer will lose its influence on the final prediction and become redundant. Therefore, when the c-th channel in the batch normalization layer has a scaling factor γ... m,l,c When γ is less than the set threshold θ (i.e., γ) m,l,c When <θ), the original channel c and the corresponding channel in the other feature map are selected and fed into the transform layer of the channel-aware network. Then, the original channel c and the corresponding channel in the other feature map are fused to obtain a new channel, and the original channel c is replaced with the new channel. This yields a cross-modal feature that integrates the information from the two feature maps, and a new feature map including the cross-modal feature is output. The new feature map is represented as follows:

[0060]

[0061] In the formula, X = x m,l,c , Y = x′ m′,l,c This represents the corresponding channel in another feature map;

[0062] Specifically, the transform layer in the channel-aware network includes three cross-attention layers, each followed by two AddNorm layers and a feedforward layer. The cross-attention layer is represented as follows:

[0063] S=crossAttention(Q,K,V)=Softmax(QK T )V (4)

[0064] In the formula, Q = XW Q Let K = YW represent the query vector. K Representing the key vector, V = YW V W represents a value vector. Q W K W VThe weights representing the learning process are represented in the AddNorm layer as follows:

[0065] Z=AddNorm(X)=LayerNorm(X+S) (5).

[0066] In one embodiment, obtaining cross-modal features that fuse information from two feature maps further includes: inputting the cross-modal features into a nonlinear activation function, and then convolving the activated cross-modal features in a self-reconstruction network of a three-dimensional variational autoencoder.

[0067] It is understandable that by inputting cross-modal features into nonlinear activation, the cross-modal features can approximate any nonlinear function, making it easier to input the activated cross-modal features into the self-reconstruction network of the 3D variational autoencoder for convolution.

[0068] In one embodiment, the self-reconstruction network of the 3D variational autoencoder further includes: the self-reconstruction network of the 3D variational autoencoder consists of an encoder network, a latent layer, and a decoder network; the encoder network is a 3D downsampling network consisting of three 3D convolutional layers, and the downsampling in the encoder network is achieved through strided convolution; the latent layer is a 3D network to preserve geometric features; and the decoder network is a 3D upsampling network consisting of three 3D convolutional layers, and the upsampling in the decoder network is achieved through fractional step convolution.

[0069] It is understandable that the encoder network and decoder network are based on the 3D U-Net architecture, which has the same structure but is reversed, and the weights are unrelated; the latent layer is a three-dimensional network to preserve global geometry as much as possible and overcome the defect of incomplete shapes lacking global geometric features.

[0070] In one embodiment, the cross-modal features are input into the encoder network, which encodes the cross-modal features into an implicit space that follows a normal distribution. In the diagram, z represents the output of the encoder network, and z′ represents the input of the decoder network. It is a multivariate Gaussian distribution, denoted as:

[0071]

[0072] In the formula, μ represents the mean and σ represents the standard deviation;

[0073] In the latent layer, from the implicit space Sample at point , to obtain the input z′ of the decoder network, where z′=μ+σ⊙ε, ε∈N(0,1) represents the product of elements;

[0074] The decoder network reconstructs the input z′, and the output of the decoder network is the reconstructed result, which yields the complete geometry. The complete geometry provides global geometric features.

[0075] It is understandable that by using a 3D variational autoencoder consisting of an encoder network, a latent layer, and a decoder network to reconstruct the incomplete shape to obtain the complete geometry, global geometric features are provided for the subsequent symmetry prediction network.

[0076] In one embodiment, the loss function of the three-dimensional variational autoencoder is defined as follows:

[0077] L = L rec +wL μ,σ2 (7)

[0078] in,

[0079] L rec =-γtlog(o)-(1-γ)(1-t)log(1-o)(8)

[0080]

[0081] In the formula, L rec L represents the local optimum of standard binary cross-entropy (BCE). μ,σ2 denoted as KL (Kullback–Leibler divergence), w is the weight balancing the reconstruction loss and KL divergence, t is the target value in {0,1}, and o is the network output at each output element in {0,1}.

[0082] It is understandable that training the decoder network based on the loss function can avoid excessive reconstruction noise or the omission of significant object details and structures, thereby improving the accuracy of the reconstruction process, obtaining a complete geometric shape, and providing global geometric features.

[0083] In one embodiment, the cross-modal features from the channel-aware network are set as F. c Set the global geometric features from the 3D autoencoder to F g The final feature F is obtained by aggregating the cross-modal features and global geometric features. A Represented as

[0084] F A =concat(F c F g (10)

[0085] The final feature F A The input symmetry prediction network performs symmetry prediction. This network consists of six 3D convolutional layers, F...A After convolution in each 3D convolutional layer, the final convolutional features are activated using modified linear units based on max pooling and leakage, resulting in the activated final features F. A The final feature F that will be activated A The parameters of the predicted symmetry plane are input into the fully connected layer to obtain the predicted symmetry of the incomplete shape.

[0086] In one embodiment, for reflection symmetry, the dense symmetry error is calculated as:

[0087]

[0088] In the formula, T ref and Point P = {P} i}, symmetry transformation of the predictive symmetry and fundamental truth symmetry of incomplete shapes i∈[1,N], ρ is the transformation from point P to The maximum distance between the planes of symmetry.

[0089] It is understandable that by calculating the dense symmetry error based on the difference between the predicted symmetry of the incomplete shape and the basic true symmetry, it is possible to determine whether the predicted symmetry of the incomplete shape is a true positive or a false positive.

[0090] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0091] In one embodiment, such as Figure 2 As shown, an incomplete shape symmetry prediction system based on multimodal feature fusion includes: a multimodal feature fusion module, a reconstruction module, and a symmetry prediction module, wherein:

[0092] The multimodal feature fusion module is used to fuse the incomplete shape RGB image and depth image input channel perception network to obtain the cross-modal features of the incomplete shape;

[0093] The reconstruction module is used to input cross-modal features into the self-reconstruction network of the 3D variational autoencoder for reconstruction to obtain the complete geometry, where the complete geometry provides global geometric features;

[0094] The symmetry prediction module is used to input cross-modal features and global geometric features into the symmetry prediction network to perform symmetry prediction and obtain the predicted symmetry of the incomplete shape.

[0095] In one embodiment, such as Figure 3 As shown, in the multimodal feature fusion module, an incomplete RGB image and a depth image are input. Two sub-networks are established to process the modal features of the RGB image and the modal features of the depth image respectively, resulting in RGB feature maps and depth feature maps. The RGB feature maps and depth feature maps are normalized according to the batch normalization layer in the channel-aware network to determine the importance of each channel in the two feature maps. Based on the importance of each channel in the two feature maps, it is determined whether the channel needs to be channel-aware. If not, the channel is output. If yes, the channel is replaced with a new channel by fusing the information of the channel and the corresponding channel in the other feature map according to the information of the channel-aware network, resulting in cross-modal features that fuse the information of the two feature maps.

[0096] In one embodiment, determining whether a channel needs to be replaced based on the importance of each channel in the two feature maps further includes: the importance of each channel in the two feature maps is represented by a scaling factor in the batch normalization layer; determining whether a channel needs to be replaced based on the scaling factor; if not, outputting the channel; if yes, replacing the channel with a new channel based on the information of the channel fused by the channel-aware network and the corresponding channel in the other feature map, thereby obtaining a cross-modal feature that fuses the information of the two feature maps.

[0097] In one embodiment, obtaining cross-modal features that fuse information from two feature maps further includes: inputting the cross-modal features into a nonlinear activation function, and then convolving the activated cross-modal features in a self-reconstruction network of a three-dimensional variational autoencoder.

[0098] In one embodiment, the self-reconstruction network of the 3D variational autoencoder further includes: the self-reconstruction network of the 3D variational autoencoder consists of an encoder network, a latent layer, and a decoder network; the encoder network is a 3D downsampling network consisting of three 3D convolutional layers, and downsampling in the encoder network is achieved through strided convolution; the latent layer is a 3D network to preserve geometric features; and the decoder network is a 3D upsampling network consisting of three 3D convolutional layers, and upsampling in the decoder network is achieved through deconvolution.

[0099] In one embodiment, such as Figure 4As shown, in the reconstruction module, the cross-modal features are input into the self-reconstruction network of the 3D variational autoencoder for reconstruction to obtain the complete geometry. The complete geometry provides global geometric features and also includes: inputting the cross-modal features; encoding the cross-modal features into a latent space according to the encoder network, the latent space following a normal distribution; sampling the latent space in the latent layer to obtain the input of the decoder network; and reconstructing the input according to the decoder network to obtain the complete geometry, where the complete geometry provides global geometric features.

[0100] In one embodiment, reconstructing the input based on the decoder network to obtain a complete geometric shape further includes: defining a loss function for a 3D variational autoencoder, training the reconstruction process of the decoder network based on the loss function to improve the accuracy of the reconstruction process, obtaining a complete geometric shape, and providing global geometric features, wherein the loss function represents the error between the complete geometric shape and the true complete shape of the incomplete shape.

[0101] In one embodiment, such as Figure 5 As shown, the symmetry prediction module inputs cross-modal features and global geometric features into the symmetry prediction network for symmetry prediction, obtaining the predicted symmetry of the incomplete shape. It also includes:

[0102] The cross-modal features and global geometric features are aggregated to obtain the final features, which are then input into a symmetry prediction network for symmetry prediction. The symmetry prediction network consists of six three-dimensional convolutional layers.

[0103] After each 3D convolutional layer, the final features are activated by the modified linear units based on max pooling and leakage to obtain the activated final features.

[0104] After convolution in each 3D convolutional layer, the final features after convolution are activated based on max pooling and leaked corrected linear units to obtain the activated final features.

[0105] In one embodiment, obtaining the predicted symmetry of the incomplete shape further includes: calculating a dense symmetry error based on the difference between the predicted symmetry of the incomplete shape and the fundamental truth symmetry, and determining the correctness of the predicted symmetry of the incomplete shape.

[0106] Specific limitations regarding the incomplete shape symmetry prediction system based on multimodal feature fusion can be found in the limitations of the incomplete shape symmetry prediction method based on multimodal feature fusion described above, and will not be repeated here. Each module in the aforementioned incomplete shape symmetry prediction system based on multimodal feature fusion can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0107] To further verify the effectiveness of the incomplete shape symmetry prediction method based on multimodal feature fusion provided by the present invention, in one embodiment, such as Figure 6 As shown, the performance of the incomplete shape symmetry prediction method based on multimodal feature fusion provided in this invention in predicting reflection symmetry was compared with RGB-D Retrieval and Geometric Fitting methods on three public datasets: ShapeNet, YCB, and ScanNet. On the ShapeNet dataset, a training set containing 100,000 training RGB-D images was used, and the test set was divided into two subsets: view-preserving and instance-preserving. On the YCB dataset, a predetermined original partition was used. On the ScanNet dataset, a training set containing 13,126 training RGB-D images was used, and the test set was divided into two subsets: view-preserving and scene-preserving. The comparison results show that the incomplete shape symmetry prediction method based on multimodal feature fusion provided in this invention achieved the best scores on all dataset subsets.

[0108] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0109] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for predicting the symmetry of incomplete shapes based on multimodal feature fusion, characterized in that, The method includes: An incomplete RGB image and a depth image are input into a channel-aware network and fused to obtain the cross-modal features of the incomplete shape. The fusion process of the channel-aware network includes: inputting the incomplete RGB image and depth image, establishing two sub-networks to process the modal features of the RGB image and the modal features of the depth image respectively, obtaining a color feature map and a depth feature map; normalizing the color feature map and the depth feature map respectively according to the batch normalization layer in the channel-aware network to determine the importance of each channel in the two feature maps; the importance of each channel in the two feature maps is represented by a scaling factor in the batch normalization layer, and determining whether the channel needs to be replaced according to the scaling factor. If not, the channel is output; if yes, the channel is fed into the transform layer in the channel-aware network with the corresponding channel in another feature map to obtain a new channel, and the channel is replaced with the new channel to obtain the cross-modal features that fuse the information of the two feature maps; wherein, the transform layer includes three cross-attention layers, each of which is followed by two AddNorm layers and a feedforward layer. The cross-modal features are input into the self-reconstruction network of the three-dimensional variational autoencoder for reconstruction to obtain the complete geometry, wherein the complete geometry provides global geometric features; The cross-modal features and the global geometric features are input into a symmetry prediction network for symmetry prediction to obtain the predicted symmetry of the incomplete shape. The prediction process of the symmetry prediction network includes: aggregating the cross-modal features and the global geometric features to obtain the final features; inputting the final features into the symmetry prediction network for symmetry prediction, wherein the symmetry prediction network consists of six three-dimensional convolutional layers; after convolution in each of the three-dimensional convolutional layers, the final features are activated according to max pooling and activation functions, and the output is the final features; the final features are input into a fully connected layer to predict the parameters of the symmetry plane to obtain the predicted symmetry of the incomplete shape.

2. The method according to claim 1, characterized in that, After obtaining the cross-modal feature that fuses the information from the two feature maps, the method further includes: The cross-modal features are input into a nonlinear activation function, and the activated cross-modal features are then convolved in the self-reconstruction network of the three-dimensional variational autoencoder.

3. The method according to claim 1, characterized in that, The self-reconstruction network of the three-dimensional variational autoencoder includes: The self-reconstruction network of the three-dimensional variational autoencoder consists of an encoder network, a latent layer, and a decoder network. The encoder network is a three-dimensional downsampling network, consisting of three three-dimensional convolutional layers. Downsampling in the encoder network is achieved through strided convolution. The potential layer is a three-dimensional network to preserve geometric features; The decoder network is a three-dimensional upsampling network, consisting of three three-dimensional convolutional layers. Upsampling in the decoder network is achieved through deconvolution.

4. The method according to claim 3, characterized in that, The cross-modal features are input into the self-reconstruction network of the 3D variational autoencoder for reconstruction to obtain a complete geometric shape, wherein the complete geometric shape provides global geometric features, including: The cross-modal features are input, and the encoder network encodes the cross-modal features into an implicit space, the implicit space of which follows a normal distribution. The implicit space is sampled in the latent layer to obtain the input of the decoder network; The input is reconstructed using the decoder network to obtain a complete geometry, wherein the complete geometry provides global geometric features.

5. The method according to claim 4, characterized in that, Before reconstructing the input based on the decoder network to obtain the complete geometry, the method further includes: Define the loss function of the three-dimensional variational autoencoder, train the reconstruction process of the decoder network according to the loss function, improve the accuracy of the reconstruction process, obtain the complete geometric shape, and provide global geometric features, wherein the loss function represents the error between the complete geometric shape and the true complete shape of the incomplete shape.

6. The method according to claim 1, characterized in that, After obtaining the predicted symmetry of the incomplete shape, the method further includes: The symmetry error is calculated based on the difference between the predicted symmetry and the true symmetry of the incomplete shape, and the accuracy of the predicted symmetry of the incomplete shape is determined.

7. A system for predicting the symmetry of incomplete shapes based on multimodal feature fusion, characterized in that, The system includes: A multimodal feature fusion module is used to input incomplete RGB images and depth images into a channel-aware network for fusion to obtain cross-modal features of the incomplete shape. The fusion process of the channel-aware network includes: inputting the incomplete RGB image and depth image, establishing two sub-networks to process the modal features of the RGB image and the modal features of the depth image respectively, obtaining color feature maps and depth feature maps; normalizing the color feature maps and the depth feature maps respectively according to the batch normalization layer in the channel-aware network to determine the importance of each channel in the two feature maps; the importance of each channel in the two feature maps is represented by a scaling factor in the batch normalization layer, determining whether the channel needs to be replaced according to the scaling factor, if not, outputting the channel; if yes, sending the channel and the corresponding channel in another feature map into the transform layer in the channel-aware network for fusion to obtain a new channel, replacing the original channel with the new channel, to obtain the cross-modal features that fuse the information of the two feature maps; wherein, the transform layer includes three cross-attention layers, each cross-attention layer is followed by two AddNorm layers and a feedforward layer. The reconstruction module is used to input the cross-modal features into the self-reconstruction network of the three-dimensional variational autoencoder for reconstruction to obtain the complete geometry, wherein the complete geometry provides global geometric features; A symmetry prediction module is used to input the cross-modal features and the global geometric features into a symmetry prediction network for symmetry prediction, thereby obtaining the predicted symmetry of the incomplete shape. The prediction process of the symmetry prediction network includes: aggregating the cross-modal features and the global geometric features to obtain a final feature; inputting the final feature into the symmetry prediction network for symmetry prediction, wherein the symmetry prediction network consists of six three-dimensional convolutional layers; after convolution in each of the three-dimensional convolutional layers, the final feature is activated according to max pooling and activation functions, and the output is the final feature; the final feature is input into a fully connected layer to predict the parameters of the symmetry plane, thereby obtaining the predicted symmetry of the incomplete shape.

Images