A deformation prediction method and system based on multi-modal feature fusion
By combining the characteristics of point cloud data and physical factor data, the problem of deformation accuracy of point cloud data under the influence of multiple factors is solved, and a higher deformation detection accuracy is achieved.
Patent Information
- Application Number
- CN202210813356.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-11
- Publication Date
- 2025-07-25
- Estimated Expiration
- 2042-07-11
AI Technical Summary
When the prior art uses deep learning methods to analyze point cloud data, it is difficult to effectively solve the material deformation problem under the influence of multiple factors, which makes it difficult to ensure the accuracy of the deformation result.
Combining point cloud data with physical factor data for feature fusion, training point cloud feature extraction model and physical factor feature extraction model, obtain point cloud feature vectors and physical factor feature vectors, and perform feature fusion to obtain deformation level.
The accuracy of deformation results has been improved, especially under different degrees of deformation categories, the classification effect has been significantly improved, and the average accuracy and overall accuracy have been improved.
Smart Images
Figure CN115147832B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of deformation detection, and particularly to a deformation prediction method and system based on multi-modal feature fusion. Background Art
[0002] With the development of technology, people have higher and higher requirements for the accuracy of manufacturing materials. Taking aircraft manufacturing as an example, during the manufacturing process of aircraft materials, the materials themselves are flexible and will deform to a certain extent under external forces, affecting the manufacturing accuracy. Therefore, it is necessary to timely detect the deformation problems of the materials. In recent years, with the development of artificial intelligence technology, deep learning has been applied to various fields, and it also shows certain advantages in the analysis of material deformation problems.
[0003] Material data in the manufacturing process is usually obtained by three-dimensional scanning and analyzed and calculated in the form of point cloud data. For the classification problem of point cloud data, nowadays, the method of studying three-dimensional point clouds based on deep learning has become the mainstream trend. Its advantage is that it can automatically extract high-level features from a large amount of input data, and can realize the automatic extraction of point cloud feature information, which can effectively avoid the influence brought by the subjective factors of operators when extracting features by traditional algorithms. In addition, the classification model of deep learning has strong generalization ability and high classification accuracy, and can be widely applied to large and complex three-dimensional scenes. However, for the possible deformation problems of point cloud data, simply analyzing and processing the data usually difficult to achieve good results, because the influencing factors causing deformation may be diverse, and the point cloud modal data can extract the deformation characteristics of the data itself and has strong non-linear solving ability, but its data generalization ability is poor, so it is difficult to guarantee the accuracy of the obtained results. Summary of the Invention
[0004] In view of this, the present invention provides a deformation prediction method and system based on multi-modal feature fusion, which combines point cloud data and physical factor data for feature fusion to obtain a deformation result, and improves the accuracy of the output deformation result by making up for each other's advantages between the point cloud data and the physical factor data.
[0005] Because there is no direct connection between the point cloud data and the physical factor data itself, multi-modal information fusion based on a collaborative architecture can achieve better results. Therefore, we propose a prediction model that can classify the degree of material deformation to achieve the goal of timely detecting the material deformation phenomenon.
[0006] To achieve the above object, the present invention provides the following solutions:
[0007] A deformation prediction method based on multi-modal feature fusion, comprising:
[0008] Obtain a training data set; the training data set includes a training point cloud data set and a training physical factor data set;
[0009] Train a point cloud feature extraction model based on the training point cloud data set to obtain a trained point cloud feature extraction model;
[0010] Train a physical feature extraction model based on the training physical factor data set to obtain a trained physical factor feature extraction model;
[0011] Obtain the point cloud data and physical factor data of the target to be detected;
[0012] Based on the point cloud data and the trained point cloud feature extraction model, obtain the point cloud feature vector of the target to be located;
[0013] Based on the physical factor data and the trained physical feature extraction model, obtain the physical factor feature vector of the target to be located;
[0014] Perform feature fusion on the point cloud feature vector and the physical factor feature vector to obtain the maximum a posteriori probability; obtain the deformation level based on the maximum a posteriori probability.
[0015] Preferably, the training point cloud data set uses 726 three-dimensional data in the Airplane class of the ModelNet40 data set as the initial training point cloud data set; perform deformation processing on the 726 three-dimensional data at degrees of 0.5%, 1%, 2%, 3%, and 5% respectively, that is, adjust the spatial positions of 0.5%, 1%, 2%, 3%, and 5% of the points in the point cloud coordinate system according to downsampling to obtain the training point cloud data set;
[0016] The training physical factor data set includes a number of training physical factor vectors, and each training physical factor vector includes environmental temperature, environmental humidity, assembly duration, and assembly stress.
[0017] Preferably, the point cloud feature extraction model selects the PointNet network, and the point cloud feature extraction model includes a first 3D spatial transformation matrix prediction network T-Net, a first feature extraction network, a second 3D spatial transformation matrix prediction network T-Net, a second feature extraction network, a maximum pooling network, and a fully connected network.
[0018] Preferably, the physical feature extraction model selects the ResDNN network, and the physical feature extraction model includes a first residual network, a second residual network, and a regression network.
[0019] The present invention also provides a deformation prediction system based on multi-modal feature fusion, including:
[0020] A data acquisition module for acquiring a training data set; the training data set includes a training point cloud data set and a training physical factor data set;
[0021] A first training module for training a point cloud feature extraction model based on the training point cloud data set to obtain a trained point cloud feature extraction model;
[0022] A second training module for training a physical feature extraction model based on the training physical factor data set to obtain a trained physical factor feature extraction model;
[0023] A data collection module for acquiring point cloud data and physical factor data of a target to be detected;
[0024] A point cloud feature extraction module for obtaining a point cloud feature vector of a target to be located based on the point cloud data and the trained point cloud feature extraction model;
[0025] A physical feature extraction module for obtaining a physical factor feature vector of a target to be located based on the physical factor data and the trained physical feature extraction model;
[0026] A feature fusion module for fusing the point cloud feature vector and the physical factor feature vector to obtain a maximum a posteriori probability; obtaining a deformation level based on the maximum a posteriori probability.
[0027] Preferably, the training point cloud data set uses 726 three-dimensional data in the Airplane class of the ModelNet40 data set as the initial training point cloud data set; perform deformation processing on the 726 three-dimensional data at 0.5%, 1%, 2%, 3%, and 5% levels respectively, that is, adjust the spatial positions of 0.5%, 1%, 2%, 3%, and 5% of the points in the point cloud coordinate system according to downsampling to obtain the training point cloud data set;
[0028] The training physical factor data set includes a number of training physical factor vectors, and each training physical factor vector includes environmental temperature, environmental humidity, assembly duration, and assembly stress.
[0029] Preferably, the point cloud feature extraction model selects the PointNet network, and the point cloud feature extraction model includes a first 3D spatial transformation matrix prediction network T-Net, a first feature extraction network, a second 3D spatial transformation matrix prediction network T-Net, a second feature extraction network, a maximum pooling network, and a fully connected network.
[0030] Preferably, the physical feature extraction model selects the ResDNN network, and the physical feature extraction model includes a first residual network, a second residual network, and a regression network.
[0031] According to the specific embodiments provided by the present invention, the following technical effects are disclosed by the present invention:
[0032] The present invention relates to the technical field of deformation detection, and particularly to a deformation prediction method and system based on multi-modal feature fusion. The method includes: obtaining a training data set; the training data set includes a training point cloud data set and a training physical factor data set; training a point cloud feature extraction model based on the training point cloud data set to obtain a trained point cloud feature extraction model; training a physical feature extraction model based on the training physical factor data set to obtain a trained physical factor feature extraction model; obtaining the point cloud data and physical factor data of the target to be detected; obtaining the point cloud feature vector of the target to be located based on the point cloud data and the trained point cloud feature extraction model; obtaining the physical factor feature vector of the target to be located based on the physical factor data and the trained physical feature extraction model; performing feature fusion on the point cloud feature vector and the physical factor feature vector to obtain the maximum posterior probability; and obtaining the deformation level based on the maximum posterior probability. The present invention makes the point cloud data and the physical factor data complement each other's advantages, thereby improving the accuracy of the output deformation result. Description of the Drawings
[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.
[0034] Figure 1 It is a flow chart of the deformation prediction method based on multi-modal feature fusion of the present invention;
[0035] Figure 2 It is a structure diagram of the point cloud feature extraction model of the present invention;
[0036] Figure 3 It is a structure diagram of the physical feature extraction model of the present invention;
[0037] Figure 4 It is a schematic diagram of the confusion matrix of the method of the present invention and the PointNet method;
[0038] Figure 5 It is a structure diagram of the deformation prediction system based on multi-modal feature fusion of the present invention.
[0039] Symbol description: 1 - data acquisition module, 2 - first training module, 3 - second training module, 4 - data collection module, 5 - point cloud feature extraction module, 6 - physical feature extraction module, 7 - feature fusion module. Detailed Implementation Manner
[0040] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.
[0041] The purpose of the present invention is to provide a deformation prediction method and system based on multi-modal feature fusion, which combines point cloud data and physical factor data for feature fusion to obtain a deformation result, and improves the accuracy of the output deformation result by making up for each other's advantages between the point cloud data and the physical factor data.
[0042] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.
[0043] Figure 1 This is a flowchart of the deformation prediction method based on multi-modal feature fusion of the present invention. As Figure 1 shown, the present invention provides a deformation prediction method based on multi-modal feature fusion, including:
[0044] Step S1, obtaining a training data set; the training data set includes a training point cloud data set and a training physical factor data set.
[0045] Specifically, the training point cloud data set uses 726 three-dimensional data in the Airplane class in the ModelNet40 data set as the initial training point cloud data set; perform deformation processing on the 726 three-dimensional data at degrees of 0.5%, 1%, 2%, 3%, and 5% respectively, that is, adjust the spatial positions of 0.5%, 1%, 2%, 3%, and 5% of the points in the point cloud coordinate system according to downsampling to obtain the training point cloud data set.
[0046] The training physical factor data set includes a number of training physical factor vectors, and each training physical factor vector includes environmental temperature, environmental humidity, assembly duration, and assembly stress.
[0047] Step S2, training a point cloud feature extraction model based on the training point cloud data set to obtain a trained point cloud feature extraction model.
[0048] As Figure 2 shown, the point cloud feature extraction model includes a first 3D spatial transformation matrix prediction network T-Net, a first feature extraction network, a second 3D spatial transformation matrix prediction network T-Net, a second feature extraction network, a maximum pooling network, and a fully connected network.
[0049] The point cloud feature extraction model extracts point cloud features by performing point cloud transformation, MLP learning, and max pooling on the training point cloud dataset in sequence.
[0050] For each training point cloud data in the training point cloud dataset, which includes n three-dimensional point clouds (n×3), a 3x3 transformation matrix is obtained through the first 3D spatial transformation matrix prediction network T-Net. The transformation matrix is multiplied by the training point cloud data to achieve data alignment, obtaining the first pair of data; the first feature extraction network includes two layers of perceptrons (MultilayerPerceptron, abbreviated as MLP). The first pair of data is used as a single point cloud unit, and feature extraction is performed through the first feature extraction network to obtain the first feature data. Each point cloud extracts 64-dimensional features. The first feature data passes through the second 3D spatial transformation matrix prediction network T-Net to obtain a 64×64 feature transformation matrix. The feature transformation matrix is multiplied by the first feature data to achieve data alignment, obtaining the second feature-aligned data; the second feature extraction network includes three layers of MLP, and the dimensions of the three layers of MLP are 64, 128, and 1024 respectively; the second feature-aligned data passes through the second feature extraction network to obtain the third feature data; the max pooling network performs max pooling on the third feature data to obtain a global feature vector. The fully connected network includes three layers of MLP, and the dimensions of the three layers of MLP are 512, 256, and 6 respectively. The global feature vector passes through the fully connected network to obtain a 6-dimensional feature M pc 。
[0051] The training point cloud dataset is randomly divided into a training set and a validation set according to a ratio of 4:1. The point cloud feature extraction model is trained to obtain a trained point cloud feature extraction model. The Adam optimizer is used in the training process, the cross-entropy is used as the loss function, the initial learning rate is set to 0.002, and it is decreased by 10 times every 30 epochs, and trained for 200 epochs.
[0052] Figure 2 In it, input represents the input, Maxpool represents max pooling, and Global Feature represents the global feature.
[0053] Step S3, based on the training physical factor dataset, the physical feature extraction model is trained to obtain a trained physical factor feature extraction model.
[0054] As Figure 3 shown, the physical feature extraction model includes a first residual network, a second residual network, and a regression network.
[0055] Both the first residual network and the second residual network include three layers of neural networks. Each neural network includes a fully connected layer and a Relu layer.
[0056] Figure 3 where input represents the input, Residual Connection represents the residual network, Fully Connection represents the fully connected layer, Relu represents the Relu function, and M phy represents the output of the physical feature extraction model, which is a 6-dimensional feature.
[0057] The training physical factor dataset is randomly divided into a training set and a validation set in a ratio of 4:1. The Adam optimizer is used during the training process, the cross-entropy is used as the loss function, the initial learning rate is set to 0.0001, and the training is carried out for 30 epochs.
[0058] Step S4: Obtain the point cloud data and physical factor data of the target to be detected.
[0059] Step S5: Based on the point cloud data and the trained point cloud feature extraction model, obtain the point cloud feature vector of the target to be located.
[0060] Step S6: Based on the physical factor data and the trained physical feature extraction model, obtain the physical factor feature vector of the target to be located.
[0061] Step S7: Perform feature fusion on the point cloud feature vector and the physical factor feature vector to obtain the maximum a posteriori probability; obtain the deformation level based on the maximum a posteriori probability.
[0062] Specifically, use the logSoftmax function to perform feature fusion based on Bayesian decision-making for M pc and M Phy , calculate the maximum class of the posterior probability in different categories for M pc and M Phy , introduce the assumption that the prior probability and the posterior probability are approximately equal to form a joint conditional classification for M pc and M Phy . The logSoftmax has better numerical stability compared to the Softmax function, and the feature fusion formula is:
[0063]
[0064] where P is the maximum a posteriori probability, M is the number of classifiers, ω k represents the k-th category, P(ω k ) represents the prior probability of the k-th class. λ1 represents the weight of the point cloud feature vector, λ2 represents the weight of the physical factor feature vector, and max is to take the maximum value.
[0065] Figure 5 This is the structural diagram of the deformation prediction system based on multi-modal feature fusion of the present invention. As Figure 5 shown, the present invention provides a deformation prediction system based on multi-modal feature fusion, including: a data acquisition module 1, a first training module 2, a second training module 3, a data collection module 4, a point cloud feature extraction module 5, a physical feature extraction module 6, and a feature fusion module 7.
[0066] The data acquisition module 1 is used to acquire a training data set; the training data set includes a training point cloud data set and a training physical factor data set.
[0067] The first training module 2 is used to train a point cloud feature extraction model based on the training point cloud data set to obtain a trained point cloud feature extraction model.
[0068] The second training module 3 is used to train a physical feature extraction model based on the training physical factor data set to obtain a trained physical factor feature extraction model.
[0069] The data collection module 4 is used to acquire the point cloud data and physical factor data of the target to be detected.
[0070] The point cloud feature extraction module 5 is used to obtain the point cloud feature vector of the target to be located based on the point cloud data and the trained point cloud feature extraction model.
[0071] The physical feature extraction module 6 is used to obtain the physical factor feature vector of the target to be located based on the physical factor data and the trained physical feature extraction model.
[0072] The feature fusion module 7 is used to perform feature fusion on the point cloud feature vector and the physical factor feature vector to obtain the maximum a posteriori probability; and obtain the deformation level based on the maximum a posteriori probability.
[0073] Optionally, the training point cloud data set uses 726 three-dimensional data in the Airplane class of the ModelNet40 data set as the initial training point cloud data set; perform deformation processing on the 726 three-dimensional data at degrees of 0.5%, 1%, 2%, 3%, and 5% respectively, that is, adjust the spatial positions of 0.5%, 1%, 2%, 3%, and 5% of the points in the point cloud coordinate system according to downsampling respectively to obtain the training point cloud data set;
[0074] The training physical factor data set includes a number of training physical factor vectors, and each training physical factor vector includes environmental temperature, environmental humidity, assembly duration, and assembly stress.
[0075] Optionally, the point cloud feature extraction model selects the PointNet network, and the point cloud feature extraction model includes a first 3D spatial transformation matrix prediction network T-Net, a first feature extraction network, a second 3D spatial transformation matrix prediction network T-Net, a second feature extraction network, a maximum pooling network, and a fully connected network.
[0076] Optionally, the physical feature extraction model selects the ResDNN network, and the physical feature extraction model includes a first residual network, a second residual network, and a regression network.
[0077] The following uses specific examples to illustrate the effects of the present invention.
[0078] The method of the present invention is compared with existing mainstream methods, and the classification accuracy is selected as the evaluation index. The obtained results are shown in Table 1.
[0079] Table 1 Classification accuracy of different methods
[0080]
[0081] The first column in Table 1 represents different methods. The first row represents different categories, which are deformation amounts of 0%, 0.5%, 1%, 2%, 3%, and 5% respectively. It can be seen from Table 1 that the present invention can achieve better classification effects under different degrees of deformation categories. Among them, the optimal effect in the current mainstream methods is still improved by 10.7% in the 0.5% Def category. Def represents the sampling rate, that is, the deformation amount.
[0082] To more intuitively evaluate the accuracy of the present invention, we selected two evaluation indexes, mean accuracy (MA) and overall accuracy (OA), to analyze the classification effect. The results are shown in Table 2.
[0083] Table 2 Mean accuracy and overall accuracy of different methods
[0084]
[0085] It can be seen from Table 2 that the MA of the present invention reaches 80.98%, and the OA reaches 84.23%, both achieving better classification accuracy. The confusion matrix of the present invention and PointNet in deformation classification is as Figure 4 shown, Figure 4 in which Target represents the index, that is, the deformation amount.
[0086] To evaluate the influence degree of each physical factor on deformation, an ablation experiment was conducted. In the training physical factor dataset, each physical factor was grouped and experiments were carried out separately. According to the experimental results, the weight of the influence of different physical factors on deformation was evaluated. The experimental results are shown in Table 3.
[0087]
[0088] In Table 3, T represents the environmental temperature in the physical factors, H represents the environmental humidity in the physical factors, M d represents the assembly duration in the physical factors, and M s represents the assembly stress in the physical factors. It can be seen from the data in Table 3 that when the environmental temperature factor was not added to the physical factor deformation dataset, the average accuracy was 79.75%. When the environmental humidity factor was not added, the average accuracy was 80.15%. When the assembly duration factor was not added, the average accuracy was 76.83%. When the assembly stress factor was not added, the average accuracy was 74.64%. Therefore, we can conclude that the assembly stress factor is the physical factor that has the greatest impact on deformation classification, while the environmental humidity has the least impact on deformation classification.
[0089] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.
[0090] Specific examples are used in this article to elaborate on the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention. At the same time, for those of ordinary skill in the art, based on the idea of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.
Claims
1. A deformation prediction method based on multi-modal feature fusion, characterized in that Including: Obtain a training data set; the training data set includes a training point cloud data set and a training physical factor data set; the training physical factor data set includes a number of training physical factor vectors, and each training physical factor vector includes environmental temperature, environmental humidity, assembly duration, and assembly stress; Train a point cloud feature extraction model based on the training point cloud data set to obtain a trained point cloud feature extraction model; Train a physical feature extraction model based on the training physical factor data set to obtain a trained physical factor feature extraction model; Obtain the point cloud data and physical factor data of the target to be detected; Based on the point cloud data and the trained point cloud feature extraction model, obtain the point cloud feature vector of the target to be located; Based on the physical factor data and the trained physical feature extraction model, obtain the physical factor feature vector of the target to be located; Fuse the point cloud feature vector and the physical factor feature vector to obtain the maximum a posteriori probability; obtain the deformation level based on the maximum a posteriori probability.
2. The deformation prediction method based on multi-modal feature fusion according to claim 1, wherein The training point cloud data set uses 726 three-dimensional data in the Airplane class of the ModelNet40 data set as the initial training point cloud data set; perform deformation processing on the 726 three-dimensional data at degrees of 0.5%, 1%, 2%, 3%, and 5% respectively, that is, adjust the spatial positions of 0.5%, 1%, 2%, 3%, and 5% of the points in the point cloud coordinate system according to downsampling to obtain the training point cloud data set.
3. The deformation prediction method based on multi-modal feature fusion according to claim 1, wherein The point cloud feature extraction model selects the PointNet network, and the point cloud feature extraction model includes a first 3D spatial transformation matrix prediction network T-Net, a first feature extraction network, a second 3D spatial transformation matrix prediction network T-Net, a second feature extraction network, a maximum pooling network, and a fully connected network.
4. The deformation prediction method based on multi-modal feature fusion according to claim 1, wherein The physical feature extraction model selects the ResDNN network, and the physical feature extraction model includes a first residual network, a second residual network, and a regression network.
5. A deformation prediction system based on multi-modal feature fusion, characterized in that, Including: A data acquisition module for obtaining a training data set; the training data set includes a training point cloud data set and a training physical factor data set; the training physical factor data set includes a number of training physical factor vectors, and each training physical factor vector includes environmental temperature, environmental humidity, assembly duration, and assembly stress; A first training module for training a point cloud feature extraction model based on the training point cloud data set to obtain a trained point cloud feature extraction model; A second training module for training a physical feature extraction model based on the training physical factor data set to obtain a trained physical factor feature extraction model; A data collection module for obtaining the point cloud data and physical factor data of the target to be detected; A point cloud feature extraction module for obtaining the point cloud feature vector of the target to be located based on the point cloud data and the trained point cloud feature extraction model; A physical feature extraction module for obtaining the physical factor feature vector of the target to be located based on the physical factor data and the trained physical feature extraction model; A feature fusion module is used to perform feature fusion on the point cloud feature vector and the physical factor feature vector to obtain the maximum a posteriori probability; and a deformation level is obtained based on the maximum a posteriori probability.
6. The deformation prediction system based on multi-modal feature fusion according to claim 5, wherein, The training point cloud dataset uses 726 three-dimensional data in the Airplane class of the ModelNet40 dataset as the initial training point cloud dataset; 726 three-dimensional data are respectively subjected to deformation processing at degrees of 0.5%, 1%, 2%, 3% and 5%, that is, according to downsampling, 0.5%, 1%, 2%, 3% and 5% of the points are respectively selected to adjust the spatial positions in the point cloud coordinate system to obtain the training point cloud dataset.
7. The deformation prediction system based on multi-modal feature fusion according to claim 5, characterized in that The point cloud feature extraction model selects the PointNet network, and the point cloud feature extraction model includes a first 3D spatial transformation matrix prediction network T-Net, a first feature extraction network, a second 3D spatial transformation matrix prediction network T-Net, a second feature extraction network, a maximum pooling network and a fully connected network.
8. The deformation prediction system based on multi-modal feature fusion according to claim 5, characterized in that The physical feature extraction model selects the ResDNN network, and the physical feature extraction model includes a first residual network, a second residual network and a regression network.