Body position classification method and system based on 3D point cloud data
By using technical means such as domain sampling, feature alignment, feature extraction, attention weighting and position classification in deep neural networks, the problem of difficult to automatically classify position in the existing technology is solved, and efficient position recognition of 3D point cloud data is achieved.
Patent Information
- Application Number
- CN202411931027.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-26
- Publication Date
- 2025-05-02
AI Technical Summary
The prior art is difficult to directly perform automatic classification of positions with high quality through end-to-end deep neural networks, especially in medically assisted diagnosis based on 3D point cloud data.
The position classification method based on 3D point cloud data is adopted to construct a deep neural network model to identify position through the steps of domain sampling, feature alignment, feature extraction, attention weighting and position classification.
It realizes the direct use of 3D point cloud data, and the current position of the 3D point cloud data can be identified after a small amount of processing, improving the accuracy and efficiency of position classification.
Smart Images
Figure CN119919715A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of point cloud processing and deep learning technology, and in particular to a body position classification method and system based on 3D point cloud data. Background Art
[0002] Optical imaging is one of the most commonly used imaging techniques in preclinical research. It is a technology that uses optical devices or equipment to form images of information such as the shape and color of an object through the reflection or refraction of light. Due to its advantages such as non-invasiveness, no radiation, and high resolution, it has been widely used in medical diagnosis and research.
[0003] The results of optical imaging are saved as 3D point cloud data, which can usually be further analyzed, such as internal organ reconstruction, surface tissue segmentation, etc. However, body position will have a certain impact on the reconstruction results. For example, the heart is usually located to the left of the center of the chest, and the stomach is usually located in the upper left of the abdominal cavity.
[0004] In recent years, deep learning has shown great promise in various aspects of medical assisted diagnosis. However, there is currently no suitable end-to-end deep neural network that can directly perform automatic classification of body positions with high quality. Summary of the invention
[0005] In order to overcome the above-mentioned defects in the prior art, the present invention provides a body position classification system based on 3D point cloud data, which can directly use the 3D point cloud data and identify the current body position of the 3D point cloud data after a small amount of processing.
[0006] To achieve the above object, the present invention adopts the following technical solutions, including:
[0007] The body position classification method based on 3D point cloud data includes:
[0008] Perform domain sampling on the target point cloud data to obtain domain-level feature data;
[0009] Perform feature alignment on the domain-level feature data to obtain aligned feature data;
[0010] Perform feature extraction on the aligned data to obtain high-level feature data;
[0011] Perform attention weighting on high-level feature data to obtain weighted feature data;
[0012] Body position classification is performed based on the weighted feature data, and the predicted body position of the target is output.
[0013] Preferably, the specific method of field sampling is as follows:
[0014] First, the point cloud is divided into N fields, each of which contains K field points; then the multi-layer perceptron MLP is used to extract features of each point in the field, and then the symmetric function Max is used to aggregate features of each point in the field to obtain feature data of each field, thereby obtaining field-level feature data.
[0015] Preferably, the specific method of the feature alignment is as follows:
[0016] The input data is processed through the T-Net network to obtain the affine transformation matrix; the affine transformation matrix is multiplied by the input data to obtain the aligned feature data.
[0017] Preferably, a multi-layer perceptron MLP is used for feature extraction, and the calculation method of the multi-layer perceptron MLP is:
[0018] Ft=f(w*X T +B);
[0019] Where X is the input data of the multilayer perceptron MLP; Out is the output data of the multilayer perceptron MLP; f(·) is the activation function; B is the bias; w is the weight; superscript T Indicates transpose.
[0020] Preferably, the specific method of attention weighting is as follows:
[0021] The high-level feature data is averaged and pooled on the channel, i.e. compressed, to obtain the channel average value of each field;
[0022] According to the channel average of each field, the attention between fields is obtained;
[0023] Calculate the attention between fields and get the weight of each field;
[0024] The high-level feature data is weighted according to the weight of each field to obtain weighted feature data.
[0025] Preferably, the specific method of body position classification is as follows:
[0026] Perform maximum pooling on the weighted feature data to obtain global aggregated feature data;
[0027] The global aggregate feature data is sequentially subjected to dimensionality reduction processing to obtain the confidence of each body position category;
[0028] The confidence of each body position category is normalized, and the body position category corresponding to the maximum confidence is selected as the predicted body position of the target.
[0029] The present invention also provides a body position classification system based on 3D point cloud data, characterized in that it is applicable to the above-mentioned body position classification method based on 3D point cloud data, and the system adopts a body position classification model; the body position classification model includes a domain sampling module, a feature alignment module, a feature extraction module, a point attention module and a feature classification module;
[0030] The domain sampling module performs domain sampling on the point cloud data of the target to obtain domain-level feature data;
[0031] The feature alignment module includes at least two feature alignment branches, at least feature alignment branch 1 and feature alignment branch 2; the input data of feature alignment branch 1 is the domain-level feature data output by the domain sampling module, and the input data of feature alignment branch 2 is the weighted feature data output by point attention branch 1; the feature alignment branch is used to perform feature alignment on the input data to obtain aligned feature data;
[0032] The feature extraction module includes at least two feature extraction branches, at least feature extraction branch one and feature extraction branch two; the input data of feature extraction branch one is the aligned feature data output by feature alignment branch one, and the input data of feature extraction branch two is the aligned feature data output by feature alignment branch two; the feature extraction branch is used to extract features from the input data to obtain high-level feature data;
[0033] The point attention module includes at least two point attention branches, at least one point attention branch and one point attention branch; the input data of the point attention branch one is the high-level feature data output by the feature extraction branch one, and the input data of the point attention branch two is the high-level feature data output by the feature extraction branch two; the point attention branch one is used to perform attention weighting on the input data to obtain weighted feature data; the weighted feature data output by the point attention branch two is input into the feature classification module;
[0034] The feature classification module is used to perform body position classification based on the weighted feature data output by the point attention branch 2, and output the predicted body position of the target.
[0035] Preferably, the feature extraction branch one includes at least two multilayer perceptrons MLP connected in sequence; the feature extraction branch two includes at least three multilayer perceptrons MLP connected in sequence; the weights of the multilayer perceptrons MLP of the two feature extraction branches are shared; the number of channels of the multilayer perceptrons MLP of the feature extraction branch one is the same; the number of channels of the multilayer perceptrons MLP of the feature extraction branch two increases in sequence.
[0036] Preferably, the point attention branch includes an average pooling layer AvgPooling, a multi-layer perceptron MLP and a weighted layer, specifically; wherein,
[0037] The average pooling layer AvgPooling is used to compress the high-level feature data output by the feature extraction branch on the channel and calculate the channel average value of each field;
[0038] Use multi-layer perceptron MLP to calculate the attention between fields based on the channel average of each field;
[0039] Use multi-layer perceptron MLP to calculate the weight of each field based on the attention between fields;
[0040] The weighted layer is used to weight the high-level feature data according to the weight of each field to obtain the weighted feature data.
[0041] Preferably, the feature classification module includes a maximum pooling layer, multiple multi-layer perceptrons (MLPs) and an output layer. The specific processing method is as follows:
[0042] The Max Pooling layer is used to perform maximum pooling processing on the weighted feature data output by the second point attention branch to obtain global aggregated feature data;
[0043] Multiple multi-layer perceptrons (MLPs) perform dimensionality reduction on the global aggregate feature data in turn to obtain the confidence level of each body position category;
[0044] The output layer first normalizes the confidence of each body position category through the SoftMax function, and then selects the body position category corresponding to the maximum confidence through the ArgMax function as the predicted body position of the target.
[0045] The advantages of the present invention are:
[0046] (1) The present invention provides a body position classification method based on 3D point cloud data, which can directly utilize 3D point cloud data and identify the current body position of the 3D point cloud data after a small amount of processing.
[0047] (2) For a classification task, the model pays more attention to the overall and larger features than to the features of a single data point. Directly extracting features from a single data point will result in a small receptive field of the model, which is not conducive to classification. Therefore, the present invention performs domain sampling on the point cloud data of the target to obtain domain-level feature data, thereby increasing the receptive field of the model and being more conducive to classification.
[0048] (3) As a geometric data, point cloud data should not change the category of each point no matter what kind of rigid body transformation is applied, such as translation and rotation of all points. In order to keep the features of data unchanged under different geometric transformations, the network can adapt to the geometric transformation of features. Therefore, the present invention performs feature alignment on the input data, aligns the input data, and converts it into a fixed angle, thereby achieving angle standardization.
[0049] (4) In order to better extract information from point cloud data, the present invention performs feature extraction to obtain high-level feature data. The number of output channels directly determines the model's ability to express features. A higher number of channels can capture more complex local patterns and improve sensitivity to subtle differences in point cloud data.
[0050] (5) The present invention can weight different data points according to their importance, pay attention to different points, and make important points obtain more weight. In the process of attention calculation, since the channel average is calculated and the weight is obtained according to the channel average, the permutation invariance of the data will not be destroyed.
[0051] (6) The points in the point cloud are not as regular as the grid in the image. No matter how the order of the points is changed, the structure of the point cloud and its visualization result are the same, that is, the point cloud data (unordered point set) has permutation invariance. Similarly, for classification tasks, no matter how the order of the points is changed, the model needs to output the same result. Therefore, in order to process the unordered point set in the point cloud data, the arrangement order of different points does not affect the output of the model result. It is necessary to use a symmetric function (specifically, using the maximum pooling layer Max Pooling) to process the features extracted above. No matter how the features are arranged, the output results after Max Pooling processing should be the same, thereby eliminating the influence of data disorder. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 Schematic diagram of the body position classification model.
[0053] Figure 2 Schematic diagram of the feature alignment module.
[0054] Figure 3 Schematic diagram of the feature extraction module.
[0055] Figure 4 Schematic diagram of the point attention module.
[0056] Figure 5 Schematic diagram of the feature classification module. DETAILED DESCRIPTION
[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0058] Example 1
[0059] The body position classification method based on 3D point cloud data includes the following steps:
[0060] Step 1: Perform domain sampling on the target point cloud data to obtain domain-level feature data.
[0061] The specific method of field sampling is:
[0062] First, the point cloud is divided into N fields, each of which contains K field points; then the multi-layer perceptron MLP is used to extract features of each point in the field, and then the symmetric function Max is used to aggregate features of each point in the field to obtain feature data of each field, thereby obtaining field-level feature data.
[0063] Step 2: Perform feature alignment on the domain-level feature data to obtain aligned feature data.
[0064] The specific method of feature alignment is:
[0065] The input data is processed through the T-Net network to obtain the affine transformation matrix; the affine transformation matrix is multiplied by the input data to obtain the aligned feature data.
[0066] Step 3: Extract features from the aligned data to obtain high-level feature data.
[0067] Multilayer perceptron MLP is used for feature extraction. The calculation method of multilayer perceptron MLP is:
[0068] Ft=f(w*X T +B);
[0069] Where X is the input data of the multilayer perceptron MLP; Out is the output data of the multilayer perceptron MLP; f(·) is the activation function; B is the bias; w is the weight; superscript T Indicates transpose.
[0070] Step 4: Perform attention weighting on the high-level feature data to obtain weighted feature data.
[0071] The specific method of attention weighting is:
[0072] The high-level feature data is averaged and pooled on the channel, i.e. compressed, to obtain the channel average value of each field;
[0073] According to the channel average of each field, the attention between fields is obtained;
[0074] Calculate the attention between fields and get the weight of each field;
[0075] The high-level feature data is weighted according to the weight of each field to obtain weighted feature data.
[0076] Step 5: Perform body position classification based on the weighted feature data and output the predicted body position of the target.
[0077] The specific classification method of body position is as follows:
[0078] Perform maximum pooling on the weighted feature data to obtain global aggregated feature data;
[0079] The global aggregate feature data is sequentially subjected to dimensionality reduction processing to obtain the confidence of each body position category;
[0080] The confidence of each body position category is normalized, and the body position category corresponding to the maximum confidence is selected as the predicted body position of the target.
[0081] Example 2
[0082] A body position classification system based on 3D point cloud data, including a body position classification model.
[0083] Depend on Figure 1 As shown, the body posture classification model adopts a deep neural network, and the body posture classification model includes a domain sampling module, a feature alignment module, a feature extraction module, a point attention module and a feature classification module.
[0084] (1) Domain Sampling Module
[0085] For a classification task, the model pays more attention to overall and larger features than to the features of a single data point. Directly extracting features from a single data point will result in a model with a small receptive field, which is not conducive to classification. Therefore, after obtaining M data points (point cloud data), the domain sampling module will further randomly sample N data points as the center point of the domain, and calculate the distance from the data point to the center point with a certain range R as the radius, and select K points as the domain points of the center point to extract local features in the domain. If the points in the domain are too sparse and less than K points, the domain sampling module will also resample to generate K domain points.
[0086] After performing domain division and domain point extraction, the domain sampling module first uses the multi-layer perceptron MLP to extract features from each point in the domain, and then uses the symmetric function Max to aggregate features of each point in the domain to obtain feature data of each domain, thereby obtaining domain-level feature data, denoted as F1∈R N×C2 , N is the number of fields; C2 is the number of channels.
[0087] Among them, the dimension of the multilayer perceptron MLP input data is N×K×C1. After feature extraction, the dimension of the multilayer perceptron MLP output data is N×K×C2. After using the symmetric function Max for feature aggregation, the dimension of the symmetric function Max output data is N×C2; wherein N is the number of center points, that is, the number of domains. In this embodiment, N=1000, K is the number of points in each domain, and C1 and C2 are both the number of channels.
[0088] (2) Feature Alignment Module
[0089] The feature alignment module includes two feature alignment branches with the same network structure, namely feature alignment branch 1 and feature alignment branch 2. The input data of feature alignment branch 1 is the domain-level feature data output by the domain sampling module, and the input data of feature alignment branch 2 is the weighted feature data output by point attention branch 1. The feature alignment branch is used to align the input data to obtain aligned feature data.
[0090] The 3D point cloud data of the target is a geometric data. No matter what kind of rigid body transformation is applied, such as translation and rotation of all points, the category of each point should not be changed. In order to keep the data features unchanged under different geometric transformations, the network can adapt to the geometric transformation of the features.
[0091] Depend on Figure 2 As shown in the figure, the processing methods of the two feature alignment branches are the same: first, the input data (domain-level feature data output by the domain sampling module or the weighted feature data output by the point attention branch 1) is input into the T-Net network to obtain a 3×3 affine transformation matrix; then the affine transformation matrix is multiplied by the input data to obtain the angle-normalized data, i.e., the aligned feature data.
[0092] This method can be used to align the input data and convert it into a fixed angle. This method, combined with the data enhancement method during model training, can also provide more different data for T-Net network learning.
[0093] Affine transformation is a spatial transformation that can achieve data rotation through matrix multiplication in order to perform angle standardization.
[0094] The feature alignment branch performs angle normalization through affine transformation, such as rotating by angle θ around the z-axis. The affine transformation matrix M is:
[0095]
[0096] For a point with coordinates [x,y,z] T , the output after affine transformation is [x',y',z'] T =M*[x,y,z] T , where the superscript T In particular, the affine transformation matrix is learned by the network itself, which can realize rotations of different angles, thereby achieving angle standardization.
[0097] (3) Feature extraction module
[0098] The feature extraction module includes two feature extraction branches, namely feature extraction branch 1 and feature extraction branch 2. The input data of feature extraction branch 1 is the aligned feature data output by feature alignment branch 1, and the input data of feature extraction branch 2 is the aligned feature data output by feature alignment branch 2. The feature extraction branch is used to extract features from the input data to obtain high-level feature data.
[0099] like Figure 3 As shown in the figure, in order to better extract the information of the target 3D point cloud data, two feature extraction branches are constructed using multi-layer perceptron MLP. Among them, feature extraction branch one includes two multi-layer perceptron MLPs connected in sequence, and the number of channels of the two multi-layer perceptron MLPs of feature extraction branch one is C1; feature extraction branch two includes three multi-layer perceptron MLPs connected in sequence, and the number of channels of the three multi-layer perceptron MLPs of feature extraction branch two is C1, C2, and C3 respectively. The weights of the multi-layer perceptron MLPs of the two feature extraction branches are shared. The number of output channels directly determines the model's ability to express features. A higher number of channels can capture more complex local patterns and improve sensitivity to subtle differences in point cloud data.
[0100] Among them, the calculation method of the multi-layer perceptron MLP is:
[0101] Ft=f(w*X T +B);
[0102] Among them, Ft is the output of the multilayer perceptron MLP; X is the input of the multilayer perceptron MLP, with a dimension of N×C1, f(·) is the activation function, specifically the Relu activation function; B is the bias, with a dimension of C2×1; w is the weight, with a dimension of C2×C1.
[0103] The aligned feature data output by the feature alignment branch 1 is extracted by two multi-layer perceptrons MLP of the feature extraction branch 1 to obtain high-level feature data, which is recorded as F3. 1 ∈R N×C1 , C1 is the number of channels. In this embodiment, the number of channels of the two multi-layer perceptrons MLP of the feature extraction branch 1 is 64, that is, C1 is 64.
[0104] The aligned feature data output by the feature alignment branch 2 is extracted by three multi-layer perceptrons MLP in the feature extraction branch 2 to obtain high-level feature data, which is recorded as F3. 2 ∈R N×C3 , C3 is the number of channels. In this embodiment, the number of channels of the three multi-layer perceptrons MLP of the feature extraction branch 2 are 64, 128, and 1024 respectively, that is, C3 is 1024.
[0105] (4) Point Attention Module
[0106] The entire data processing process of the present invention is random. For classification, different data points have different importance. In order to make the model weighted according to the importance of different data points, a point attention module is added. Similar to the spatial attention on the image, the point attention module can pay attention to the importance of different points (spatial positions) so that important points can obtain more weight.
[0107] The point attention module includes two point attention branches, namely point attention branch 1 and point attention branch 2. The input data of point attention branch 1 is the high-level feature data output by feature extraction branch 1, and the input data of point attention branch 2 is the high-level feature data output by feature extraction branch 2. Point attention branch 1 is used to perform attention weighting on the input data to obtain weighted feature data. The weighted feature data output by point attention branch 2 is input into the feature classification module.
[0108] like Figure 4 As shown, the attention branch 1 or 2 is used for the high-level feature data (F3 1 ∈R N×C1 or F3 2 ∈R N×C3 ), firstly, perform a Squeeze operation on the channel to obtain the channel average value of each field and obtain the field feature; then, according to the proportion of the field feature in all field features, obtain the weighted weight; finally, extract the high-level feature data (F3 1 ∈R N×C1 or F3 2 ∈R N×C3) is weighted to obtain the final data after point attention calculation, that is, the weighted feature data. Since the channel average is calculated and the weight is obtained based on the channel average, the permutation invariance of the data will not be destroyed.
[0109] The point attention branch includes an average pooling layer AvgPooling, two multi-layer perceptrons MLP and a weighted layer. The specific processing process is as follows:
[0110] First, use the average pooling layer AvgPooling to extract the high-level feature data (F3 1 ∈R N×C1 or F3 2 ∈R N×C3 ) compresses the channels, calculates the channel average of each field, and obtains the field features. The output data of the average pooling layer AvgPooling is denoted as G, with a dimension of N×1.
[0111] Then, the first multi-layer perceptron, MLP1, is used to calculate the attention between the fields, and the second multi-layer perceptron, MLP2, is used to calculate the importance of each field, i.e., the weight, as follows:
[0112] s=f(W1*G+B1);
[0113] W = SoftMax(W2*s+B2);
[0114] Among them, W1 and W2 are the weights of multilayer perceptrons MLP1 and MPL2 respectively; B1 and B2 are the biases of multilayer perceptrons MLP1 and MPL2 respectively; f is the activation function of the first multilayer perceptron MLP1, specifically the Relu activation function; s is the output result of the first multilayer perceptron MLP1; SofMax is the activation function of the second multilayer perceptron MLP2; W is the output result of the second multilayer perceptron MLP2, that is, the weight of each field.
[0115] Finally, the weighted layer is used to extract the high-level feature data (F3) output by branch one or branch two according to the weight of each field. 1 ∈R N×C1 or F3 2 ∈R N×C3 ) to perform weighted calculation, and the weighted feature data is output as F4 1 =W*F3 1 ∈R N×C1 or F4 2 =W*F3 2 ∈R N×C1 .
[0116] Among them, the weighted feature data of the point attention branch 1 is F41 ∈R N×C1 The weighted feature data obtained by attention branch 2 is F4 2 ∈R N×C3 .
[0117] (5) Feature classification module
[0118] The points in the point cloud are not as regular as the grid in the image. No matter how the order of the points is changed, the structure of the point cloud and its visualization result are the same, that is, the point set of the point cloud data has permutation invariance. Similarly, for tasks such as classification, no matter how the order of the points is changed, the network model needs to output the same result.
[0119] In order to process the unordered point set in the 3D point cloud data, a symmetric function is needed to process the features extracted above so that the arrangement order of different points does not affect the output of the model results.
[0120] like Figure 5 As shown, the feature classification module first processes the input data (the weighted feature data F4 output by the attention branch 2) 2 ∈R N×C3 ) is processed using the maximum pooling layer Max Pooling. Regardless of the order of the features, the output results after MaxPooling processing should be the same, thereby eliminating the influence of data disorder. At this time, the features processed by Max Pooling are the global aggregate features of the features of all points in each dimension. After extracting the global aggregate feature data, in order to fuse the information between different features, multiple multi-layer perceptrons MLP are used to perform dimensionality reduction processing on the global aggregate feature data in turn to obtain the confidence of each body position category. The confidence of each body position category is then normalized using the SoftMax function, and the body position category corresponding to the maximum confidence is selected using the ArgMax function, and the predicted body position of the target is finally output.
[0121] The specific processing method of the feature classification module is as follows:
[0122] Out=ArgMax(SoftMax(MLP(MaxPooling(X))))
[0123] For input data F4 2 ∈R N×C3Using MaxPooling on dimension N, a vector of dimension C3 can be obtained. Then, multiple multi-layer perceptrons (MLPs) are used to reduce the dimension in turn, and the output dimension is gradually reduced to 512, 256, and k, where k is the number of body position categories to be classified. The confidence of the classification is normalized by the SoftMax function, and then the index corresponding to the maximum confidence is taken by the ArgMax function. By comparing the corresponding relationship between the specified index and the body position category, the classification result can be obtained, and the predicted body position of the target is finally output.
[0124] During the training process of the posture classification model, cross entropy is used as the loss function, the Adam optimization algorithm is adopted, the initial learning rate is set to 0.001, and the learning rate adjustment strategy is step decay.
[0125] In this embodiment, mice are used as the research object (target), and the body positions of mice are classified. During the training process of the body position classification model, a mouse optical contour data set with body position labels can be obtained from any appropriate source, such as a research institution, a public data set, etc., to construct a sample set. In this embodiment, two body positions, abdomen facing outward and back facing outward, are used as the gold standard for the required classification. The sample set is divided into a training set, a validation set, and a test set according to an appropriate ratio, such as 5:2:3. The gold standard refers to the authoritative result obtained by manual annotation of the data.
[0126] In order to make the reasoning results of the posture classification model more robust and adaptable to postures of different sizes and slight displacements, the present invention applies a variety of data enhancement methods, including: randomly scaling the point cloud data with its center of mass as the center point at different proportions, performing certain rotations at small angles in the sagittal and horizontal planes, and performing random translation operations.
[0127] Since the program runs very fast and the model size is very small, in order to obtain higher robustness during model testing, the original point cloud data of the target optical profile (far more than the required 5,000 points) can be sampled multiple times and combined into a batch for simultaneous reasoning. Finally, the final prediction result of the data is voted in a similar way to Bagging for ensemble learning.
[0128] The body posture classification system of the above-mentioned embodiment 2 may further include a preprocessing module.
[0129] The preprocessing module is used for the optical contour image data of the target, extracts the point cloud of the optical contour and the Mesh (triangular patch) data constructed by the point cloud through triangles, and then calculates the normal vector of the Mesh data. The normal vector of each point cloud can be calculated based on the normal vector of the Mesh data to obtain the 3D point cloud data of the target, denoted as x, where x is an M×6 matrix, M is the number of points, and 6 is the number of channels, including the three-dimensional coordinates of the point cloud and the three-dimensional normal vector.
[0130] In order to speed up the reasoning of the subsequent posture classification model, this embodiment randomly samples 5000 points (about 15% of the data points) as the input of the posture classification model, that is, M=5000 in this embodiment, while ensuring that the optical contour of the target is not affected, the input data is reduced as much as possible.
[0131] Example 3
[0132] Using the body position classification system of the above-mentioned embodiment 2, the body position classification method based on 3D point cloud data specifically includes the following steps:
[0133] S1, use the preprocessing module to preprocess the optical contour image data of the target and extract the point cloud data of the target; or directly obtain the point cloud data of the target. The point cloud data is recorded as x∈R M×6 , M is the number of points, 6 is the number of channels, and it is composed of the three-dimensional coordinates and three-dimensional normal vectors of the point cloud.
[0134] S2, the domain sampling module first divides the point cloud into N domains, each of which contains K domain points; then uses the multi-layer perceptron MLP to extract features from each point in the domain, and then uses the symmetric function Max to aggregate features of each point in the domain to obtain feature data of each domain, thereby obtaining domain-level feature data, denoted as F1∈R N×C2 , N is the number of fields; C2 is the number of channels.
[0135] S3, feature alignment branch 1 inputs the domain-level feature data output by the domain sampling module into the T-Net network to obtain the affine transformation matrix, and then multiplies the affine transformation matrix with the domain-level feature data to obtain the aligned feature data, denoted as F2 1 ∈R N×C1 , C1 is the number of channels.
[0136] Feature alignment branch 2 inputs the weighted feature data output by the point attention branch 1 into the T-Net network to obtain the affine transformation matrix, and then multiplies the affine transformation matrix with the weighted feature data to obtain the aligned feature data, denoted as F2 2 ∈R N ×C1 , C1 is the number of channels.
[0137] S4, feature extraction branch 1 includes two multi-layer perceptrons MLP, the number of channels of each multi-layer perceptron MLP is C1, the aligned feature data output by feature alignment branch 1 is extracted by the two multi-layer perceptrons MLP of feature extraction branch 1, and high-level feature data is obtained, which is recorded as F3 1 ∈R N×C1 , C1 is the number of channels.
[0138] The feature extraction branch 2 includes three multi-layer perceptrons (MLPs). The number of channels of the three multi-layer perceptrons (MLPs) are C1, C2, and C3, respectively. The aligned feature data output by the feature alignment branch 2 is extracted by the three multi-layer perceptrons (MLPs) of the feature extraction branch 2 to obtain high-level feature data, which is recorded as F3. 2 ∈R N×C3 , C3 is the number of channels.
[0139] S5, the attention branch 1 first compresses the high-level feature data output by the feature extraction branch 1 on the channel, and calculates the channel average of each field; then calculates the attention between fields based on the channel average of each field; then calculates the weight of each field based on the attention between fields; finally, the high-level feature data output by the feature extraction branch 1 is weighted according to the weight of each field to obtain the weighted feature data, which is recorded as F4 1 ∈R N×C1 , C1 is the number of channels.
[0140] The second attention branch first compresses the high-level feature data output by the feature extraction branch 2 on the channel and calculates the channel average of each field; then calculates the attention between fields based on the channel average of each field; then calculates the weight of each field based on the attention between fields; finally, weights the high-level feature data output by the feature extraction branch 1 according to the weight of each field to obtain the weighted feature data, which is recorded as F4. 2 ∈R N×C3 , C3 is the number of channels.
[0141] S6, the feature classification module first uses the maximum pooling layer Max Pooling to perform maximum pooling processing on the weighted feature data output by the point attention branch 2 to obtain the global aggregated feature data; then uses multiple multi-layer perceptrons MLP to perform dimensionality reduction processing on the global aggregated feature data in turn to obtain the confidence of each body position category; then the confidence of each body position category is normalized through the SoftMax function, and the body position category corresponding to the maximum confidence is selected through the ArgMax function, and finally the predicted body position of the target is output.
[0142] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A body position classification method based on 3D point cloud data, characterized in that: include: Perform domain sampling on the target point cloud data to obtain domain-level feature data; Perform feature alignment on the domain-level feature data to obtain aligned feature data; Perform feature extraction on the aligned data to obtain high-level feature data; Perform attention weighting on high-level feature data to obtain weighted feature data; Body position classification is performed based on the weighted feature data, and the predicted body position of the target is output.
2. The body position classification method based on 3D point cloud data according to claim 1, characterized in that: The specific method of field sampling is as follows: First, the point cloud is divided into N fields, each of which contains K field points; then the multi-layer perceptron MLP is used to extract features of each point in the field, and then the symmetric function Max is used to aggregate features of each point in the field to obtain feature data of each field, thereby obtaining field-level feature data.
3. The body position classification method based on 3D point cloud data according to claim 1, characterized in that: The specific method of feature alignment is as follows: The input data is processed through the T-Net network to obtain the affine transformation matrix; the affine transformation matrix is multiplied by the input data to obtain the aligned feature data.
4. The body position classification method based on 3D point cloud data according to claim 1, characterized in that: Multilayer perceptron MLP is used for feature extraction. The calculation method of multilayer perceptron MLP is: Ft=f(w*X T +B); Where X is the input data of the multilayer perceptron MLP; Out is the output data of the multilayer perceptron MLP; f(·) is the activation function; B is the bias; w is the weight; superscript T Indicates transpose.
5. The body position classification method based on 3D point cloud data according to claim 1, characterized in that: The specific method of attention weighting is as follows: The high-level feature data is averaged and pooled on the channel, i.e. compressed, to obtain the channel average value of each field; According to the channel average of each field, the attention between fields is obtained; Calculate the attention between fields and get the weight of each field; The high-level feature data is weighted according to the weight of each field to obtain weighted feature data.
6. The body position classification method based on 3D point cloud data according to claim 1, characterized in that: The specific method of body position classification is as follows: Perform maximum pooling on the weighted feature data to obtain global aggregated feature data; The global aggregate feature data is sequentially subjected to dimensionality reduction processing to obtain the confidence of each body position category; The confidence of each body position category is normalized, and the body position category corresponding to the maximum confidence is selected as the predicted body position of the target.
7. A body position classification system based on 3D point cloud data, characterized in that: The body position classification method based on 3D point cloud data is applicable to any one of claims 1 to 6 above, wherein the system adopts a body position classification model; the body position classification model includes a domain sampling module, a feature alignment module, a feature extraction module, a point attention module and a feature classification module; The domain sampling module performs domain sampling on the point cloud data of the target to obtain domain-level feature data; The feature alignment module includes at least two feature alignment branches, at least feature alignment branch 1 and feature alignment branch 2; the input data of feature alignment branch 1 is the domain-level feature data output by the domain sampling module, and the input data of feature alignment branch 2 is the weighted feature data output by point attention branch 1; the feature alignment branch is used to perform feature alignment on the input data to obtain aligned feature data; The feature extraction module includes at least two feature extraction branches, at least feature extraction branch one and feature extraction branch two; the input data of feature extraction branch one is the aligned feature data output by feature alignment branch one, and the input data of feature extraction branch two is the aligned feature data output by feature alignment branch two; the feature extraction branch is used to extract features from the input data to obtain high-level feature data; The point attention module includes at least two point attention branches, at least one point attention branch and one point attention branch; the input data of the point attention branch one is the high-level feature data output by the feature extraction branch one, and the input data of the point attention branch two is the high-level feature data output by the feature extraction branch two; the point attention branch one is used to perform attention weighting on the input data to obtain weighted feature data; the weighted feature data output by the point attention branch two is input into the feature classification module; The feature classification module is used to perform body position classification based on the weighted feature data output by the point attention branch 2, and output the predicted body position of the target.
8. The body position classification system based on 3D point cloud data according to claim 7, characterized in that: The feature extraction branch 1 includes at least two multi-layer perceptrons MLP connected in sequence; the feature extraction branch 2 includes at least three multi-layer perceptrons MLP connected in sequence; The weights of the multilayer perceptrons MLP of the two feature extraction branches are shared; the number of channels of the multilayer perceptrons MLP of the feature extraction branch one is the same; and the number of channels of the multilayer perceptrons MLP of the feature extraction branch two increases sequentially.
9. The body position classification system based on 3D point cloud data according to claim 7, characterized in that: The point attention branch includes an average pooling layer AvgPooling, a multi-layer perceptron MLP and a weighted layer, specifically; wherein, The average pooling layer AvgPooling is used to compress the high-level feature data output by the feature extraction branch on the channel and calculate the channel average value of each field; Use multi-layer perceptron MLP to calculate the attention between fields based on the channel average of each field; Use multi-layer perceptron MLP to calculate the weight of each field based on the attention between fields; The weighted layer is used to weight the high-level feature data according to the weight of each field to obtain the weighted feature data.
10. The body position classification system based on 3D point cloud data according to claim 7, characterized in that: The feature classification module includes a maximum pooling layer, multiple multi-layer perceptrons (MLPs) and an output layer. The specific processing method is as follows: The Max Pooling layer is used to perform maximum pooling processing on the weighted feature data output by the second point attention branch to obtain global aggregated feature data; Multiple multi-layer perceptrons (MLPs) perform dimensionality reduction on the global aggregate feature data in turn to obtain the confidence level of each body position category; The output layer first normalizes the confidence of each body position category through the SoftMax function, and then selects the body position category corresponding to the maximum confidence through the ArgMax function as the predicted body position of the target.