A Visual Analysis Method for the Interpretability of PointCNN

Abstract

The present invention relates to a visual analysis method for the interpretability of PointCNN, belonging to the field of visualization technology. The method includes: S1: Inputting the three-dimensional coordinates of the three-dimensional human body point cloud into the target neural network model to be studied for prediction to obtain the data to be studied; the target neural network is a neural network model that uses the PointCNN network structure to eliminate ambiguous points for the segmentation of human body point cloud parts; S2: Extracting the data to be studied, specifically including network parameters of each layer, feature information, point cloud data of each layer, and detailed information of the predicted result point cloud; S3: Associating the obtained point cloud, feature information, and network parameters, and outputting and storing the relationship matrix; S4: Presenting the associated point cloud, feature information, and network parameters in a visual manner to analyze PointCNN from multiple aspects. The present invention is beneficial to better understanding the operation mechanism of PointCNN and the details in the network.

Description

Technical Field

[0001] The present invention belongs to the technical field of visualization, and relates to a visual analysis method for the interpretability of PointCNN. Background Art

[0002] In recent years, with the rapid update of three-dimensional acquisition devices and the advent of the big data era, the amount of point cloud data collected is increasing and the collected data is becoming more and more complex. In many scenarios, traditional methods for point cloud segmentation and registration are difficult to solve some problems. With the development of deep learning, especially after CNN has made great progress in images, more and more researchers have tried to apply the modified CNN to point clouds. PointCNN is a leader in point cloud segmentation and classification in recent years. However, since the input of CNN is a two-dimensional image and the input of PointCNN is a three-dimensional point cloud, many researchers do not understand its internal operation mechanism, that is, how a two-dimensional neural network is applied to a task with three-dimensional input. Visual analysis technology realizes a graphical visualization model through visualization algorithms to display multi-dimensional or high-dimensional data. The visualization model combined with human-computer interaction can also perform dynamic multi-angle analysis.

[0003] The visual analysis research on the interpretability of deep learning is an important method for further in-depth research on deep learning models. The research on the interpretability of deep learning models is mainly divided into methods based on static expressions, methods based on the internal structure of the model, methods based on input-output data and model performance statistical data, and methods based on surrogate models. Use one of them or combine some of them to do some extensions to develop a visual analysis system for a specific model to analyze the model.

[0004] PointCNN shows excellent features in many scenarios, but many researchers still do not understand how its core X convolution works, how ordinary CNN is applied to three-dimensional point clouds, and do not have a clear understanding of the outputs of each layer in the network. For many researchers, PointCNN is still a deep learning black box. By visualizing the internal structure details and data flow of PointCNN, and visualizing in combination with multiple aspects of information, further in-depth understanding of the internal operation mechanism of PointCNN and what needs to be improved can be achieved. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a visual analysis method for the interpretability of PointCNN. Aiming at the problem that there is a lack of visual analysis of 3D point cloud deep learning in existing deep learning visual analysis systems, a neural network with the PointCNN as the main architecture is taken as the research object, and network parameter extraction, point cloud data extraction, and feature data extraction are carried out, and its operation mode and more intermediate details of the neural network are presented in a visual way to better understand PointCNN and provide certain reference for improving PointCNN.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A visual analysis method for the interpretability of PointCNN specifically includes the following steps:

[0008] S1: Input the 3D coordinates of the 3D human body point cloud into the target neural network model to be studied for prediction to obtain the required research data; the target neural network is a neural network model that uses the PointCNN network structure to achieve ambiguity point elimination for the segmentation of human body point cloud parts;

[0009] S2: Extract the required research data, specifically including network parameters of each layer, feature information, point cloud data of each layer, and detailed information of the predicted result point cloud;

[0010] S3: Associate the obtained point cloud, feature information, and network parameters, and output and store the relationship matrix;

[0011] S4: Present the associated point cloud, feature information, and network parameters in a visual way, and analyze PointCNN from multiple aspects.

[0012] Further, in step S1, the target neural network to be studied is a neural network model that uses the PointCNN network architecture to achieve ambiguity point elimination for the segmentation of human body point cloud parts, and the ambiguity point is a point that makes no contribution to the part segmentation in the part segmentation.

[0013] Each layer of the PointCNN network is similar to the convolutional layer of the CNN. It is modified on the basis of the conv layer, and PointCNN calls it X-Conv. This is also the core for PointCNN to process 3D point cloud input; specifically includes the following steps:

[0014] (1) Input K, p, P, F into the PointCNN network; where K is the value of K for K-nearest neighbors and is also the convolutional kernel size; p is the feature aggregation point, P is the set of K-nearest neighbors of p, and F is the feature matrix of point p. For the first layer of the network, F is the 3D coordinate matrix of point p;

[0015] (2) For p′, use MLP to generate the feature matrix F δ ; Where p′ is the new three-dimensional coordinate set generated by subtracting the coordinates of point p from the three-dimensional coordinates of point P;

[0016] (3) F and F δ Spliced into F * ;

[0017] (4) Learn the X transformation matrix through MLP;

[0018] F X =X×F * ;

[0019] F p =Conv(K,F x )

[0020] To sum up, it can be expressed as:

[0021] F p =X-Conv(K,p,P,F)=Conv(K,MLP(Pp)×[MLP δ (Pp),F])

[0022] Among them, F p The feature learned by point p through X-Conv is represented by the X-Conv operation. The principle is to aggregate the point cloud features to a small number of points through X-Conv, learn the features from their K-nearest neighbors, and then restore the features to each point through X-Deconv (deconvolution). Finally, segmentation or classification is performed through the softmax layer.

[0023] Furthermore, in step S2, the network parameters include the number of neural network layers and the number of channels in each layer; the feature information data includes the feature matrix; the point cloud data of each layer is the three-dimensional coordinates of the feature point cloud of each layer; the prediction result detailed data includes the three-dimensional coordinates of the prediction result point cloud, the accuracy of each point and the labels of each point in the manually annotated original point cloud.

[0024] Furthermore, in step S4, the associated point cloud, feature information and network parameters are presented in a visual manner, specifically including: using a three-dimensional scatter plot to represent the feature aggregation points of each layer, the ambiguous point division results, the incorrectly segmented point cloud, and the k-nearest neighbor point cloud (features are extracted from the k-nearest neighbor point cloud); using a pie chart to represent the proportion of incorrectly segmented point clouds; using a bar chart to represent the number of incorrectly segmented point clouds with different degrees of discrimination; using a pixel matrix to represent the features learned by the feature points; and using a Bezier curve to represent the data flow.

[0025] Furthermore, in step S4, visualization is achieved using Echarts.js, Echarts.GL.js, and SVG, analyzing PointCNN from multiple perspectives. Specifically, the following operations are performed: A 3D scatter plot is used to display the original point cloud uploaded to the network and the ambiguous point segmentation results to understand the neural network prediction effect; a 3D scatter plot is used to display the incorrectly segmented point cloud to understand where the incorrectly segmented point cloud is located on the human body; a bar chart is used to display the number of incorrectly segmented point clouds with different discrimination intervals, where the discrimination is the difference between the predicted probability and 0.5; and a pie chart is used to display the proportion of incorrectly segmented point clouds, providing a reference for subsequent optimization of the PointCNN feature extraction method; a 3D scatter plot is used to display the feature clusters of each layer; clicking on a feature point displays the highlighted k-nearest neighbor points and the features represented by the pixel matrix; and a Bezier curve is used to represent the data flow in the network. This comprehensive analysis allows researchers to further understand the operating mechanism of PointCNN and provide a reference for optimizing PointCNN.

[0026] The beneficial effect of the present invention is that: the present invention uses the idea of data visual analysis and certain visualization symbols to intuitively display the relationship between neural network data, thereby deepening the user's understanding of the operation mechanism of the neural network. The research object in the present invention is a neural network that uses a PointCNN network structure to segment human body part point clouds. The segmentation network is representative of the PointCNN network segmentation task. By analyzing the data flow and combining the point cloud linkage to display the K nearest neighbor point cloud, the operation mechanism of the PointCNN core step X convolution is obtained. By visualizing the detailed information of the point cloud prediction results, the problems existing in the network in human body part segmentation are obtained. By comparing the model prediction effect point cloud display, the influence of the network architecture on the prediction results is obtained. By combining multiple aspects for visualization and analysis, it can help researchers to timely understand the operation mechanism and existing problems of PointCNN, and provide a certain reference for optimizing and improving PointCNN.

[0027] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0029] Figure 1 Flowchart of the visual analysis method for PointCNN interpretability of the present invention;

[0030] Figure 2 The original point cloud shown using a three-dimensional scatter plot and the effect diagram of the segmented point cloud;

[0031] Figure 3 The effect diagram showing the feature aggregation points of each layer using a three-dimensional scatter plot;

[0032] Figure 4 The display diagram of the feature points and the newly linked k-nearest neighbor points;

[0033] Figure 5 The effect diagram of the features extracted by the pixel map linked display

[0034] Figure 6 The point cloud with segmentation errors shown using a three-dimensional scatter plot. Specific implementation manners

[0035] The following uses specific specific examples to illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0036] Among them, the attached drawings are only for illustrative purposes, showing only schematic diagrams, rather than physical diagrams, and should not be construed as a limitation to the present invention; in order to better illustrate the embodiments of the present invention, some components in the attached drawings will be omitted, enlarged or reduced, and do not represent the dimensions of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the attached drawings may be omitted.

[0037] In the attached drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "rear", etc. indicating the orientation or positional relationship, it is based on the orientation or positional relationship shown in the attached drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the positional relationship in the attached drawings are only for illustrative purposes and should not be construed as a limitation to the present invention. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to specific situations.

[0038] Please refer to Figures 1 to 4, the present invention provides an interpretable visual analysis method for PointCNN, as Figure 1 shown, the method specifically includes the following steps:

[0039] S1: Predict the three-dimensional coordinates of the three-dimensional human body point cloud through the target neural network model to be studied. In this embodiment, the target neural network is a neural network model that uses the PointCNN network structure to remove the ambiguity points in the segmentation of human body point cloud parts, and obtain the data to be studied;

[0040] S2: Extract the data to be studied;

[0041] S3: Associate the obtained point cloud, feature information, and network parameters, and output and store the relationship matrix;

[0042] S4: Present the associated point cloud, features, and network parameters in a visual way, and analyze PointCNN from multiple aspects;

[0043] In this embodiment, the neural network to be studied is a neural network model that uses the PointCNN network architecture to eliminate the ambiguity points in the segmentation of human body part point clouds. The ambiguity points are the points that do not contribute to the part segmentation in the part segmentation.

[0044] Each layer of the PointCNN network is similar to the convolutional layer of the CNN, and is modified on the basis of the conv layer. PointCNN calls it X-Conv, which is also the core for PointCNN to process three-dimensional point cloud inputs. Specifically, it includes the following steps:

[0045] (1) Input K, p, P, F into the PointCNN network; where K is the value of K in the K-nearest neighbor, and is also the convolutional kernel size; p is the feature aggregation point, P is the set of K-nearest neighbors of p, and F is the feature matrix of point p. In the first layer of the network, F is the three-dimensional coordinate matrix of point p;

[0046] (2) Use MLP to generate the feature matrix F for p′ δ ; where p′ is a new set of three-dimensional coordinates generated by subtracting the coordinates of point p from the three-dimensional coordinates of points in P;

[0047] (3) Concatenate F and F δ into F * ;

[0048] (4) Learn the X transformation matrix through MLP;

[0049] F X = X × F * ;

[0050] F p = Conv(K, FX )

[0051] To sum up, it can be expressed as:

[0052] F p =X-Conv(K,p,P,F)=Conv(K,MLP(Pp)×[MLP δ (Pp),F])

[0053] Among them, F p The feature learned by point p through X-Conv is represented by the X-Conv operation. The principle is to aggregate the point cloud features to a small number of points through X-Conv, learn the features from their K-nearest neighbors, and then restore the features to each point through X-Deconv (deconvolution). Finally, segmentation or classification is performed through the softmax layer.

[0054] In this embodiment, the information to be extracted specifically includes network parameters of each layer, feature information, point cloud data of each layer, and detailed information of the prediction result point cloud;

[0055] In this embodiment, the extracted information is associated, the relationship between the features and the point cloud and network parameters is analyzed, and a relationship matrix is established.

[0056] In this embodiment, Echarts.js, Echarts.GL.js, and SVG visual analysis technology are finally used to analyze PointCNN from multiple aspects, including the following:

[0057] like Figure 2 As shown, a three-dimensional scatter plot is used to display the original point cloud and the segmented point cloud.

[0058] like Figure 3 As shown in the figure, a three-dimensional scatter plot is used to show the feature aggregation points of each layer.

[0059] like Figure 4 As shown in the figure, click the feature point with the mouse to highlight the k-nearest neighbor points, and at the same time, the new coordinate system of the new k-nearest neighbor points with the feature point as the reference point will be displayed.

[0060] like Figure 5 As shown in the figure, when you click the feature point with the mouse, the pixel map will display the extracted features, and the Bezier curve will indicate the data flow.

[0061] like Figure 6 As shown in the figure, the three-dimensional scatter plot shows the point cloud with segmentation errors, from which we can find out which parts of the human body the points with segmentation errors are distributed. The discrimination is the difference between the prediction accuracy and 0.5. The pie chart shows the proportion of the incorrectly segmented point cloud in different discrimination intervals, and the bar chart shows the number of incorrectly segmented points in different discrimination intervals.

[0062] For the above-mentioned interpretable visual analysis method of PointCNN, the theoretical knowledge is based on data visualization and point cloud deep learning models. First, it is necessary to upload the human body point cloud to the studied model for prediction. By extracting the point cloud coordinates output, feature information, network parameters, detailed prediction results, original point cloud and its labels in the network, and then associating them to establish a relationship matrix, analyze PointCNN from multiple aspects to better enable researchers to understand the operation mechanism and existing problems of PointCNN, and at the same time provide certain references for optimizing and improving PointCNN.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the purpose and scope of the present technical solution, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A visual analysis method for PointCNN interpretability, characterized by: The method specifically comprises the following steps: S1: Input the 3D coordinates of the 3D human body point cloud into the target neural network model to be studied for prediction to obtain the required research data; the target neural network is a neural network model that uses the PointCNN network structure to segment the human body point cloud and eliminate ambiguous points; Each layer of the PointCNN network is X-Conv, which is used to process 3D point cloud input; it specifically includes the following steps: (1) Input K, p, P, and F into the PointCNN network; K is the K value of the K nearest neighbor, which is also the size of the convolution kernel; p is the feature cluster point, P is the set of K nearest neighbors of p, and F is the feature matrix of point p. The F of the first layer of the network is the three-dimensional coordinate matrix of point p. (2) For p', use MLP to generate the feature matrix F δ ; Where p' is the new three-dimensional coordinate set generated by subtracting the three-dimensional coordinates of point P from the coordinates of point p; (3) F and F δ Spliced into F * ; (4) Learn the X transformation matrix through MLP; F X =X×F * F p =Conv(K,F X ) To sum up, it can be expressed as: F p =X-Conv(K,p,P,F)=Conv(K,MLP(P-p)×[MLP δ (P-p),F]) Among them, F p Represents the features learned by point p through X-Conv; S2: Extract the required research data, including network parameters, feature information, point cloud data of each layer, and detailed information of the predicted point cloud; the network parameters include the number of neural network layers and the number of channels of each layer; the feature information data includes the feature matrix; the point cloud data of each layer is the three-dimensional coordinates of the feature point cloud of each layer; the detailed prediction result data includes the three-dimensional coordinates of the predicted point cloud, the accuracy of each point, and the labels of each point in the manually annotated original point cloud; S3: Associate the obtained point cloud, feature information and network parameters, output and store the relationship matrix; S4: The associated point cloud, feature information, and network parameters are presented in a visual manner, and PointCNN is analyzed from multiple aspects.

2. The visual analysis method for PointCNN interpretability according to claim 1, characterized in that In step S4, the associated point cloud, feature information and network parameters are presented in a visual manner, including: using a three-dimensional scatter plot to represent the feature aggregation points of each layer, the ambiguous point division results, the incorrectly segmented point cloud, and the k-nearest neighbor point cloud; using a pie chart to represent the proportion of incorrectly segmented point clouds; using a bar chart to represent the number of incorrectly segmented point clouds with different degrees of discrimination; using a pixel matrix to represent the features learned by the feature points; and using a Bezier curve to represent the data flow.

3. The visual analysis method for PointCNN interpretability according to claim 2, characterized in that In step S4, PointCNN is analyzed from multiple aspects, including: using a three-dimensional scatter plot to display the original point cloud uploaded to the network and the ambiguous point segmentation results to understand the prediction effect of the neural network; a three-dimensional scatter plot to display the incorrectly segmented point cloud to understand which parts of the human body the incorrectly segmented point cloud is located in, a bar chart to display the number of incorrectly segmented point clouds with different discrimination intervals, where the discrimination is the difference between the predicted probability and 0.5, and a pie chart to display the proportion of incorrectly segmented point clouds; a three-dimensional scatter plot to display the feature aggregation points of each layer, and clicking on a feature point will display the highlighted k-nearest neighbor points and the features represented by the pixel matrix; a Bezier curve to represent the data flow in the network.

Images