Multi-modal target part detection method based on point cloud diversity representation and pointrcnn

Abstract

The application discloses a multi-modal target part detection method based on point cloud diversity representation and PointRCNN, which combines cylinder representation and graph representation of point cloud data, fully utilizes respective advantages to improve feature expression capability of the whole algorithm, and constructs relative position and structure information between points in the point cloud by combining a manifold self-attention mechanism, so as to finally improve accuracy and efficiency of multi-modal target part detection, and comprises the following steps: through data preprocessing, original point cloud data is converted into a data format required by a 2D backbone network; the obtained data is input into a point cloud diversity representation module and a manifold grouping feature sampling module for processing, diversified features and multi-layer image features are obtained; the diversified features and the multi-layer image features are input into a double-layer feature fusion module for fusion and enhancement processing, and multi-modal fusion features are output; and 3D backbone network is used for target part detection on the multi-modal fusion features, so as to obtain a detection result of the target part.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of mechanical engineering, intelligent manufacturing technology, etc., and in particular to a multi-modal target part detection method based on point cloud diversity representation and PointRCNN. BACKGROUND

[0002] Multi-modal target detection is a method that integrates data from multiple sensors such as lidar, cameras, and millimeter-wave radars to achieve target detection and recognition in three-dimensional environments. Compared with single-modal target detection, this method provides higher accuracy and robustness. By combining data from different modalities, the limitations of a single modality in specific scenarios can be effectively overcome, thereby improving the accuracy and stability of object detection in practical applications. Early research, such as MV3D and AVOD, typically represents objects as multiple two-dimensional images from different perspectives and performs target detection through multi-view fusion techniques. However, due to the differences in angle and resolution of images from different perspectives, the fusion process is often affected by inconsistent information. In addition, methods such as Transfusion, 3D-CVF, and CLOCs, which are based on image and LiDAR fusion, utilize the semantic information of images and the spatial features of point cloud data to improve the accuracy and robustness of target detection. However, these methods mostly rely on simple feature fusion techniques such as concatenation or weighted summation, and fail to fully exploit the correlation and complementarity between different feature types. Therefore, enhancing the fusion process to better utilize the synergy between different modal features has become an important direction of research. In addition, point cloud multi-modal target detection also faces some challenges. Different data representation methods have their own advantages and disadvantages, but research combining multiple representation methods is relatively less; the processing and feature fusion of multi-modal data require a large amount of computational resources; and single multi-modal data fusion methods, such as direct concatenation or the use of attention mechanisms, become a bottleneck for multi-modal target detection, limiting its adaptability in diverse scenarios and application requirements.

[0003] The PointRCNN algorithm can not process the point cloud data, thereby retaining more original information, and thus performs better in target detection accuracy. However, the algorithm has a strong dependence on the quality and density of the input point cloud data. When the point cloud data is sparse, incomplete or contains a large amount of noise, the detection effect is often unsatisfactory. This problem is mainly due to the direct use of point cloud data in the feature extraction stage of PointRCNN, which makes it susceptible to irrelevant points and noise. In addition, the PointNet++ network is highly dependent on the quality and density of the point cloud data, making it difficult to effectively capture key features in sparse environments. The working process of PointRCNN can be divided into two main stages: bottom-up 3D candidate box generation and standardized 3D detection box refinement. The specific steps are as follows: first, a point cloud encoding network based on the PointNet++ network is used to extract a feature vector for each point from the three-dimensional point cloud; then, a foreground point segmentation network is used to generate a segmentation mask to mark the foreground points in the point cloud, and then a small number of high-quality 3D candidate boxes are generated in a bottom-up manner based on these foreground points. Secondly, after obtaining the 3D candidate boxes, each box is expanded to encode its context information and retain all points located within the expanded box. Finally, the aggregated points of each candidate box are converted to a canonical coordinate system, the local features of the target are learned, and the local features and global semantic features are combined to achieve more accurate bounding box refinement and confidence prediction. SUMMARY

[0004] The purpose of the present application is to provide a multi-modal target part detection method based on point cloud diversity representation and PointRCNN, which combines the cylinder representation and graph representation of point cloud data, fully utilizes the advantages of each to improve the overall feature expression ability of the algorithm, and constructs the relative position and structure information between points in the point cloud by combining the manifold self-attention mechanism, finally improves the accuracy and efficiency of multi-modal target part detection.

[0005] The present application is realized by the following technical scheme: a multi-modal target part detection method based on point cloud diversity representation and PointRCNN, comprising the following steps:

[0006] 1) Through data preprocessing, the original point cloud data is converted into a data format required by a 2D backbone network;

[0007] 2) The data obtained in step 1) is input into a point cloud diversity representation module for processing, so as to improve the accuracy and efficiency of point cloud target part detection by utilizing the advantages of cylinder representation and graph representation, and obtain diversified features;

[0008] 3) input the data obtained in step 1) into a manifold grouping feature sampling module to effectively calculate the mutual relationship between points in the point cloud, and obtain multi-layer image features, so that the detection method can more accurately capture the context information and global features in the point cloud data, and the manifold grouping feature sampling module not only improves the understanding ability of the algorithm for complex spatial structures, but also significantly enhances the generalization ability of the algorithm;

[0009] 4) After step 3), the output features (diversified features) of the point cloud diversity representation module and the output features (multi-layer image features) of the manifold grouping feature sampling module are input into a double-layer feature fusion module for fusion and enhancement processing; that is, the point cloud and image features are fused in the double-layer feature fusion module to obtain comprehensive feature representation, and on the basis of preliminary fusion, more rich feature expression is obtained by enhancing the semantic features of the point cloud, so as to improve the detection efficiency, and output multi-modal fusion features;

[0010] 5) using a 3D backbone network to detect the multi-modal fusion features output by the double-layer feature fusion module, and obtaining the detection result of the target part through the 3D point cloud detection head in the 3D backbone network.

[0011] In order to better realize the multi-modal target part detection method based on point cloud diversity representation and PointRCNN, the following settings are particularly adopted: the 2D backbone network in step 1) uses a pre-trained Swin-Transformer model as the detection model (backbone network) of the 2D backbone network for image feature extraction, so as to fully exert the advantages of self-attention; an important difference between RGB images and laser radar point cloud data is that the former can provide rich color and texture information, which helps to distinguish different target part categories; in the method, multi-level semantic features in the RGB image are integrated into the point cloud features when the 2D backbone network pre-processes the data; compared with the traditional convolutional neural network, the self-attention mechanism more effectively captures the long-range dependency relationship between any two positions in the image, and collects the global information and context of the image by exploring the correlation of the whole image; in order to further enrich the feature expression, a feature pyramid network structure is introduced into the 2D backbone network, and an additional pooling layer is added on the basis of the original four layers, so as to generate five layers of image features, represented as F image ={f I 1 ,f I 2 ,f I 3 ,f I 4 ,f I 5The algorithm employs pooling operations on the top-level features to obtain the fifth layer of features, thereby introducing features at more scales and further enhancing the detection algorithm's ability to perceive target parts. In the feature pyramid stage, the number of input channels for each layer is set to 96, 192, 384, and 768, while the number of output channels is fixed at 96. This configuration allows the algorithm to effectively extract features at different scales, enabling it to capture fine-grained and deep information from the input image, with the image resolution set to 2048×618 pixels.

[0012] To further improve the multimodal target part detection method based on point cloud diversity representation and PointRCNN described in this invention, the following configuration is specifically adopted: The point cloud diversity representation module in step 2) combines the advantages of cylinder representation and graph representation, providing flexible resolution while retaining detailed information. Dynamic cylinder feature encoding and dynamic graph network encoding are used to process the data output in step 1). Dynamic cylinder feature encoding can effectively reduce data dimensionality, thereby saving computational resources. Unlike traditional methods, the dynamic cylinder feature encoding does not sample the point cloud into a fixed number of cylinders with a fixed volume, but rather retains the complete mapping relationship between points and cylinders: the original point cloud data is represented as P = {p1, p2, p3, ..., p...} n}, where p i ={(x i ,y i ,z i )|i=1,2,3,...,n},(x i ,y i ,z i Let ) represent the three-dimensional coordinates of point i, and r represent the reflectivity intensity. Dividing the point cloud space into three-dimensional cylindrical spaces, each cylinder has a specified size [Δx, Δy, Δz]. A three-dimensional cylinder can then be represented by the following formula:

[0013]

[0014] in, This represents the rounding operation. (i,j,k) represents the coordinates of the center point of the 3D cylinder. For each cylinder's center point, the point cloud data is mapped to the corresponding cylinder, and the average value of the point cloud within each cylinder is calculated to obtain the corresponding cylinder value. This process also generates the columnar features F of the point cloud. pillar ={f p1 ,f p2 ,f p3 …f pn}

[0015] Dynamic graph network encoding can preserve more point cloud information without loss of detail and can handle point cloud data of arbitrary shapes, increasing shape diversity. Furthermore, dynamic graph networks have stronger representational power and scalability, enabling the extraction of higher-level semantic information, thereby improving the accuracy of target part detection. The dynamic graph network encoding in the point cloud diversity representation module is used for cylinder features F... pillar Based on this, the point cloud image encoding features F are further extracted and aggregated through three feature extraction layers. g ′ raph Multi-layer feature extraction helps the module learn hierarchical feature representations of point cloud data step by step. Each layer focuses on capturing abstract features at different levels in the point cloud, making the final representation richer and more expressive. As the number of layers increases, the receptive field of each layer also expands accordingly, enabling the module to better understand the global and local point cloud structure during the learning process, which helps to better capture the overall shape and contextual information of the target part.

[0016] Furthermore, the multi-layer feature extraction in dynamic graph network encoding introduces more non-linear transformations, which helps the module learn complex relationships in point cloud data. For target part detection tasks, more in-depth feature extraction networks are often needed to better simulate the complexity of point cloud data. For each cylinder, dynamic graph network is used to extract graph encoding features F. graph =(f g1 ,f g2 ,f g3 ,...,f gj ), where f gj Let the graph-based features of the j-th cylinder be represented by the parameter Θ of the neural network. Then, the feature extraction process can be expressed by the following formula:

[0017] f gj =f Θ (F pj ,{f pi} i∈N(j) );

[0018] Among them, F pj This represents all point cloud data contained in the j-th cylinder, {f pi} i∈N(j) The feature representation of other prisms adjacent to the j-th prism is given, and N(j) is the neighborhood set of the j-th prism. The k-nearest neighbor algorithm is used to capture the local relationships and similarities between nodes in the graph structure. This invention combines discriminative k-nearest neighbor and fuzzy k-nearest neighbor algorithms at different levels to improve the algorithm's adaptability to different data distributions and features. A feature aggregation layer is added in the third layer to aggregate the graph encoding features F of each prism. graph The aggregation is performed to obtain the aggregated point cloud image encoding features F′. graph.

[0019] Preferably, the cylinder encoding input of the point cloud diversity representation module consists of 17600 points sampled from four dimensions, the range of the point cloud is [0, -40, -3, 70.4, 40, 1], the encoding is performed by dynamic cylinder feature encoding alignment, the size of the cylinder is set to [0.05, 0.05, 0.1], the module integrates a feature fusion layer, and five layers of image feature mapping are used, wherein the image channel is 256, the point cloud channel is 64, and the output channel is 128.

[0020] Preferably, the graph encoding of the point cloud diversity representation module extracts features from four dimensions of the point cloud, uses three graph feature extraction layers, each layer includes a grouping operation, each group contains 20 samples, the first layer uses a discriminative k-nearest neighbor algorithm, and the other two layers use a fuzzy k-nearest neighbor algorithm, and the hyperparameter of the activation function is set to 0.2.

[0021] Further, in order to better realize the multi-modal target part detection method based on point cloud diversity representation and PointRCNN, the following setting mode is particularly used: the manifold grouping feature sampling module in the step 3) can unitize the input feature vector f of each point in the point cloud to obtain a feature representation mapped into a manifold space, so as to use a self-attention mechanism in the manifold space, and the calculation formula is described as follows:

[0022]

[0023] wherein ||·|| represents the Frobenius norm, represents dividing the input feature vector f by its Frobenius norm, thereby realizing the unitization of the input feature vector f, so that the length of each vector is 1. Through this process, the input feature vector of each point in the point cloud can be mapped to a unit length sphere, realizing the mapping of the point cloud in the manifold space;

[0024] In the manifold grouping feature sampling module, when the point cloud is mapped to the manifold space, the shortest path between two points can be determined by the included angle represented by the two point vectors, by calculating the included angle values between these unit vectors and using a softmax function for normalization, finally the manifold self-attention score can be obtained, and the formula is as follows:

[0025]

[0026] In the formula, E(p i ) represents the point set around the point p i , x j is the point set around the point p i) where n denotes the dimension of the attention query vector Q and the attention key vector K, V denotes the attention value vector, diag(·) denotes the diagonal values of the matrix, represents the element-wise product of two matrices, f i attn represents the input point p i The output feature obtained after the manifold attention weighting calculation.

[0027] Further, in order to better realize the multi-modal target part detection method based on point cloud diversity representation and PointRCNN, the following setting mode is particularly adopted: the manifold grouping feature sampling module in the step 3) introduces a grouping self-attention mechanism, the feature channels are grouped and the attention weights are shared within the group, so that the number of parameters and the calculation complexity of the module are reduced, and the overfitting phenomenon is alleviated. The grouping self-attention mechanism inherits the advantages of vector attention and multi-head attention, can more effectively learn deep feature representation, and thus improves the processing ability and generalization ability of the module to complex data. In the grouping self-attention, the channels of the value vector are evenly divided into k groups, and 1≤k≤c. Therefore, the output of the weight encoding layer is not a complete attention vector of c channels, but a grouped attention vector of k channels, wherein the channels of each group share one scalar attention weight, and the grouping linear transformation function τ is:

[0028]

[0029] where rel is the relationship vector between the attention query vector Q and the attention key vector K, p1, p2, …, p k are parameters of the grouping linear transformation, wherein each point p i is a grouping matrix for independently transforming the input relationship vector rel; the zero matrix is used to isolate different groups in the transformation. In order to enhance the role of position information in the attention mechanism, an additional position encoding multiplier is introduced to improve the influence of position information on weight encoding, so that the module can better learn and understand the relative positions between points in three-dimensional space, thereby improving its understanding and processing ability of complex spatial structures.

[0030] The manifold grouping feature sampling module promotes information exchange between groups and improves the nonlinear expression ability of the module by normalizing, activating and further linearly transforming each group of relationship vectors rel subjected to the grouping linear transformation function τ, and finally forms the grouping weight encoding function of the relationship vector rel as:

[0031] ω(rel)=Norm(Relu(Linear(ζ mul (p i -p j )⊙τ(rel)+ζbias (p i -p j ));

[0032] where p i and p j are the coordinates of spatial points and neighboring points respectively, ζ mul and ζ bias are two position encoding functions realized by fully connected layers, which are used to enhance the understanding and representation ability of the module for the complex spatial relationship in the point cloud data, Linear(·) is a linear transformation layer realized by a fully connected layer, which is responsible for further feature conversion between groups; Relu(·) is an activation function, which is used to introduce nonlinearity to better learn complex features; Norm(·) is a normalization layer, which is used to speed up the training speed and improve the stability of the module, and finally the calculation formula of the manifold grouping self-attention feature vector is as follows:

[0033]

[0034] where l represents the group index, V j lc / k+m represents the mth feature in the lth group in the value vector grouping. By applying the grouping attention mechanism, the weighted and aggregated context information of the spatial point coordinates p i is realized, and f i attn not only contains the information of the spatial point coordinates p i itself, but also integrates the information of the surrounding neighborhood, significantly improving the efficiency and representation ability of the module, so as to more effectively extract features in sparse point clouds.

[0035] Further, in order to better realize the multi-modal target part detection method based on point cloud diversity representation and PointRCNN described in the present application, the following setting mode is particularly adopted: the double-layer feature fusion module in the step 4) is divided into a point cloud branch and an image branch, wherein the point cloud branch transforms and aggregates the diversified features in the step 2), the image branch aligns and aggregates the multi-layer image features in the step 3), and finally the features aggregated by the point cloud branch and the image branch are fused;

[0036] In the point cloud branch, the diversified features obtained by the point cloud diversity representation module are transformed using a fully connected layer and normalization processing, and unified point cloud features are obtained and represented as where represents the 3D feature of the nth point; then the unified point cloud features are grouped and aggregated, and the K-dimensional tree method is used to divide the point cloud features F pc based on its local region; for the point cloud features F pca single feature vector representing the region is obtained by averaging the feature vectors within the region; each point cloud feature F pc The new feature vector of each point cloud feature F

[0037] F p ′ c = Mean(KD(F pc ));

[0038] where F p ′ c is the point cloud feature after grouping aggregation, and Mean(·) represents the mean aggregation method;

[0039] In the image branch, the multi-layer image features obtained from the manifold grouping feature sampling module are aligned and transformed to obtain image feature representations with the same dimension as the point cloud features F pc In the point cloud branch, the multi-layer image features are aligned with the point cloud features F pc by the alignment sub-module, and the extracted multi-layer image features and the point cloud features F pc are normalized, the multi-layer image features are aligned with the point cloud features F pc using the iterative closest point algorithm, and their spatial relationship is optimized, and the aligned multi-layer image features are obtained by convolution operation and enhancement transformation to obtain image features F img ;

[0040] Then the image features F img are grouped and aggregated, for each image feature F img , its surrounding image features are found using radius neighborhood search, the found surrounding image features are aggregated, the features within the neighborhood are aggregated into a representative feature using the pooling operation, the aggregated features are fused with the initial image features F img to form new image features F i ′ mg ;

[0041] Finally, the point cloud features F p ′ c after grouping aggregation are fused with the new image features F i ′ mg to obtain multi-modal fusion features F fusion .

[0042] The double-layer feature fusion module adopts a sparse 3D-U-shaped network to fuse the multi-modal fusion features F fusionA more fine-grained processing is performed, and semantic features are used to enhance it to obtain better target part detection results. fusion are input into the 3D-U-shaped network, and spatial features F spatial and semantic features F semantic are further obtained. final The two features are combined to obtain the final fusion feature F final as shown in the following formula:

[0043] F spatial = Concat(F semantic , Conv(F ij ));

[0044] wherein Concat(·) represents a feature concatenation operation, and Conv(·) is used to expand the features to the same dimension for concatenation using a sparse 3D-U-shaped network.

[0045] To better implement the point cloud diversity representation and PointRCNN-based multi-modal target part detection method described in the application, the following settings are particularly adopted: the 3D backbone network in step 5) adopts the concept of spatial autocorrelation as a measure of the spatial dispersion degree of the point cloud. Spatial autocorrelation describes the influence of random variable values at different positions in space on the random variable values at nearby positions, showing a certain spatial trend. Data with high spatial autocorrelation indicates a certain degree of aggregation and trend. On the contrary, data with low spatial autocorrelation is more dispersed and may be interference noise. The 3D backbone network calculates the spatial correlation between each point and its nearby points to determine whether the point cloud is dispersed, and removes the point cloud with low spatial autocorrelation, thereby reducing the number of point clouds while retaining key information.

[0046] Each point cloud data is represented as (x, y, z, r) in a Cartesian coordinate system. The spatial autocorrelation algorithm converts the point cloud data set from the Cartesian coordinate system to the spherical coordinate system to obtain the distance r and the pitch angle for calculating the weight value and the spatial autocorrelation value between points. The specific calculation formula is:

[0047]

[0048] After inputting the original point cloud data, the entire point cloud space is divided into multiple grids, and the spatial autocorrelation value of each grid is calculated.

[0049] Since different points in space contribute differently to the calculation of the spatial autocorrelation value, the angle information between points can be used as relative position information of the data points in three-dimensional space. The 3D backbone network calculates the spatial angle distance w ij, to measure the relevance between two points; the greater the relevance between two points, the greater the weight given, the spatial angular distance w ij The calculation formula is:

[0050]

[0051] Where i represents the processing point, and j represents the surrounding point of the processing point. In the spatial autocorrelation measure, autocorrelation is used to evaluate the spatial relationship between a point and other points, and the relative position of the point to itself is always zero, so when i=j, w ij is set to zero, and the calculation formula of the spatial autocorrelation value I in each grid is:

[0052]

[0053] Where, is the average value of all distances, is the sum of all spatial angular distances w ij , and N represents the number of point clouds in a grid. After obtaining the spatial autocorrelation value of each grid, the grid point cloud with I less than the threshold value is removed according to the set threshold value, so as to obtain the point cloud data after removing noise points and irrelevant points, so that the subsequent 3D point cloud detection head can better focus on the feature detection of the target part.

[0054] Further, in order to better realize the multi-modal target part detection method based on point cloud diversity representation and PointRCNN described in the present application, the following setting method is particularly adopted: the 3D point cloud detection head in the 3D backbone network in step 5) uses the following bounding box encoding function for the detection object:

[0055]

[0056] Where x, y, and z are the center coordinates; w, l, and h are the width, length, and height, respectively; theta is the yaw angle around the z axis; x gt and x a are the ground truth and anchor box, respectively; is the diagonal line of the anchor box bottom edge.

[0057] Further, in order to better realize the multi-modal target part detection method based on point cloud diversity representation and PointRCNN described in the present application, the following setting method is particularly adopted: the loss function of the 3D backbone network in step 5) is set to have: foreground and background point classification loss function, interval-based positioning loss function, and total loss function in the refinement stage;

[0058] Because the number of foreground points (target object points) and background points (non-target object points) is unbalanced in point cloud processing, the number of background points is much larger than that of foreground points, therefore the foreground and background point classification loss function adopts a focal loss function to reduce the weight of negative samples in training:

[0059] Wherein, p represents the foreground prediction probability of each point, and a and g are hyperparameters of the focal loss function, and the algorithm sets these parameters to be the same as the standard model PointRCNN;

[0060] The interval-based positioning loss function is:

[0061] Wherein, pos represents a set of positive sample points; N pos represents the number of positive sample points; is an interval-based target bounding box position regression loss, and is an interval-based target bounding box size regression loss, and is the predicted discrete interval of the foreground point in dimension u e {x, z, q}, and Wherein, u p is the real center coordinate of the object, u (p) is the real coordinate of the corresponding foreground point, is the search range on the X and Z axes, and h is a uniform interval length; is the residual error of the predicted discrete interval of the foreground point and the real position, and Wherein, C is a constant for normalization, to ensure that the residual error calculation is not affected by the size of the interval. Considering that the center position of most objects in the vertical direction Y axis does not change much, and the difference between the object real center coordinate y p and the foreground point coordinate y (p) is used to calculate the residual error on the Y axis: Cross-entropy loss function is used to measure the consistency of the predicted discrete interval and the real interval; is a smooth L1 loss function, used to reduce the difference between the predicted residual error and the real residual error;

[0062] In order to obtain more accurate target part detection frame, the 3D backbone network needs to ensure that the network can correctly classify each detected object in the candidate frame refinement stage, and accurately predict the position and size of the object bounding box, and the total loss function of the refinement stage is:

[0063]

[0064] Wherein, the total number of sample points representing a processing batch, the total number of positive sample points in a processing batch, prob is a predicted label, and label represents a true label, and respectively correspond to the position regression and size regression of the generated candidate box after refinement. This method allows the 3D backbone network to process continuous attributes and discretize them, which can take advantage of the simplicity of the classification task and retain the accuracy of the regression task, thereby better predicting the detection box of the target part.

[0065] Further, in order to better realize the multi-modal target part detection method based on point cloud diversity representation and PointRCNN described in the application, the following setting mode is particularly adopted: the positioning loss of the 3D point cloud detection head is defined as:

[0066] L loc =∑ b∈(x,y,z,w,l,h,θ) SmoothL1(Δb);

[0067] Since the angle positioning loss cannot distinguish the flipped bounding box, a Softmax classification loss L loc is used to learn the discrete direction, and an object classification loss using focal loss is used:

[0068] L cls =-α a (1-p a ) γ log p a ;

[0069] where p a represents the class probability of the anchor point, a takes the default value 0.25, and gamma takes the default value 3. The loss function is optimized using the AdamW optimizer, and is decayed by 0.1 every 15 periods.

[0070] Compared with the prior art, the application has the following advantages and beneficial effects:

[0071] The application combines the cylinder representation and graph representation of point cloud data, fully utilizes the respective advantages to improve the overall feature expression ability of the algorithm, and combines the manifold self-attention mechanism to construct the relative position and structure information between points in the point cloud, thereby overcoming the inherent limitations of traditional point cloud target detection methods that rely on single representation features, solving the problem of low target detection rate in the case of sparse point cloud data or a large number of noise points, and ultimately improving the accuracy and efficiency of multi-modal target part detection. BRIEF DESCRIPTION OF DRAWINGS

[0072] Figure 1 is a structure diagram of a point cloud diversity representation module.

[0073] Figure 2 Flow chart for manifold grouping self-attention mechanism.

[0074] Figure 3 Structural diagram of manifold grouping feature sampling module.

[0075] Figure 4 Flow chart for point cloud branch and image branch.

[0076] Figure 5 Structural diagram of double-layer feature fusion module. DETAILED DESCRIPTION

[0077] The application will be further described below in connection with embodiments, but the embodiments of the application are not limited thereto.

[0078] To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0079] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise explicitly and specifically limited.

[0080] Noun explanation:

[0081] PointRCNN: is a point cloud-based object detection algorithm, which draws on the idea of region convolutional neural network (RCNN) in 2D object detection and applies it to point cloud data. The core idea of PointRCNN is to regard point cloud data as a point set, generate proposal boxes through sampling and alignment operations on the point set, and finally identify target objects in the point cloud.

[0082] Swin-Transformer Model: The Swin-Transformer model is a deep learning model specifically designed for computer vision tasks. It gradually reduces the spatial resolution of feature maps across multiple stages while increasing the number of channels, allowing the model to effectively capture features at different scales. It introduces a windowed multi-head self-attention mechanism, where the input feature map is divided into non-overlapping windows, and self-attention is computed within each window. The shift window strategy allows elements in adjacent windows to interact, helping to build deeper image feature representations.

[0083] MV3D: MV3D (Multi-View 3D) is a deep learning framework for 3D object detection. It extracts features from both lidar point clouds and RGB images. For lidar data, a voxelization method is used to convert 3D point clouds into 2D or 3D grids. For images, a convolutional neural network is used to extract visual features. The features from both sources are fused to obtain more comprehensive and representative feature descriptions, ensuring that features from different modalities can be effectively combined. Based on the fused feature maps, a series of candidate regions are generated, and a regressor is applied within each proposal region to determine specific 3D bounding box parameters. A classifier is also used to determine whether an object exists within the region and its class.

[0084] tAVOD: tAVOD (Temporal Awareness Vehicle Object Detection) is a deep learning model for 3D object detection that considers temporal information. It compares and correlates features from the current frame with historical frames to construct a feature representation that includes the time dimension. By effectively integrating information along the time axis, the model can predict the positions and trends of objects within a certain time frame, enhancing its ability to adapt to rapidly changing environments.

[0085] Transfusion: Transfusion is a deep learning framework for multi-modal data fusion. It combines data from different sensors (such as lidar and cameras) to improve the accuracy and robustness of 3D object detection and tracking. It uses a soft association mechanism instead of the traditional hard association mechanism, making the model more robust to degraded image quality and sensor misalignment. The model uses a Transformer decoder-based detection head to achieve adaptive feature fusion between images and point clouds, limiting the spatial range of cross-attention to the initial bounding box to allow the model to better access relevant locations. This model provides richer image features for object detection and is more robust to poor image conditions.

[0086] 3D-CVF: 3D-CVF (3D Camera-LiDAR Fusion) is a multi-sensor fusion technology for autonomous driving, which combines three-dimensional data from cameras and LiDAR to improve the understanding of the surrounding environment and target detection accuracy. The visual information of the camera and the three-dimensional spatial information of the LiDAR complement each other, making the target detection and scene understanding of the model more accurate. In complex environments or poor lighting conditions, a single sensor may be limited. 3D-CVF enhances the robustness and reliability of the model by combining data from different sensors, enabling more accurate positioning, environmental perception, and decision-making.

[0087] CLOCs: CLOCs (Camera-LiDAR Object Candidates) combine color and texture information provided by cameras with depth and spatial location information provided by LiDAR, using the output of 3D and 2D detectors before non-maximum suppression, while utilizing their geometric and semantic consistency to achieve more accurate detection accuracy. Late fusion strategy is adopted to integrate the detection results of different modalities at the decision level, significantly improving the performance of three-dimensional target detection of the model and enhancing the perception ability and target recognition ability of the model in complex environments.

[0088] LiDAR: LiDAR (Light Detection and Ranging) refers to "laser radar", which is a remote sensing technology that determines distance by emitting laser pulses and measuring the time it takes for these pulses to reflect back from target objects, combined with angle information and other auxiliary data, to generate detailed three-dimensional spatial information, including terrain, buildings, and other features, facilitating subsequent data analysis and processing. LiDAR systems can create high-resolution digital surface models and three-dimensional point cloud maps, widely used in geographic information systems, autonomous driving, three-dimensional target detection, and other fields.

[0089] PointNet++ network: PointNet++ network is a deep neural network specifically designed for processing point cloud data, extracting features from different scales of local regions to capture different levels of geometric information, dynamically selecting neighbor points based on distance between points to form local regions, thus more flexibly adapting to different geometric structures. To effectively organize point cloud data and construct local regions, PointNet++ uses a query sphere-based sampling and grouping strategy, for each center point, a fixed radius sphere is defined, within which neighbor points are searched to form a local region, ensuring that points within the local region have similar spatial positions, significantly improving the network's understanding of point cloud data.

[0090] The present application is based on the following theoretical basis:

[0091] Compared with other point cloud object detection methods, PointRCNN algorithm can directly process raw point cloud data, avoiding information loss that may occur during data conversion. The algorithm uses a bottom-up strategy to generate high-quality 3D candidate boxes, significantly reducing the search space, and in the second stage, it adjusts the candidate boxes to fine boundary boxes, which makes it particularly suitable for processing complex three-dimensional environments. However, due to the sparsity of point cloud data and the presence of a large number of noise, PointRCNN faces challenges in extracting effective point cloud features. The algorithm introduces a spatial autocorrelation method, which not only reduces the size of the input data, but also eliminates the interference of noise points and irrelevant points on feature learning during model training, helping the model to converge better. Compared with the two-dimensional convolution module used in the backbone network of PointRCNN, the proposed multi-modal target part detection algorithm based on point cloud diversity representation and PointRCNN can effectively calculate the mutual relationship between points in the point cloud using self-attention mechanism. This method can more accurately capture the context information and global features in the point cloud data, thereby enhancing the algorithm's understanding of complex spatial structures and significantly improving its generalization ability. Compared with traditional convolutional networks, this self-attention-based strategy exhibits higher efficiency and accuracy in handling the unordered and uneven nature of point cloud data, providing a more efficient and accurate detection method for point cloud target part detection.

[0092] In sparse scenes, PointRCNN cannot extract sufficient effective point cloud features, resulting in insufficient feature information and affecting target detection accuracy. To improve detection performance, the proposed multi-modal target part detection method based on point cloud diversity representation and PointRCNN improves the point cloud encoding network of PointRCNN and introduces a spatial autocorrelation algorithm for point cloud preprocessing during the training phase. Although PointRCNN's point cloud encoding network uses PointNet to directly extract features, traditional convolutional neural networks are mainly designed to process regularly arranged image data, while three-dimensional point cloud data is essentially a non-regular point set embedded in continuous space, which makes convolutional neural networks less effective when directly applied to point cloud data processing. Especially when extracting local features in complex three-dimensional spatial structures, convolutional neural networks often fail to fully capture rich spatial relationships due to their fixed-size convolution kernels and dependence on regular grid structures. In addition, the receptive field limitation of convolutional neural networks makes them less adaptable to the density changes and irregular distribution of point cloud data. To overcome these limitations, the method introduces the self-attention mechanism in the Transformer into the point cloud encoding network of PointRCNN. Unlike convolutional neural networks, the self-attention mechanism can naturally handle irregular and unordered data structures, dynamically adjusting aggregation weights based on data features and relative positions, thereby more effectively capturing complex spatial relationships.

[0093] Traditional dot product self-attention mainly calculates the relationship between features in Euclidean space, but for point cloud data with complex structure and distributed in non-Euclidean space, the traditional dot product self-attention may not effectively capture its internal geometric structure. Therefore, in order to better adapt to the spatial characteristics of point cloud data, the dot product operation in the self-attention mechanism is changed to the shortest path calculation in the manifold space, which can more accurately construct the relative position and structure information between points in the point cloud, so as to help the model better understand the spatial relationship and geometric properties in the point cloud data. By introducing the grouping self-attention mechanism, the number of weight encoding parameters is reduced, which effectively prevents overfitting and enhances the ability of the model in capturing data features. The manifold grouping feature sampling module in the multi-modal target part detection method based on point cloud diversity representation and PointRCNN combines the manifold self-attention mechanism and the grouping self-attention mechanism, and improves the point cloud encoding network of PointRCNN, providing the PointRCNN algorithm with stronger three-dimensional spatial feature extraction capability.

[0094] Embodiment 1:

[0095] The present application designs a multi-modal target part detection method based on point cloud diversity representation and PointRCNN, which combines the cylinder representation and graph representation of point cloud data, fully utilizes the advantages of each to improve the overall feature expression ability of the algorithm, and combines the manifold self-attention mechanism to construct the relative position and structure information between points in the point cloud, finally improves the accuracy and efficiency of multi-modal target part detection, including the following steps:

[0096] 1) Through data preprocessing, the original point cloud data is converted into the data format required by the 2D backbone network;

[0097] 2) The data obtained in step 1) is input into the point cloud diversity representation module for processing, so as to utilize the advantages of cylinder representation and graph representation to improve the accuracy and efficiency of point cloud target part detection, and obtain diversified features;

[0098] 3) The data obtained in step 1) is input into the manifold grouping feature sampling module to effectively calculate the mutual relationship between the points inside the point cloud, and obtain multi-layer image features, so that the detection method can more accurately capture the context information and global features in the point cloud data. The manifold grouping feature sampling module not only improves the understanding ability of the algorithm for complex spatial structure, but also significantly enhances the generalization ability of the algorithm;

[0099] 4) After step 3), the output features of the point cloud diversity representation module (diversified features) and the output features of the manifold grouping feature sampling module (multi-layer image features) are input into the double-layer feature fusion module for fusion and enhancement processing; that is, the point cloud and image features are fused in the double-layer feature fusion module to obtain comprehensive feature representation, and on the basis of preliminary fusion, the semantic features of the point cloud are enhanced to obtain more rich feature expression, thereby improving the detection efficiency, and outputting multi-modal fusion features;

[0100] 5) The 3D backbone network is used to detect the target parts of the multi-modal fusion features output by the double-layer feature fusion module, and the detection result of the target parts is obtained through the 3D point cloud detection head in the 3D backbone network.

[0101] Embodiment 2:

[0102] This embodiment is further optimized on the basis of the above-mentioned embodiments, and the same as the previous technical solutions will not be repeated here. In order to better realize the multi-modal target part detection method based on point cloud diversity representation and PointRCNN described in the present application, the following setting method is particularly adopted: the 2D backbone network in step 1) uses a pre-trained Swin-Transformer model as the detection model (backbone network) of the 2D backbone network for image feature extraction, so as to fully exert the advantages of self-attention; an important difference between RGB images and laser radar point cloud data is that the former can provide rich color and texture information, which helps to distinguish different target part categories; in this method, the multi-level semantic features in the RGB image are integrated into the point cloud features when the 2D backbone network pre-processes the data. Compared with the traditional convolutional neural network, the self-attention mechanism more effectively captures the long-range dependency between any two positions in the image, and through exploring the correlation of the whole image, the global information and context of the image are collected; in order to further enrich the feature expression, a feature pyramid network structure is introduced into the 2D backbone network, and an additional pooling layer is added on the basis of the original four layers, thereby generating five layers of image features, represented as F image I 1 I 2 I 3 I 4 I 5 ​​​​​The algorithm employs pooling operations on the top-level features to obtain the fifth layer of features, thereby introducing features at more scales and further enhancing the detection algorithm's ability to perceive target parts. In the feature pyramid stage, the number of input channels for each layer is set to 96, 192, 384, and 768, while the number of output channels is fixed at 96. This configuration allows the algorithm to effectively extract features at different scales, enabling it to capture fine-grained and deep information from the input image, with the image resolution set to 2048×618 pixels.

[0103] Example 3:

[0104] This embodiment is a further optimization based on any of the above embodiments. The similarities with the aforementioned technical solutions will not be repeated here. Furthermore, to better realize the multimodal target part detection method based on point cloud diversity representation and PointRCNN described in this invention, the following configuration is specifically adopted: the point cloud diversity representation module in step 2) combines the advantages of cylindrical representation and graph representation, such as... Figure 1 As shown, while preserving detailed information, flexible resolution is provided. Dynamic column feature encoding and dynamic graph network encoding are used to process the data output in step 1). The dynamic graph network encoding consists of a dynamic graph convolutional network, which further extracts graph encoding features through three feature extraction layers. Each feature extraction layer consists of a graph convolutional module. A feature aggregation layer is added in the third feature extraction layer to aggregate the graph encoding features of each column, thereby obtaining the aggregated point cloud graph encoding features.

[0105] Dynamic cylinder feature encoding can effectively reduce data dimensionality, thereby saving computational resources. Unlike traditional methods, dynamic cylinder feature encoding does not sample point clouds into a fixed number of cylinders of fixed volume, but rather preserves the complete mapping relationship between points and cylinders: the original point cloud data is represented as P = {p1, p2, p3, ..., p...} n}, where p i ={(x i ,y i ,z i )|i=1,2,3,...,n},(x i ,y i ,z i Let ) represent the three-dimensional coordinates of point i, and r represent the reflectivity intensity. Dividing the point cloud space into three-dimensional cylindrical spaces, each cylinder has a specified size [Δx, Δy, Δz]. A three-dimensional cylinder can then be represented by the following formula:

[0106]

[0107] in, denotes an integer operation, (i, j, k) denotes the center point coordinates of a three-dimensional cylinder, for each center point of the cylinder, the point cloud data is mapped into the corresponding cylinder, and the average value of the point cloud in each cylinder is calculated to obtain the corresponding cylinder value At the same time, this process generates the cylinder feature F of the point cloud pillar = {f p1 ,f p2 ,f p3 …f pn}.

[0108] The dynamic graph network coding can preserve more point cloud information without losing details, and can process point cloud data of any shape, increasing the diversity of shapes. In addition, the dynamic graph network has stronger representation ability and scalability, and can extract higher-level semantic information, thereby improving the accuracy of target part detection. The dynamic graph network coding in the point cloud diversity representation module further extracts and aggregates the point cloud graph coding feature F pillar based on the cylinder feature F g ′ raph of the point cloud. Multi-layer feature extraction can help the module to learn the hierarchical feature representation of the point cloud data step by step, and each layer focuses on capturing different levels of abstract features in the point cloud, so that the final representation is more rich and expressive; with the increase of the number of layers, the receptive field of each layer also expands accordingly, so that the module can better understand the global and local point cloud structure in the learning process, which helps to better capture the overall shape and context information of the target part.

[0109] In addition, the multi-layer feature extraction in the dynamic graph network coding also introduces more nonlinear transformations, which helps the module to learn the complex relationships in the point cloud data. For the target part detection task, a deeper feature extraction network is often needed to better simulate the complexity of the point cloud data. For each cylinder, the graph coding feature F graph = (f g1 ,f g2 ,f g3 ,...,f gj ) is extracted using the dynamic graph network, where f gj represents the graph-based feature of the jth cylinder. Let the parameters of the neural network be Θ, then the feature extraction process can be represented by the following formula:

[0110] f gj = f Θ (F pj ,{f pi} i∈N(j) );

[0111] where F pj represents all point cloud data contained in the jth cylinder, and {fpi} i∈N(j) The feature representation of other column bodies adjacent to the jth column body is represented, and N(j) is the neighborhood set of the jth column body, which captures the local relationship and similarity between nodes in the graph structure by using the k-neighbor algorithm. The present application combines discriminant k-neighbor and fuzzy k-neighbor algorithms in different levels to improve the adaptability of the algorithm to different data distribution and features, and adds a feature aggregation layer in the third layer to obtain the graph encoding feature F graph is aggregated to obtain the aggregated point cloud graph encoding feature F g ′ raph .

[0112] Preferably, the point cloud diversity representation module introduces a residual structure to reduce network parameters, accelerate network training, and improve module performance and efficiency. The dynamic graph network coding can compensate for the limitations of voxelization to some extent. However, the graph coding representation may introduce higher computational complexity and memory requirements. Therefore, the method introduces residual connection in the point cloud diversity representation module to reduce the number of parameters and memory occupation of the module, thereby alleviating the problem of gradient disappearance and speeding up the training speed of the network. The column features F pillar and the point cloud graph encoding features F g ′ raph are processed to finally obtain the diversified features F diversity output by the point cloud diversity representation module, as shown in the following formula:

[0113] F diversity = Res(F pillar ,F g ′ raph );

[0114] Wherein, Res(·) represents the residual connection operation.

[0115] Preferably, the column encoding input of the point cloud diversity representation module is composed of 17600 points sampled from four dimensions, and the range of the point cloud is [0, -40, -3, 70.4, 40, 1]. The dynamic column feature coding alignment is used for coding, and the column size is set to [0.05, 0.05, 0.1]. The module integrates a feature fusion layer and uses five layers of image feature mapping, wherein the image channel is 256, the point cloud channel is 64, and the output channel is 128.

[0116] Preferably, the graph encoding of the point cloud diversity representation module extracts features from four dimensions of the point cloud, uses three graph feature extraction layers, each layer contains a grouping operation, each group contains 20 samples, the first layer adopts the discriminative k-nearest neighbor algorithm, and the other two layers use the fuzzy k-nearest neighbor algorithm, and the hyperparameter of the activation function is set to 0.2.

[0117] Embodiment 4:

[0118] This embodiment is further optimized on the basis of any of the above embodiments, and the same as the foregoing technical solutions will not be repeated here. In order to better realize the multi-modal target part detection method based on point cloud diversity representation and PointRCNN, the following setting mode is particularly adopted: the manifold grouping feature sampling module in the step 3) can unitize the input feature vector f of each point in the point cloud to obtain the feature representation mapped into the manifold space, so as to use the self-attention mechanism in the manifold space, and the calculation formula is described as follows:

[0119]

[0120] where ||·|| represents the Frobenius norm, represents dividing the input feature vector f by its Frobenius norm, thereby realizing the unitization of the input feature vector f, so that the length of each vector is 1. Through this process, the input feature vector of each point in the point cloud can be mapped to a unit length sphere, realizing the mapping of the point cloud in the manifold space;

[0121] In the manifold grouping feature sampling module, when the point cloud is mapped to the manifold space, the shortest path between them can be determined by the included angle represented by the two point vectors. By calculating the included angle value between these unit vectors and using the softmax function for normalization, the manifold self-attention score can be finally obtained, and the formula is as follows:

[0122]

[0123] where E(p i ) represents the point set around point p i , x j is a point in E(p i ), n represents the dimension of the attention query vector Q and the attention key vector K, V represents the attention value vector, diag(·) represents the diagonal value of the matrix, and represents the element-wise product of two matrices, f i attn represents the output feature obtained after the manifold attention weighting calculation of the input point p i .

[0124] Embodiment 5:

[0125] The embodiment is further optimized on the basis of any of the above embodiments, and the same as the foregoing technical solutions will not be described here. To better implement the multi-modal target part detection method based on point cloud diversity representation and PointRCNN, the following setting mode is particularly adopted: the manifold grouping feature sampling module in the step 3) introduces a grouping self-attention mechanism, which reduces the number of parameters and the computational complexity of the module by grouping the feature channels and sharing the attention weights within the group, and alleviates the overfitting phenomenon. The grouping self-attention mechanism inherits the advantages of vector attention and multi-head attention, and can more effectively learn deep feature representation, thereby improving the processing ability and generalization ability of the module for complex data. In the grouping self-attention, the channels of the value vector are evenly divided into k groups, and 1≤k≤c. Therefore, the output of the weight encoding layer is not a complete attention vector of c channels, but a grouped attention vector of k channels, where the channels of each group share a scalar attention weight, and the grouping linear transformation function τ is:

[0126]

[0127] where rel is the relationship vector between the attention query vector Q and the attention key vector K, p1, p2, …, p k are parameters of the grouping linear transformation, where each point p i is a grouping matrix used for independent transformation of the input relationship vector rel; the zero matrix is used to isolate different groups in the transformation. In order to enhance the role of position information in the attention mechanism, an additional position encoding multiplier is introduced to improve the influence of position information on weight encoding, so that the module can better learn and understand the relative positions between points in three-dimensional space, thereby improving its understanding and processing ability of complex spatial structures.

[0128] The manifold grouping feature sampling module normalizes, activates and further linearly transforms each group of relationship vectors rel that has undergone the grouping linear transformation function τ, promotes information exchange between groups, and improves the nonlinear expression ability of the module, and finally forms the grouped weight encoding function of the relationship vector rel as:

[0129] ω(rel)=Norm(Relu(Linear(ζ mul (p i -p j )⊙τ(rel)+ζ bias (p i -p j ));

[0130] where p i and p jp j and p lc / k+m are the spatial point coordinates and the neighboring point coordinates, respectively, ζ mul and ζ bias are two position encoding functions implemented by fully connected layers, which are used to enhance the understanding and representation ability of the module for the complex spatial relationship in the point cloud data, Linear(·) is a linear transformation layer implemented by a fully connected layer, which is responsible for further feature conversion between groups; Relu(·) is an activation function, which is used to introduce nonlinearity to better learn complex features; Norm(·) is a normalization layer, which is used to speed up the training and improve the stability of the module, and the final calculation formula of the manifold grouping self-attention feature vector is as follows:

[0131]

[0132] where l represents the group index, V j lc / k+m represents the mth feature in the lth group in the value vector grouping. By applying the grouping attention mechanism, the weighted and aggregated context information of the spatial point coordinates p i is realized, f i attn not only contains the information of the spatial point coordinates p i itself, but also fuses the information of the surrounding neighborhood, significantly improving the efficiency and representation ability of the module, thereby more effectively extracting features in sparse point clouds.

[0133] The process of the manifold grouping self-attention mechanism in the manifold grouping feature sampling module is shown in Figure 2 , f i and f j represent the spatial point coordinate features and the neighboring point coordinate features, respectively, which are converted into query vectors Q, key vectors K and value vectors V through convolution operations, p i and p j are the spatial point coordinates and the neighboring point coordinates, respectively, Δp represents the spatial coordinate difference value of the two points, the included angle value between these unit vectors is calculated, and the softmax function is used for normalization, and finally the manifold self-attention score can be obtained; The grouping operation sub-module is used to perform grouping self-attention operation on the value vector V, so that it corresponds to the relationship vector which is also grouped, and finally the feature vector based on the manifold grouping self-attention is calculated.

[0134] The overall structure of the manifold grouping feature sampling module is shown in Figure 3As shown, the module receives original point cloud data, embeds original features into a higher-dimensional feature space through a patch embedding method, fuses scattered information in the original point cloud data, generates high-level and discriminative features, so that the subsequent module can learn more effectively. The high-dimensional features after patch embedding are captured through a multi-layer manifold grouping self-attention sub-module to capture the relationship between each point in the point cloud, encode the spatial structure features of each point, and extract global features through a max-pooling layer to obtain the overall context information of the point cloud data. The global features will be combined with the local features and encoded again through the manifold grouping self-attention sub-module to obtain the comprehensive features of each point. This combination method not only retains the attributes of each point itself, but also fuses the spatial information and context information learned from the data, significantly enhancing the feature expression ability of the algorithm for point cloud data.

[0135] The manifold grouping feature sampling module uses the manifold grouping self-attention mechanism to map the features to the manifold space and obtain the spatial correlation of the point cloud, thereby improving the learning ability of the method for point cloud features. At the same time, the grouping self-attention mechanism reduces the number of weight encoding parameters, effectively prevents overfitting, and enhances the generalization ability of the algorithm. The point cloud encoding network in the manifold grouping feature sampling module is mainly composed of multiple down-sampling layers and up-sampling layers. Each down-sampling layer is composed of a point cloud sampling layer and a manifold grouping self-attention sub-module. In the point cloud sampling layer, the farthest point sampling algorithm is used to obtain the key points in the point set, and then based on a fixed radius, the local domain is formed, and other points close to the sampling points are grouped together. The up-sampling layer performs accurate original point mapping by using 3D linear interpolation to maximize the retention of the detailed features of the point cloud data. In order to effectively maintain these detailed information, the output of the up-sampling layer is cascaded with the corresponding down-sampling layer through a skip connection. This structure design enables the manifold grouping feature sampling module to effectively combine low-level features and high-level features, thereby enhancing the learning and processing ability of the module for detailed information.

[0136] Embodiment 6

[0137] This embodiment is further optimized on the basis of any of the above embodiments, and the same as the foregoing technical solutions will not be repeated here. In order to better implement the multi-modal target part detection method based on point cloud diversity representation and PointRCNN described in the present application, the following setting method is particularly adopted: the double-layer feature fusion module in step 4) is divided into a point cloud branch and an image branch, as shown in Figure 4 The point cloud branch transforms and aggregates the diversified features in step 2), and the image branch aligns and transforms the multi-layer image features in step 3), and finally fuses the features aggregated by the point cloud branch and the image branch.

[0138] In the point cloud branch, the diversified features obtained by the point cloud diversity representation module are transformed using a fully connected layer and normalization processing to obtain unified point cloud features and represent them as wherein represents the 3D feature of the nth point; then the unified point cloud features are grouped and aggregated using a K-dimensional tree method to obtain the point cloud features F pc based on their local regions; the point cloud features F pc are spatially divided; for each local region of the point cloud features F pc , the feature vectors within the region are averaged to obtain a single feature vector representing the region; the new feature vector of each point cloud feature F p is determined by the feature vector of the corresponding local region to which it belongs, and the calculation process is as follows:

[0139] F ′ c = Mean(KD(F pc ));

[0140] wherein F p ′ c is the point cloud feature after grouping and aggregation, and Mean(·) represents the mean aggregation method;

[0141] In the image branch, the multi-layer image features obtained from the manifold grouping feature sampling module are aligned and transformed to obtain image feature representations with the same dimension as the point cloud features F pc in the point cloud branch; the multi-layer image features are aligned with the point cloud features F pc by the alignment submodule, the extracted multi-layer image features and the point cloud features F pc are normalized, and the nearest point iteration algorithm is used to align the multi-layer image features with the point cloud features F pc and optimize their spatial relationship; the aligned multi-layer image features are obtained by convolution operation and enhancement transformation to obtain image features F img ;

[0142] Then the image features F img are grouped and aggregated; for each image feature F img , its surrounding image features are found using radius neighborhood search, the found surrounding image features are aggregated, and the features within the neighborhood are aggregated into a representative feature using the pooling operation; the aggregated features are fused with the initial image features F img to form new image features F i ′ mg ;

[0143] Finally, the grouped and aggregated point cloud features F p′ c With new image features F i ′ mg Fusion yields multimodal fusion features F fusion .

[0144] The dual-layer feature fusion module is as follows: Figure 5 As shown, the features after aggregating the point cloud branch and the image branch are fused using a sparse 3D-U-shaped network to fuse the initially fused multimodal fusion features F. fusion To achieve finer-grained processing, skip connection features are used to directly pass low-level features to the high-level decoder, accelerating the training process. Simultaneously, features from previous layers are directly passed, preserving rich local information and spatial structure. Semantic and spatial features are then used to enhance these features, resulting in better target part detection results. Multimodal fusion features F... fusion The input is fed into the 3D-U-shaped network to further obtain the spatial features F. spatial and semantic features F semantic These two features are combined to obtain the final fused feature F. final As shown in the following formula:

[0145] F final =Concat(F spatial ,Conv(F semantic ));

[0146] Here, Concat(·) represents the feature concatenation operation, and Conv(·) is used to expand features to the same dimension for concatenation using a sparse 3D-U-shaped network.

[0147] Example 7:

[0148] This embodiment is a further optimization based on any of the above embodiments. The similarities with the aforementioned technical solutions will not be repeated here. Furthermore, to better realize the multimodal target part detection method based on point cloud diversity representation and PointRCNN described in this invention, the following configuration is specifically adopted: In step 5), the 3D backbone network uses the concept of spatial autocorrelation as a measure of the spatial dispersion of the point cloud. Spatial autocorrelation describes the influence of random variable values ​​at different locations in space on the values ​​of random variables at their neighboring locations, exhibiting a certain spatial trend. Data with high spatial autocorrelation indicates a certain degree of clustering and trend; conversely, data with low spatial autocorrelation is more dispersed and may be interfering noise. The 3D backbone network determines whether the point cloud is discrete by calculating the spatial autocorrelation between each point and its neighboring points, and removes point clouds with low spatial autocorrelation, thereby reducing the number of point clouds while retaining key information.

[0149] Each point cloud data is represented as (x, y, z, r) in Cartesian coordinate system, and the spatial autocorrelation algorithm converts the point cloud data set from Cartesian coordinate system to spherical coordinate system to obtain distance r and pitch angle θ, azimuth angle for calculating the weight value and spatial autocorrelation value between points; the specific calculation formula is:

[0150]

[0151] After inputting the original point cloud data, the entire point cloud space will be divided into multiple grids, and the spatial autocorrelation value of each grid is calculated respectively.

[0152] Since different points in space have different contributions to the calculation of spatial autocorrelation value, the angle information between points can be used as the relative position information of data points in three-dimensional space. The 3D backbone network measures the relevance between two points by calculating the spatial angle distance w ij between two points; the greater the relevance between two points, the greater the weight given, and the spatial angle distance w ij The calculation formula is:

[0153]

[0154] Where i represents the processing point, and j represents the surrounding point of the processing point. In the spatial autocorrelation measurement, autocorrelation is used to evaluate the spatial relationship between a point and other points, and the relative position of the point to itself is always zero, so when i=j, w ij is set to zero, and the calculation formula of the spatial autocorrelation value I in each grid is:

[0155]

[0156] Where, is the average value of all distances, is the sum of all spatial angle distances w ij , and N represents the number of point clouds in a grid. After obtaining the spatial autocorrelation value of each grid, the grid points with I less than the threshold value are removed according to the set threshold value, thereby obtaining the point cloud data after removing noise points and irrelevant points, so that the subsequent 3D point cloud detection head can better focus on feature detection of the target part.

[0157] Example 8:

[0158] The embodiment is further optimized on the basis of any of the above embodiments, and the same as the foregoing technical solutions will not be described here. In order to better realize the multi-modal target part detection method based on point cloud diversity representation and PointRCNN, the following setting mode is particularly adopted: the 3D point cloud detection head in the 3D backbone network in the step 5) uses the following bounding box encoding function for the detection object:

[0159]

[0160] Wherein, x, y, z are the center coordinates; w, l, h are the width, length and height respectively; theta is the yaw angle around the z axis; x gt and x a are the ground truth and anchor box respectively; is the diagonal line of the anchor box bottom side.

[0161] The loss function of the 3D backbone network in the step 5) is provided with: a foreground and background point classification loss function, an interval-based positioning loss function, and a total loss function in the refinement stage.

[0162] Since the number of foreground points (target object points) and background points (non-target object points) is unbalanced in point cloud processing, the number of background points is much larger than that of foreground points, so the foreground and background point classification loss function adopts a focal loss function to reduce the weight of negative samples in training:

[0163] Wherein, p represents the foreground prediction probability of each point, and alpha and gamma are hyperparameters of the focal loss function. The parameters of the algorithm are set to be the same as those of the standard model PointRCNN;

[0164] In order to more accurately capture the position of the object center from the discrete point cloud, the 3D backbone network divides each point in the point cloud into a series of discrete intervals along the X and Z axes, which can more accurately locate the center of the object. The search range on the X and Z axes is evenly divided into intervals with a uniform length of eta, which is used to represent the position of different object centers on the X-Z plane. The discrete interval of the foreground point on the dimension u e {x, z, theta} is calculated Wherein, u p is the real center coordinate of the object, and u (p) is the real coordinate of the corresponding foreground point. Through the above process, the continuous position is converted into discrete category information to determine that the foreground point belongs to a certain interval on the X and Z axes respectively.

[0165] In order to refine the exact position of the foreground point in the interval, the residual is calculated by the following formula:

[0166]

[0167] where C is a constant for normalization, ensuring that the residual calculation is not affected by the size of the interval; considering that most objects have little change in the center position of the vertical direction Y axis, the real center coordinate y p The difference between the foreground point coordinate y (p) and the residual on the Y axis is calculated:

[0168]

[0169] Based on the above formula, the interval-based target bounding box position regression loss and size regression loss can be obtained:

[0170]

[0171] where, is the predicted discrete interval of the foreground point in dimension u∈{x,z,θ}, the cross-entropy loss function is used to measure the consistency of the predicted interval and the real interval, and the smooth L1 loss function is used to reduce the difference between the predicted residual and the real residual. Combining and , the final interval-based positioning loss function is:

[0172]

[0173] where pos represents the set of positive sample points, N pos represents the number of positive sample points.

[0174] In order to obtain more accurate target part detection frame, the 3D backbone network needs to ensure that the network can correctly classify each detected object and accurately predict the object's bounding box position and size in the candidate frame refinement stage. The total loss function of the refinement stage is:

[0175]

[0176] where, represents the total number of sample points in a processing batch, represents the total number of positive sample points in a processing batch, prob is the predicted label, and label represents the real label, and correspond to the position regression and size regression of the refined candidate frame, respectively. This method allows the 3D backbone network to process continuous attributes and discretize them, which can take advantage of the simplicity of classification tasks and retain the accuracy of regression tasks, thereby better predicting the detection frame of the target part.

[0177] Embodiment 9:

[0178] The embodiment is further optimized on the basis of any of the above embodiments, and the same parts as the foregoing technical solutions will not be described again. To better implement the multi-modal target part detection method based on point cloud diversity representation and PointRCNN, the following setting method is particularly adopted: the positioning loss of the 3D point cloud detection head is defined as:

[0179] L loc =∑ b∈(x,y,z,w,l,h,θ) SmoothL1(Δb);

[0180] Since the angle positioning loss cannot distinguish the flipped bounding box, the Softmax classification loss L loc is used to learn the discrete direction, and the object classification loss using the focal loss is:

[0181] L cls =-α a (1-p a ) γ log p a ;

[0182] wherein p a represents the class probability of the anchor point, a takes the default value 0.25, γ takes the default value 3, the loss function is optimized using the AdamW optimizer, and is decayed by 0.1 times every 15 periods.

[0183] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Any simple modification or equivalent change made according to the technical essence of the present application to the above embodiment falls within the protection scope of the present application.

Claims

1. A multimodal target part detection method based on point cloud diversity representation and PointRCNN, characterized in that: Comprising the following steps: 1) converting the original point cloud data into the data format required by the 2D backbone network through data preprocessing; 2) inputting the data obtained in step 1) into the point cloud diversity representation module for processing to obtain diversified features; 3) inputting the data obtained in step 1) into the manifold grouping feature sampling module to calculate the mutual relationship between the points inside the point cloud, and obtain multi-layer image features; The manifold grouping feature sampling module can sample the input feature vector of each point in the point cloud Unit vectorization is performed to obtain a feature representation mapped into a manifold space, so as to use a self-attention mechanism in the manifold space. The calculation formula is described as follows: ； wherein, denotes Frobenius a norm, denotes dividing the input feature vector by its Frobenius norm, thereby achieving unitization of the input feature vector such that the length of each vector is 1; In the manifold grouping feature sampling module, when the point cloud is mapped to the manifold space, the shortest path between them is determined by the included angle of two-point vector representation, the included angle value between these unit vectors is calculated, and the function is used for normalization, and finally the manifold self-attention score is obtained, and the formula is formed as follows: softmax The function is used for normalization, and finally the manifold self-attention score is obtained, and the formula is formed as follows: ； wherein representative point a set of points around is a point in representative attention query vector Q with attention key vector K dimension of V representative attention value vector representative diagonal value of extraction matrix representative element-wise product of two matrices representative input point output feature obtained after manifold attention weighting calculation The manifold grouping feature sampling module introduces a grouping self-attention mechanism, in which the channels of the value vector are evenly divided into groups, and where the channels of each group share a scalar attention weight, and the grouping linear transformation function is: ； wherein, is an attention query vector Q is a relationship vector between the attention key vector K ; is a parameter of a group linear transformation, where each point is a group matrix for independently transforming the input relationship vector ; a zero matrix is used to isolate different groups in the transformation; 4) after step 3), input the output features of the point cloud diversity representation module and the output features of the manifold grouping feature sampling module into the double-layer feature fusion module for fusion and enhancement processing, and output multi-modal fusion features; 5) using a 3D backbone network to detect the multi-modal fusion features output by the double-layer feature fusion module to obtain the detection result of the target part through the 3D point cloud detection head in the 3D backbone network.

2. The multi-modal target part detection method based on point cloud diversity representation and PointRCNN according to claim 1, characterized in that: The 2D backbone network in the step 1) uses a pre-trained Swin-Transformer model as a detection model of the 2D backbone network for image feature extraction; the 2D backbone network integrates multi-level semantic features in the RGB image into point cloud features when pre-processing data; a feature pyramid network structure is introduced in the 2D backbone network, and an additional pooling layer is added on the basis of the original four layers, so as to generate five layers of image features, represented as The fifth layer of features is obtained by performing a pooling operation on the top layer of features.

3. The multi-modal target part detection method based on point cloud diversity representation and PointRCNN according to claim 1, characterized in that: The point cloud diversity representation module in step 2) combines columnar representation and graph representation, and uses dynamic columnar feature encoding and dynamic graph network encoding to process the data output from step 1); the dynamic columnar feature encoding represents the original point cloud data as... ,in , Indicates the first The three-dimensional coordinates of a point Reflectivity intensity is represented by dividing the point cloud space into three-dimensional cylindrical spaces, each cylinder having a specified size. A three-dimensional cylinder can be represented by the following formula: ； wherein, denotes an integer operation, denotes the center point coordinates of the three-dimensional cylinder, for each center point of the cylinder, the point cloud data is mapped into the corresponding cylinder, and the average value of the point cloud in each cylinder is calculated to obtain the corresponding cylinder value , and this process generates the cylinder features of the point cloud ; Dynamic graph network coding on the basis of cylinder features Further extraction and aggregation of point cloud graph coding features are obtained through three feature extraction layers .

4. The multi-modal target part detection method based on point cloud diversity representation and PointRCNN according to claim 1, characterized in that: The double-layer feature fusion module in step 4) is divided into a point cloud branch and an image branch, wherein the point cloud branch transforms and aggregates the diversified features in step 2), and the image branch aligns and transforms the multi-layer image features in step 3), and finally fuses the features aggregated by the point cloud branch and the image branch; In the point cloud branch, the diverse features obtained through the point cloud diversity representation module are transformed using a fully connected layer and normalization to obtain unified point cloud features, which are then represented as follows: ,in Indicates the first 3D features of individual points; then, the unified point cloud features are grouped and aggregated, using... The dimensional tree method is based on the feature of point clouds in their local regions. Perform spatial partitioning; for point cloud features For each local region, the feature vectors within that region are averaged to obtain a single feature vector representing that region; each point cloud feature... The new feature vector is determined by the feature vector of its corresponding local region, and the calculation process is shown in the following formula: ； wherein, is a point cloud feature after grouping aggregation, denotes a mean aggregation method; In the image branch, the multi-layer image features obtained from the manifold grouping feature sampling module are aligned and transformed to obtain image features of the same dimension as the point cloud features in the point cloud branch; the multi-layer image features are aligned with the point cloud features by an alignment submodule , the extracted multi-layer image features and the point cloud features are normalized , and the multi-layer image features are aligned with the point cloud features using a nearest point iteration algorithm , and their spatial relationship is optimized, and the aligned multi-layer image features are obtained through convolution operation and enhancement transformation to obtain image features ; Then the image features are grouped and aggregated For each image feature , its surrounding image features are found using a radius neighborhood search, the found surrounding image features are aggregated, the features within the neighborhood are aggregated into one representative feature using a pooling operation, the aggregated feature is fused with the initial image feature to form a new image feature ; The point cloud features aggregated after grouping the packets with the new image features fusion, obtaining a multi-modal fusion feature .

5. The multi-modal target part detection method based on point cloud diversity representation and PointRCNN according to claim 1, characterized in that: The 3D backbone network in the step 5) adopts the concept of spatial autocorrelation as a measure of the spatial dispersion degree of the point cloud, the 3D backbone network judges whether the point cloud is dispersed by calculating the spatial correlation between each point and its nearby points, and the point cloud with low spatial autocorrelation is removed, and each point cloud data is expressed as in the Cartesian coordinate system, the spatial autocorrelation algorithm converts the point cloud data set from the Cartesian coordinate system to the spherical coordinate system, and obtains the distance and the pitch angle and the azimuth angle , which are used to calculate the weight value and the spatial autocorrelation value between points; the specific calculation formula is: ； ； 。 6. The multi-modal target part detection method based on point cloud diversity representation and PointRCNN according to claim 1, characterized in that: The loss function of the 3D backbone network in step 5) is provided with: a foreground and background point classification loss function, an interval-based positioning loss function, and a total loss function in the refinement stage; The foreground and background point classification loss function uses a focal loss function to reduce the weight of negative samples in training: ; wherein, represents the foreground prediction probability of each point, and is a hyperparameter of the focal loss function; The interval-based positioning loss function is: ; wherein, represents a set of positive sample points; represents the number of positive sample points; is an interval-based target bounding box position regression loss, and ; is an interval-based target bounding box size regression loss, and , is a predicted discrete interval of a foreground point in dimension , and ; wherein, is a real center coordinate of an object, is a real coordinate of a corresponding foreground point, is a search range on and axes, is a uniform interval length; is a residual error of a predicted discrete interval of a foreground point and a real position, and wherein, is a constant for normalization, and the residual error on the axis is calculated by a difference between the real center coordinate of the object and the foreground point coordinate : ; a cross-entropy loss function is used to measure the consistency of the predicted discrete interval and the real interval; is a smooth L1 loss function used to reduce the difference between the predicted residual error and the real residual error; The total loss function of the refinement stage is: ; wherein, represents the total number of sample points of a processing batch, represents the total number of positive sample points in a processing batch, is a predicted label, represents a true label, and respectively correspond to the position regression and size regression of the refined candidate box generated.

7. The multi-modal target part detection method based on point cloud diversity representation and PointRCNN according to claim 1, characterized in that: The positioning loss of the 3D point cloud detection head is defined as: ； and a Softmax classification loss learning discrete directions, object classification loss using focal loss: ； wherein, class probability representing an anchor point, takes the default value 0.25, takes the default value 3, the loss function is optimized using the AdamW optimizer and is decayed by a factor of 0.1 every 15 periods.

Images