A Multi-View Collaborative Cross-Modal 3D Perception Model Training Method Based on Data Filtering and Optimization

Abstract

A multi-view collaborative cross-modal 3D perception model training method based on data filtering optimization belongs to the field of cross-modal learning in computer vision. This invention addresses the problems of poor data quality, large inter-domain differences, and spatial information loss in existing cross-modal learning methods. The method includes: filtering the original point cloud data to obtain filtered point cloud data; obtaining a set of multi-view rendering images for each filtered point cloud data set; training the point cloud encoder using the filtered point cloud data; training the multi-view encoder using the multi-view rendering images; aligning the encoded point cloud features, encoded multi-view features, and encoded text features to a shared representation space; minimizing contrast loss to make the multi-modal features of the same object as close as possible in the shared representation space, while keeping the multi-modal features of different objects separated in the shared representation space, and adjusting the network parameters. This invention can be applied to the optimization and improvement of 3D perception technology.

Description

Technical Field

[0001] This invention relates to a multi-view collaborative cross-modal 3D perception model training method based on data filtering and optimization, belonging to the field of cross-modal learning in computer vision. Background Technology

[0002] With the rapid development of autonomous driving, augmented reality / virtual reality (AR / VR), intelligent manufacturing, robotics, and other fields, 3D visual perception technology, as one of the core technologies, has been widely applied to tasks such as environmental perception, object recognition, and spatial modeling. With the rapid development of multimodal learning and 3D perception technology, how to effectively utilize multimodal information such as vision and language to improve the accuracy and stability of 3D perception models has become a critical issue that urgently needs to be addressed. To solve this problem, an effective approach is to apply existing successful vision-language (VL) pre-trained models to 3D vision tasks.

[0003] Currently, mainstream cross-modal learning methods for 3D perception can be broadly categorized as follows:

[0004] Traditional manual feature extraction methods process 3D data by manually designing features (such as surface normals and curvature). While this can achieve some results in simple scenarios, its generalization ability is poor, and it cannot effectively handle noise and incomplete data in complex environments. Therefore, this method performs poorly when dealing with large-scale and complex scenarios and cannot meet the needs of big data and multimodal data fusion.

[0005] Deep learning-based point cloud processing methods: With the development of deep learning, point cloud-based deep neural networks (such as PointNet, PointNet++, DGCNN, etc.) have become the mainstream methods in the field of 3D perception. These methods can effectively capture the geometric features of point clouds by directly learning from point cloud data, thus improving the robustness and accuracy of the model. However, these methods rely on large-scale labeled datasets, and the currently available 3D datasets are limited in size and cannot cover all complex scenes, which restricts the generalization ability of the model.

[0006] Cross-modal learning methods: In recent years, cross-modal learning has made significant progress in fields such as computer vision and natural language processing. In particular, visual-language pre-trained models such as CLIP (Contrastive Language-Image Pretraining) have improved the generalization ability of models and achieved effective alignment of cross-modal features by combining large-scale image and text data. These methods have been successfully applied to tasks such as image classification and image generation. Therefore, to address the problem of insufficient training data in the pre-training of deep learning-based point cloud processing methods, an effective approach is to apply existing successful visual-language (VL) pre-trained models (CLIP) to 3D vision tasks. Because these pre-trained models are trained on massive amounts of data and possess rich prior knowledge, applying them to the 3D domain allows the use of this prior knowledge, thus reducing the reliance on large-scale 3D datasets.

[0007] Typical examples of cross-modal learning methods include:

[0008] PointCLIP: PointCLIP projects points from 3D point cloud data into a 2D image and then uses CLIP for feature analysis. Although this method successfully introduces a vision-language model into the 3D domain, it loses some spatial information during the point cloud data conversion process, thus failing to fully leverage the advantages of point cloud data.

[0009] CLIP2Point: CLIP2Point generates a 2D depth map through multi-view rendering, and then aligns it using a depth encoder and CLIP's image encoder. Although the rendering process retains some geometric information, it often ignores some spatial characteristics of the 3D point cloud due to its reliance on feature extraction from the depth map, resulting in the loss of spatial information.

[0010] ULIP: ULIP directly uses a 3D encoder and clip for alignment, aiming to align 3D data with language and image features through a unified cross-modal learning framework. However, when processing 3D point clouds, ULIP suffers from a significant gap between low-resolution point cloud data and 2D visual priors, resulting in limited feature alignment effectiveness.

[0011] OpenShape: OpenShape is a state-of-the-art cross-modal learning framework. Its training framework is very similar to ULIP, but its performance is significantly better. OpenShape's significant advantage lies in its use of a larger dataset, enabling the training of superior models. However, the quality of its dataset remains an issue, and there are significant inter-domain differences between 3D data and 2D visual priors (such as CLIP), limiting further performance improvements.

[0012] Therefore, although cross-modal learning has made good progress in 3D perception tasks,

[0013] While cross-modal learning has made good progress in 3D perception tasks, it still faces the following major challenges:

[0014] 1. Data quality and generalization issues: Existing datasets suffer from low data quality, especially some user-generated 3D datasets, which lack sufficient accuracy and consistency. This prevents the model from obtaining high-quality feature inputs during training, affecting the model's training performance and generalization ability.

[0015] 2. Cross-modal alignment problem: There are significant inter-domain differences between 3D point cloud data and modalities such as images and text. Most existing mainstream methods project or render 3D data into 2D images and then use CLIP models for analysis. However, the projection or rendering process loses 3D spatial information, negating the advantages of point cloud data. Alternatively, a 3D encoder can be used to directly align point cloud features with CLIP features, but the low resolution of 3D data and the significant inter-domain differences between real images render the two-dimensional prior of CLIP ineffective. Summary of the Invention

[0016] To address the problems of poor data quality, large inter-domain differences, and loss of spatial information in existing cross-modal learning methods, this invention provides a multi-view collaborative cross-modal 3D perception model training method based on data filtering and optimization.

[0017] The present invention provides a multi-view collaborative cross-modal 3D perception model training method based on data filtering and optimization, comprising:

[0018] Acquire 3D datasets, and obtain multiple sets of raw point cloud data for various objects from the 3D datasets;

[0019] Flattened point cloud data are removed from each group of original point cloud data to obtain filtered point cloud data groups; each filtered point cloud data in the filtered point cloud data group is rendered to obtain a set of multi-view rendering images for each filtered point cloud data.

[0020] A point cloud encoder is used to encode each filtered point cloud data in the filtered point cloud data group to obtain the encoded point cloud features.

[0021] A multi-view encoder is used to encode each set of multi-view rendering images to obtain the encoded multi-view features.

[0022] Simultaneously, a text encoder is used to encode the text corresponding to the filtered point cloud data to obtain encoded text features; the encoded point cloud features, encoded multi-view features, and encoded text features constitute multimodal features;

[0023] The multimodal features are aligned to a shared representation space through contrastive learning; the contrastive loss is minimized to make the multimodal features of the same object as close as possible in the shared representation space, and to keep the multimodal features of different objects separate in the shared representation space; the network parameters of the point cloud encoder and the multiview encoder are adjusted based on the contrastive loss, and then the next set of original point cloud data is processed. The point cloud encoder and the multiview encoder are iteratively trained until the end, and the trained point cloud encoder and the trained multiview encoder are obtained.

[0024] According to the data filtering and optimization-based multi-view collaborative cross-modal 3D perception model training method of the present invention, the flattening of point cloud data in each group of original point cloud data includes RANSAC in-point percentage evaluation for each group of original point cloud data:

[0025] Using RANSAC, a principal plane is fitted based on a set of original point cloud data, and the percentage P of interior points of each original point cloud data relative to the principal plane is calculated. inlier :

[0026]

[0027] In the formula, N is the total number of points in each original point cloud data, and I(·) is an indicator function that returns 1 if the condition is met, and 0 otherwise; (x i ,y i ,z i ) represents the three-dimensional coordinates of the i-th point in the original point cloud data, (a,b,c,d) are all plane fitting parameters, and ∈ is the threshold for the distance from the point to the principal plane;

[0028] Set the interior point percentage threshold to τ. If:

[0029] P inlier >τ,

[0030] The original point cloud data is then identified as flat point cloud data and removed, resulting in point cloud data after one removal.

[0031] According to the multi-view collaborative cross-modal 3D perception model training method based on data filtering optimization of the present invention, the removal of flattened point cloud data from each group of original point cloud data also includes calculating the projected area of ​​each point cloud data after one removal:

[0032] A proj =Area(ConvHull(P proj )),

[0033] In the formula A proj P represents the projected area of ​​the point cloud data onto the principal plane after one round of culling. proj ConvHull represents the set of projected points of the point cloud data onto the principal plane after one round of culling. proj The convex hull is the smallest closed region of the two-dimensional projection point;

[0034] P proj ={(x′,y′)∣(x′,y′,z′)=Projection(x,y,z)},

[0035] In the formula, (x,y,z) represents the three-dimensional coordinates of the point cloud data after one removal, (x′,y′,z′) represents the projected coordinates of (x,y,z) on the principal plane, and z′=0; Projection represents the projection operation;

[0036] Set the projection area threshold τ A ,like:

[0037] A proj >τ A ,

[0038] The point cloud data after the first removal is then determined to be flat point cloud data and removed, resulting in the point cloud data after the second removal.

[0039] According to the multi-view collaborative cross-modal 3D perception model training method based on data filtering optimization of the present invention, the removal of flattened point cloud data from each group of original point cloud data also includes ring object recognition from the removed original point cloud data:

[0040] Calculate the average distance from each projection point of each removed original point cloud data point in the principal plane to the center of all projection points.

[0041]

[0042] In the formula, M represents the total number of projected points for each original point cloud data after removal, and P represents the total number of projected points. m For the m-th projection point, P j Let j be the j-th projection point; ||·|| is the Euclidean distance;

[0043] Based on average Calculate the variance σ of the distance from the projection point to the center of the projection point. 2 :

[0044]

[0045] Set the filtering threshold τ for distance variance σ ,like:

[0046] σ 2 <τ σ ,

[0047] The corresponding original point cloud data that has been removed is retained to obtain the retained point cloud data.

[0048] According to the multi-view collaborative cross-modal 3D perception model training method based on data filtering optimization of the present invention, the set of point cloud data after secondary removal and the set of retained point cloud data are used as the filtered point cloud data group, and F is adopted. w This represents the initial feature set of the w-th data point in each set of original point cloud data, with dimension c. n w This represents the total number of data points in each group of original point cloud data; using P w This represents the three-dimensional coordinates of the w-th data point in each set of original point cloud data.

[0049] Will be on (P) w ,F w The removal of flattened point cloud data is represented as (P) w ,F w ) * The filtered point cloud data is represented as follows In the formula This represents the initial feature set of the u-th filtered point cloud data obtained from each set of original point cloud data. Represents the 3D coordinates of the u-th filtered point cloud data obtained from each set of original point cloud data:

[0050]

[0051] in

[0052] In the formula This means that the value is 0 if the condition in parentheses is true, and 1 otherwise.

[0053] According to the data filtering and optimization-based multi-view collaborative cross-modal 3D perception model training method of the present invention, the training datasets of the point cloud encoder, multi-view encoder, and text encoder are represented as follows:

[0054]

[0055] In the formula T u Let represent the text corresponding to the u-th filtered point cloud data. Then, the u-th training data sample is represented as...

[0056] According to the multi-view collaborative cross-modal 3D perception model training method based on data filtering optimization of the present invention, a point cloud encoder is used to encode each filtered point cloud data in the filtered point cloud data group to obtain the encoded point cloud feature representation as f. p :

[0057]

[0058] In the formula, PointEncoder represents a point cloud encoder.

[0059] According to the multi-view collaborative cross-modal 3D perception model training method based on data filtering optimization of the present invention, a set of multi-view rendering maps obtained by rendering each filtered point cloud data is represented as follows:

[0060]

[0061] In the formula I V This represents the Vth rendered image, and Render() represents the rendering function, used to render based on... Generate rendered images from V specified viewpoints; V represents the number of specified viewpoints.

[0062] Use Vision Transformer to extract the feature g of each rendered image. v :

[0063] g v =ViT(I v ), v = 1, 2, 3, ... V,

[0064] In the formula, ViT represents Vision Transformer;

[0065] For all rendered graph features g v The encoded multi-view features f are generated using an aggregation method based on a self-attention mechanism, where v = 1, 2, 3, ... V. i :

[0066]

[0067] In the formula W q To query the projection matrix, W k W is the key projection matrix. Q The projection matrix is ​​the value. This is the scaling factor.

[0068] According to the multi-view collaborative cross-modal 3D perception model training method based on data filtering optimization of the present invention, a text encoder is used to encode the text corresponding to the filtered point cloud data to obtain the encoded text feature representation as f. t :

[0069] f t = CLIPTextEncoder(T u ), u=1,2,…,U,

[0070] In the formula, CLIPTextEncoder represents the text encoder of the CLIP model.

[0071] According to the data-filtered optimization-based multi-view collaborative cross-modal 3D perception model training method of the present invention, the total contrastive loss function is expressed as L:

[0072] L = L p-t +L i-t ,

[0073] In the formula L p-t For point cloud-text contrast loss, L i-t For multi-view image-text contrast loss;

[0074]

[0075] In the formula f pu Let f be the encoded point cloud feature corresponding to the u-th filtered point cloud data. tu f represents the encoded text feature corresponding to the u-th filtered point cloud data. tJ f represents the encoded text feature corresponding to the J-th filtered point cloud data. iu Let τ0 be the encoded multi-view feature corresponding to the u-th filtered point cloud data, where τ0 is the temperature coefficient.

[0076]

[0077] The beneficial effects of this invention are as follows: The method of this invention is designed for learning from 3D point cloud data and can be applied to the optimization and improvement of 3D perception technology, including fields such as autonomous driving, augmented / virtual reality, robot navigation, and intelligent manufacturing. By effectively optimizing dataset quality, achieving collaborative fusion of multi-view information, and performing cross-modal alignment based on a vision-language pre-trained model, it effectively overcomes the shortcomings of existing technologies, such as poor data quality, large inter-domain differences, and loss of spatial information.

[0078] This invention comprises two parts: a data augmentation strategy for 3D datasets and a multi-view collaborative cross-modal learning algorithm based on a vision-language pre-trained model. The first part, through a selection and optimization mechanism, significantly improves the quality of the 3D point cloud dataset, providing a more accurate and consistent data foundation for model training. The second part combines multi-view feature fusion with point cloud feature encoding, using both multi-view rendered images and point clouds to represent 3D information. Cross-modal comparative learning is then performed, and the optimized data is used for training, achieving efficient alignment of point cloud, image, and text features in a unified representation space, thus improving the performance and robustness of 3D perception tasks. These two parts work together, from data optimization to model learning, to construct a complete 3D perception solution.

[0079] The method of this invention combines data filtering and multi-view collaborative cross-modal learning based on a vision-language pre-trained model, demonstrating excellent performance in multimodal data fusion and 3D point cloud classification.

[0080] The data filtering process is efficient and accurate: it can automatically filter point clouds, avoiding the need for a large amount of manpower to judge data quality; the filtering steps can effectively preserve complex geometric structures (such as ring objects) rather than simply removing flat and abnormal point clouds, thus achieving precise preservation of geometric shapes.

[0081] In multi-view collaborative cross-modal learning, a 3D representation closer to 2D is obtained through a multi-view encoder, which, together with point cloud features, represents 3D objects. This approach preserves 3D spatial information (point cloud encoder) and bridges the inter-domain differences between the low resolution of 3D data and real images (multi-view encoder). It effectively utilizes the geometric information of point clouds and image information from different perspectives to capture rich 3D features. Attached Figure Description

[0082] Figure 1 This is a flowchart for filtering raw point cloud data;

[0083] Figure 2 This is a training block diagram of the multi-view collaborative cross-modal 3D perception model training method based on data filtering and optimization described in this invention. Detailed Implementation

[0084] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0085] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0086] The present invention will be further described below with reference to the accompanying drawings, but this should not be construed as limiting the invention.

[0087] Specific Implementation Method 1: Combination Figure 1 and Figure 2 As shown, this invention provides a multi-view collaborative cross-modal 3D perception model training method based on data filtering and optimization, including:

[0088] Acquire 3D datasets, and obtain multiple sets of raw point cloud data for various objects from the 3D datasets;

[0089] Flattened point cloud data are removed from each group of original point cloud data to obtain filtered point cloud data groups; each filtered point cloud data in the filtered point cloud data group is rendered to obtain a set of multi-view rendering images for each filtered point cloud data.

[0090] A point cloud encoder is used to encode each filtered point cloud data in the filtered point cloud data group to obtain the encoded point cloud features.

[0091] A multi-view encoder is used to encode each set of multi-view rendering images to obtain the encoded multi-view features.

[0092] Simultaneously, a text encoder is used to encode the text corresponding to the filtered point cloud data to obtain encoded text features; the encoded point cloud features, encoded multi-view features, and encoded text features constitute multimodal features;

[0093] The multimodal features are aligned to a shared representation space through contrastive learning; the contrastive loss is minimized to make the multimodal features of the same object as close as possible in the shared representation space, and to keep the multimodal features of different objects separate in the shared representation space; the network parameters of the point cloud encoder and the multiview encoder are adjusted based on the contrastive loss, and then the next set of original point cloud data is processed. The point cloud encoder and the multiview encoder are iteratively trained until the end, and the trained point cloud encoder and the trained multiview encoder are obtained.

[0094] Furthermore, data augmentation strategies for 3D datasets:

[0095] Drawing on the performance advantages of OpenShape, the performance of deep learning models is significantly constrained by the quantity and quality of training data. However, much of the current dataset consists of data uploaded by internet users without expert verification, resulting in inconsistent quality and issues such as distorted point cloud shapes. This implementation method optimizes point cloud data quality through 3D data augmentation strategies, ensuring higher accuracy and consistency in model training data, thereby improving the performance of 3D perception models. Data augmentation includes RANSAC inlier percentage evaluation, projected area calculation, and ring recognition.

[0096] Combination Figure 1 As shown, the process of flattening point cloud data for each set of original point cloud data includes evaluating the percentage of in-point points in RANSAC for each set of original point cloud data:

[0097] RANSAC inlier percentage assessment is a key step in identifying flat point clouds. This method, based on the Random Sample Consensus (RANSAC) algorithm, determines the geometric complexity of a point cloud by fitting a principal plane and calculating the proportion of inliers. For 3D point cloud data, flat point clouds typically exhibit small height variations, simple distribution, and a lack of complex geometric characteristics. These point clouds often fail to effectively represent the shape features of objects and therefore need to be eliminated in the initial screening. RANSAC is a commonly used model fitting method, with the advantage of robustly fitting the optimal model to noisy point cloud data. For each point cloud object, RANSAC is used to fit the principal plane, and the inlier percentage metric quantifies the degree of fit between the point cloud and the plane, thereby filtering out flat point clouds as candidates for unacceptable point clouds.

[0098] Using RANSAC, a principal plane is fitted based on a set of original point cloud data, and the percentage P of interior points of each original point cloud data relative to the principal plane is calculated. inlier This represents the proportion of points within a certain height threshold of the plane:

[0099]

[0100] In the formula, N is the total number of points in each original point cloud data, and I(·) is an indicator function that returns 1 if the condition is met, and 0 otherwise; (x i ,y i ,z i ) represents the 3D coordinates of the i-th point in the original point cloud data, (a,b,c,d) are all plane fitting parameters, and ∈ is the threshold for the distance from the point to the principal plane, used to determine interior points;

[0101] Set the interior point percentage threshold to τ. If:

[0102] P inlier >τ,

[0103] The original point cloud data is then identified as flat point cloud data and removed, resulting in point cloud data after one removal.

[0104] As an example, the interior point percentage threshold τ can be set to 0.8 or 80%.

[0105] The process of flattening point cloud data for each set of original point cloud data also includes calculating the projected area of ​​each point cloud data after one round of flattening:

[0106] This implementation projects point cloud data onto the fitted principal plane, calculates the projected area, and determines whether to retain the point cloud based on an area threshold. The reason for this is that, based on the RANSAC in-point percentage evaluation, a projected area calculation method is introduced to further eliminate invalid point clouds with flat shapes but large area footprints. Flat point clouds typically exhibit small height variations and simple geometric shapes, but these point clouds can be divided into two categories: one is invalid terrain point clouds, which often have large projected areas; the other is small objects with flat shapes but belonging to the valid category, such as scissors or lollipops, which have small projected areas. Therefore, the RANSAC in-point percentage alone cannot effectively distinguish between these two types of point clouds; further calculation of their projected areas is necessary for differentiation.

[0107] A proj =Area(ConvHull(P proj )),

[0108] In the formula A proj The area represents the projected area of ​​the point cloud data onto the principal plane after one round of culling, obtained by calculating the convex hull area of ​​the point cloud; Area represents the area, P proj ConvHull represents the set of projected points of the point cloud data onto the principal plane after one round of culling. proj The convex hull is the smallest closed region of the two-dimensional projection point. The size of the projection area is used to determine whether the point cloud meets the shape requirements.

[0109] P proj ={(x′,y′)∣(x′,y′,z′)=Projection(x,y,z)},

[0110] In the formula, (x,y,z) represents the three-dimensional coordinates of the point cloud data after one removal, (x′,y′,z′) represents the projected coordinates of (x,y,z) on the principal plane, and z′=0; Projection represents the projection operation;

[0111] Set the projection area threshold τ A , indicating the upper limit of the filtering conditions, used to filter out point clouds with excessively large projected areas. If:

[0112] A proj>τ A ,

[0113] The point cloud data after the first removal is then determined to be flat point cloud data and removed, resulting in the point cloud data after the second removal.

[0114] The process of flattening point cloud data for each set of original point cloud data also includes identifying ring objects in the removed original point cloud data:

[0115] After two rounds of screening, a large number of flat, large-area point clouds that do not meet the requirements have been removed. However, some special valid objects may still be mistakenly screened, such as ring-shaped objects like bracelets and belts. These objects, due to their flat shape and large convex hull area, may be incorrectly labeled as invalid point clouds during the removal process. However, such objects often have important classification value in practical tasks, thus requiring further screening and retention. This implementation introduces a ring-shaped object recognition mechanism to effectively identify these special point clouds, ensuring the integrity and accuracy of the dataset. In this process, the geometric distribution of projection points is calculated to determine whether an object is ring-shaped. Ring-shaped objects typically have circular or annular geometric features; therefore, they are identified by calculating the variance of the distance from the geometric center of the projection points to the center.

[0116] Calculate the average distance from each projection point of each removed original point cloud data point in the principal plane to the center of all projection points.

[0117]

[0118] In the formula, M represents the total number of projected points for each original point cloud data after removal, and P represents the total number of projected points. m For the m-th projection point, P j Let be the j-th projection point; ||·|| is the Euclidean distance, also known as the L2 norm;

[0119] Based on average Calculate the variance σ of the distance from the projection point to the center of the projection point. 2 :

[0120]

[0121] Set the filtering threshold τ for distance variance σ ,like:

[0122] σ 2 <τ σ ,

[0123] The corresponding original point cloud data that has been removed is retained to obtain the retained point cloud data.

[0124] This implementation combines three judgment conditions: interior point ratio, projected area, and ring features to ensure the quality optimization of point clouds in various scenarios, thereby enhancing the adaptability of point clouds in complex tasks.

[0125] Furthermore, the set of point cloud data after the second round of removal and the set of retained point cloud data are used as the filtered point cloud data group, and F is adopted. w This represents the initial feature set of the w-th data point in each set of original point cloud data, with dimension c. n w This represents the total number of data points in each group of original point cloud data; using P w This represents the three-dimensional coordinates of the w-th data point in each set of original point cloud data. Initial feature set F w Including normal vectors, color, etc.;

[0126] Will be on (P) w ,F w The removal of flattened point cloud data is represented as (P) w ,F w ) * The filtered point cloud data is represented as follows In the formula This represents the initial feature set of the u-th filtered point cloud data obtained from each set of original point cloud data. Represents the 3D coordinates of the u-th filtered point cloud data obtained from each set of original point cloud data:

[0127]

[0128] in

[0129] In the formula This means that the value is 0 if the condition in parentheses is true, and 1 otherwise.

[0130] Multi-view collaborative cross-modal learning based on vision-language pre-trained models:

[0131] This implementation method is based on a vision-language pre-trained model and multi-view feature fusion, and completes cross-modal learning of 3D point clouds by training two encoders. Specifically, the point cloud encoder is used to extract 3D point cloud features, and the multi-view encoder is used to process 2D image features rendered from multiple perspectives as supplementary 3D features. A contrastive learning method is used to align these two encoders with CLIP's text encoder, so that the feature vectors output by the two encoders are aligned in a joint representation space, thereby improving the model's ability to perceive complex tasks.

[0132] The input for model training includes two modalities of data: 3D point cloud data and text descriptions.

[0133] The training datasets for the point cloud encoder, multi-view encoder, and text encoder are represented as follows:

[0134]

[0135] In the formula T u Let represent the text corresponding to the u-th filtered point cloud data. Then, the u-th training data sample is represented as... U represents the total number of filtered point cloud data in the filtered point cloud data group.

[0136] Furthermore, combining Figure 2 As shown, a point cloud encoder is used to encode each filtered point cloud data in the filtered point cloud data group, and the encoded point cloud feature representation is f. p :

[0137]

[0138] In the formula, PointEncoder represents a point cloud encoder, which can be any point cloud encoder, such as PointNet, PointNet++, Transformer, etc.

[0139] For each filtered point cloud data, multi-view rendering is performed, and then the features of each image are extracted using Vision Transformer (ViT).

[0140] The set of multi-view rendering images obtained from each filtered point cloud data rendering is represented as follows:

[0141]

[0142] In the formula I V This represents the Vth rendered image, and Render() represents the rendering function, used to render based on... Generate rendered images from V specified viewpoints; V represents the number of specified viewpoints.

[0143] Use Vision Transformer to extract the feature g of each rendered image. v :

[0144] g v =ViT(I v ), v = 1, 2, 3, ... V,

[0145] In the formula, ViT represents Vision Transformer;

[0146] For all rendered graph features g vThe encoded multi-view features f are generated using an aggregation method based on a self-attention mechanism, where v = 1, 2, 3, ... V. i That is, the global features after multi-view aggregation:

[0147]

[0148] In the formula W q To query the projection matrix, W k W is the key projection matrix. Q The projection matrix is ​​the value. This is a scaling factor used to stabilize numerical values.

[0149] A text encoder is used to encode the text corresponding to the filtered point cloud data, resulting in the encoded text feature representation f related to the point cloud. t :

[0150] f t = CLIPTextEncoder(T u ), u=1,2,…,U,

[0151] In the formula, CLIPTextEncoder represents the text encoder of the CLIP model.

[0152] In this embodiment, point cloud features, multi-view features, and text features are aligned to a shared representation space through contrastive learning. By minimizing the contrastive loss, different modal features (point cloud, multi-view, and text) of the same object are made as close as possible in the shared representation space, while the modal features of different objects remain separate.

[0153] The total contrastive loss function is expressed as L:

[0154] L = L p-t +L i-t ,

[0155] L is the loss that needs to be minimized throughout the entire training process, where L p-t For point cloud-text contrast loss, L i-t For multi-view image-text contrast loss;

[0156]

[0157] In the formula f pu Let f be the encoded point cloud feature corresponding to the u-th filtered point cloud data. tu f represents the encoded text feature corresponding to the u-th filtered point cloud data. tJ f represents the encoded text feature corresponding to the J-th filtered point cloud data. iuLet τ0 be the encoded multi-view feature corresponding to the u-th filtered point cloud data, and let τ0 be the temperature coefficient used to control the smoothness of the distribution.

[0158]

[0159] This invention introduces a self-attention mechanism into the multi-view feature aggregation process. By calculating the similarity between viewpoint features and assigning weights, it intelligently aggregates multi-view information. The self-attention mechanism captures complementary information between features from different viewpoints, solving the feature redundancy problem that may be caused by simple mean pooling or weighted averaging. The aggregated global features can more comprehensively represent the geometric shape and semantic information of the point cloud.

[0160] By optimizing the point cloud-text contrast loss and the image-text contrast loss, a unified representation of multimodal features is achieved.

[0161] Example: The Objaverse dataset was selected as the original dataset. Objaverse is a multimodal 3D object dataset containing rich 3D point clouds and textual descriptions. Due to its large data volume, it can train models with good generalization capabilities. However, the data in the Objaverse dataset are all manually modeled and uploaded, and many point cloud data do not meet standards, exhibiting flat or distorted shapes (mostly terrain point clouds), which greatly affects the quality of the dataset and the training effect of the model. After data augmentation of the Objaverse dataset using the method of this invention, the augmented point clouds are used with CLIP to train a multi-view collaborative cross-modal learning algorithm based on a vision-language pre-trained model, constructing an intelligent model that can be used for augmented reality (AR) scene navigation. The model can achieve object recognition, semantic annotation, and interactive applications with the real world.

[0162] For example, in cultural sites or museums, visitors can capture target objects, such as statues or artifacts, using their mobile phones or AR glasses. The system will then identify the objects in real time and generate relevant semantic descriptions with high accuracy.

[0163] The specific steps are as follows:

[0164] Step 1: RANSAC in-point percentage assessment;

[0165] First, the RANSAC algorithm is used to fit the principal plane of the point cloud. Through iterative random sampling and model optimization, the best-fit plane is obtained even with a complex point cloud distribution. During the fitting process, the RANSAC algorithm automatically removes outliers and generates planar model parameters and a set of interior points.

[0166] Next, the percentage of inliers in the point cloud is calculated based on the fitting results. By setting a distance threshold, it is determined whether each point is an inlier, and the ratio of the number of inliers that meet the criteria to the total number of points in the point cloud is calculated. This ratio is used to measure whether the point cloud meets the preset shape requirements.

[0167] In this step, the robustness of the RANSAC algorithm can adapt to the noise and distribution characteristics of different point cloud data. Even when the point cloud data is noisy or unevenly distributed, it can stably generate a planar model and interior point set that conforms to geometric rules.

[0168] Step 2: Calculate the projected area;

[0169] To further improve the filtering accuracy of point cloud data, based on RANSAC plane fitting, abnormal point clouds that do not meet the shape requirements are efficiently removed by calculating the convex hull area after the point cloud is projected onto the principal plane.

[0170] The specific implementation method is as follows: First, the points in the point cloud selected by the RANSAC algorithm are projected onto the fitted principal plane to form a two-dimensional projected point set. Then, the minimum bounding region of this point set is generated using the convex hull algorithm in computational geometry, and the area of ​​this convex hull region is calculated as the projected area of ​​the point cloud.

[0171] Step 3: Identification and Retention of Circular Objects;

[0172] To ensure that valid point clouds with special geometric features (such as ring-shaped objects) are not mistakenly filtered out during the data filtering process, a method for ring-shaped object identification and retention is used. This method accurately identifies ring-shaped objects by analyzing the geometric distribution of projection points and retains them in the dataset.

[0173] Specifically, the process involves: First, calculating the geometric center of the point cloud projection point set after projection area filtering. Then, calculating the distance variance of the point cloud based on the distance distribution from the projection points to the geometric center. For ring-shaped objects, the distance distribution between their projection points and the geometric center typically has a larger variance, while the variance of the distance distribution for non-ring-shaped objects is relatively smaller.

[0174] This method effectively preserves data on ring-shaped objects such as bracelets and tires, improving data diversity and completeness without introducing additional noise, and providing richer geometric feature support for subsequent model training. This step compensates for the shortcomings of traditional geometric screening methods that may miss objects with special shapes, further optimizing the overall quality of the dataset.

[0175] Next, we will train the multi-view collaborative cross-modal learning algorithm:

[0176] Point cloud feature extraction;

[0177] In this step, PointBERT is used as a point cloud encoder to extract features from the input 3D point cloud data. PointBERT is a point cloud feature extraction network based on the Transformer architecture, designed with a different approach from traditional convolutional neural networks, specifically for handling irregular and unordered point cloud data. Specifically, PointBERT first embeds the 3D point cloud data, mapping the spatial coordinates of each point to a high-dimensional space. Then, multiple Transformer encoder layers process the point cloud data, capturing the spatial relationships and geometric features between points. Unlike other methods, PointBERT's advantage lies in its ability to adaptively model the contextual information of each point in the point cloud through a self-attention mechanism, effectively extracting local and global geometric features regardless of the point cloud's distribution or density. In this way, PointBERT can generate point cloud feature vectors with rich geometric and spatial information, providing crucial geometric information support for subsequent cross-modal learning tasks such as alignment with multi-view image and text features. PointBERT is used as an example here, but any point cloud encoder can be used in practice.

[0178] Multi-view rendering and feature extraction:

[0179] The input 3D point cloud data is rendered from multiple perspectives to generate several 2D projected images from different angles. By observing the same object from different viewpoints, such as front, side, and top views, various appearance information of the object is captured. The rendered 2D images contain rich visual information, which helps the model understand the shape and structure of the object.

[0180] The image from each viewpoint is converted into a feature vector, representing the visual features of the object from that viewpoint. Then, the feature vectors from all viewpoints are fused through a self-attention mechanism to comprehensively consider the local information from each viewpoint and generate a global, multi-view feature representation.

[0181] Text feature extraction:

[0182] Features of textual descriptions related to point cloud data are extracted using a pre-trained visual-language model (CLIP). These textual descriptions typically include semantic information such as object categories and attributes. The core purpose of this process is to facilitate the alignment of geometric and visual features extracted by the point cloud encoder and multi-view encoder with textual features during subsequent contrastive learning, thereby providing consistent semantic support for subsequent cross-modal learning.

[0183] In this step, the main role of contrastive learning is to align point cloud features, multi-view features, and text features into a shared representation space. Through contrastive learning, the model is trained to make different modalities (point cloud, image, and text) from the same object as close as possible in this space, while features from different objects are pushed further apart. This process ensures the semantic and geometric consistency between point cloud and image features and text descriptions by maximizing the similarity between features of the same modality while minimizing the similarity between features of different objects.

[0184] Finally, the trained model is applied in practice: After training, the model can be deployed to a VR device for recognition tasks. In the recognition process, features are extracted using a point cloud encoder and a multi-view image encoder, then weighted and fused together, and matched with text information in a text library to achieve a comprehensive multimodal match.

[0185] The fused multimodal features are represented as f combined :

[0186] f combined =w p ·f p +w i ·f i ,

[0187] In the formula w p w i Let w be the weighting coefficient. p +w i =1.

[0188] In summary, this invention proposes a complete learning framework from point cloud optimization to multimodal alignment, which is highly modular and flexible. The framework can adapt to various task scenarios, such as 3D object classification and cross-modal retrieval, and different modules can be expanded and replaced to meet different needs. Specifically, the point cloud optimization part ensures high-quality input data; the feature extraction module performs multimodal feature extraction on point clouds, multi-view images, and text; the multi-view aggregation module introduces a self-attention mechanism to enhance the feature complementarity between viewpoints; and the contrastive learning part realizes a unified representation of multimodal features.

[0189] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A multi-view collaborative cross-modal 3D perception model training method based on data filtering and optimization, characterized in that, include, Acquire 3D datasets, and obtain multiple sets of raw point cloud data for various objects from the 3D datasets; Flattened point cloud data is removed from each group of original point cloud data to obtain filtered point cloud data groups; Render each filtered point cloud data in the filtered point cloud data group to obtain a set of multi-view rendering images for each filtered point cloud data. A point cloud encoder is used to encode each filtered point cloud data in the filtered point cloud data group to obtain the encoded point cloud features. A multi-view encoder is used to encode each set of multi-view rendering images to obtain the encoded multi-view features. Simultaneously, a text encoder is used to encode the text corresponding to the filtered point cloud data to obtain encoded text features; the encoded point cloud features, encoded multi-view features, and encoded text features constitute multimodal features; The multimodal features are aligned to a shared representation space through contrastive learning; the contrastive loss is minimized to make the multimodal features of the same object as close as possible in the shared representation space, and to keep the multimodal features of different objects separate in the shared representation space; the network parameters of the point cloud encoder and the multiview encoder are adjusted based on the contrastive loss, and then the next set of original point cloud data is processed. The point cloud encoder and the multiview encoder are iteratively trained until the end, and the trained point cloud encoder and the trained multiview encoder are obtained.

2. The multi-view collaborative cross-modal 3D perception model training method based on data filtering and optimization according to claim 1, characterized in that, Flattening the point cloud data for each set of raw point cloud data includes evaluating the percentage of inliers in each set of raw point cloud data using RANSAC: Using RANSAC, a principal plane is fitted based on a set of original point cloud data, and the percentage P of interior points of each original point cloud data relative to the principal plane is calculated. inlier : In the formula, N is the total number of points in each original point cloud data, and I(·) is an indicator function that returns 1 if the condition is met, and 0 otherwise; (x i ,y i ,z i ) represents the three-dimensional coordinates of the i-th point in the original point cloud data, (a,b,c,d) are all plane fitting parameters, and ∈ is the threshold for the distance from the point to the principal plane; Set the interior point percentage threshold to τ. If: P inlier >t, The original point cloud data is then identified as flat point cloud data and removed, resulting in point cloud data after one removal.

3. The multi-view collaborative cross-modal 3D perception model training method based on data filtering and optimization according to claim 2, characterized in that, The process of flattening point cloud data for each set of original point cloud data also includes calculating the projected area of ​​each point cloud data after one round of flattening: A proj =Area(ConvHull(P proj )), In the formula A proj P represents the projected area of ​​the point cloud data onto the principal plane after one round of culling. proj ConvHull represents the set of projected points of the point cloud data onto the principal plane after one round of culling. proj The convex hull is the smallest closed region of the two-dimensional projection point; P proj ={(x′,y′)∣(x′,y′,z′)=Projection(x,y,z)}, In the formula, (x,y,z) represents the three-dimensional coordinates of the point cloud data after one removal, (x′,y′,z′) represents the projected coordinates of (x,y,z) on the principal plane, and z′=0; Projection represents the projection operation; Set the projection area threshold τ A ,like: A proj >t A , The point cloud data after the first removal is then determined to be flat point cloud data and removed, resulting in the point cloud data after the second removal.

4. The multi-view collaborative cross-modal 3D perception model training method based on data filtering and optimization according to claim 3, characterized in that, The process of flattening point cloud data for each set of original point cloud data also includes identifying ring objects in the removed original point cloud data: Calculate the average distance from each projection point of each removed original point cloud data point in the principal plane to the center of all projection points. In the formula, M represents the total number of projected points for each original point cloud data after removal, and P represents the total number of projected points. m For the m-th projection point, P j Let j be the j-th projection point; ||·|| is the Euclidean distance; Based on average Calculate the variance σ of the distance from the projection point to the center of the projection point. 2 : Set the filtering threshold τ for distance variance σ ,like: s 2 <t σ , The corresponding original point cloud data that has been removed is retained to obtain the retained point cloud data.

5. The multi-view collaborative cross-modal 3D perception model training method based on data filtering and optimization according to claim 4, characterized in that, The set of point cloud data after secondary removal and the set of retained point cloud data are used as the filtered point cloud data group, and F is used. w This represents the initial feature set of the w-th data point in each set of original point cloud data, with dimension c. n w This represents the total number of data points in each group of original point cloud data; using P w This represents the three-dimensional coordinates of the w-th data point in each set of original point cloud data. Will be on (P) w ,F w The removal of flattened point cloud data is represented as (P) w ,F w ) * The filtered point cloud data is represented as follows In the formula This represents the initial feature set of the u-th filtered point cloud data obtained from each set of original point cloud data. Represents the 3D coordinates of the u-th filtered point cloud data obtained from each set of original point cloud data: in In the formula This means that the value is 0 if the condition in parentheses is true, and 1 otherwise.

6. The multi-view collaborative cross-modal 3D perception model training method based on data filtering and optimization according to claim 5, characterized in that, The training datasets for the point cloud encoder, multi-view encoder, and text encoder are represented as follows: In the formula T u Let represent the text corresponding to the u-th filtered point cloud data. Then, the u-th training data sample is represented as...

7. The multi-view collaborative cross-modal 3D perception model training method based on data filtering and optimization according to claim 6, characterized in that, A point cloud encoder is used to encode each filtered point cloud data in the filtered point cloud data group, and the encoded point cloud feature representation is f. p : In the formula, PointEncoder represents a point cloud encoder.

8. The multi-view collaborative cross-modal 3D perception model training method based on data filtering and optimization according to claim 7, characterized in that, The set of multi-view rendering images obtained from each filtered point cloud data rendering is represented as follows: In the formula I V This represents the Vth rendered image, and Render() represents the rendering function, used to render based on... Generate rendered images from V specified viewpoints; V represents the number of specified viewpoints. Use Vision Transformer to extract the feature g of each rendered image. v : g v =ViT(I v ),v=1,2,3,…V, In the formula, ViT represents Vision Transformer; For all rendered graph features g v The encoded multi-view features f are generated using an aggregation method based on a self-attention mechanism, where v = 1, 2, 3, ... V. i : In the formula W q To query the projection matrix, W k W is the key projection matrix. Q The projection matrix is ​​the value. This is the scaling factor.

9. The multi-view collaborative cross-modal 3D perception model training method based on data filtering and optimization according to claim 8, characterized in that, The text corresponding to the filtered point cloud data is encoded using a text encoder to obtain the encoded text feature representation f. t : f t =CLIPTextEncoder(T u ),u=1,2,…,U, In the formula, CLIPTextEncoder represents the text encoder of the CLIP model.

10. The multi-view collaborative cross-modal 3D perception model training method based on data filtering and optimization according to claim 9, characterized in that, The total contrastive loss function is expressed as L: L=L p-t +L i-t , In the formula L p-t For point cloud-text contrast loss, L i-t For multi-view image-text contrast loss; In the formula f pu f is the encoded point cloud feature corresponding to the u-th filtered point cloud data. tu f represents the encoded text feature corresponding to the u-th filtered point cloud data. tJ f represents the encoded text feature corresponding to the J-th filtered point cloud data. iu Let τ0 be the encoded multi-view feature corresponding to the u-th filtered point cloud data, where τ0 is the temperature coefficient.

Images