Multi-view collaborative cross-modal 3D perception model training method based on data screening optimization

Abstract

The invention discloses a multi-view collaborative cross-modal 3D perception model training method based on data screening optimization, and belongs to the field of cross-modal learning in computer vision. The method aims at solving the problems that in an existing cross-modal learning method, the data quality is poor, the inter-domain difference is large, and space information is lost. Comprising the following steps: screening original point cloud data to obtain screened point cloud data; obtaining a group of multi-view rendering images of each screened point cloud data; training a point cloud encoder by adopting the screened point cloud data; training a multi-view-angle encoder by adopting the multi-view-angle rendering graph; aligning the encoded point cloud features, the encoded multi-view features and the encoded text features to a shared representation space; the comparison loss is minimized, the multi-modal features of the same object are made to be as close as possible in the shared representation space, the multi-modal features of different objects are made to be separated in the shared representation space, and network parameters are adjusted. The method can be applied to optimization and improvement of the three-dimensional perception technology.

Description

Technical Field

[0001] The present invention relates to a multi-view collaborative cross-modal 3D perception model training method based on data screening and optimization, and belongs to the field of cross-modal learning in computer vision. Background Art

[0002] With the rapid development of autonomous driving, augmented reality / virtual reality (AR / VR), intelligent manufacturing, robotics and other fields, three-dimensional (3D) visual perception technology, as one of the core technologies, has been widely used in tasks such as environmental perception, object recognition, and spatial modeling. With the rapid development of multimodal learning and 3D perception technology, how to effectively use multimodal information such as vision and language to improve the accuracy and stability of 3D perception models has become a key issue that needs to be solved. To solve this problem, an effective method is to apply the existing successful vision-language (VL) pre-training model to 3D vision tasks.

[0003] Currently, 3D perception technologies in mainstream cross-modal learning methods can be roughly divided into the following categories:

[0004] Traditional manual feature extraction method: 3D data is processed by manually designed features (such as surface normals, curvature, etc.). Although it can achieve certain effects in some simple scenarios, its generalization ability is poor, and it cannot effectively deal with noise and incomplete data in complex environments. Therefore, this method performs poorly when processing large-scale and complex scenes, and cannot adapt to the needs of big data and multimodal data fusion.

[0005] Point cloud processing methods based on deep learning: With the development of deep learning, point cloud-based deep neural networks (such as PointNet, PointNet++, DGCNN, etc.) have become the mainstream methods in the field of 3D perception. These methods can effectively capture the geometric features of point clouds by directly learning from point cloud data, improving the robustness and accuracy of the model. However, these methods rely on large-scale annotated datasets. The currently available 3D datasets are limited in size and cannot cover all complex scenes, resulting in limited generalization capabilities of the model.

[0006] Cross-modal learning methods: In recent years, cross-modal learning has made significant progress in fields such as computer vision and natural language processing. In particular, visual-language pre-training models such as CLIP (Contrastive Language-Image Pretraining) have improved the generalization ability of the model and achieved effective alignment of cross-modal features by combining large-scale image and text data. These methods have been successfully applied to tasks such as image classification and image generation. Therefore, in order to solve the problem of lack of sufficient training data in the pre-training of point cloud processing methods based on deep learning, an effective method is to apply the existing successful visual-language (VL) pre-training model (CLIP) to 3D vision tasks. Because this pre-training model has been trained with massive data and has rich prior knowledge, it can be applied to the 3D field. This prior knowledge can be used. This method can reduce dependence on large-scale 3D datasets.

[0007] Typical representatives of cross-modal learning methods include:

[0008] PointCLIP: PointCLIP projects the points of 3D point cloud data into 2D images and uses CLIP for feature analysis. Although this method successfully introduces the visual-language model into the 3D field, it loses some spatial information during the conversion of point cloud data, making it impossible to fully utilize the advantages of point cloud data.

[0009] CLIP2Point: CLIP2Point generates a 2D depth map through multi-view rendering, and then uses the depth encoder and CLIP's image encoder for alignment. Although the rendering process retains certain geometric information, due to the reliance on feature extraction of the depth map, the spatial characteristics of some 3D point clouds are often ignored, resulting in loss of spatial information.

[0010] ULIP: ULIP directly uses the 3D encoder to align with the clip, and its goal is to align 3D data with language features and image features through a unified cross-modal learning framework. However, when ULIP processes 3D point clouds, the gap between low-resolution point cloud data and 2D visual priors is large, resulting in limited feature alignment effects.

[0011] OpenShape: OpenShape is a recent cross-modal learning framework with the best results. Its training framework is very similar to ULIP, but its results are significantly better than ULIP. The significant advantage of OpenShape is that it uses a larger dataset and can train a better model. However, there are still problems with the quality of its dataset, and there are significant inter-domain differences between 3D data and 2D visual priors (such as CLIP), which restricts further performance improvement.

[0012] Therefore, although cross-modal learning has made good progress in 3D perception tasks,

[0013] In the existing technology, although cross-modal learning has made good progress in 3D perception tasks, the following major problems still exist:

[0014] 1. Data quality and generalization issues: Existing datasets have low data quality, especially some 3D datasets generated by users, which lack sufficient accuracy and consistency. This results in the inability to obtain high-quality feature inputs during model training, affecting the training effect and generalization ability of the model.

[0015] 2. Cross-modal alignment problem: There are large domain differences between 3D point cloud data and modalities such as images and text. Most of the existing mainstream methods project or render 3D data into 2D images and then use the CLIP model for analysis. However, 3D spatial information is lost during the projection or rendering process, which loses the advantages of point cloud data. Alternatively, a 3D encoder is used to directly align point cloud features with CLIP features. However, there are significant domain differences between the low resolution of 3D data and real images, making the two-dimensional prior of CLIP ineffective. Summary of the invention

[0016] In response to the problems of poor data quality, large inter-domain differences, and loss of spatial information in existing cross-modal learning methods, the present invention provides a multi-view collaborative cross-modal 3D perception model training method based on data screening and optimization.

[0017] The present invention provides a multi-view collaborative cross-modal 3D perception model training method based on data screening optimization, comprising:

[0018] Acquire a 3D data set, and obtain multiple sets of original point cloud data of various objects from the 3D data set;

[0019] The flat point cloud data of each group of original point cloud data are eliminated to obtain a filtered point cloud data group; each filtered point cloud data in the filtered point cloud data group is rendered to obtain a group of multi-view rendering images of each filtered point cloud data;

[0020] A point cloud encoder is used to encode each filtered point cloud data in the filtered point cloud data group to obtain encoded point cloud features;

[0021] A multi-view encoder is used to perform multi-view encoding on each group of multi-view rendering images to obtain encoded multi-view features;

[0022] At the same time, a text encoder is used to encode the text corresponding to the filtered point cloud data to obtain encoded text features; the encoded point cloud features, encoded multi-view features and encoded text features constitute multimodal features;

[0023] The multimodal features are aligned to a shared representation space through contrastive learning; the contrast loss is minimized to make the multimodal features of the same object as close as possible in the shared representation space, and the multimodal features of different objects are kept separated in the shared representation space; the network parameters of the point cloud encoder and the multi-view encoder are adjusted based on the contrast loss, and then the next set of original point cloud data is processed, and the point cloud encoder and the multi-view encoder are continued to be iteratively trained until the end, so as to obtain the trained point cloud encoder and the trained multi-view encoder.

[0024] According to the multi-view collaborative cross-modal 3D perception model training method based on data screening optimization of the present invention, the flat point cloud data of each group of original point cloud data is removed, including performing RANSAC internal point percentage evaluation on each group of original point cloud data:

[0025] Use RANSAC to fit the main plane based on a set of original point cloud data, and calculate the percentage of internal points P of each original point cloud data and the main plane inlier :

[0026]

[0027] Where N is the total number of points of each original point cloud data, I(·) is the indicator function, which returns 1 if the condition is met, otherwise returns 0; (x i ,y i ,z i ) is the 3D coordinate of the i-th point in the original point cloud data, (a, b, c, d) are all plane fitting parameters, ∈ is the threshold of the distance from the point to the main plane;

[0028] Set the internal point percentage threshold to τ, if:

[0029] P inlier >τ,

[0030] The corresponding original point cloud data is determined to be flat point cloud data and is eliminated to obtain point cloud data after elimination.

[0031] According to the multi-view collaborative cross-modal 3D perception model training method based on data screening optimization of the present invention, the elimination of flat point cloud data for each group of original point cloud data also includes calculating the projection area of ​​each point cloud data after elimination:

[0032] A proj =Area(ConvHull(P proj )),

[0033] Where A proj It represents the projection area of ​​the point cloud data on the main plane after one elimination. Area represents the area. P proj ConvHull represents the projection point set P after the point cloud data is removed. proj The convex hull of is the minimum enclosed area of ​​the two-dimensional projection points;

[0034] P proj ={(x′,y′)∣(x′,y′,z′)=Projection(x,y,z)},

[0035] Where (x, y, z) represents the three-dimensional coordinates of the point cloud data after one elimination, (x′, y′, z′) represents the projection coordinates of (x, y, z) on the main plane, z′=0; Projection represents the projection operation;

[0036] Set the projection area threshold τ A ,like:

[0037] A proj >τ A ,

[0038] The point cloud data after the first elimination is determined to be flat point cloud data and eliminated, thereby obtaining the point cloud data after the second elimination.

[0039] According to the multi-view collaborative cross-modal 3D perception model training method based on data screening optimization of the present invention, the removal of flat point cloud data from each group of original point cloud data also includes the recognition of ring objects on the removed original point cloud data:

[0040] Calculate the average distance from each projection point of each eliminated original point cloud data in the main plane to the center of all projection points

[0041]

[0042] Where M is the total number of projection points of each original point cloud data removed, P m is the mth projection point, P j is the jth projection point; ||·|| is the Euclidean distance;

[0043] Based on average Calculate the distance variance σ from the projection point to the center of the projection point 2 :

[0044]

[0045] Set the screening threshold τ of distance variance σ ,like:

[0046] σ 2 <τ σ ,

[0047] The corresponding removed original point cloud data is retained to obtain the retained point cloud data.

[0048] According to the multi-view collaborative cross-modal 3D perception model training method based on data screening optimization of the present invention, the set of point cloud data after secondary elimination and the set of retained point cloud data are used as the screened point cloud data set, and F w Represents the initial feature set of the wth data in each set of original point cloud data, with a dimension of c. n w Represents the total number of data in each group of original point cloud data; P w Represents the three-dimensional coordinates of the wth data in each set of original point cloud data,

[0049] Will (P w ,F w ) to remove the flat point cloud data is expressed as (P w ,F w ) * , the filtered point cloud data is expressed as In the formula represents the initial feature set of the u-th filtered point cloud data obtained from each set of original point cloud data, Represents the three-dimensional coordinates of the u-th filtered point cloud data obtained from each set of original point cloud data:

[0050]

[0051] in

[0052] In the formula It means that when the condition in the brackets is true, the value is 0; otherwise it is 1.

[0053] According to the multi-view collaborative cross-modal 3D perception model training method based on data screening optimization of the present invention, the training data set of the point cloud encoder, the multi-view encoder and the text encoder is expressed as

[0054]

[0055] In the formula T u represents the text corresponding to the u-th filtered point cloud data, then the u-th training data sample is expressed as

[0056] According to the multi-view collaborative cross-modal 3D perception model training method based on data screening optimization of the present invention, a point cloud encoder is used to encode each filtered point cloud data in the filtered point cloud data group, and the encoded point cloud feature representation is obtained as f p :

[0057]

[0058] Where PointEncoder represents the point cloud encoder.

[0059] According to the multi-view collaborative cross-modal 3D perception model training method based on data screening optimization of the present invention, a set of multi-view rendering images obtained by rendering each filtered point cloud data is represented as follows:

[0060]

[0061] Where I V represents the Vth rendered image, Render(·) represents the rendering function, which is used to Generates a rendering from V specified viewpoints; V represents the number of specified viewpoints;

[0062] Use Vision Transformer to extract features g for each rendering v :

[0063] g v =ViT(I v ), v = 1, 2, 3, ... V,

[0064] Where represents Vision Transformer;

[0065] For all rendering features g v , v = 1, 2, 3, ... V uses an aggregation method based on the self-attention mechanism to generate the encoded multi-view features f i :

[0066]

[0067] Where W q To query the projection matrix, W k is the key projection matrix, W Q is the value projection matrix, is the scaling factor.

[0068] According to the multi-view collaborative cross-modal 3D perception model training method based on data screening optimization of the present invention, a text encoder is used to encode the text corresponding to the screened point cloud data, and the encoded text feature representation is obtained as f t :

[0069] f t = CLIPTextEncoder(T u ),u=1,2,...,U,

[0070] Where CLIPTextEncoder represents the text encoder of the CLIP model.

[0071] According to the multi-view collaborative cross-modal 3D perception model training method based on data screening optimization of the present invention, the total contrast loss function is expressed as L:

[0072] L=L p-t +L i-t ,

[0073] Where L p-t is the point cloud-text contrast loss, L i-t is the multi-view image-text contrast loss;

[0074]

[0075] In the formula is the encoded point cloud feature corresponding to the u-th filtered point cloud data, is the encoded text feature corresponding to the u-th filtered point cloud data, is the encoded text feature corresponding to the Jth filtered point cloud data, is the encoded multi-view feature corresponding to the u-th filtered point cloud data, τ0 is the temperature coefficient;

[0076]

[0077] Beneficial effects of the invention: The method of the invention is oriented to learning from three-dimensional point cloud data and can be applied to the optimization and improvement of three-dimensional perception technology, including autonomous driving, augmented / virtual reality, robot navigation, intelligent manufacturing and other fields. By effectively optimizing the quality of the data set, realizing the coordinated fusion of multi-view information, and performing cross-modal alignment based on the visual-language pre-training model, the defects of the existing technology such as poor data quality, large inter-domain differences and loss of spatial information are effectively overcome.

[0078] The method of the present invention consists of two parts: a data enhancement strategy for 3D data sets and a multi-view collaborative cross-modal learning algorithm based on a visual-language pre-training model. The first part significantly improves the quality of 3D point cloud data sets through a screening optimization mechanism, providing a more accurate and consistent data basis for model training; the second part combines multi-view feature fusion with point cloud feature encoding, and uses multi-view renderings and point clouds to represent 3D information, and then conducts cross-modal comparative learning, using optimized data for training, achieving efficient alignment of point cloud, image and text features in a unified representation space, and improving the performance and robustness of 3D perception tasks. The two parts of the method work together to build a complete 3D perception solution from data optimization to model learning.

[0079] The method of the present invention combines data screening with multi-view collaborative cross-modal learning based on a vision-language pre-training model, and shows excellent performance in multimodal data fusion and 3D point cloud classification:

[0080] The data screening process is efficient and accurate: it can automatically screen point clouds, avoiding a lot of manpower to determine data quality; the screening step can effectively retain complex geometric structures (such as ring objects) rather than simply removing flat abnormal point clouds, thereby achieving accurate geometric shape retention.

[0081] In multi-view collaborative cross-modal learning, a 3D representation that is closer to 2D is obtained through a multi-view encoder, and 3D objects are represented together with point cloud features. This can not only retain the 3D spatial information (point cloud encoder), but also bridge the inter-domain difference between the low resolution of 3D data and the real image (multi-view encoder), effectively utilizing the geometric information of the point cloud and image information from different perspectives to capture rich 3D features. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 It is a flowchart for screening the original point cloud data;

[0083] Figure 2 It is a training block diagram of the multi-view collaborative cross-modal 3D perception model training method based on data screening and optimization described in the present invention. DETAILED DESCRIPTION

[0084] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0085] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0086] The present invention will be further described below in conjunction with the accompanying drawings, but is not intended to be a limitation of the present invention.

[0087] Specific implementation method 1. Combination Figure 1 and Figure 2 As shown, the present invention provides a multi-view collaborative cross-modal 3D perception model training method based on data screening optimization, comprising:

[0088] Acquire a 3D data set, and obtain multiple sets of original point cloud data of various objects from the 3D data set;

[0089] The flat point cloud data of each group of original point cloud data are eliminated to obtain a filtered point cloud data group; each filtered point cloud data in the filtered point cloud data group is rendered to obtain a group of multi-view rendering images of each filtered point cloud data;

[0090] A point cloud encoder is used to encode each filtered point cloud data in the filtered point cloud data group to obtain encoded point cloud features;

[0091] A multi-view encoder is used to perform multi-view encoding on each group of multi-view rendering images to obtain encoded multi-view features;

[0092] At the same time, a text encoder is used to encode the text corresponding to the filtered point cloud data to obtain encoded text features; the encoded point cloud features, encoded multi-view features and encoded text features constitute multimodal features;

[0093] The multimodal features are aligned to a shared representation space through contrastive learning; the contrast loss is minimized to make the multimodal features of the same object as close as possible in the shared representation space, and the multimodal features of different objects are kept separated in the shared representation space; the network parameters of the point cloud encoder and the multi-view encoder are adjusted based on the contrast loss, and then the next set of original point cloud data is processed, and the point cloud encoder and the multi-view encoder are continued to be iteratively trained until the end, so as to obtain the trained point cloud encoder and the trained multi-view encoder.

[0094] Further, the data enhancement strategy for 3D datasets:

[0095] Referring to the performance advantages of OpenShape, the performance of deep learning models is largely restricted by the quantity and quality of training data. However, much of the data in the current data sets is uploaded by Internet users and has not been verified by experts. The quality is uneven and there are problems such as point cloud shape deformity. This implementation method optimizes the quality of point cloud data through a 3D data enhancement strategy to ensure that the model training data has higher accuracy and consistency, thereby improving the performance of the 3D perception model. Data enhancement is divided into RANSAC internal point percentage evaluation, projection area calculation, and ring recognition.

[0096] Combination Figure 1 As shown, the removal of flat point cloud data for each set of original point cloud data includes evaluating the percentage of RANSAC internal points for each set of original point cloud data:

[0097] RANSAC inlier percentage evaluation is one of the key steps to identify flat point clouds. This method is based on the random sample consensus (RANSAC) algorithm, which judges the geometric complexity of the point cloud by fitting the main plane and calculating the proportion of inliers in the point cloud. For 3D point cloud data, flat point clouds usually have small height changes, simple distribution, and lack of complex geometric characteristics. These point clouds usually cannot effectively characterize the shape characteristics of objects, so they need to be eliminated in the preliminary screening. RANSAC is a commonly used model fitting method, and its advantage is that it can robustly fit the optimal model in point cloud data containing noise. For each point cloud object, RANSAC is used to fit the main plane, and the degree of fit between the point cloud and the plane is quantified by the inlier percentage indicator, so as to screen out flat point clouds as candidates for unqualified point clouds.

[0098] Use RANSAC to fit the main plane based on a set of original point cloud data, and calculate the percentage of internal points P of each original point cloud data and the main plane inlier , which represents the proportion of points within the height threshold from the plane:

[0099]

[0100] Where N is the total number of points of each original point cloud data, I(·) is the indicator function, which returns 1 if the condition is met, otherwise returns 0; (x i ,y i ,z i ) is the 3D coordinate of the i-th point in the original point cloud data, (a, b, c, d) are all plane fitting parameters, ∈ is the threshold of the distance from the point to the main plane, which is used to determine the internal point;

[0101] Set the internal point percentage threshold to τ, if:

[0102] P inlier >τ,

[0103] The corresponding original point cloud data is determined to be flat point cloud data and is eliminated to obtain point cloud data after elimination.

[0104] As an example, the inlier percentage threshold τ may be set to 0.8 or 80%.

[0105] The elimination of flat point cloud data for each set of original point cloud data also includes the calculation of the projection area of ​​each point cloud data after elimination:

[0106] In this implementation, the point cloud data is projected onto the fitting main plane, the projection area is calculated, and whether to retain the point cloud is determined based on the area threshold. The specific reason is: on the basis of the RANSAC inlier percentage evaluation, in order to further eliminate invalid point clouds that are flat in shape but occupy a large area, a projection area calculation method is introduced. Flat point clouds usually have small height changes and simple geometric shapes, but these point clouds can be divided into two categories: one is invalid terrain point clouds, which often have a large projection area; the other is small objects that are flat in shape but belong to the valid category, such as scissors, lollipops, etc., and the projection area of ​​these objects is small. Therefore, it is impossible to effectively distinguish these two types of point clouds only through the RANSAC inlier percentage, and it is necessary to further calculate their projection areas for distinction.

[0107] A proj =Area(ConvHull(P proj )),

[0108] Where A proj represents the projection area of ​​the point cloud data on the main plane after one elimination, which is obtained by calculating the convex hull area of ​​the point cloud; Area represents the area, P proj ConvHull represents the projection point set P after the point cloud data is removed. proj The convex hull is the minimum closed area of ​​the two-dimensional projection point. The size of the projection area is used to determine whether the point cloud meets the morphological requirements;

[0109] P proj ={(x′,y′)∣(x′,y′,z′)=Projection(x,y,z)},

[0110] Where (x, y, z) represents the three-dimensional coordinates of the point cloud data after one elimination, (x′, y′, z′) represents the projection coordinates of (x, y, z) on the main plane, z′=0; Projection represents the projection operation;

[0111] Set the projection area threshold τ A , represents the upper limit of the screening condition, which is used to filter out point clouds with too large projection area. If:

[0112] A proj>τ A ,

[0113] The point cloud data after the first elimination is determined to be flat point cloud data and eliminated, thereby obtaining the point cloud data after the second elimination.

[0114] Eliminating flat point cloud data from each set of original point cloud data also includes identifying ring objects from the eliminated original point cloud data:

[0115] After two screenings, a large number of flat large-area point clouds that do not meet the requirements have been eliminated, but there may still be some special valid objects that are misscreened, such as ring-shaped objects such as bracelets and belts. These objects may be mistakenly marked as invalid point clouds in the elimination step due to their flat shape and large convex hull area. However, such objects usually have important classification value in actual tasks, so they need to be further screened and retained. This embodiment introduces an annular object recognition mechanism to effectively identify these special point clouds and ensure the integrity and accuracy of the data set. In this process, the geometric distribution of the projection points is calculated to determine whether the object is an annular object. Annular objects usually have circular or ring-shaped geometric features, so they are identified by calculating the variance of the geometric center of the projection point and the distance from the point to the center.

[0116] Calculate the average distance from each projection point of each eliminated original point cloud data in the main plane to the center of all projection points

[0117]

[0118] Where M is the total number of projection points of each original point cloud data removed, P m is the mth projection point, P j is the j-th projection point; ||·|| is the Euclidean distance, also known as the L2 norm;

[0119] Based on average Calculate the distance variance σ from the projection point to the center of the projection point 2 :

[0120]

[0121] Set the screening threshold τ of distance variance σ ,like:

[0122] σ 2 <τ σ ,

[0123] The corresponding removed original point cloud data is retained to obtain the retained point cloud data.

[0124] This implementation combines three judgment conditions, interior point ratio, projection area and annular feature to ensure quality optimization of point clouds in various scenarios, thereby enhancing the adaptability of point clouds in complex tasks.

[0125] Furthermore, the set of the point cloud data after secondary elimination and the retained point cloud data is taken as the point cloud data group after screening, and the F w Represents the initial feature set of the wth data in each set of original point cloud data, with a dimension of c. n w Represents the total number of data in each group of original point cloud data; P w Represents the three-dimensional coordinates of the wth data in each set of original point cloud data, Initial feature set F w Including normal vector, color, etc.;

[0126] Will (P w ,F w ) to remove the flat point cloud data is expressed as (P w ,F w ) * , the filtered point cloud data is expressed as In the formula represents the initial feature set of the u-th filtered point cloud data obtained from each set of original point cloud data, Represents the three-dimensional coordinates of the u-th filtered point cloud data obtained from each set of original point cloud data:

[0127]

[0128] in

[0129] In the formula It means that when the condition in the brackets is true, the value is 0; otherwise it is 1.

[0130] Multi-view collaborative cross-modal learning based on vision-language pre-training model:

[0131] This implementation is based on the visual-language pre-training model and multi-view feature fusion, and completes cross-modal learning of 3D point clouds by training two encoders. Specifically, the point cloud encoder is used to extract 3D point cloud features, and the multi-view encoder is used to process 2D image features rendered from multiple perspectives as supplementary 3D features. The two encoders are aligned with the text encoder of CLIP using the contrastive learning method, so that the feature vectors output by the two encoders will be aligned in a joint representation space, thereby improving the model's perception of complex tasks.

[0132] The input for model training includes two modal data, 3D point cloud data and text description.

[0133] The training data sets of point cloud encoder, multi-view encoder and text encoder are represented as

[0134]

[0135] In the formula T u represents the text corresponding to the u-th filtered point cloud data, then the u-th training data sample is expressed as U is the total number of filtered point cloud data in the filtered point cloud data group.

[0136] Going a step further, combined Figure 2 As shown, a point cloud encoder is used to encode each filtered point cloud data in the filtered point cloud data group, and the encoded point cloud feature is represented as f p :

[0137]

[0138] Where PointEncoder represents a point cloud encoder, which can be any point cloud encoder, such as PointNet, PointNet++, Transformer, etc.

[0139] Perform multi-view rendering on each filtered point cloud data, and then use Vision Transformer (ViT) to extract the features of each image;

[0140] A set of multi-view renderings obtained by rendering each filtered point cloud data is represented as

[0141]

[0142] Where I V represents the Vth rendered image, Render(·) represents the rendering function, which is used to Generates a rendering from V specified viewpoints; V represents the number of specified viewpoints;

[0143] Use Vision Transformer to extract features g for each rendering v :

[0144] g v =ViT(I v ), v = 1, 2, 3, ... V,

[0145] Where represents Vision Transformer;

[0146] For all rendering features g v, v = 1, 2, 3, ... V uses an aggregation method based on the self-attention mechanism to generate the encoded multi-view features f i , that is, the global features after multi-view aggregation:

[0147]

[0148] Where W q To query the projection matrix, W k is the key projection matrix, W Q is the value projection matrix, is a scaling factor used to stabilize the value.

[0149] The text corresponding to the filtered point cloud data is encoded using a text encoder to obtain the encoded text feature related to the point cloud represented as f t :

[0150] f t = CLIPTextEncoder(T u ),u=1,2,...,U,

[0151] Where CLIPTextEncoder represents the text encoder of the CLIP model.

[0152] In this implementation, point cloud features, multi-view features, and text features are aligned into a shared representation space through contrastive learning. By minimizing the contrast loss, different modal features of the same object (point cloud, multi-view, text) are made as close as possible in the shared representation space, while modal features of different objects remain separate.

[0153] Denote the total contrast loss function as L:

[0154] L=L p-t +L i-t ,

[0155] L is the loss that needs to be minimized during the entire training process, where L p-t is the point cloud-text contrast loss, L i-t is the multi-view image-text contrast loss;

[0156]

[0157] In the formula is the encoded point cloud feature corresponding to the u-th filtered point cloud data, is the encoded text feature corresponding to the u-th filtered point cloud data, is the encoded text feature corresponding to the Jth filtered point cloud data, is the encoded multi-view feature corresponding to the u-th filtered point cloud data, τ0 is the temperature coefficient, which is used to control the smoothness of the distribution;

[0158]

[0159] The method of the present invention introduces a self-attention mechanism in the process of multi-view feature aggregation, and intelligently aggregates multi-view information by calculating the similarity between view features and assigning weights. The self-attention mechanism captures the complementary information between different view features and solves the problem of feature redundancy that may be caused by simple mean pooling or weighted averaging. The aggregated global features can more comprehensively characterize the geometric shape and semantic information of the point cloud.

[0160] By optimizing the point cloud-text contrast loss and image-text contrast loss, a unified representation of multimodal features is achieved.

[0161] Embodiment: The Objaverse dataset is selected as the original dataset. Objaverse is a multimodal three-dimensional object dataset, which contains rich three-dimensional point clouds and text description information. Because of the large amount of data, a model with better generalization can be trained. However, the data in the Objaverse dataset are all uploaded after artificial modeling. There are many point cloud data that do not meet the standards and have flat or deformed features (mostly terrain point clouds), which greatly affects the quality of the dataset and the training effect of the model. After the Objaverse dataset is enhanced using the method of the present invention, the enhanced point cloud and CLIP are used to train a multi-perspective collaborative cross-modal learning algorithm based on a visual-language pre-training model to construct an intelligent model that can be used for augmented reality (AR) scene navigation. The model can realize object recognition, semantic annotation, and interactive applications with the real world.

[0162] For example, in cultural relics sites or museums, tourists capture target objects such as statues and cultural relics through mobile phones or AR glasses. The system will recognize the objects in real time and generate relevant semantic descriptions with high accuracy.

[0163] The specific steps are as follows:

[0164] Step 1: RANSAC inlier percentage evaluation;

[0165] First, the RANSAC algorithm is used to fit the main plane of the point cloud. Through iterative random sampling and model optimization, the best fitting plane is obtained in the case of complex distribution of point clouds. During the fitting process, the RANSAC algorithm can automatically remove outliers and generate plane model parameters and internal point sets.

[0166] Next, the percentage of inliers in the point cloud is calculated based on the fitting results. By setting a distance threshold, each point is determined to be an inlier, and the ratio of the number of inliers that meet the conditions to the total number of points in the point cloud is counted. This ratio is used to measure whether the point cloud meets the preset morphological requirements.

[0167] In this step, the robustness of the RANSAC algorithm can adapt to the noise and distribution characteristics of different point cloud data, and can stably generate plane models and internal point sets that conform to geometric rules even when the point cloud data has high noise or uneven distribution.

[0168] Step 2: Calculate the projection area;

[0169] In order to further improve the screening accuracy of point cloud data, based on RANSAC plane fitting, the convex hull area of ​​the point cloud after projecting it onto the main plane is calculated to efficiently eliminate abnormal point clouds that do not meet the morphological requirements.

[0170] The specific implementation method is as follows: First, the inner points of the point cloud screened by the RANSAC algorithm are projected onto the fitted principal plane to form a two-dimensional projection point set. Then, the convex hull algorithm in computational geometry is used to generate the minimum outer enclosing area of ​​the point set, and the area of ​​the convex hull area is calculated as the projection area of ​​the point cloud.

[0171] Step 3: Annular object recognition and retention;

[0172] In order to ensure that valid point clouds with special geometric features (such as ring objects) are not mistakenly screened out during the data screening process, the ring object recognition and retention method is used. This method accurately identifies ring objects by analyzing the geometric distribution of projection points and retains them in the data set.

[0173] Specifically, first, the geometric center of the point cloud projection point set after the projection area screening is calculated. Then, the distance variance of the point cloud is calculated based on the distance distribution from the projection point to the geometric center. For annular objects, the distance distribution between their projection points and the geometric center usually has a large variance, while the distance distribution variance of non-annular objects is relatively small.

[0174] This method can effectively retain the data of annular objects such as bracelets and tires, improve the diversity and integrity of the data without introducing additional noise, and provide richer geometric feature support for subsequent model training. This step makes up for the shortcomings of traditional geometric screening methods that may miss objects with special shapes, and further optimizes the overall quality of the dataset.

[0175] Next, we train the multi-view collaborative cross-modal learning algorithm:

[0176] Point cloud feature extraction;

[0177] In this step, PointBERT is used as a point cloud encoder to extract features from the input 3D point cloud data. PointBERT is a point cloud feature extraction network based on the Transformer architecture. It is designed to use a different method from traditional convolutional neural networks and is specifically designed to process irregular and disordered point cloud data. Specifically, PointBERT first embeds the 3D point cloud data and maps the spatial coordinates of each point to a high-dimensional space. Then, the point cloud data is processed through multiple Transformer encoder layers to capture the spatial relationship and geometric features between points. Unlike other methods, the advantage of PointBERT is that it can adaptively model the contextual information of each point in the point cloud through the self-attention mechanism, and can effectively extract local and global geometric features regardless of the distribution or density of the point cloud. In this way, PointBERT can generate point cloud feature vectors with rich geometric and spatial information, which provide key geometric information support for subsequent cross-modal learning tasks such as alignment with multi-view image features and text features. PointBERT is used as an example here, and any point cloud encoder can be used in actual use.

[0178] Multi-view rendering and feature extraction:

[0179] The input 3D point cloud data is rendered from multiple perspectives to generate two-dimensional projection images from multiple angles. By observing the same object from different perspectives such as front view, side view, and top view, various appearance information of the object can be captured. The rendered two-dimensional image contains rich visual information, which helps the model understand the shape and structure of the object.

[0180] The image of each view is converted into a feature vector, which represents the visual features of the object at that view. Then, the feature vectors of all views are fused through the self-attention mechanism to comprehensively consider the local information of each view and generate a global, multi-view feature representation.

[0181] Text feature extraction:

[0182] The features of text descriptions related to point cloud data are extracted through a pre-trained visual-language model (CLIP). These text descriptions usually include semantic information such as the category and attributes of the object. The core purpose of this process is to facilitate the subsequent contrastive learning process so that the geometric and visual features extracted by the point cloud encoder and the multi-view encoder can be aligned with the text features, thereby providing consistent semantic support for subsequent cross-modal learning.

[0183] In this step, the main role of contrastive learning is to align point cloud features, multi-view features, and text features into the same shared representation space. Through contrastive learning, the training model makes different modal features (point cloud, image, and text) from the same object as close as possible in this space, while features from different objects are pushed further away. This process ensures the consistency of point cloud and image features with text descriptions at the semantic and geometric levels by maximizing the similarity between features of the same modality and minimizing the similarity between features of different object modalities.

[0184] Finally, the practical application of the trained model: After the training is completed, the model can be deployed to the VR device and then the recognition task can be carried out. In the recognition process, the features are extracted by the point cloud encoder and the multi-view image encoder, and then weighted fusion is performed to match the text information in the text library, so as to make the final match by integrating multi-modal information.

[0185] The fused multimodal features are represented as f combined :

[0186] f combined =w p ·f p +w i ·f i ,

[0187] Where w p , w i is the weight coefficient, satisfying w p +w i =1.

[0188] In summary, the present invention proposes a complete learning framework from point cloud optimization to multimodal alignment, which is highly modular and flexible. The framework can adapt to a variety of task scenarios, such as three-dimensional object classification, cross-modal retrieval, etc., and different modules can be expanded and replaced to meet different needs. The point cloud optimization part ensures the high quality of input data; the feature extraction module extracts multimodal features from point clouds, multi-view images and texts; the multi-view aggregation module introduces a self-attention mechanism to enhance the feature complementarity between views; the contrastive learning part realizes the unified representation of multimodal features.

[0189] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that the features described in conjunction with a single embodiment may be used in other described embodiments.

Claims

1. A multi-view collaborative cross-modal 3D perception model training method based on data screening and optimization, characterized in that: include, Acquire a 3D data set, and obtain multiple sets of original point cloud data of various objects from the 3D data set; The flat point cloud data of each group of original point cloud data is eliminated to obtain a filtered point cloud data group; Rendering each filtered point cloud data in the filtered point cloud data group to obtain a set of multi-view rendering images of each filtered point cloud data; A point cloud encoder is used to encode each filtered point cloud data in the filtered point cloud data group to obtain encoded point cloud features; A multi-view encoder is used to perform multi-view encoding on each group of multi-view rendering images to obtain encoded multi-view features; At the same time, a text encoder is used to encode the text corresponding to the screened point cloud data to obtain encoded text features; the encoded point cloud features, the encoded multi-view features and the encoded text features constitute multimodal features; The multimodal features are aligned to a shared representation space through contrastive learning; the contrast loss is minimized to make the multimodal features of the same object as close as possible in the shared representation space, and the multimodal features of different objects are kept separated in the shared representation space; the network parameters of the point cloud encoder and the multi-view encoder are adjusted based on the contrast loss, and then the next set of original point cloud data is processed, and the point cloud encoder and the multi-view encoder are continued to be iteratively trained until the end, so as to obtain the trained point cloud encoder and the trained multi-view encoder.

2. The multi-view collaborative cross-modal 3D perception model training method based on data screening optimization according to claim 1 is characterized in that: The removal of flat point cloud data for each set of original point cloud data includes evaluating the percentage of RANSAC internal points for each set of original point cloud data: Use RANSAC to fit the main plane based on a set of original point cloud data, and calculate the percentage of internal points P of each original point cloud data and the main plane inlier : Where N is the total number of points of each original point cloud data, I(·) is the indicator function, which returns 1 if the condition is met, otherwise returns 0; (x i ,y i ,z i ) is the 3D coordinate of the i-th point in the original point cloud data, (a, b, c, d) are all plane fitting parameters, ∈ is the threshold of the distance from the point to the main plane; Set the internal point percentage threshold to τ, if: P inlier >t, The corresponding original point cloud data is determined to be flat point cloud data and is eliminated to obtain point cloud data after elimination.

3. The multi-view collaborative cross-modal 3D perception model training method based on data screening optimization according to claim 2 is characterized in that: The elimination of flat point cloud data for each set of original point cloud data also includes the calculation of the projection area of ​​each point cloud data after elimination: A proj =Area(ConvHull(P proj )), Where A proj It represents the projection area of ​​the point cloud data on the main plane after one elimination. Area represents the area. P proj ConvHull represents the projection point set P after the point cloud data is removed. proj The convex hull of is the minimum enclosed area of ​​the two-dimensional projection points; P proj ={(x′,y′)∣(x′,y′,z′)=Projection(x,y,z)}, Where (x, y, z) represents the three-dimensional coordinates of the point cloud data after one elimination, (x′, y′, z′) represents the projection coordinates of (x, y, z) on the main plane, z′=0; Projection represents the projection operation; Set the projection area threshold τ A ,like: A proj >t A , The point cloud data after the first elimination is determined to be flat point cloud data and eliminated, thereby obtaining the point cloud data after the second elimination.

4. The multi-view collaborative cross-modal 3D perception model training method based on data screening optimization according to claim 3 is characterized in that: Eliminating flat point cloud data from each set of original point cloud data also includes identifying ring objects from the eliminated original point cloud data: Calculate the average distance from each projection point of each eliminated original point cloud data in the main plane to the center of all projection points Where M is the total number of projection points of each original point cloud data removed, P m is the mth projection point, P j is the j-th projection point; ||·|| is the Euclidean distance; Based on average Calculate the distance variance σ from the projection point to the center of the projection point 2 : Set the screening threshold τ of distance variance σ ,like: s 2 <t σ , The corresponding removed original point cloud data is retained to obtain the retained point cloud data.

5. The multi-view collaborative cross-modal 3D perception model training method based on data screening optimization according to claim 4 is characterized in that: The set of point cloud data after secondary elimination and retained point cloud data is taken as the point cloud data group after screening, and F is used to w Represents the initial feature set of the wth data in each set of original point cloud data, with a dimension of c. n w Represents the total number of data in each group of original point cloud data; P w Represents the three-dimensional coordinates of the wth data in each set of original point cloud data, Will (P w ,F w ) to remove the flat point cloud data is expressed as (P w ,F w ) * , the filtered point cloud data is expressed as In the formula represents the initial feature set of the u-th filtered point cloud data obtained from each set of original point cloud data, Represents the three-dimensional coordinates of the u-th filtered point cloud data obtained from each set of original point cloud data: in In the formula It means that when the condition in the brackets is true, the value is 0; otherwise it is 1.

6. The multi-view collaborative cross-modal 3D perception model training method based on data screening optimization according to claim 5 is characterized in that: The training data sets of point cloud encoder, multi-view encoder and text encoder are represented as In the formula T u represents the text corresponding to the u-th filtered point cloud data, then the u-th training data sample is expressed as 7. The multi-view collaborative cross-modal 3D perception model training method based on data screening optimization according to claim 6 is characterized in that: The point cloud encoder is used to encode each filtered point cloud data in the filtered point cloud data group, and the encoded point cloud feature representation is f p : Where PointEncoder represents the point cloud encoder.

8. The multi-view collaborative cross-modal 3D perception model training method based on data screening optimization according to claim 7 is characterized in that: A set of multi-view renderings obtained by rendering each filtered point cloud data is represented as Where I V represents the Vth rendered image, Render(·) represents the rendering function, which is used to Generates a rendering from V specified viewpoints; V represents the number of specified viewpoints; Use Vision Transformer to extract features g for each rendering v : g v =ViT(I v ),v=1,2,3,…V, Where represents Vision Transformer; For all rendering features g v , v = 1, 2, 3, ... V uses an aggregation method based on the self-attention mechanism to generate the encoded multi-view features f i : Where W q To query the projection matrix, W k is the key projection matrix, W Q is the value projection matrix, is the scaling factor.

9. The multi-view collaborative cross-modal 3D perception model training method based on data screening optimization according to claim 8 is characterized in that: The text corresponding to the filtered point cloud data is encoded using a text encoder, and the encoded text feature representation is f t : f t =CLIPTextEncoder(T u ),u=1,2,...,U, Where CLIPTextEncoder represents the text encoder of the CLIP model.

10. The multi-view collaborative cross-modal 3D perception model training method based on data screening optimization according to claim 9 is characterized in that: Denote the total contrast loss function as L: L=L p-t +L i-t , Where L p-t is the point cloud-text contrast loss, L i-t is the multi-view image-text contrast loss; In the formula is the encoded point cloud feature corresponding to the u-th filtered point cloud data, is the encoded text feature corresponding to the u-th filtered point cloud data, is the encoded text feature corresponding to the Jth filtered point cloud data, f i u is the encoded multi-view feature corresponding to the u-th filtered point cloud data, τ0 is the temperature coefficient;

Images