A Method for Understanding Open Scenes of 3D Point Clouds Based on Dual-Stream Feature Fusion

Abstract

This invention discloses a method for understanding open scenes in 3D point clouds based on dual-stream feature fusion, belonging to the field of multimodal learning. The method includes data preparation, collecting 3D point cloud data V, labeling the data to obtain a 3D point cloud open scene understanding dataset A, and partitioning the obtained dataset A; constructing 3D point cloud open scene supervision information, constructing foreground interest regions and open semantic information, and fusing dual-stream feature information; and designing a method including a 3D point cloud feature extractor E and a pre-trained text feature extractor θ. text Model M is obtained by training model M with the multimodal discriminator D. * In model M * The invention employs a three-dimensional point cloud open scene understanding method based on dual-stream feature fusion, which achieves three-dimensional point cloud scene understanding on open sets and significantly improves the accuracy of tasks such as open set semantic segmentation.

Description

Technical Field

[0001] This invention relates to the field of multimodal learning, and in particular to a method for understanding open scenes of 3D point clouds based on dual-stream feature fusion. Background Technology

[0002] Visual tasks related to 3D point clouds are characterized by high costs for data acquisition using LiDAR equipment, high costs for manual annotation of large-scale data, and a late start in the development of 3D point cloud technologies, resulting in a lag in the development of computer vision compared to the field of 2D images.

[0003] The field of 2D images has achieved rapid development in recent years due to its low data acquisition and annotation costs and long history. In recent years, a series of multimodal models based on text and image data pairs have been developed. Through the alignment and training of text and image pairs, related tasks of open scene understanding in the field of 2D images have been realized, such as image classification, segmentation, and object detection of open vocabulary.

[0004] However, the success of open scene understanding in the 2D image domain is based on a large amount of training data of text and image pairs. The 3D point cloud domain lacks and it is difficult to obtain such a large amount of data. How to transfer the ability of open scene understanding from the 2D image domain to the 3D point cloud domain based on unlabeled 3D point cloud data has become an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide a method for understanding open scenes of 3D point clouds based on dual-stream feature fusion, in order to solve the problem that existing 3D point cloud open scene understanding technologies do not fully utilize the differences between 3D point cloud modalities and 2D point cloud modalities, and the complementary and mutually reinforcing enhancement features for extraction and understanding.

[0006] To achieve the above objectives, this invention provides a method for understanding open scenes of 3D point clouds based on dual-stream feature fusion, comprising the following steps:

[0007] S1. Data preparation and dataset partitioning;

[0008] S2. Construct 3D point cloud open scene supervision information; including:

[0009] S21. Constructing foreground interest region information from the bottom up;

[0010] S22. Construct foreground interest region information from top to bottom;

[0011] S23. Construct open scene semantic information from the foreground interest region;

[0012] S24. Integrate bottom-up and top-down dual-flow feature information.

[0013] S3. Design a model that includes a 3D point cloud feature extractor E and a pre-trained text feature extractor θ. text Model M of the multimodal discriminator D;

[0014] S4. Train the model, train model M to obtain model M. * ;

[0015] S5, Test Model.

[0016] Preferably, S1 specifically includes:

[0017] S11. Data Preparation: Collect 3D point cloud data V. Each 3D point cloud data V has a corresponding 2D image sequence P. Each frame of the image sequence P... i By mapping the world coordinates to camera coordinates onto the 3D point cloud scene using a transformation matrix, the features in the 3D point cloud are aligned one-to-one with the features in the 2D image. The calculation formula is as follows:

[0018]

[0019] Where (u,v) represents the position of a pixel in the 2D image in camera coordinates, and (x,y,z) represents the position of a point in the 3D point cloud in world coordinates. It is the camera's intrinsic parameter matrix, [R 3×3 T 3×1 ] is the camera extrinsic parameter matrix; the transformation process from the camera coordinate system of the 2D image to the world coordinate system of the 3D point cloud is denoted as R. 2D→3D (·), the transformation process from the world coordinate system of a 3D point cloud to the camera coordinate system of a 2D image is denoted as R. 3D→2D (·); Then, the 3D point cloud data is labeled to obtain the 3D point cloud open scene understanding dataset A, where the annotation content is the object category label in the 3D point cloud, using C class Represents the set of all object labels;

[0020] S12. Divide the dataset: Divide the obtained 3D point cloud open scene understanding dataset A into a training set C. train Validation set C val and test set C test Furthermore, based on the object category labels, the labeled category labels should be further divided into known classes C. know With unknown class C unknow , represented as:

[0021] A = {C train C val C test};

[0022] C class ={Cknow C unknow};

[0023] Given class C know The labels are used in training and testing, and the unknown class C unknow The labels are not used in training, but need to be predicted during testing.

[0024] Preferably, S21 is as follows:

[0025] Based on the acquired two-dimensional image modalities, a sliding window or a pre-trained model θ is used. 2D For example, the open-set 2D detection network DINO and the open-set 2D segmentation network SAM predict the foreground region from image data and obtain the corresponding 3D point cloud coordinate range of the foreground region through a transformation matrix. The process is represented as follows:

[0026] C bottom-up,2D ={θ 2D (P i )};

[0027] C bottom-up,3D =R 2D→3D (C bottom-up,2D );

[0028] Among them, C bottom-up,2D C represents the set of foreground regions obtained through a bottom-up approach. bottom-up,3D This represents the set obtained by projecting the foreground region acquired by the bottom-up method onto 3D space. The bottom-up method is better able to extract small objects in the scene and establish feature information of small objects through the texture information in the image. However, it is difficult to capture complete feature information for objects located at the edge of the image frame.

[0029] Preferably, S22 is as follows:

[0030] Based on the acquired 3D point cloud modalities, a pre-trained model θ is used. 3D For example, the open-set 3D detection network OV3D predicts the foreground region from point cloud data. It maps the region of the 3D point cloud to a 2D image space using a camera pose rotation matrix. This process can be represented as follows:

[0031] C top-down,3D ={θ 3D (V i )};

[0032] C top-down,2D =R 3D→2D (C top-down,3D );

[0033] Among them, C top-down,3D and C top-down,2DV represents the sets of foreground regions in three-dimensional and two-dimensional space obtained by the top-down method, respectively. i For a scene representing a 3D point cloud, a top-down approach starts from the 3D point cloud, focuses on 3D structural information, and can more accurately construct the boundary information of 3D objects and establish large object information across image frames.

[0034] Preferably, S23 is as follows:

[0035] The obtained foreground interest region C is integrated bottom-up,2D With C top-down,2D The foreground region C of the two streams is obtained. TDBU,2D And use pre-trained images to describe model θ cap Obtain the image text description information of the foreground interest region as an open scene description tag set C cap ; through pre-trained large language model θ LLM Get the open scene description tag set C cap The set of foreground entity noun information C entity This is used to construct bottom-up and top-down semantic prototypes, represented as:

[0036] C TDBU,2D ={C top-down,2D C bottom-up,2D};

[0037] C TDBU,3D ={C top-down,3D C bottom-up,3D};

[0038] C cap =θ cap (C TDBU,2D );

[0039] C entity =θ LLM (C cap );

[0040] Among them, C TDBU,3D C represents TDBU,2D Region in 3D space; C cap For semantic description information; C entity This refers to entity noun information.

[0041] Preferably, S24 is as follows:

[0042] The obtained foreground region C of the two streams TDBU,2D Using a pre-trained image encoder θ image For example, the CLIP image encoder obtains image features C. imageThe obtained subclass features of the same entity are used to establish class prototype centers, that is, to establish the image feature prototype C of the class. prototype The open scene description label set C is filtered by using the cosine distance between image features and their corresponding feature prototypes. cap High-quality samples are used to construct the training set C. train pseudo-label semantic supervision information C pseudo Using the relationship C corresponding to the foreground region in the 3D point cloud space. TDBU,3D Establish 3D point cloud and semantic supervision information C pseudo Based on the correspondence, the open semantic feature dataset of 3D point clouds is constructed, and the process is represented as follows:

[0043] C image =θ image (C TDBU,2D );

[0044] C prototype ={AVG(C image,i )|i∈C entity};

[0045] C pseudo ={C cap if cos(C image,i C prototype,i )<ε|i∈C entity};

[0046] Where AVG(·) is the average function, cos(·) is the cosine distance function, and ε is the threshold for cosine distance filtering.

[0047] Preferably, the specific steps of S3 are as follows:

[0048] S31. The overall model is denoted as M, which includes a 3D point cloud feature extractor E and a pre-trained text feature extractor θ. text With a multimodal discriminator D, where θ text The CLIP text encoder was employed. The model input was a multimodal 3D point cloud dataset A; the 3D point cloud feature extractor E received 3D point cloud data V and output 3D point cloud features. X is represented as:

[0049] X = E(V);

[0050] Pre-trained text feature extractor θ text Responsible for receiving open-scene text semantic supervision information C after dual-stream fusion pseudo The encoded output is the text feature Y of the target region;

[0051] Y = θ text (Cpseudo );

[0052] To alleviate the problem of significant distribution differences between 3D point cloud features X and text features Y, where features from different modalities emphasize characteristics of different modalities, a cross-modal Transformer is used to bridge the gap in feature distribution. Furthermore, an attention mechanism is employed to enhance the feature representation of the target region. The cross-modal Transformer is defined as: Q X =XP q K Y =YP k V Y =YP v Q X K Y 、V Y These represent the query, key, and value, respectively.

[0053]

[0054] Among them, P q P k P v Z represents the learnable parameter, where Z represents the enhancement feature from text modality to 3D point cloud modality;

[0055] S32. Using contrastive learning, the open text description features are distilled into a 3D point cloud network. The enhanced features Z are processed through a multi-layer MLP and then fed into a multimodal discriminator D along with the text features Y. The multimodal discriminator outputs the cosine similarity prediction score s between each point in the enhanced features Z and the text features Y. Cross-entropy is used to calculate the relationship between the output s and the text feature region C. pseudo The matching relationship between them is defined as:

[0056] s = D(Z,Y);

[0057]

[0058] Where g is C pseudo Match the one-hot encoded vector of the corresponding region. BCELoss(·,·) is the cross-entropy loss, calculated as follows:

[0059]

[0060] in, y is the predicted value, and y is the true value of the label;

[0061] To further reduce the similarity of cross-modal features of objects of the same class and increase the difference between features of different classes, contrastive learning is used to enhance the loss between two open scene features. The calculation is as follows:

[0062]

[0063] Among them, y i z is the i-th feature vector in the text features Y of the open scene. i It is the enhancement feature Z and y i Positive sample regions corresponding to text features, z j This represents the negative sample region corresponding to the current sample, N is the total number of text features, and τ is the temperature parameter.

[0064] For training set C train Class C is known in class For objects, supervised learning of known classes is performed using real object labels and cross-entropy, calculated as follows:

[0065] s know =D(Z) know C know );

[0066]

[0067] Among them, Z know It is to extract the output features contained in the known class C. know Feature points within the range, s know It is the similarity prediction score of the known class, g know It is the one-hot encoded vector of the corresponding region;

[0068] S33. Calculate the total training loss of model M. Wherein, λ1, λ2, and λ3 are the cross-entropy losses of open text features, respectively. Comparative learning loss Learning loss with known classes The weights of model M are updated using gradient descent, and the operation S3 is repeated several times to obtain a model with high prediction performance in open scene understanding tasks.

[0069] The gradient descent method works as follows: after calculating... Based on this, calculate gradient with respect to parameter θ And update the parameter θ in the gradient direction, as follows:

[0070]

[0071] Where α represents the learning rate.

[0072] Preferably, the specific steps of S4 are as follows:

[0073] S41, The training set C train With known class C knowThe labeled information is input into the model M designed by S3, and the model is trained using the gradient descent method;

[0074] S42. During the training phase, the validation set C is used simultaneously. val With all object label set C class Validate the model training effect, that is, when model M is on the validation set C val It has gained a better understanding of open-world scenarios and is capable of handling unknown C-class applications. unknow With known class C know When the prediction performance reaches a good level of accuracy, and this accuracy can no longer be improved during training, training is stopped, and finally, model M is obtained after training is completed. * .

[0075] Preferably, the specific steps of S5 are as follows:

[0076] S51, Input test dataset C test It includes 3D point cloud data V and the text conditions L to be queried; the model used in the testing phase is model M. * Feature extractor E, pre-trained text feature extractor θ test Multimodal discriminator D;

[0077] S52, Test set C test Model M obtained from input S4 * The test set C is obtained from the middle. test The open scene prediction results include predictions for known class C. know and unknown class C unknow The predictive performance.

[0078] Therefore, the present invention employs the above-mentioned method for understanding open scenes of 3D point clouds based on dual-stream feature fusion, which has the following beneficial effects:

[0079] (1) Using a dual-stream approach of top-down and bottom-up to obtain open semantic features can not only make full use of geometric information in three-dimensional space and local details in two-dimensional images, but also generate high-quality text descriptions and semantic features by combining image description models and CLIP models, thereby significantly improving the performance of object detection and semantic understanding.

[0080] (2) A large language model is used to extract and filter entity nouns from the text description, and the newly added entity nouns are taken as unknown class targets. Then, the average clustering method is used to establish unknown class centers and filter the image features of the candidate foreground region to filter out inaccurate candidate foreground regions and obtain candidate foreground regions with higher accuracy and corresponding text description information.

[0081] (3) The open scene understanding method for 3D point cloud based on prototype contrastive learning and cross-modal knowledge distillation significantly improves the ability to understand open scenes by optimizing feature extraction, strengthening sample matching, realizing cross-modal knowledge transfer and improving multimodal fusion performance.

[0082] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0083] Figure 1 This is a flowchart of a three-dimensional point cloud open scene understanding method based on dual-stream feature fusion according to the present invention.

[0084] Figure 2 This is a network diagram illustrating a three-dimensional point cloud open scene understanding method based on dual-stream feature fusion according to the present invention. Detailed Implementation

[0085] The following detailed description of embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0086] Example

[0087] like Figure 1 As shown, a method for understanding open scenes of 3D point clouds based on dual-stream feature fusion includes the following steps:

[0088] S1. Data preparation and dataset partitioning;

[0089] S11. Data Preparation: Collect a large amount of 3D point cloud data V, where each 3D point cloud data Y has a corresponding 2D image sequence P, and each frame of the image sequence P... i The transformation matrix between world coordinates and camera coordinates is used to map the data to the 3D point cloud scene, and vice versa. The features in the 3D point cloud are aligned one-to-one with the features in the 2D image. The calculation formula is as follows:

[0090]

[0091] Where (u,v) represents the position of a pixel in the 2D image in camera coordinates, and (x,y,z) represents the position of a point in the 3D point cloud in world coordinates. It is the camera's intrinsic parameter matrix, [R 3×3 T 3×1 ] is the camera extrinsic parameter matrix; the transformation process from the camera coordinate system of the 2D image to the world coordinate system of the 3D point cloud is denoted as R. 2D→3D(·), the transformation process from the world coordinate system of a 3D point cloud to the camera coordinate system of a 2D image is denoted as R. 3D→2D (·); Then, the 3D point cloud data is labeled to obtain the 3D point cloud open scene understanding dataset A, where the annotation content is the object category label in the 3D point cloud, using C class Represents the set of all object labels;

[0092] S12. Divide the dataset: Divide the obtained 3D point cloud open scene understanding dataset A into a training set C. train Validation set C val and test set C test It is necessary to further divide the labeled category into known classes C based on the object category labels. know With unknown class C unknow , represented as:

[0093] A = {C train C val C test};

[0094] C class ={C know C unknow};

[0095] Given class C know The labels are used in training and testing, and the unknown class C unknow The labels are not used in training, but need to be predicted during testing.

[0096] S2. Construct 3D point cloud open scene supervision information; including:

[0097] S21. Construct foreground interest region information from the bottom up.

[0098] Based on the acquired two-dimensional image modalities, a sliding window or a pre-trained model θ is used. 2D For example, the open-set 2D detection network DINO and the open-set 2D segmentation network SAM predict the foreground region from image data and obtain the corresponding 3D point cloud coordinate range of the foreground region through a transformation matrix. The process is represented as follows:

[0099] C bottom-up,2D ={θ 2D (P i )};

[0100] C bottom-up,3D =R 2D→3D (C bottom-up,2D );

[0101] Among them, C bottom-up,2D C represents the set of foreground regions obtained through a bottom-up approach. bottom-up,3DThis represents the set obtained by projecting the foreground region acquired by the bottom-up method onto 3D space. The bottom-up method is better able to extract small objects in the scene and establish feature information of small objects through the texture information in the image. However, it is difficult to capture complete feature information for objects located at the edge of the image frame.

[0102] S22. Construct foreground interest region information from top to bottom.

[0103] Based on the acquired 3D point cloud modalities, a pre-trained model θ is used. 3D For example, the open-set 3D detection network OV3D predicts the foreground region from point cloud data. It maps the region of the 3D point cloud to a 2D image space using a camera pose rotation matrix. This process can be represented as follows:

[0104] C top-down,3D ={θ 3D (V i )};

[0105] C top-down,2D =R 3D→2D (C top-down,3D );

[0106] Among them, C top-down,3D and C top-down,2D V represents the sets of foreground regions in three-dimensional and two-dimensional space obtained by the top-down method, respectively. i For a scene representing a 3D point cloud, a top-down approach starts from the 3D point cloud, focuses on 3D structural information, and can more accurately construct the boundary information of 3D objects and establish large object information across image frames.

[0107] S23. Construct open scene semantic information from the foreground interest region.

[0108] The obtained foreground interest region C is integrated bottom-up,2D With C top-down,2D The foreground region C of the two streams is obtained. TDBU,2D And use pre-trained images to describe model θ cap Obtain the image text description information of the foreground interest region as an open scene description tag set θ cap By pre-training the large language model θ LLM For example, ChatGPT extracts and filters entity noun information set C from description information. entity Construct bottom-up and top-down semantic prototypes, represented as:

[0109] C TDBU,2D ={C top-down,2D C bottom-up,2D};

[0110] C TDBU,3D ={Ctop-down,3D C bottom-up,3D};

[0111] C cap =θ cap (C TDBU,2D );

[0112] C entity =θ LLM (C cap ).

[0113] S24. Integrate bottom-up and top-down dual-flow feature information.

[0114] The obtained foreground region C of the two streams TDBU,2D Using a pre-trained image encoder θ image For example, the CLIP image encoder obtains image features C. image The obtained subclass features under the same entity are used to establish class prototype centers through averaging, thus establishing the image feature prototype C of the class. prototype Since there may be some misclassified foreground regions, this embodiment calculates the cosine distance between image features and their corresponding feature prototypes to remove outliers and misclassified samples, thus constructing a more accurate training set C for the open scene understanding task. train Pseudo-label semantic supervision information C pseudo Using the relationship C corresponding to the foreground region in the 3D point cloud space. TDBU,3D Establish 3D point cloud and semantic supervision information C pseudo Based on the correspondence, the open semantic feature dataset of 3D point clouds is constructed, and the process is represented as follows:

[0115] C image =θ image (C TDBU,2D );

[0116] C prototype ={AVG(C image,i )|i∈C entity};

[0117] C pseudo ={C cap if cos(C image,i C prototype,i )<ε|i∈C entity};

[0118] Where AVG(·) is the average function, cos(·) is the cosine distance function, and ε is the threshold for cosine feature selection.

[0119] S3, Design Model.

[0120] S31. The overall model is denoted as M, which includes a 3D point cloud feature extractor E and a pre-trained text feature extractor θ. text With a multimodal discriminator D, where θ text The CLIP text encoder was employed. The model input was a multimodal 3D point cloud dataset A; the 3D point cloud feature extractor E received 3D point cloud data V and output 3D point cloud features. X is represented as:

[0121] X = E(V);

[0122] Pre-trained text feature extractor θ text Responsible for receiving open-scene text semantic supervision information C after dual-stream fusion pseudo The encoded output is the text feature Y of the target region:

[0123] Y = θ text (C pseudo );

[0124] For 3D point cloud features X and text features Y, their distributions differ significantly because features from different modalities emphasize the characteristics of different modalities. To alleviate this problem, this embodiment utilizes a cross-modal Transformer to bridge the gap in feature distributions and employs an attention mechanism to enhance the feature representation of the target region; the cross-modal Transformer is defined as: Q X =XP q K Y =YP k V Y =YP v Q X K Y V Y These represent the query, key, and value, respectively.

[0125]

[0126] Among them, P q P k P v Z is a learnable parameter, representing the enhanced features from the text modality to the 3D point cloud modality. During the 3D point cloud encoding process, descriptive text is injected into the 3D point cloud features through the cross-modal Transformer mechanism, and the foreground region features are enhanced through the attention mechanism to obtain multimodal feature output.

[0127] S32. Using contrastive learning, the open text description features are distilled into a 3D point cloud network. The enhanced features Z are processed through a multi-layer MLP and then fed into a multimodal discriminator D along with the text features Y. The multimodal discriminator outputs the cosine similarity prediction score s between each point in the enhanced features Z and the text features Y. Cross-entropy is used to calculate the relationship between the output s and the text feature region C. pseudo The matching relationship between them is defined as:

[0128] s = D(Z,Y);

[0129]

[0130] Where g is C pseudo Match the one-hot encoded vector of the corresponding region. BCELoss(·,·) is the cross-entropy loss, calculated as follows:

[0131]

[0132] in, y is the predicted value, and y is the true value of the label.

[0133] To further reduce the similarity of cross-modal features of objects of the same class and increase the difference between features of different classes, contrastive learning is used to enhance the loss between two open scene features. The calculation is as follows:

[0134]

[0135] Among them, y i z is the i-th feature vector in the text features Y of the open scene. i It is the enhancement feature Z and y i Positive sample regions corresponding to text features, z j This represents the negative sample region corresponding to the current sample, N is the total number of text features, and τ is the temperature parameter.

[0136] For training set C train Class C is known in class For objects, supervised learning of known classes is performed using real object labels and cross-entropy, with limited annotation information used for supervision. This enhances the feature extraction capability of 3D point clouds and compensates for the limitations of using open text features. pseudo The resulting problems, such as slow training and insufficient feature supervision information, are calculated as follows:

[0137] s know =D(Z) know C know );

[0138]

[0139] Among them, Zknow It is to extract the output features contained in the known class C. know Feature points within the range, s know It is the predicted similarity score of the known classes, g know It is the one-hot encoded vector of the corresponding region;

[0140] S33. Calculate the total training loss of model M. Wherein, λ1, λ2, and λ3 are the cross-entropy losses of open text features, respectively. Comparative learning loss Learning loss with known classes The weights of model M are updated using gradient descent, and the operation S3 is repeated multiple times to obtain a model with high prediction performance in open scene understanding tasks.

[0141] The gradient descent method works as follows: after calculating... Based on this, calculate gradient with respect to parameter θ And update the parameter θ in the gradient direction, as follows:

[0142]

[0143] Where α represents the learning rate.

[0144] S4, Training the model.

[0145] S41, The training set C train With known class C know The labeled information is input into the model M designed by S3, and the model is trained using the gradient descent method;

[0146] S42. During the training phase, the validation set C is used simultaneously. val With all object label set C class Validate the model training effect, that is, when model M is on the validation set C val It has gained a better understanding of open-world scenarios and is capable of handling unknown C-class applications. unknow With known class C know When the prediction performance reaches a good level of accuracy, and this accuracy can no longer be improved during training, training is stopped, and finally, model M is obtained after training is completed. * .

[0147] S5, Test Model.

[0148] S51, Input test dataset C test It includes 3D point cloud data V and the text conditions L to be queried; the model used in the testing phase is model M. * Feature extractor E, pre-trained text feature extractor θtext Multimodal discriminator D;

[0149] S52, Test set C test Model M obtained from input S4 * The test set C is obtained from the middle. test The open scene prediction results include predictions for known class C. know and unknown class C unknow The predictive performance.

[0150] Therefore, this invention employs the aforementioned two-stream feature fusion-based 3D point cloud open scene understanding method, achieving 3D point cloud scene understanding on open sets and significantly improving the accuracy of tasks such as open set semantic segmentation. Specifically, it uses top-down and bottom-up two-stream methods combined with prototype clustering to extract richer and more accurate textual description information from unlabeled data, alleviating the problems of insufficient feature extraction and misleading features in existing methods. Through prototype contrastive learning and cross-modal knowledge distillation, the open scene description features of the 2D image modality are transferred to the 3D point cloud modality, further reducing the differences in features between modalities and improving the ability to express open scene features in the 3D point cloud modality, thereby enhancing the performance of open scene understanding tasks.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for understanding open scenes of 3D point clouds based on dual-stream feature fusion, characterized in that, Includes the following steps: S1. Data preparation and dataset partitioning; S2. Construct 3D point cloud open scene supervision information; including: S21. Constructing bottom-up foreground interest region information; specifically: Based on the acquired 2D image modalities, a sliding window or a pre-trained model is used. Predicting the foreground region and obtaining the corresponding 3D point cloud coordinate range of the foreground region through a transformation matrix is ​​represented as follows: ; ; in, This represents the set of foreground regions obtained through a bottom-up approach. This represents the set obtained by projecting the foreground region obtained by the bottom-up method onto three-dimensional space; S22. Constructing top-down foreground interest region information; specifically: Based on the acquired 3D point cloud modalities, a pre-trained model is used. Predicting the foreground region involves mapping the region of the 3D point cloud to a 2D image space using a camera pose rotation matrix. This process can be represented as follows: ; ; in, and These represent the sets of foreground regions in three-dimensional and two-dimensional space obtained by the top-down method, respectively. A scene representing a certain 3D point cloud data; S23. Construct open scene semantic information from the foreground interest region; specifically: Integrated acquisition of foreground interest regions and The foreground region of the two streams is obtained. And use pre-trained images to describe the model. Obtain the image text description information of the foreground interest region as an open scene description tag set. ; through pre-trained large language models Get the set of open scene description tags The set of foreground entity noun information in This is used to construct bottom-up and top-down semantic prototypes, represented as: ； ； ； ； in, express Areas in 3D space; It is semantic description information; This is entity noun information; S24. Integrate bottom-up and top-down dual-flow feature information; specifically: The obtained foreground region of the two streams Using a pre-trained image encoder Obtaining image features The obtained subclass features of the same entity are averaged to establish class prototype centers, which are then used to establish the image feature prototypes of the classes. By calculating the cosine distance between image features and their corresponding feature prototypes, a set of open scene description labels is selected. High-quality samples are used to construct the training set. pseudo-label semantic supervision information Using the relationship between the foreground region and the corresponding 3D point cloud space Establishing 3D point cloud and semantic supervision information Based on the correspondence, the open semantic feature dataset of 3D point clouds is constructed, and the process is represented as follows: ； ； ； in, It is an average function. It is a cosine distance function. It is the threshold for cosine distance filtering; S3. Design model; the specific steps are as follows: S31, the overall model is denoted as , Includes a 3D point cloud feature extractor Pre-trained text feature extractor With multimodal discriminator ,in The CLIP text encoder was used, and the model input was a multimodal 3D point cloud dataset. 3D point cloud feature extractor Receive 3D point cloud data Output 3D point cloud features , Represented as: ； Pre-trained text feature extractor Used to receive open-scene text semantic supervision information after dual-stream fusion. The encoded output is the text feature of the target region. ; A cross-modal Transformer is used to bridge the gap between feature distributions, and an attention mechanism is employed to enhance the feature representation of the target region. The cross-modal Transformer is defined as follows: , , , , , These represent the query, key, and value, respectively. ； in, , , All of these are learnable parameters. Represents enhancement features from text modality to 3D point cloud modality; S32. Using contrastive learning, open text description features are distilled into a 3D point cloud network to enhance the features. After multi-layer MLP processing, combined with text features Together, they are fed into the multimodal discriminator D, which outputs enhanced features. Each point in the text features Cosine similarity prediction score The output is calculated using cross-entropy. With text feature regions The matching relationship between them is defined as: ； ； in, for Match the one-hot encoded vector of the corresponding region. It is the cross-entropy loss, calculated as follows: ； in, It is a predicted value. It is the truth value of the label; Use contrastive learning to enhance the loss between two open scene features. The calculation is as follows: ； in, It is a text feature of open scene The Middle 1 eigenvector It is an enhanced feature Zhongyu Positive sample regions corresponding to text features This is the negative sample region corresponding to the current sample. It is the total number of text features. It is a temperature parameter; For the training set Known classes For objects, supervised learning of known classes is performed using real object labels and cross-entropy, calculated as follows: ； ； in, It takes the output features contained in the known class. Feature points within the range, It is a similarity prediction score for known classes. It is the one-hot encoded vector of the corresponding region; S33. Calculate the total training loss of model M. ,in, , and These are open text feature cross-entropy loss. Comparative learning loss Learning loss with known classes The weights of model M are updated using gradient descent, and the operation S3 is repeated several times to obtain a model with high prediction performance in open scene understanding tasks. The gradient descent method works as follows: after calculating... Based on this, calculate For parameters gradient And update the parameters in that gradient direction. The formula is: ； in, Indicates the learning rate; S4, Training the model; S5, Test Model.

2. The method for understanding open scenes of 3D point clouds based on dual-stream feature fusion according to claim 1, characterized in that, S1 specifically includes: S11. Data Preparation: Collect 3D point cloud data Each 3D point cloud data There is a corresponding two-dimensional image sequence. Each frame of an image sequence By mapping the world coordinates to camera coordinates onto the 3D point cloud scene using a transformation matrix, the features in the 3D point cloud are aligned one-to-one with the features in the 2D image. The calculation formula is as follows: ； in, It refers to the position of a pixel in a two-dimensional image in camera coordinates. It refers to the position of a point in a 3D point cloud in world coordinates. It is the camera's intrinsic parameter matrix. It is the camera extrinsic parameter matrix; the transformation process from the camera coordinate system of the 2D image to the world coordinate system of the 3D point cloud is denoted as... The process of transforming the world coordinate system of a 3D point cloud to the camera coordinate system of a 2D image is denoted as... Then, the 3D point cloud data is labeled to obtain a 3D point cloud open scene understanding dataset. The annotations are the object category labels in the 3D point cloud, using... Represents the set of all object labels; S12. Divide the dataset: Divide the obtained 3D point cloud open scene understanding dataset. Divided into training set Validation set and test set Furthermore, based on the object category labels, the labeled category labels should be further divided into known classes. With unknown class , represented as: ; 。 3. The method for understanding open scenes of 3D point clouds based on dual-stream feature fusion according to claim 1, characterized in that, The specific steps of S4 are as follows: S41, Transfer the training set With known classes The annotation information is input into the model designed by S3. In this process, gradient descent is used for model training. S42. The validation set is used simultaneously during the training phase. With all object tag sets Verify the model training effect, and finally obtain the model after training is completed. .

4. The method for understanding open scenes of 3D point clouds based on dual-stream feature fusion according to claim 3, characterized in that, The specific steps of S5 are as follows: S51, Input test dataset Includes 3D point cloud data With the text conditions to be queried The model used in the testing phase is a model. Feature extractor in Pre-trained text feature extractor Multimodal discriminator D; S52, Test set Model obtained by inputting S4 The test set was obtained from the middle. The open scene prediction results include predictions for known classes. and unknown class The predictive performance.

Images