Three-dimensional point cloud open scene understanding method based on double-flow feature fusion

Abstract

The invention discloses a three-dimensional point cloud open scene understanding method based on double-flow feature fusion, and belongs to the field of multi-modal learning, and the method comprises the steps: preparing data, collecting three-dimensional point cloud data V, carrying out the data marking, obtaining a three-dimensional point cloud open scene understanding data set A, and carrying out the data set division of the obtained three-dimensional point cloud open scene understanding data set A; constructing three-dimensional point cloud open scene supervision information, constructing a foreground interest region, constructing open semantic information, and fusing double-flow feature information; designing a model M comprising a three-dimensional point cloud feature extractor E, a pre-trained text feature extractor theta text and a multi-modal discriminator D; training the model M to obtain a model M *; testing on the model M *; according to the three-dimensional point cloud open scene understanding method based on double-flow feature fusion, three-dimensional point cloud scene understanding on an open set is achieved, and the precision of tasks such as open set semantic segmentation is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of multimodal learning, and in particular to a three-dimensional point cloud open scene understanding method based on dual-stream feature fusion. Background Art

[0002] Visual tasks related to three-dimensional point clouds have the characteristics of high cost of data collection by lidar equipment, high cost of manual annotation of large-scale data, and late start of development of related technologies for three-dimensional point clouds. The development of computer vision lags behind that of two-dimensional images.

[0003] The field of two-dimensional images has achieved rapid development in recent years due to its low cost of data acquisition and annotation and its long development history. In recent years, a series of multimodal models based on text and image data pairs have been produced. Through the alignment and training of text and image pairs, related tasks of open scene understanding in the field of two-dimensional images have been realized, such as open vocabulary image classification, segmentation, target detection and other tasks.

[0004] However, the success of open scene understanding in the field of two-dimensional images is based on a large amount of training data of text and image pairs. The field of three-dimensional point clouds lacks such large-scale data and it is difficult to obtain it. How to transfer the ability of open scene understanding from the two-dimensional image field to the three-dimensional point cloud field based on unlabeled three-dimensional point cloud data has become an urgent problem that needs to be solved. Summary of the invention

[0005] The purpose of the present invention is to provide a three-dimensional point cloud open scene understanding method based on dual-stream feature fusion, so as to solve the problem that the existing three-dimensional point cloud open scene understanding technology does not fully utilize the differences between the three-dimensional point cloud modality and the two-dimensional point cloud modality to enhance the feature extraction and understanding capabilities.

[0006] To achieve the above object, the present invention provides a three-dimensional point cloud open scene understanding method based on dual-stream feature fusion, comprising the following steps:

[0007] S1. Data preparation and data set division;

[0008] S2. Constructing 3D point cloud open scene supervision information; including:

[0009] S21, constructing bottom-up foreground interest area information;

[0010] S22, constructing top-down foreground interest area information;

[0011] S23, constructing open scene semantic information from foreground interest regions;

[0012] S24. Fusion of bottom-up and top-down dual-stream feature information.

[0013] S3. Design model, including a 3D point cloud feature extractor E and a pre-trained text feature extractor θ text Model M with multimodal discriminator D;

[0014] S4, training model, training model M to obtain model M * ;

[0015] S5. Test model.

[0016] Preferably, S1 specifically includes:

[0017] S11. Data preparation: Collect 3D point cloud data V. Each 3D point cloud data V has a corresponding 2D image sequence P. Each frame image P in the image sequence i The transformation matrix between the world coordinates and the camera coordinates is mapped to the 3D point cloud scene, and the features in the 3D point cloud are aligned one by one with the features in the 2D image. The calculation formula is as follows:

[0018]

[0019] Among them, (u,v) is the position of the pixel in the two-dimensional image in the camera coordinates, (x,y,z) is the position of the point in the three-dimensional point cloud in the world coordinates, is the camera’s intrinsic parameter matrix, [R 3×3 T 3×1 ] is the camera external parameter matrix; the conversion process from the camera coordinate system of the two-dimensional image to the world coordinate system of the three-dimensional point cloud is recorded as R 2D→3D (·), the conversion process from the world coordinate system of the 3D point cloud to the camera coordinate system of the 2D image is recorded as R 3D→2D (·); Then the 3D point cloud data is annotated to obtain the 3D point cloud open scene understanding dataset A, where the annotation content is the object category label in the 3D point cloud, and C class Represents the set of all annotated object labels;

[0020] S12. Divide the dataset: Divide the obtained 3D point cloud open scene understanding dataset A into training set C train , validation set C val and test set C test ; According to the object category label, the annotated category label is further divided into known categories C know With unknown class C unknow , expressed as:

[0021] A={C train ,C val ,C test};

[0022] C class ={Cknow ,C unknow};

[0023] Known Class C know The labels of participate in training and testing, the unknown class C unknow The labels do not participate in training, but need to be predicted during testing.

[0024] Preferably, S21 is specifically:

[0025] Based on the acquired 2D image modality, using a sliding window or pre-trained model θ 2D , such as the open set two-dimensional detection network DINO and the open set two-dimensional segmentation network SAM, predict the foreground area from the image data, and obtain the three-dimensional point cloud coordinate range corresponding to the foreground area through the transformation matrix. The process is expressed as:

[0026] C bottom-up,2D ={θ 2D (P i )};

[0027] C bottom-up,3D =R 2D→3D (C bottom-up,2D );

[0028] Among them, C bottom-up,2D represents the set of foreground regions obtained through the bottom-up method, C bottom-up,3D It represents the set obtained by projecting the foreground area obtained by the bottom-up method into the three-dimensional space. The bottom-up method is more capable of mining small objects in the scene through the texture information in the image and establishing the feature information of small objects. However, it is difficult to capture the complete feature information of objects at the edge of the image frame.

[0029] Preferably, S22 is specifically:

[0030] Based on the collected 3D point cloud modality, using the pre-trained model θ 3D For example, the open set 3D detection network OV3D predicts the foreground area from the point cloud data, and maps the 3D point cloud area to the 2D image space through the camera pose rotation matrix. The process is expressed as:

[0031] C top-down,3D ={θ 3D (V i )};

[0032] C top-down,2D =R 3D→2D (C top-down,3D );

[0033] Among them, C top-down,3D and C top-down,2DRespectively represent the set of foreground regions in 3D space and 2D space obtained by the top-down method, V i To represent a scene with 3D point cloud data, the top-down method starts from the 3D point cloud and focuses on the 3D structure information. It can more accurately construct the boundary information of 3D objects and establish the information of large objects across image frames.

[0034] Preferably, S23 is specifically:

[0035] Integrate the obtained foreground interest area C bottom-up,2D With C top-down,2D , get the foreground area C of the dual stream TDBU,2D , and use the pre-trained image description model θ cap Get the image text description information of the foreground interest area as the open scene description label set C cap ; Through pre-training large language model θ LLM Get the open scene description label set C cap The foreground entity noun information set C in entity , in order to construct bottom-up and top-down semantic prototypes, expressed as:

[0036] C TDBU,2D ={C top-down,2D ,C bottom-up,2D};

[0037] C TDBU,3D ={C top-down,3D ,C bottom-up,3D};

[0038] C cap =θ cap (C TDBU,2D );

[0039] C entity =θ LLM (C cap );

[0040] Among them, C TDBU,3D Represents C TDBU,2D Area in 3D space; C cap is the semantic description information; C entity It is the entity noun information.

[0041] Preferably, S24 is specifically:

[0042] The obtained dual-stream foreground area C TDBU,2D Use the pre-trained image encoder θ image For example, the CLIP image encoder obtains the image feature C image, the subclass features of the same entity are obtained by averaging to establish the class prototype center, that is, to establish the image feature prototype C of the class prototype , filter the open scene description label set C by the cosine distance between the image feature and the corresponding feature prototype cap Medium and high quality samples, construct training set C train The pseudo-label semantic supervision information C pseudo , using the corresponding relationship C of the foreground area in the three-dimensional point cloud space TDBU,3D , build 3D point cloud and semantic supervision information C pseudo The corresponding relationship is used to complete the construction of the open semantic feature dataset of the 3D point cloud. The process is expressed as follows:

[0043] C image =θ image (C TDBU,2D );

[0044] C prototype ={AVG(C image,i )|i∈C entity};

[0045] C pseudo ={C cap ,if cos(C image,i ,C prototype,i )<ε|i∈C entity};

[0046] Among them, AVG(·) is the average function, cos(·) is the cosine distance function, and ε is the threshold for cosine distance screening.

[0047] Preferably, the specific steps of S3 are:

[0048] S31, the overall model is denoted as M, M includes a 3D point cloud feature extractor E, a pre-trained text feature extractor θ text and the multimodal discriminator D, where θ text The text encoder of CLIP is used. The model input is a multimodal 3D point cloud dataset A; the 3D point cloud feature extractor E receives the 3D point cloud data V and outputs the 3D point cloud features X is represented by:

[0049] X = E(V);

[0050] Pre-trained text feature extractor θ text Responsible for receiving the open scene text semantic supervision information C after dual-stream fusion pseudo , the encoding output is the target area text feature Y;

[0051] Y=θ text (Cpseudo );

[0052] In order to alleviate the problem of obvious distribution gap between 3D point cloud feature X and text feature Y, since the features of different modalities focus on expressing the characteristics of different modalities, a cross-modal Transformer is used to fill the gap between feature distributions, and an attention mechanism is used to strengthen the feature expression of the target area; the cross-modal Transformer is defined as: Q X =XP q , K Y =YP k , V Y =YP v , Q X , K Y 、V Y Represents query, key, and value respectively;

[0053]

[0054] Among them, P q , P k , P v is a learnable parameter, and Z represents the enhanced features from the text modality to the 3D point cloud modality;

[0055] S32. Use contrastive learning to distill the open text description features into a three-dimensional point cloud network. After the enhanced feature Z is processed by a multi-layer MLP, it is sent to the multimodal discriminator D together with the text feature Y. The multimodal discriminator outputs the cosine similarity prediction score s between each point in the enhanced feature Z and the text feature Y. The cross entropy is used to calculate the output s and the text feature area C. pseudo The matching relationship between them is defined as:

[0056] s=D(Z,Y);

[0057]

[0058] Where g is C pseudo Match the one hot encoding vector of the corresponding region, BCELoss(·,·) is the cross entropy loss, calculated as:

[0059]

[0060] in, is the predicted value, y is the true value of the label;

[0061] In order to further narrow the similarity of cross-modal features of the same type of objects and widen the difference between features of different types, contrastive learning is used to enhance the loss between features of two open scenes. Calculated as:

[0062]

[0063] Among them, y i is the i-th feature vector in the text feature Y of the open scene, z i is the enhanced feature Z and y i The positive sample area corresponding to the text feature, z j is the negative sample area corresponding to the current sample, N is the total number of text features, and τ is the temperature parameter;

[0064] For the training set C train Known class C class For objects, the real object labels and cross entropy are used to supervise the learning of known classes, which is calculated as:

[0065] s know =D(Z know ,C know );

[0066]

[0067] Among them, Z know It is to take the output features contained in the known class C know Feature points within the range, s know is the similarity prediction score of known classes, g know is the one hot encoding vector of the corresponding region;

[0068] S33. Calculate the total training loss of model M Among them, λ 1 , 2 and λ 3 They are open text feature cross entropy loss Contrastive Learning Loss and known class learning loss weights; update the weights of model M using the gradient descent method, and repeat operation S3 several times to obtain a model with high prediction performance in the open scene understanding task;

[0069] The process of gradient descent is as follows: Based on The gradient of the parameter θ And update the parameter θ in the gradient direction, the formula is:

[0070]

[0071] Among them, α represents the learning rate.

[0072] Preferably, the specific steps of S4 are:

[0073] S41, training set C train With known class C know The annotation information is input into the model M designed by S3, and the model is trained using the gradient descent method;

[0074] S42, the validation set C is used during the training phase val With all object label sets C class Verify the model training effect, that is, when the model M is in the verification set C val We can obtain better open scene understanding ability and unknow With known class C know When the prediction performance of the models reaches a good accuracy and the accuracy cannot be improved with the training process, the training is stopped and the model M is obtained after the training is completed. * .

[0075] Preferably, the specific steps of S5 are:

[0076] S51. Input test data set C test , including 3D point cloud data V and text conditions L to be queried; the model used in the test phase is model M * The feature extractor E in the pre-trained text feature extractor θ test , multimodal discriminator D;

[0077] S52, test set C test Input S4 to obtain the model M * To get the test set C test The open scene prediction results include the prediction results for the known class C know and unknown class C unknow prediction performance.

[0078] Therefore, the present invention adopts the above-mentioned three-dimensional point cloud open scene understanding method based on dual-stream feature fusion, which has the following beneficial effects:

[0079] (1) The top-down and bottom-up dual-stream methods are used to obtain open semantic features. This not only makes full use of the geometric information in the three-dimensional space and the local details in the two-dimensional image, but also generates high-quality text descriptions and semantic features through the combination of the image description model and the CLIP model, thereby significantly improving the performance of object detection and semantic understanding.

[0080] (2) A large language model is used to extract and filter entity nouns from text descriptions, and the newly added entity nouns are used as unknown class targets. Then, the unknown class center is established using the average clustering method, and the image features of the candidate foreground areas are screened to filter out inaccurate candidate foreground areas, thereby obtaining candidate foreground areas with higher accuracy and the corresponding text description information.

[0081] (3) The 3D point cloud open scene understanding method based on prototype contrast learning and cross-modal knowledge distillation significantly improves the ability of open scene understanding by optimizing feature extraction, strengthening sample matching, realizing cross-modal knowledge transfer, and improving multimodal fusion performance.

[0082] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 This is a flow chart of a three-dimensional point cloud open scene understanding method based on dual-stream feature fusion in the present invention;

[0084] Figure 2 This is a network schematic diagram of a three-dimensional point cloud open scene understanding method based on dual-stream feature fusion in the present invention. DETAILED DESCRIPTION

[0085] The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] Example

[0087] like Figure 1 As shown, a three-dimensional point cloud open scene understanding method based on dual-stream feature fusion includes the following steps:

[0088] S1. Data preparation and data set division;

[0089] S11. Data preparation: Collect a large amount of 3D point cloud data V, where each 3D point cloud data Y has a corresponding 2D image sequence P. Each frame image P in the image sequence i The transformation matrix between the world coordinates and the camera coordinates is mapped to the 3D point cloud scene, and vice versa. The features in the 3D point cloud are aligned one by one with the features in the 2D image. The calculation formula is as follows:

[0090]

[0091] Among them, (u,v) is the position of the pixel in the two-dimensional image in the camera coordinates, (x,y,z) is the position of the point in the three-dimensional point cloud in the world coordinates, is the camera’s intrinsic parameter matrix, [R 3×3 T 3×1 ] is the camera external parameter matrix; the conversion process from the camera coordinate system of the two-dimensional image to the world coordinate system of the three-dimensional point cloud is recorded as R 2D→3D(·), the conversion process from the world coordinate system of the 3D point cloud to the camera coordinate system of the 2D image is recorded as R 3D→2D (·); Then the 3D point cloud data is annotated to obtain the 3D point cloud open scene understanding dataset A, where the annotation content is the object category label in the 3D point cloud, and C class Represents the set of all annotated object labels;

[0092] S12. Divide the dataset: Divide the obtained 3D point cloud open scene understanding dataset A into training set C train , validation set C val and test set C test ; It is necessary to further divide the labeled category labels into known categories C according to the object category labels know With unknown class C unknow , expressed as:

[0093] A={C train ,C val ,C test};

[0094] C class ={C know ,C unknow};

[0095] Known Class C know The labels of participate in training and testing, the unknown class C unknow The labels do not participate in training, but need to be predicted during testing.

[0096] S2. Constructing 3D point cloud open scene supervision information; including:

[0097] S21. Construct bottom-up foreground area of ​​interest information.

[0098] Based on the acquired 2D image modality, using a sliding window or pre-trained model θ 2D , such as the open set two-dimensional detection network DINO and the open set two-dimensional segmentation network SAM, predict the foreground area from the image data, and obtain the three-dimensional point cloud coordinate range corresponding to the foreground area through the transformation matrix. The process is expressed as:

[0099] C bottom-up,2D ={θ 2D (P i )};

[0100] C bottom-up,3D =R 2D→3D (C bottom-up,2D );

[0101] Among them, C bottom-up,2D represents the set of foreground regions obtained through the bottom-up method, C bottom-up,3DIt represents the set obtained by projecting the foreground area obtained by the bottom-up method into the three-dimensional space. The bottom-up method is more capable of mining small objects in the scene through the texture information in the image and establishing the feature information of small objects. However, it is difficult to capture the complete feature information of objects at the edge of the image frame.

[0102] S22. Constructing top-down foreground area of ​​interest information.

[0103] Based on the collected 3D point cloud modality, using the pre-trained model θ 3D For example, the open set 3D detection network OV3D predicts the foreground area from the point cloud data, and maps the 3D point cloud area to the 2D image space through the camera pose rotation matrix. The process is expressed as:

[0104] C top-down,3D ={θ 3D (V i )};

[0105] C top-down,2D =R 3D→2D (C top-down,3D );

[0106] Among them, C top-down,3D and C top-down,2D Respectively represent the set of foreground regions in 3D space and 2D space obtained by the top-down method, V i To represent a scene with 3D point cloud data, the top-down method starts from the 3D point cloud and focuses on the 3D structure information. It can more accurately construct the boundary information of 3D objects and establish the information of large objects across image frames.

[0107] S23. Construct open scene semantic information from foreground interest regions.

[0108] Integrate the obtained foreground interest area C bottom-up,2D With C top-down,2D , get the foreground area C of the dual stream TDBU,2D , and use the pre-trained image description model θ cap , obtain the image text description information of the foreground interest area as the open scene description label set θ cap , by pre-training a large language model θ LLM , such as ChatGPT extracts and filters entity noun information set C from description information entity , construct bottom-up and top-down semantic prototypes, expressed as:

[0109] C TDBU,2D ={C top-down,2D ,C bottom-up,2D};

[0110] C TDBU,3D ={Ctop-down,3D ,C bottom-up,3D};

[0111] C cap =θ cap (C TDBU,2D );

[0112] C entity =θ LLM (C cap ).

[0113] S24. Fusion of bottom-up and top-down dual-stream feature information.

[0114] The obtained dual-stream foreground area C TDBU,2D Use the pre-trained image encoder θ image , such as CLIP image encoder to obtain image features C image , the subclass features of the same entity are obtained by averaging to establish the class prototype center, and the image feature prototype C of the class is established prototype Since there may be some wrongly predicted foreground areas, this embodiment calculates the cosine distance between the image features and the corresponding feature prototypes, removes the wrongly classified samples of outliers, and constructs a more accurate training set C for the open scene understanding task. train Pseudo-label semantic supervision information C pseudo , using the corresponding relationship C of the foreground area in the three-dimensional point cloud space TDBU,3D , build 3D point cloud and semantic supervision information C pseudo The corresponding relationship is used to complete the construction of the open semantic feature dataset of the 3D point cloud. The process is expressed as follows:

[0115] C image =θ image (C TDBU,2D );

[0116] C prototype ={AVG(C image,i )|i∈C entity};

[0117] C pseudo ={C cap ,if cos(C image,i ,C prototype,i )<ε|i∈C entity};

[0118] Among them, AVG(·) is the average function, cos(·) is the cosine distance function, and ε is the threshold for cosine feature screening.

[0119] S3. Design model.

[0120] S31, the overall model is denoted as M, M includes a 3D point cloud feature extractor E, a pre-trained text feature extractor θ text and the multimodal discriminator D, where θ text The text encoder of CLIP is used. The model input is a multimodal 3D point cloud dataset A; the 3D point cloud feature extractor E receives the 3D point cloud data V and outputs the 3D point cloud features X is represented by:

[0121] X = E(V);

[0122] Pre-trained text feature extractor θ text Responsible for receiving the open scene text semantic supervision information C after dual-stream fusion pseudo , the encoding output is the target area text feature Y:

[0123] Y=θ text (C pseudo );

[0124] For the 3D point cloud feature X and the text feature Y, since the features of different modalities focus on expressing the characteristics of different modalities, their distribution gap is obvious. To alleviate this problem, this embodiment uses a cross-modal Transformer to make up for the gap between feature distributions and uses an attention mechanism to strengthen the feature expression of the target area; the cross-modal Transformer is defined as: Q X =XP q , K Y =YP k , V Y =YP v , Q X , K Y 、V Y Represents query, key, and value respectively;

[0125]

[0126] Among them, P q , P k , P v is a learnable parameter, and Z represents the enhanced features from the text modality to the 3D point cloud modality. In the process of 3D point cloud encoding, the description text is injected into the 3D point cloud features through the cross-modal Transformer mechanism, and the foreground area features are enhanced through the attention mechanism to obtain multi-modal feature output.

[0127] S32. Use contrastive learning to distill the open text description features into a three-dimensional point cloud network. After the enhanced feature Z is processed by a multi-layer MLP, it is sent to the multimodal discriminator D together with the text feature Y. The multimodal discriminator outputs the cosine similarity prediction score s between each point in the enhanced feature Z and the text feature Y. The cross entropy is used to calculate the output s and the text feature area C. pseudo The matching relationship between them is defined as:

[0128] s=D(Z,Y);

[0129]

[0130] Where g is C pseudo Match the one hot encoding vector of the corresponding region, BCELoss(·,·) is the cross entropy loss, calculated as:

[0131]

[0132] in, is the predicted value and y is the true label value.

[0133] In order to further narrow the similarity of cross-modal features of the same type of objects and widen the difference between features of different types, contrastive learning is used to enhance the loss between features of two open scenes. Calculated as:

[0134]

[0135] Among them, y i is the i-th feature vector in the text feature Y of the open scene, z i is the enhanced feature Z and y i The positive sample area corresponding to the text feature, z j is the negative sample area corresponding to the current sample, N is the total number of text features, and τ is the temperature parameter.

[0136] For the training set C train Known class C class The real object labels and cross entropy are used to supervise the learning of known classes. A small amount of annotation information is used to strengthen the feature extraction ability of 3D point clouds, making up for the lack of open text features C pseudo The resulting slow training and insufficient feature supervision information are calculated as:

[0137] s know =D(Z know ,C know );

[0138]

[0139] Among them, Zknow It is to take the output features contained in the known class C know Feature points within the range, s know is the similarity prediction score of the known class, g know is the one hot encoding vector of the corresponding region;

[0140] S33. Calculate the total training loss of model M Among them, λ 1 , 2 and λ 3 They are open text feature cross entropy loss Contrastive Learning Loss and known class learning loss weights; update the weights of model M using the gradient descent method, and repeat operation S3 multiple times to obtain a model with high prediction performance in the open scene understanding task;

[0141] The process of gradient descent is as follows: Based on The gradient of the parameter θ And update the parameter θ in the gradient direction, the formula is:

[0142]

[0143] Among them, α represents the learning rate.

[0144] S4. Train the model.

[0145] S41, training set C train With known class C know The annotation information is input into the model M designed by S3, and the model is trained using the gradient descent method;

[0146] S42, the validation set C is used during the training phase val With all object label sets C class Verify the model training effect, that is, when the model M is in the verification set C val We can obtain better open scene understanding ability and unknow With known class C know When the prediction performance of the models reaches a good accuracy and the accuracy cannot be improved with the training process, the training is stopped and the model M is obtained after the training is completed. * .

[0147] S5. Test model.

[0148] S51. Input test data set C test , including 3D point cloud data V and text conditions L to be queried; the model used in the test phase is model M* The feature extractor E in the pre-trained text feature extractor θ text , multimodal discriminator D;

[0149] S52, test set C test Input S4 to obtain the model M * To get the test set C test The open scene prediction results include the prediction results for the known class C know and unknown class C unknow prediction performance.

[0150] Therefore, the present invention adopts the above-mentioned three-dimensional point cloud open scene understanding method based on dual-stream feature fusion, realizes three-dimensional point cloud scene understanding on open sets, and significantly improves the accuracy of tasks such as open set semantic segmentation. In particular, the top-down and bottom-up dual-stream methods and the prototype clustering method are used to extract more abundant and accurate text description information from unlabeled data to alleviate the problems of insufficient feature extraction and misleading erroneous features in the original method. Through prototype contrast learning and cross-modal knowledge distillation, the open scene description features of the two-dimensional image modality are migrated to the three-dimensional point cloud modality, and the difference in features between the modalities is further reduced, thereby improving the open scene feature expression capability in the three-dimensional point cloud modality, thereby improving the performance of the open scene understanding task.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A three-dimensional point cloud open scene understanding method based on dual-stream feature fusion, characterized in that: The following steps are involved: S1. Data preparation and data set division; S2. Constructing 3D point cloud open scene supervision information; including: S21, constructing bottom-up foreground interest area information; S22, constructing top-down foreground interest area information; S23, constructing open scene semantic information from foreground interest regions; S24. Fusion of bottom-up and top-down dual-stream feature information. S3, design model; S4, training model; S5. Test model.

2. The method for understanding a three-dimensional point cloud open scene based on dual-stream feature fusion according to claim 1, characterized in that: S1 specifically includes: S11. Data preparation: Collect 3D point cloud data V. Each 3D point cloud data V has a corresponding 2D image sequence P. Each frame image P in the image sequence i The transformation matrix between the world coordinates and the camera coordinates is mapped to the 3D point cloud scene, and the features in the 3D point cloud are aligned one by one with the features in the 2D image. The calculation formula is as follows: Among them, (u,v) is the position of the pixel in the two-dimensional image in the camera coordinates, (x,y,z) is the position of the point in the three-dimensional point cloud in the world coordinates, is the camera’s intrinsic parameter matrix, [R 3×3 T 3×1 ] is the camera external parameter matrix; the conversion process from the camera coordinate system of the two-dimensional image to the world coordinate system of the three-dimensional point cloud is recorded as R 2D→3D (·), the conversion process from the world coordinate system of the 3D point cloud to the camera coordinate system of the 2D image is recorded as R 3D→2D (·); Then the 3D point cloud data is annotated to obtain the 3D point cloud open scene understanding dataset A, where the annotation content is the object category label in the 3D point cloud, and C class Represents the set of all annotated object labels; S12. Divide the dataset: Divide the obtained 3D point cloud open scene understanding dataset A into training set C train , validation set C val and test set C test ; According to the object category label, the annotated category label is further divided into known categories C know With unknown class C unknow , expressed as: A={C train ,C val ,C test }; C class ={C know ,C unknow }。 3. The method for understanding a three-dimensional point cloud open scene based on dual-stream feature fusion according to claim 1, characterized in that: S21 is specifically: Based on the acquired 2D image modality, using a sliding window or pre-trained model θ 2D Predict the foreground area and obtain the three-dimensional point cloud coordinate range corresponding to the foreground area through the transformation matrix. The process is expressed as: C bottom-up,2D J{θ 2D (P i )}R C bottom-up,3D =R 2D→3D (C bottom-up,2D ); Among them, C bottom-up,2D represents the set of foreground regions obtained through the bottom-up method, C bottom-up,3D Represents the set obtained by projecting the foreground area obtained by the bottom-up method into the three-dimensional space.

4. The method for understanding a three-dimensional point cloud open scene based on dual-stream feature fusion according to claim 3, characterized in that: S22 is specifically: Based on the collected 3D point cloud modality, using the pre-trained model θ 3D Predict the foreground area and map the three-dimensional point cloud area to the two-dimensional image space through the camera position rotation matrix. The process is expressed as: C top-down,3D ={θ 3D (V i )}; C top-down,2D =R 3D→2D (C top-down,3D ); Among them, C top-down,3D and C top-down,2D Respectively represent the set of foreground regions in 3D space and 2D space obtained by the top-down method, V i A scene representing a 3D point cloud data.

5. The method for understanding a three-dimensional point cloud open scene based on dual-stream feature fusion according to claim 4, characterized in that: S23 is specifically: Integrate the obtained foreground interest area C bottom-up,2D With C top-down,2D , get the foreground area C of the dual stream TDBU,2D , and use the pre-trained image description model θ cap Get the image text description information of the foreground interest area as the open scene description label set C cap ; Through pre-training large language model θ LLM Get the open scene description label set C cap The foreground entity noun information set C in entity , used to construct bottom-up and top-down semantic prototypes, expressed as: C TDBU,2D ={C top-down,2D ,C bottom-up,2D }; C TDBU,3D ={C top-down,3D ,C bottom-up,3D }; C cap =θ cap (C TDBU,2D ); C entity =θ LLM (C cap ); Among them, C TDBU,3D Represents C TDBU,2D Area in 3D space; C cap is the semantic description information; C entity It is the entity noun information.

6. The method for understanding a three-dimensional point cloud open scene based on dual-stream feature fusion according to claim 5, characterized in that: S24 is specifically: The obtained dual-stream foreground area C TDBU,2D Use the pre-trained image encoder θ image Get image feature C image , the sub-class features of the same entity are obtained by averaging to establish the class prototype center, which is used to establish the image feature prototype C of the class prototype , by calculating the cosine distance between the image features and the corresponding feature prototypes, the open scene description label set C is screened cap Medium and high quality samples, construct training set C train The pseudo-label semantic supervision information C pseudo , using the corresponding relationship C of the foreground area in the three-dimensional point cloud space TDBU,3D , build 3D point cloud and semantic supervision information C pseudo The corresponding relationship is used to complete the construction of the open semantic feature dataset of the 3D point cloud. The process is expressed as follows: C image =θ image (C TDBU,2D ); C prototype ={AVG(C image,i )|i∈C entity }; C pseudo ={C cap ,if cos(C image,i ,C prototype,i )<ε|i∈C entity }; Among them, AVG(·) is the average function, cos(·) is the cosine distance function, and ε is the threshold for cosine distance screening.

7. The method for understanding a three-dimensional point cloud open scene based on dual-stream feature fusion according to claim 2, characterized in that: The specific steps of S3 are: S31, the overall model is denoted as M, M includes a 3D point cloud feature extractor E, a pre-trained text feature extractor θ text and the multimodal discriminator D, where θ text The text encoder of CLIP is used. The model input is a multimodal 3D point cloud dataset A. The 3D point cloud feature extractor E receives the 3D point cloud data V and outputs the 3D point cloud features. X is represented by: X = E(V); Pre-trained text feature extractor θ text Used to receive the open scene text semantic supervision information C after dual-stream fusion pseudo , the encoding output is the target area text feature Y; Use cross-modal Transformer to bridge the gap between feature distributions, and use attention mechanism to strengthen the feature expression of the target area; The cross-modal Transformer is defined as: Q X =XP q , k Y =YP k , V Y =YP v , Q X , K Y 、V Y Represents query, key and value respectively; Among them, P q , P k , P v All are learnable parameters, and Z represents the enhanced features from text modality to 3D point cloud modality; S32. Use contrastive learning to distill the open text description features into a three-dimensional point cloud network. After the enhanced feature Z is processed by a multi-layer MLP, it is sent to the multimodal discriminator D together with the text feature Y. The multimodal discriminator outputs the cosine similarity prediction score s between each point in the enhanced feature Z and the text feature Y. The cross entropy is used to calculate the output s and the text feature area C. pseudo The matching relationship between them is defined as: s=D(Z,Y); Where g is C pseudo Match the one hot encoding vector of the corresponding region, BCELoss(·,·) is the cross entropy loss, calculated as: in, is the predicted value, y is the true value of the label; Using contrastive learning to enhance the loss between two open scene features Calculated as: Among them, y i is the i-th feature vector in the text feature Y of the open scene, z i is the enhanced feature Z and y i The positive sample area corresponding to the text feature, z j is the negative sample area corresponding to the current sample, N is the total number of text features, and τ is the temperature parameter; For the training set C train Known class C class For objects, the real object labels and cross entropy are used to supervise the learning of known classes, which is calculated as: s know =D(Z know ,C know ); Among them, Z know It is to take the output features contained in the known class C know Feature points within the range, s know is the similarity prediction score of the known class, g know is the one hot encoding vector of the corresponding region; S33. Calculate the total training loss of model M Among them, λ1, λ2 and λ3 are the cross entropy losses of open text features respectively. Contrastive Learning Loss and known class learning loss weights; update the weights of model M using the gradient descent method, and repeat operation S3 several times to obtain a model with high prediction performance in the open scene understanding task; The process of gradient descent is as follows: Based on The gradient of the parameter θ And update the parameter θ in the gradient direction, the formula is: Among them, α represents the learning rate.

8. The method for understanding a three-dimensional point cloud open scene based on dual-stream feature fusion according to claim 7, characterized in that: The specific steps of S4 are: S41, training set C train With known class C know The annotation information is input into the model M designed by S3, and the model is trained using the gradient descent method; S42, the validation set C is used during the training phase val With all object label sets C class Verify the model training effect, and finally complete the training to obtain model M * .

9. The method for understanding a three-dimensional point cloud open scene based on dual-stream feature fusion according to claim 8, characterized in that: The specific steps of S5 are: S51. Input test data set C test , including 3D point cloud data V and text conditions L to be queried; the model used in the test phase is model M * The feature extractor E in the pre-trained text feature extractor θ text , multimodal discriminator D; S52, test set C test Input S4 to obtain the model M * To get the test set C test The open scene prediction results include the prediction results for the known class C know and unknown class C unknow prediction performance.

Images