An open-vocabulary based complex scene understanding method and system

Abstract

The application discloses a complex scene understanding method and system based on an open vocabulary, and is realized based on an RGB-D camera, and the method comprises the following steps: acquiring point cloud data of a complex scene to be studied; inputting a text of object names possibly existing in the complex scene to be studied, and extracting text features; extracting 3D features of the point cloud data through a trained 3D feature extraction model; performing point multiplication on the transposition of the 3D features and the text features, obtaining semantic label values according to the point multiplication results, and determining objects contained in the complex scene to be studied; and the training steps of the 3D feature extraction model comprise the following steps: designing a 2D-3D feature fusion network, jointly training the 2D-3D feature fusion network and the 3D feature extraction model through a loss function, guiding the fused features to be close to 2D features, having open vocabulary capability, and having a controllable gap with 3D features, so that structure information and semantic information are avoided to be lost, small object weighting optimization is carried out, and the recognition accuracy of small object categories is improved.

Description

Technical Field

[0001] This invention belongs to the field of complex scene understanding, specifically relating to a method and system for complex scene understanding based on open vocabulary. Background Technology

[0002] With the rapid development of computer vision and artificial intelligence technologies, existing methods are no longer sufficient to meet human needs for robots. Complex scene understanding has wide and in-depth applications in intelligent robots, object grasping, and autonomous driving. Robots not only need to understand what objects are in a scene, but also their positions, material properties, and functional attributes. Unlike single-object function segmentation, complex scenes involve a wide variety of objects with complex functions; multiple objects may have the same function, and the same object may have different functions. Understanding and segmenting complex scenes is a challenging task. Currently, most methods rely on datasets for object detection or segmentation, but building a dataset for each task is extremely costly, and segmentation accuracy is limited by the object categories, quantity, and quality in the dataset. Furthermore, complex scenes often involve intricate geometric relationships between objects and occlusion, further complicating the understanding of complex scenes. Therefore, researching semantic segmentation, material segmentation, and function segmentation in complex scenes is of great significance.

[0003] By leveraging pre-trained visual language models, a correspondence between text and images can be established, enabling arbitrary text segmentation of 2D images—a process known as open vocabulary learning. Based on the achievements of 2D open vocabulary learning, some progress has also been made in open vocabulary segmentation for 3D scenes. However, due to localization and pose estimation errors, 2D image features are difficult to align with 3D point cloud features. Furthermore, in complex scenes, image features are sparse due to occlusion of the viewpoint, leading to lower segmentation accuracy. In practical applications, most unmanned devices (such as drones and autonomous vehicles) lack high-precision LiDAR, making it difficult to obtain dense point cloud features. Therefore, research on understanding complex scenes with sparse features remains of significant importance. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and to propose a method and system for understanding complex scenes based on open vocabulary.

[0005] In view of this, the present invention proposes a method for understanding complex scenes based on open vocabulary, implemented using an RGB-D camera, the method comprising:

[0006] Step 1: Obtain point cloud data of the complex scene to be studied;

[0007] Step 2: Input the text of the names of objects that may exist in the complex scene to be studied, and extract the text features;

[0008] Step 3: Extract 3D features from the point cloud data using the pre-trained 3D feature extraction model;

[0009] Step 4: Perform a dot product on the transpose of the 3D features from Step 3 and the text features from Step 2, and obtain the semantic label value based on the dot product result, thereby determining the objects contained in the complex scene to be studied.

[0010] The training steps of the 3D feature extraction model include: designing a 2D-3D feature fusion network, jointly training the 2D-3D feature fusion network and the 3D feature extraction model through a designed loss function, guiding the features fused by the 2D-3D feature fusion network to move closer to the 2D features so as to have open vocabulary capability and controllable gap with 3D features, avoiding loss of structural and semantic information, and then improving the recognition accuracy of small object categories through small object weighting optimization.

[0011] Preferably, the 3D feature extraction model is the MinkUNet18A network.

[0012] Preferably, the method further includes a training step for the 3D feature extraction model and the 2D-3D feature fusion network, including:

[0013] Step S1: Acquire paired multi-view images and point clouds based on the RGB-D camera;

[0014] Step S2: For multi-view images, extract 2D features using the OpenSeg network and average them to obtain the result for any pixel. 2D features ;

[0015] Step S3: Extract 3D features from the point cloud using a 3D feature extraction model to obtain the 3D features of any point. 3D features ;

[0016] Step S4: Use the CLIP text feature extraction model to extract the text features of the input text t. ;

[0017] Step S5: For any point , Based on the pinhole imaging model, the corresponding projected coordinates are obtained, and the point is obtained from the projected coordinates. 2D features on an image ;

[0018] Step S6: [The sentence is incomplete and requires more context to be translated accurately.] , Input the 2D-3D feature fusion network, calculate the loss function, and determine whether it is a small object. If it is, multiply it by the weight coefficient. Continue until the training requirements are met to obtain the trained 3D feature extraction model and 2D-3D feature fusion network.

[0019] Preferably, the 2D-3D feature fusion network includes: a multi-head cross-attention mechanism, a normalization module, a feedforward network, residual connections, and a normalization module; using 3D features 2D features as query vectors As a key / value vector, the output is the fused feature. The multi-head cross-attention mechanism uses linear attention.

[0020] Preferably, the feedforward network includes a linear mapping, a ReLU activation function, a linear mapping, and a Dropout module connected in sequence.

[0021] Preferably, the loss function includes:

[0022] calculate and Cosine similarity: It is used to guide the features fused by the 2D-3D feature fusion network to be as close as possible to the 2D features, and has open vocabulary capabilities;

[0023] calculate and Cosine similarity: This is used to ensure that the difference between the features fused by the 2D-3D feature fusion network and the 3D features is controllable, and to avoid losing structural and semantic information.

[0024] Calculate cross-entropy loss If it is judged to be a small object, then... Multiply by predefined weights to suffer losses ;

[0025] Assign different weights respectively According to the following formula:

[0026] ;

[0027] Obtain the total loss function .

[0028] Preferably, the calculation of cross-entropy loss include:

[0029] Calculate the similarity between 2D features and text features, and select the category with the highest confidence as the pseudo-label. ;

[0030] Calculate the similarity between 3D features and text features, then multiply by a temperature coefficient to obtain the first similarity. i 3D feature similarity ;

[0031] The cross-entropy loss is obtained from the following formula. :

[0032]

[0033] in, n This indicates the total number of categories.

[0034] On the other hand, the present invention provides a complex scene understanding system based on an open vocabulary, implemented using an RGB-D camera, the system comprising:

[0035] The point cloud acquisition module is used to acquire point cloud data of the complex scene to be studied;

[0036] The text feature extraction module is used to extract text features from the text of the names of objects that may exist in the complex scene to be studied.

[0037] The 3D feature extraction module is used to extract 3D features from point cloud data using a pre-trained 3D feature extraction model; and

[0038] The dot product understanding module is used to perform dot product on the transpose of 3D features and text features, and obtain semantic label values ​​based on the dot product results, thereby determining the objects contained in the complex scene to be studied;

[0039] The training steps of the 3D feature extraction model include: designing a 2D-3D feature fusion network; jointly training the 2D-3D feature fusion network and the 3D feature extraction model using a designed loss function; guiding the fused features of the 2D-3D feature fusion network to move closer to 2D features, so as to have open vocabulary capability and controllable gap with 3D features, avoiding loss of structural and semantic information; and then improving the recognition accuracy of small object categories through small object weighting optimization. Compared with the existing technology, the advantages of this invention are:

[0040] 1. To address the issue of limited accuracy in 3D segmentation guided by 2D features when viewpoint information is limited, a 2D-3D cross-attention fusion network was designed. By using the 2D-3D cross-attention mechanism, the limited 2D features can be fully utilized to guide 3D features to better learn 2D feature information.

[0041] 2. Since some small objects account for a small proportion in point clouds, it is easy to fail to extract 2D features due to problems such as downsampling and lack of viewpoints. Therefore, a small object weighted optimization is proposed to improve the segmentation accuracy by increasing the attention to difficult categories. Attached Figure Description

[0042] Figure 1(a) is a flowchart of the application scenario of the complex scene understanding method based on open vocabulary of the present invention;

[0043] Figure 1(b) is a flowchart of the training scenario for the complex scene understanding method based on open vocabulary of the present invention;

[0044] Figure 2 It is a 2D-3D feature fusion network structure;

[0045] Figure 3 It is a feedforward network structure;

[0046] Figure 4 This is a flowchart of the weighted optimization for small objects;

[0047] Figure 5 This is an example of the input point cloud visualization result;

[0048] Figure 6 Here is an example of a visualization of the truth values ​​of a point cloud;

[0049] Figure 7 This is an example of a prediction result obtained using the method of the present invention. Detailed Implementation

[0050] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a semantic map matching method based on genetic algorithms.

[0051] The purpose of this invention is to solve the challenge of understanding complex scenes, and proposes a method for understanding complex scenes based on open vocabulary. By cross-fusion of 2D and 3D features and designing a small object weighted optimization mechanism, high-precision open vocabulary segmentation in complex scenes is achieved.

[0052] Figure 1(a) shows a flowchart of the application scenario of the method of the present invention, which includes the following steps:

[0053] A method for understanding complex scenes based on an open vocabulary, implemented using an RGB-D camera, the method comprising:

[0054] Step 1: Obtain point cloud data of the complex scene to be studied;

[0055] Step 2: Input the text of the names of objects that may exist in the complex scene to be studied, and extract the text features;

[0056] Step 3: Extract 3D features from the point cloud data using the pre-trained 3D feature extraction model;

[0057] Step 4: Perform a dot product on the transpose of the 3D features from Step 3 and the text features from Step 2, and determine the objects contained in the complex scene to be studied based on the magnitude of the dot product.

[0058] The training steps of the 3D feature extraction model include: designing a 2D-3D feature fusion network, jointly training the 2D-3D feature fusion network and the 3D feature extraction model through a designed loss function, guiding the features fused by the 2D-3D feature fusion network to move closer to the 2D features so as to have open vocabulary capability and controllable gap with 3D features, avoiding loss of structural and semantic information, and improving the recognition accuracy of small object categories through small object weighting optimization.

[0059] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0060] Example 1

[0061] Embodiment 1 of the present invention proposes a method for understanding complex scenes based on open vocabulary. As shown in Figure 1(b), the flowchart of the training scene includes the following steps:

[0062] 1. 2D Feature Extraction: A pre-trained model is used to extract 2D features from multi-view images, and the 2D features from multiple views are averaged to obtain the 2D feature of each pixel. Any pixel The features can be represented as .

[0063] 2.3D Feature Extraction: A 3D feature extraction model is used to extract 3D features from the point cloud, obtaining the 3D features of each point. Any point The features can be represented as .

[0064] 3. Text Feature Extraction: We use a pre-trained text feature extractor to extract the features of the input text. For any text t, its features are represented as follows: .

[0065] 4. Feature Projection: We use the camera pose matrix... The projected coordinates of a point on the image can be calculated: for any point , Its projected coordinates on the image are represented as follows: According to the pinhole imaging model , .according to The coordinates of the point can be used to obtain the point. 2D features on an image .

[0066] 5.2D-3D Feature Fusion: After obtaining the 2D and 3D features of point P, we aim to better fuse these features, especially to make the 3D features similar to the 2D features. Therefore, a 2D-3D cross-fusion network was designed. The network uses a multi-head cross-attention mechanism, with 3D features as the query vector and 2D features as the key / value vector. After passing through a feedforward network and residual connections, the fused features are obtained, represented as follows: 2D-3D feature fusion network structure as follows: Figure 2 As shown, Figure 3 It is a feedforward network structure. Figure 4 This is a flowchart of weighted optimization for small objects.

[0067] 6. Optimize the feature extraction model: We need to design some loss functions to guide the 3D feature extraction network and the 2D-3D cross-fusion network, so that 3D features can be accurately semantically segmented. We designed the following three loss functions:

[0068] (1) First, calculate and Cosine similarity: This guides the fused features to be as close as possible to 2D features in order to have open vocabulary capabilities.

[0069] (2) Then, calculate and Cosine similarity: This is to ensure that the fused features and 3D features do not differ too much, which could lead to the loss of structural and semantic information.

[0070] 7. Weighted Optimization for Small Objects: Finally, since some small objects occupy a small proportion in the point cloud, their 2D features are easily lost due to issues such as downsampling and lack of viewpoints. We aim to increase the attention given to small objects during training, thereby improving the recognition accuracy of small object categories. We use 2D features as soft labels to guide 3D features to gradually approach them, and in this process, we assign high weights to small object categories.

[0071] In one example of an indoor scene, we artificially define certain categories as small objects, such as pictures, refrigerators, shower curtains, and showerheads. These objects appear infrequently and occupy fewer point clouds and pixels.

[0072] The specific process is as follows:

[0073] (1) Calculate the similarity between 2D features and text features, and take the category with the highest confidence as the pseudo-label. ;

[0074] (2) Calculate the similarity between 3D features and text features, and then multiply it by a temperature coefficient to obtain the first similarity. i 3D feature similarity In one embodiment, the temperature coefficient is 0.07.

[0075] (3) Calculate the cross-entropy loss using the following formula:

[0076]

[0077] (4) Check whether the category belongs to the small object category. If it does, multiply it by a predefined weight. Otherwise, directly provide the loss function. Finally, obtain the total loss:

[0078] Overall The function can be represented as:

[0079]

[0080] In one embodiment, .

[0081] Training process: We used three RTX 4090 GPUs. We used MinkUNet18A as the skeleton of the 3D network and OpenSeg as the 2D model. During training, we used the Adam optimizer with a learning rate of 0.0001 for 100 epochs. The data was downsampled by 8cm.

[0082] Example 2

[0083] Embodiment 2 of the present invention provides a complex scene understanding system based on an open vocabulary, implemented using an RGB-D camera, and employing the method of Embodiment 1. The system includes:

[0084] The point cloud acquisition module is used to acquire point cloud data of the complex scene to be studied;

[0085] The text feature extraction module is used to extract text features from the text of the names of objects that may exist in the complex scene to be studied.

[0086] The 3D feature extraction module is used to extract 3D features from point cloud data using a pre-trained 3D feature extraction model.

[0087] The dot product understanding module is used to perform dot product on the transpose of 3D features and text features, and obtain semantic label values ​​based on the dot product results, thereby determining the objects contained in the complex scene to be studied;

[0088] The training steps of the 3D feature extraction model include: designing a 2D-3D feature fusion network; jointly training the 2D-3D feature fusion network and the 3D feature extraction model using a designed loss function; guiding the fused features of the 2D-3D feature fusion network to move closer to 2D features to enable open vocabulary capabilities and keep the difference from 3D features controllable, avoiding the loss of structural and semantic information; and then improving the recognition accuracy of small object categories through small object weighting optimization.

[0089] It is worth noting that in the embodiments of the above system, the modules included are divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional module are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0090] Simulation example:

[0091] like Figure 5 This is an example of the input point cloud visualization result; you can see objects such as tables, walls, and floors in the image. This image has no semantic labels, so all objects are represented by the same color.

[0092] Figure 6 This is an example of a visualization of the ground truth of point clouds; the ground truth is a scene semantic map that is manually annotated. Different colors represent different objects, such as green for the floor, pink for the table, blue for the wall, purple for the window, and yellow-green for the chair.

[0093] Figure 7 This is an example of the prediction results obtained using the method of this invention. It can be seen that... Figure 5 They are very similar. Objects such as walls, floors, tables, chairs, and windows are very similar to the true values ​​in terms of position and structure, which indicates that our algorithm has very high precision and accuracy.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for understanding complex scenes based on open vocabulary, implemented using an RGB-D camera, the method comprising: Step 1: Obtain point cloud data of the complex scene to be studied; Step 2: Input the text of the names of objects that may exist in the complex scene to be studied, and extract the text features; Step 3: Extract 3D features from the point cloud data using the pre-trained 3D feature extraction model; Step 4: Perform a dot product on the transpose of the 3D features from Step 3 and the text features from Step 2, and obtain the semantic label value based on the dot product result, thereby determining the objects contained in the complex scene to be studied. The training steps of the 3D feature extraction model include: designing a 2D-3D feature fusion network, jointly training the 2D-3D feature fusion network and the 3D feature extraction model through a designed loss function, guiding the features fused by the 2D-3D feature fusion network to move closer to the 2D features so as to have open vocabulary capability and controllable gap with 3D features, avoiding loss of structural and semantic information, and then improving the recognition accuracy of small object categories through small object weighting optimization; The method also includes training steps for a 3D feature extraction model and a 2D-3D feature fusion network, including: Step S1: Acquire paired multi-view images and point clouds based on the RGB-D camera; Step S2: For multi-view images, extract 2D features using the OpenSeg network and average them to obtain the result for any pixel. 2D features ; Step S3: Extract 3D features from the point cloud using a 3D feature extraction model to obtain the 3D features of any point. 3D features ; Step S4: Use the CLIP text feature extraction model to extract the text features of the input text t. ; Step S5: For any point , Based on the pinhole imaging model, the corresponding projected coordinates are obtained, and the point is obtained from the projected coordinates. 2D features on an image ; Step S6: [The sentence is incomplete and requires more context to be translated accurately.] , Input the 2D-3D feature fusion network, calculate the loss function, and determine whether it is a small object. If it is, multiply it by the weight coefficient. Continue until the training requirements are met to obtain the trained 3D feature extraction model and 2D-3D feature fusion network.

2. The method for understanding complex scenes based on open vocabulary according to claim 1, characterized in that, The 3D feature extraction model is the MinkUNet18A network.

3. The method for understanding complex scenes based on open vocabulary according to claim 1, characterized in that, The 2D-3D feature fusion network includes: a multi-head cross-attention mechanism, a normalization module, a feedforward network, residual connections, and a normalization module; using 3D features... 2D features as query vectors As a key / value vector, the output is the fused feature. The multi-head cross-attention mechanism uses linear attention.

4. The method for understanding complex scenes based on open vocabulary according to claim 3, characterized in that, The feedforward network comprises: a linear mapping, a ReLU activation function, a linear mapping, and a Dropout module connected in sequence.

5. The method for understanding complex scenes based on open vocabulary according to claim 3, characterized in that, The loss function includes: calculate and Cosine similarity: It is used to guide the features fused by the 2D-3D feature fusion network to be as close as possible to the 2D features, and has open vocabulary capabilities; calculate and Cosine similarity: This is used to ensure that the difference between the features fused by the 2D-3D feature fusion network and the 3D features is controllable, and to avoid losing structural and semantic information. Calculate cross-entropy loss If it is judged to be a small object, then... Multiply by predefined weights to suffer losses ; Assign different weights respectively According to the following formula: ； Obtain the total loss function .

6. The method for understanding complex scenes based on open vocabulary according to claim 5, characterized in that, The calculation of cross-entropy loss include: Calculate the similarity between 2D features and text features, and select the category with the highest confidence as the pseudo-label. ; Calculate the similarity between 3D features and text features, then multiply by a temperature coefficient to obtain the first similarity. i 3D feature similarity ; The cross-entropy loss is obtained from the following formula. : ； in, n This indicates the total number of categories.

7. A complex scene understanding system based on an open vocabulary, implemented using an RGB-D camera, characterized in that, The system includes: The point cloud acquisition module is used to acquire point cloud data of the complex scene to be studied; The text feature extraction module is used to extract text features from the text of the names of objects that may exist in the complex scene to be studied. The 3D feature extraction module is used to extract 3D features from point cloud data using a pre-trained 3D feature extraction model; and The dot product understanding module is used to perform dot product on the transpose of 3D features and text features, and obtain semantic label values ​​based on the dot product results, thereby determining the objects contained in the complex scene to be studied; The training steps of the 3D feature extraction model include: designing a 2D-3D feature fusion network, jointly training the 2D-3D feature fusion network and the 3D feature extraction model through a designed loss function, guiding the features fused by the 2D-3D feature fusion network to move closer to the 2D features so as to have open vocabulary capability and controllable gap with 3D features, avoiding loss of structural and semantic information, and then improving the recognition accuracy of small object categories through small object weighting optimization; The training steps for the 3D feature extraction model and the 2D-3D feature fusion network include: Step S1: Acquire paired multi-view images and point clouds based on the RGB-D camera; Step S2: For multi-view images, extract 2D features using the OpenSeg network and average them to obtain the result for any pixel. 2D features ; Step S3: Extract 3D features from the point cloud using a 3D feature extraction model to obtain the 3D features of any point. 3D features ; Step S4: Use the CLIP text feature extraction model to extract the text features of the input text t. ; Step S5: For any point , Based on the pinhole imaging model, the corresponding projected coordinates are obtained, and the point is obtained from the projected coordinates. 2D features on an image ; Step S6: [The sentence is incomplete and requires more context to be translated accurately.] , Input the 2D-3D feature fusion network, calculate the loss function, and determine whether it is a small object. If it is, multiply it by the weight coefficient. Continue until the training requirements are met to obtain the trained 3D feature extraction model and 2D-3D feature fusion network.

Images