Combined top-down and bottom-up three-dimensional visual search method and system

Abstract

The application discloses a combined top-down and bottom-up three-dimensional visual retrieval method and system, and belongs to the technical field of three-dimensional visual retrieval. The method comprises the following steps: calculating three-dimensional point cloud features of a three-dimensional point cloud, and obtaining query language features of a query text; performing language-guided object proposal by fusing the three-dimensional point cloud features and the query language features; constructing a graph structure to learn the correlation between the object proposals, so as to obtain updated object proposal features; and calculating the matching scores of the updated object proposals and the query text according to the updated object proposal features and the query language features, thereby obtaining the three-dimensional visual retrieval result of the query text. The application can not only reduce the redundancy caused by the top-down method, significantly improve the calculation efficiency, but also capture the relationship information between the proposals and finely adjust the relationship information, so that the problem of the rough proposals generated by the bottom-up method can be solved.

Description

Technical Field

[0001] This invention relates to the field of 3D visual retrieval technology, and in particular to a combined top-down and bottom-up 3D visual retrieval method and system. Background Technology

[0002] 3D visual retrieval is a fundamental and crucial task in 3D understanding, attracting widespread attention in recent years, with applications spanning robotics and AR / VR systems. The goal of this task is to retrieve target objects within a 3D point cloud scene based on a given free-form query text description. Unlike previous mature 2D retrieval methods, 3D visual retrieval presents two more significant challenges: First, it takes sparse, noisy, and information-dense 3D point clouds as input, making it more difficult to extract visual information. Second, the relationships between objects and between objects and the scene in 3D space are far more complex than in 2D images, further increasing the difficulty of the 3D visual retrieval task.

[0003] Existing methods for 3D vision retrieval tasks can be mainly divided into two categories based on their model design:

[0004] (1) Top-down approach: such as Figure 1 As shown, these methods typically first generate a large number of candidate proposals across the entire point cloud scene using a pre-trained 3D object detection or segmentation model, and then select the proposal that best matches the semantic meaning of the query text. Although they can obtain high-quality proposals, they need to enumerate all possible objects to ensure that the generated proposals contain the objects required by the text, resulting in a large number of redundant proposals.

[0005] (2) Bottom-up approach: such as Figure 2 As shown, these methods typically begin by interacting the point set with the text description through an early fusion strategy, and then predict the target bounding box directly from the learned query-relevant point features. Compared to top-down methods, they do not rely on complex proposal generation and selection, thus enabling efficient end-to-end training.

[0006] However, both approaches have their advantages and disadvantages. Top-down methods typically produce high-quality proposals and facilitate better capture of relationships between a large number of proposals. However, their drawback is that a large number of proposals need to be generated to ensure they include those required by the query text, and the vast majority of these proposals are irrelevant to the description in the query text, reducing the method's efficiency. Conversely, reducing the number of generated proposals risks ignoring the objects required by the query. Bottom-up methods, on the other hand, achieve higher computational efficiency because they avoid generating and processing excessive redundant proposals. Furthermore, since the bounding boxes are directly regressed and not limited by proposals generated by the pre-trained model, these methods can capture smaller objects that the pre-trained model might overlook. However, they neglect the rich information between global points because they struggle to model object-level interrelationships. Therefore, the predicted object proposals they obtain are relatively coarse, lacking further refinement. Summary of the Invention

[0007] To address the aforementioned problems, this invention proposes a combined top-down and bottom-up 3D visual retrieval method and system, which reduces their limitations by integrating the advantages of both methods.

[0008] To achieve the above-mentioned objectives, the technical solution of this invention includes the following:

[0009] A combined top-down and bottom-up 3D visual retrieval method, the method comprising:

[0010] Calculate the 3D point cloud features of the 3D point cloud and obtain the query language features of the query text;

[0011] This involves language-guided object proposals achieved by fusing 3D point cloud features and query language features; the information in the object proposal includes: the bounding box center, the bounding box size, and the center feature of the object proposal.

[0012] Construct a graph structure to learn the relationships between object proposals in order to obtain updated object proposal features;

[0013] Based on the updated object proposal features and query language features, the matching score between the updated object proposal and the query text is calculated, thereby obtaining the three-dimensional visual retrieval results of the query text.

[0014] Furthermore, the language-guided object proposal, which integrates 3D point cloud features and query language features, includes:

[0015] Based on 3D point cloud features and query language features, points in the 3D point cloud with a correlation greater than a set value with the query text are selected, and the feature V of the filtered point set is calculated. F ;

[0016] For the filtered points, the center coordinates of the corresponding object are generated by predicting the object to which each point belongs;

[0017] Combining the features V of the filtered point set F Based on the center coordinates of each object, obtain the candidate region for object proposals.

[0018] Furthermore, based on the features of the 3D point cloud and the query language features, points in the 3D point cloud with a correlation greater than a set value with the query text are selected, and the feature V of the filtered point set is calculated. F ,include:

[0019] Based on multimodal attention fusion of 3D point cloud features and query language features, feature V is obtained. T ;

[0020] The 3D point cloud is filtered based on the attention score generated by the cross-attention mechanism.

[0021] According to the feature V T The attention score and the number of points in the filtered point set are used to obtain the feature V of the filtered point set. F .

[0022] Furthermore, the feature V of the combined filtered point set F Based on the center coordinates of each object, obtain the candidate region for object proposals, including:

[0023] Based on the center coordinates, several object proposal centers were obtained using sampling from the farthest point. i ;

[0024] Based on the distance c from the proposal center of each object i Features corresponding to points within a certain distance Calculate the central features of the proposed object.

[0025] Based on the characteristics of the center To predict the bounding box center of the corresponding object proposal and bounding box size

[0026] Furthermore, the relationships between object proposals are learned based on the graph structure to obtain updated object proposal features, including:

[0027] Construct a fully connected graph between object proposals;

[0028] Based on the central features of the object proposal Calculate the semantic relevance strength between object proposals;

[0029] Calculate the spatial correlation between object proposals based on the bounding box center and bounding box size of the object proposals;

[0030] Based on the semantic relevance strength and spatial relevance between object proposals, the edge weight W between object proposals is defined. uv ;

[0031] Using the edge weight W uv By aggregating the features of adjacent nodes, the updated object proposal features are obtained.

[0032] Further, the step of calculating the matching score between the updated object proposal and the query text based on the updated object proposal features and query language features, thereby obtaining the three-dimensional visual retrieval result of the query text, includes:

[0033] The updated object proposal features and query language features are combined and concatenated.

[0034] The matching score between the updated object proposal and the query text is calculated based on the splicing results;

[0035] The object proposal with the highest matching score is selected as the 3D visual retrieval result for the query text.

[0036] Furthermore, before selecting the object proposal with the highest matching score as the 3D visual retrieval result of the query text, the process also includes:

[0037] Based on the updated object proposal features, refine the bounding box center and bounding box size;

[0038] Use the refined bounding box center and bounding box size as the object proposal.

[0039] A combined top-down and bottom-up 3D visual retrieval system, the system comprising:

[0040] The feature generation module is used to calculate the 3D point cloud features of the 3D point cloud and obtain the query language features of the query text.

[0041] The bottom-up computation module is used to generate language-guided object proposals by fusing 3D point cloud features and query language features; wherein, the information of the object proposal includes: bounding box center, bounding box size, and the center feature of the object proposal.

[0042]

[0043] The top-down computation module is used to build a graph structure to learn the relationships between object proposals in order to obtain updated object proposal features;

[0044] The retrieval results generation module is used to calculate the matching score between the updated object proposal and the query text based on the updated object proposal features and query language features, thereby obtaining the three-dimensional visual retrieval results of the query text.

[0045] Furthermore, the loss used to train the system includes: center loss. Reference loss Object detection loss and object classification loss Wherein, the central loss It is achieved by calculating the center coordinates c of the 3D visual retrieval results. f Corresponding to the actual ground center coordinates c gt The L1 loss between them is obtained, and the reference loss is obtained. It is obtained by selecting the best object proposal through querying the text.

[0046] A computer device, characterized in that the computer device comprises: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the combined top-down and bottom-up three-dimensional visual retrieval method described above.

[0047] Compared to existing technologies, this invention reveals that the advantages and disadvantages of top-down and bottom-up methods are actually complementary. Specifically, the bottom-up method can generate proposals closely related to the query text, thereby reducing redundancy caused by the top-down method and significantly improving computational efficiency. On the other hand, the top-down method, by capturing and fine-tuning the relationship information between proposals, can alleviate the problem of coarse proposals generated by the bottom-up method. Attached Figure Description

[0048] Figure 1 Flowchart of the top-down approach.

[0049] Figure 2 Flowchart of the bottom-up approach.

[0050] Figure 3 The flowchart of the method proposed in this invention.

[0051] Figure 4 A flowchart of an embodiment of the present invention.

[0052] Figure 5 A comparison chart of the visualization results of the method in this invention and the existing 3D-SPS method. Detailed Implementation

[0053] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below.

[0054] This invention generates initial proposals in a bottom-up manner and then integrates these proposals on a graph using a top-down approach, combining the advantages of both bottom-up and top-down methods while mitigating their limitations. Specifically, as... Figure 3 As shown, this invention comprises two phases. In the first phase, the invention designs a proposal generation module similar to a bottom-up approach. This module aggregates information from 3D point clouds and query text, enabling the invention to directly predict bounding boxes of objects highly relevant to the query text based on these features and extract corresponding object features. This allows the invention to utilize query text information as guidance, avoiding the inefficiency caused by simultaneously detecting and analyzing a large number of redundant objects, and identifying objects that might be ignored by pre-trained detectors. In the second phase, the invention addresses the problem of coarse proposals generated by bottom-up methods by developing a graph-based proposal integration module inspired by top-down approaches. This module further captures the relationships between objects and between objects and the scene, and updates the object features accordingly. Subsequently, the invention predicts more refined bounding boxes of objects based on these updated features. The invention was evaluated on the widely adopted ScanRefer dataset, and experiments demonstrate that the proposed method achieves state-of-the-art performance compared to existing methods.

[0055] Before introducing the main steps of the method, this invention will first provide an overview of the 3D vision retrieval task. This invention first defines the 3D vision retrieval task as follows: Given a point cloud P∈R... N×(3+F) and free-form language query text Where N is the number of points, F is the dimension of the point cloud's additional features (such as color and normal vector), and W represents the number of input words. The objective of this invention is to predict the 3D bounding boxes of objects that match the input description.

[0056] (1) Algorithm framework:

[0057] like Figure 4 As shown, this invention uses a pre-trained encoder to encode input 3D point clouds and text, comprising two phases: a bottom-up phase and a top-down phase. In the bottom-up phase, the module of this invention fuses these features to make language-guided object proposals. In the top-down phase, the optimization module of this invention enhances these proposals with graph-based features and then selects the best-matching bounding box by predicting a matching score.

[0058] Specifically, this invention uses a pre-trained encoder to independently encode 3D point clouds and query text information. In the bottom-up phase, these two types of features are input into the proposed bottom-up proposal generation module for feature fusion and updating, resulting in language-guided object proposals. Subsequently, in the top-down phase, these proposals are input into the proposed top-down proposal merging module for further optimization, thereby obtaining improved proposals. Finally, this invention predicts the matching scores between these proposals and the query language, and selects the bounding box that best matches the query language as the final output.

[0059] (2) Bottom-up stage

[0060] To locate objects, this invention proposes a simple and lightweight bottom-up proposal generation module that generates candidate proposals guided by query text. This approach achieves higher efficiency and is simpler than previous bottom-up methods.

[0061] To achieve this goal, this invention first utilizes a Transformer module based on multimodal attention to fuse linguistic and 3D visual information and eliminate points irrelevant to a given linguistic query. By leveraging the features generated by the Transformer, this invention predicts the center coordinates of the object to which each point in the point cloud belongs. Subsequently, based on each center, this invention employs a voting aggregation module to combine features of neighboring points belonging to the same object. Using these aggregated features, this invention can regress linguistically guided object proposals.

[0062] For the Transformer module based on multimodal attention, this invention employs two independent self-attention layers to encode the context in 3D point cloud features and query language features, while using a cross-attention layer to encode the correspondence between the two modalities. The self-attention mechanism enables the model to capture the correlations and dependencies within each modality, while the cross-attention mechanism helps to adjust the language and visual features, allowing the model to focus on the information most relevant to the query language. The output of the Transformer module is represented as follows: The calculation process can be represented as follows:

[0063] V T =Transformer(V,L).

[0064] Here, V and L are the outputs of the point cloud and the text encoder, respectively. By examining the attention score generated by the cross-attention mechanism, this invention can filter out points that are weakly associated with a given query language. The attention score between V and L calculated by cross-attention is denoted as att. langThis score represents the degree of relevance between each point and the query text. This invention obtains the filtered point set through the following calculations. M here f and C t These are the number of points after filtering and the feature dimension of the points:

[0065] V F =V T [argtop k (Mean(att lang ),M f )).

[0066] Subsequently, for each point obtained above, the present invention uses a feature V as a basis. F A center predictor is used to predict the object to which a point belongs. This process generates the center coordinates of the corresponding object for each point, represented as... Here, c f M represents f The three-dimensional coordinates of each individual center point are used. The center predictor can be implemented using a multilayer perceptron (MLP). Then, in order to obtain object candidate regions from these points, the present invention first obtains several candidate proposal centers based on these center coordinates using farthest point sampling. Where K is the number of candidates. Since points belonging to the same object have closely related center coordinates, farthest point sampling helps to select points belonging to different objects. After obtaining the selected points using farthest point sampling, this invention performs max pooling on points within a certain distance of each selected center. Let's say from point c... i The set of points within a certain distance r is Their characteristics are represented as Then point c i Features It can be calculated as follows:

[0067]

[0068] Aggregation features The results are then fed into a proposal predictor, which uses a multilayer perceptron to regress the initial proposal outcome. The proposal predictor predicts the center bounding boxes of points in the 3D scene corresponding to different objects' proposal regions. and bounding box size

[0069]

[0070] (3) Top-down stage

[0071] Since the target proposal is not always accurate—it may only contain a portion of an object or include surrounding objects within its bounding box—refinement is necessary. However, previous bottom-up approaches directly output regression proposals as the final result, which can easily lead to inaccurate localization. While top-down approaches attempt to correlate all proposals to select the best one, they still do not refine the proposals.

[0072] To address these issues, this invention proposes constructing a graph structure to learn the relationships between proposals. Unlike previous methods, this invention additionally develops a weighted edge, used only to associate related proposals. Furthermore, this invention designs a novel proposal integration strategy that enriches and refines the information in each proposal based on the context of related proposals belonging to the same object.

[0073] Specifically, this invention refines each proposal using a graph-based method that leverages information from its neighboring and related proposals. To achieve this, the invention first constructs a fully connected graph among the generated proposals, where each node in the graph represents a proposal. However, this invention recognizes that the relevance between different nodes (proposals) in the graph can vary significantly. Objects that are spatially close and of similar categories typically have stronger relationships. To fully utilize the relevance between these nodes while reducing interference from unrelated nodes, this invention defines an edge weight W between proposals. uv as follows:

[0074]

[0075] Here, α and β are hyperparameters that quantify the weights of the two terms. The first term represents the semantic relevance strength between the two proposals, measured by the cosine similarity of their features. The second term quantifies the spatial relevance between the two proposals based on the intersection-union ratio (IU) of their initial bounding boxes. The IU can be calculated using the centers of the two proposals. and size Calculated.

[0076] Then, this invention updates the feature V of the proposal using edge weights through a weighted summation. A :

[0077]

[0078] Through this step, the present invention optimizes the representation of proposals, taking into account their semantic and spatial associations with other relevant proposals in the scenario.

[0079] (4) Training objectives

[0080] Based on the characteristics V of the proposal A This invention utilizes a proposal predictor to analyze the bounding box information of each proposal. Further refinement is needed. Then, this invention will refine V. A The text feature L is concatenated with the text feature L and a score predictor is used to predict the matching score between the proposal and the query text. Then, the present invention selects the proposal with the highest matching score as the final result.

[0081] To ensure the model achieves satisfactory results, this invention employs a combination of multiple loss functions to supervise the entire process. In the initial proposal generation phase, this invention uses center loss. This is used to supervise the center predictor and ensure the correct prediction of the object's center coordinates. The center loss is calculated by determining the predicted center coordinates c. f Corresponding to the actual ground center coordinates c gt Calculated using L1 loss between:

[0082]

[0083] To obtain the true object center of a point, this invention utilizes the object centers in the dataset. For each point, this invention calculates its distance to all true object centers in the dataset. Then, the point is assigned to the object with the nearest center, making it the true object center corresponding to that point.

[0084] In the final selection phase, this invention uses reference loss. This is used to supervise the selection of the best matching proposal for the query text. For each proposal, it has a matching score s. i and the corresponding bounding box This invention uses the following method to determine its corresponding label. First, the intersection-union ratio (IUR) between the predicted bounding box and the ground truth bounding box is calculated. Then, if the IUR value of the proposal is the largest and greater than a specified threshold, the proposal y is... i The label for [the proposal] is set to 1. For all other proposals, the label is set to 0.

[0085] Reference loss The loss is calculated by measuring the cross-entropy between the predicted matching score *s* and the corresponding label *y*. The formula for the reference loss is as follows:

[0086]

[0087] Furthermore, this invention employs object detection loss. This is used to supervise the object detection process. This loss function helps the model accurately locate and identify objects in the scene.

[0088] This invention utilizes language-to-object classification loss. To aid in language comprehension.

[0089] By combining these different loss functions, the proposed method aims to achieve better performance in 3D vision localization tasks. In summary, the total loss used during training can be expressed as:

[0090]

[0091] λ1, λ2, λ3, and λ4 are the weights assigned to different types of loss.

[0092] (5) Experimental conditions

[0093] To verify the effectiveness of the method of this invention and compare it with existing methods, experiments were conducted on the commonly used ScanRefer dataset. The evaluation metric used in this invention is Acc@kIoU, where k can be 0.25 or 0.5, representing the proportion of samples whose intersection-union ratio (IU) between the final localization result and the ground truth bounding box is greater than k; a higher metric is better. The experiments were trained for 32 epochs on four NVIDIA RTX3090 GPUs using the AdamW optimizer. The input to the model of this invention includes the coordinates, color, normal vector of the 3D point cloud, and 2D multi-view features. The experimental results are shown in Table 1. In the table, unique and multiple represent two subsets of the dataset, where unique represents a sample in the scene with only one object of the same class as the target object, and multiple represents a sample in the scene with multiple objects of the same class as the target object.

[0094]

[0095] Table 1: Results of different 3D visual retrieval methods on the Scanrefer dataset. Higher values ​​for all metrics are better.

[0096]

[0097] Table 2: Residual experiment results for the top-down and bottom-up modules. Accuracy with an intersection-union ratio (IU) greater than 0.25 and 0.5 was used as the metric.

[0098]

[0099] Table 3: Residual experimental results of the number of points K selected in the farthest point sampling step in the bottom-up proposal generation module.

[0100]

[0101] Table 4: Ablation experiment results based on graph-based information aggregation iteration number n.

[0102] Figure 5This is a comparison chart of the visualization results of the method in this invention and existing 3D-SPS methods. For example... Figure 5 As shown, the first column displays the actual bounding boxes provided by the ScanRefer dataset. The second and third columns represent the output of the bottom-up proposal generation module and the final output of the entire model, respectively. The last column displays the results obtained from the 3D-SPS method.

[0103] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the principles and scope of the present invention. The scope of protection of the present invention should be determined by the claims.

Claims

1. A combined top-down and bottom-up three-dimensional visual retrieval method, characterized in that, The method includes: Calculate the 3D point cloud features of the 3D point cloud and obtain the query language features of the query text; This involves language-guided object proposals that fuse 3D point cloud features with query language features; the information in the object proposal includes: bounding box center, bounding box size, and the center feature of the object proposal. ; Construct a graph structure to learn the relationships between object proposals in order to obtain updated object proposal features; Based on the updated object proposal features and query language features, the matching score between the updated object proposal and the query text is calculated, thereby obtaining the three-dimensional visual retrieval results of the query text. The object proposal, which integrates 3D point cloud features and query language features for language guidance, includes: Based on 3D point cloud features and query language features, points in the 3D point cloud with a correlation greater than a set value with the query text are selected, and the features of the filtered point set are calculated. ; For the filtered points, the center coordinates of the corresponding object are generated by predicting the object to which each point belongs; Combining the features of the filtered point set Based on the center coordinates of each object, obtain the candidate region for object proposals; The method involves using 3D point cloud features and query language features to filter points in the 3D point cloud that have a correlation with the query text greater than a set value, and then calculating the features of the filtered point set. ,include: Based on multimodal attention fusion of 3D point cloud features and query language features, the features are obtained. ; The 3D point cloud is filtered based on the attention score generated by the cross-attention mechanism. According to the features The features of the filtered point set are obtained by considering the attention score and the number of points in the filtered point set. ; The graph structure is used to learn the relationships between object proposals to obtain updated object proposal features, including: Construct a fully connected graph of object proposals, where each node in the graph represents an object proposal. Based on the central features of the object proposal Calculate the semantic relevance strength between object proposals; Calculate the spatial correlation between object proposals based on the bounding box center and bounding box size of the object proposals; Based on the semantic relevance and spatial relevance between object proposals, the edge weights between object proposals are defined. ; Utilizing the edge weight By aggregating the features of adjacent nodes, the updated object proposal features are obtained.

2. The method according to claim 1, characterized in that, The features of the combined filtered point set Based on the center coordinates of each object, obtain the candidate region for object proposals, including: Several object proposal centers were obtained by sampling from the farthest point based on the center coordinates. ; Based on the distance to each object's proposal center Features corresponding to points within a certain distance Calculate the central features of the proposed object. ; Based on the characteristics of the center To predict the bounding box center of the corresponding object proposal and bounding box size .

3. The method according to claim 1, characterized in that, The step involves calculating the matching score between the updated object proposal and the query text based on the updated object proposal features and query language features, thereby obtaining the 3D visual retrieval results of the query text, including: The updated object proposal features and query language features are combined and concatenated. The matching score between the updated object proposal and the query text is calculated based on the splicing results; The object proposal with the highest matching score is selected as the 3D visual retrieval result for the query text.

4. The method according to claim 3, characterized in that, Before selecting the object proposal with the highest matching score as the 3D visual retrieval result of the query text, the process also includes: Based on the updated object proposal features, refine the bounding box center and bounding box size; Use the refined bounding box center and bounding box size as the object proposal.

5. A combined top-down and bottom-up three-dimensional visual retrieval system applying the method of any one of claims 1-4, characterized in that, The system includes: The feature generation module is used to calculate the 3D point cloud features of the 3D point cloud and obtain the query language features of the query text. The bottom-up computation module is used to perform language-guided object proposals by fusing 3D point cloud features and query language features; wherein, the information of the object proposal includes: bounding box center, bounding box size, and the center feature of the object proposal. ; The top-down computation module is used to build a graph structure to learn the relationships between object proposals in order to obtain updated object proposal features; The retrieval results generation module is used to calculate the matching score between the updated object proposal and the query text based on the updated object proposal features and query language features, thereby obtaining the three-dimensional visual retrieval results of the query text.

6. The system according to claim 5, characterized in that, The loss for training the system includes: center loss. Reference loss Object detection loss and object classification loss ; wherein, the central loss It is achieved by calculating the center coordinates of the 3D visual retrieval results. Corresponding ground center coordinates The L1 loss between them is obtained, and the reference loss is obtained. It is obtained by selecting the best object proposal through querying the text.

7. A computer device, characterized in that, The computer device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the combined top-down and bottom-up three-dimensional visual retrieval method as described in any one of claims 1-4.

Images