Zero-sample 3D scene understanding method based on optimal viewpoint selection

Abstract

The invention discloses a zero-sample 3D scene understanding method based on optimal viewpoint selection, and relates to the field of 3D scene understanding, and the method comprises the steps: screening out a preliminary view set containing all query association instances; filtering redundant images with similar visual angles to obtain candidate view subsets with diversified visual angles; establishing a mapping relationship between the 2D instance and the 3D object; performing comprehensive calculation on all instance relation matching results of each candidate view, and selecting a plurality of views with the highest view total relation score as an optimal view set; and inputting the optimal view set and the original text query into the multi-modal large language model to complete a downstream understanding task in the 3D scene. According to the method, effectiveness and diversity of views are guaranteed through an instance complete screening and view angle optimization mechanism, accuracy and consistency of scene understanding are improved through 3D mapping and spatial relation verification, and efficient cross-modal pushing under the condition of no large-scale annotation data is achieved in a complex 3D scene.

Description

Technical Field

[0001] This invention relates to the field of 3D scene understanding technology, and more specifically to a zero-sample 3D scene understanding method based on optimal viewpoint selection. Background Technology

[0002] Zero-shot 3D scene understanding, a core technology in computer vision, aims to achieve target localization, spatial relationship reasoning, and scene interpretation in complex 3D scenes by fusing multi-view 2D visual information with textual semantic descriptions without requiring large-scale 3D labeled data. This technology is widely used in key areas such as robot interaction and autonomous driving environmental perception, serving as a crucial bridge connecting visual perception and high-level decision-making.

[0003] With the rapid development of multimodal large language models (MLLMs), zero-shot 3D understanding technology has achieved a significant breakthrough. Current mainstream methods transform 3D scenes into 2D visual inputs that MLLMs can process, and combine this with semantic guidance from text descriptions to achieve cross-modal reasoning. However, existing technologies still suffer from two major bottlenecks: First, insufficient instance completeness of views. Existing methods often select views that only focus on the target object itself, lacking complete coverage of surrounding objects. This results in MLLMs being unable to acquire sufficient contextual information, making it difficult to support complex cross-object spatial relationship reasoning. Second, a lack of spatial consistency between the view and the speaker's perspective. Because the speaker's observation perspective corresponding to the text description is not considered, there is a misalignment between the views selected by existing methods and the spatial representation of the semantic description, leading to misjudgments of spatial relationships during 3D scene interpretation and severely affecting the accuracy of understanding.

[0004] Therefore, overcoming the lack of instance integrity and spatial consistency in views and improving the accuracy of zero-sample 3D scene understanding is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a zero-sample 3D scene understanding method based on optimal viewpoint selection. Through an innovative two-stage view planning mechanism: the first stage uses instance-level text-image matching and camera pose constraints to filter out candidate views that contain all query-related instances and have diverse perspectives; the second stage uses geometric modeling to predict the camera pose consistent with the text space relationship, and selects the optimal view that is closest to the speaker's perspective from the candidate views, providing accurate and comprehensive cross-modal reasoning basis, effectively overcoming the inherent defects of existing technologies in instance completeness and spatial consistency.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: This invention discloses a zero-shot 3D scene understanding method based on optimal viewpoint selection, comprising the following steps: S1. The input multi-view 2D images are processed one by one using the GroundingDINO instance detection model to obtain the 2D bounding boxes and detection confidence of the query-related instances, and the local visual information of the instances is obtained by cropping according to the bounding boxes; the matching degree between the local visual information of each instance and the text instance concept is calculated by the BLIP vision-language model, and the image-level matching result is obtained by combining the detection confidence, and a preliminary view set containing all query-related instances is selected. S2. Extract the camera shooting direction and position information corresponding to each view in the preliminary view set, calculate the view similarity between views, and calculate the redundancy score of each view in one step; calculate the final screening score of each view based on the image-level matching result and the redundancy score of each view, and sort and filter redundant images with similar viewpoints to obtain a candidate view subset with diverse viewpoints. S3. For each view in the candidate view subset, a 2D mask corresponding to the target instance is generated using the SAM2 image segmentation model; the pixels corresponding to the 2D mask are back-projected to 3D space using a back-projection formula based on image depth information and camera intrinsic and extrinsic parameters, and the 3D center position of the instance is calculated; the 3D center position is matched with the preset 3D object layout information through nearest neighbor search to establish a mapping relationship between 2D instances and 3D objects. S4. Decompose the spatial relationship description in the text query into simple relationship combinations, calculate the relative position vector between instances in 3D space, match it with the preset spatial relationship standard vector, and obtain the relationship matching result of each instance; perform a comprehensive calculation on the relationship matching results of all instances of each candidate view, and select the views with the highest total relationship score as the optimal view set. S5. Input the optimal view set and the original text query into the multimodal large language model to complete downstream understanding tasks such as target localization and spatial relationship reasoning in the 3D scene.

[0007] Furthermore, the formula for calculating the matching degree between the local visual information of each instance and the text instance concept is as follows: ; in, Indicates the first In the 2D image, the first Local visual information of an instance, Indicates the first The concept of a text instance. express and Instance-level matching score; and They are respectively and Characteristic norm, for The feature transpose; The formula for the image-level matching result is: ; in, Indicates the first Image-level matching results for a 2D image; Indicates the first In the 2D image, the first The detection confidence of each instance. For the first Total number of instances in a 2D image.

[0008] Furthermore, the calculation of viewpoint similarity between views includes calculating the similarity of shooting directions and spatial distance, specifically using the following formula: ; ; Among them, it means and Representing views respectively and view The shooting direction vector, for and Similarity of shooting directions; The dot product of the shooting direction vectors, These represent the norms of the camera's shooting direction vector, respectively. and Representing views respectively and view Spatial location, express and Spatial distance between them; The square of the Euclidean distance in space; The attenuation coefficient; The formula for the redundancy score is: ; in, Represents a view Redundancy fraction; This indicates retrieving the maximum value from all views except the current view; The formula for the final screening score is: ; in, Represents a view The final screening score; Represents the set of preliminary views Image-level matching results Sort in descending order to obtain a view of the sorted sequence of image-level matching results. Image-level matching results.

[0009] Furthermore, the back projection formula is: ; ; in, Represents pixels 3D coordinates of the camera coordinate system; Represents the depth information of views within the candidate view subset; The inverse matrix of the intrinsic parameter matrix; Represents the homogeneous coordinates of a pixel; Represents pixels in the world coordinate system 3D coordinates; and Here, represents the camera's extrinsic parameters, and represents the rotation matrix and translation vector, respectively; the intrinsic parameter matrix... The formula is: ; in, and For camera focal length, and The coordinates of the camera's principal point; The formula for calculating the center position of the 3D instance is as follows: ; in, Indicates the first The 3D center location of an instance. Indicates the first 3D point cloud after back projection of an instance express The number of points in the middle.

[0010] Furthermore, 3D object layout information is obtained through an open-source 3D semantic segmentation model. , For the first in the scene The center coordinates of a 3D object; find the nearest neighbor with... To establish a mapping relationship between the 2D instance and the 3D object, find the nearest 3D object, using the following formula: ; in, Index the matched 3D objects. The distance is Euclidean.

[0011] Furthermore, the formula for calculating the relative position vectors between instances in 3D space is: ; ; in, Representing the subject and object The relative position vectors between them This is the normalized relative position vector; The formula for the relationship matching result is: ; in, Indicates the first A pre-defined spatial relationship standard vector, express and The matching score; The formula for comprehensively calculating the instance relationship matching results for each candidate view is as follows: ; ; in, Representation of instances The final relation score, Representation of instances The set of relational triples involved, Representation of instances The number of relational triples involved. Representing relation triples The collection of instances contained therein; This represents the total relation score of the view. Indicates the current view Total number of instances.

[0012] Furthermore, S5 is represented as: ; in, For the optimal set of views, For the current text query, For the current downstream task types, This represents the inference results for a multimodal large model.

[0013] As can be seen from the above technical solution, compared with the prior art, the present invention provides a zero-sample 3D scene understanding method based on optimal viewpoint selection, which has the following beneficial effects: This invention employs S1 to filter a preliminary view set containing all query-related instances, achieving separation between target instances and redundant backgrounds and providing a complete core visual foundation for subsequent view optimization. Building upon this, S2 combines camera shooting direction and position information to calculate view similarity, filtering redundant images with similar viewpoints and fusing matching results to optimize view diversity, significantly reducing unnecessary computational load and substantially improving the efficiency and information complementarity of view filtering. Simultaneously, S3 establishes a mapping relationship between 2D instances and 3D objects through image segmentation and 3D back-projection technology, transforming two-dimensional visual information into three-dimensional spatial coordinates, providing a crucial bridge for spatial relationship verification. S4 decomposes textual spatial relationships into simple combinations, and through matching calculations of the relative positions between instances with preset standards, filters out the optimal view with the highest relationship matching degree, ensuring spatial logic consistency between the view and text description, effectively solving the core problem of spatial expression misalignment in existing technologies. Finally, S5 inputs the optimal view and the original text query into a multimodal large language model, achieving accurate implementation of tasks such as target localization and spatial relationship reasoning in 3D scenes.

[0014] This invention, through its innovative design of phased view filtering and cross-modal collaborative reasoning, fully meets the practical needs of zero-shot 3D understanding in scenarios such as robot interaction and autonomous driving environmental perception. The complete instance filtering and viewpoint optimization mechanism ensures the effectiveness and diversity of views, while 3D mapping and spatial relationship verification improve the accuracy and consistency of scene understanding. It achieves efficient cross-modal reasoning in complex 3D scenes without large-scale labeled data, which is of great significance for promoting the engineering application of zero-shot 3D understanding technology and has strong practicality. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the overall process of the present invention.

[0017] Figure 2 This is a schematic diagram of the network structure provided by the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] This invention discloses a zero-shot 3D scene understanding method based on optimal viewpoint selection, such as... Figure 1 and Figure 2 As shown, it includes the following steps: S1. Process the input multi-view 2D images one by one using the instance detection model to obtain the 2D bounding boxes and detection confidence of the query related instances, and crop the instances according to the bounding boxes to obtain the local visual information of the instances; calculate the matching degree between the local visual information of each instance and the text instance concept through the vision-language model, and obtain the image-level matching result by combining the detection confidence, and filter out the preliminary view set containing all query related instances. S2. Extract the camera shooting direction and position information corresponding to each view in the preliminary view set, calculate the view similarity between views, and calculate the redundancy score of each view in one step; based on the image-level matching results and redundancy scores of each view, calculate the final screening score of each view, and sort and filter redundant images with similar viewpoints to obtain a candidate view subset with diverse viewpoints. S3. For each view in the candidate view subset, use the SAM2 image segmentation model to generate a 2D mask corresponding to the target instance; use the back projection formula based on image depth information and camera intrinsic and extrinsic parameters to backproject the pixels corresponding to the 2D mask to 3D space and calculate the 3D center position of the instance; use nearest neighbor search to match the 3D center position with the preset 3D object layout information to establish the mapping relationship between 2D instances and 3D objects. S4. Decompose the spatial relationship description in the text query into simple relationship combinations, calculate the relative position vector between instances in 3D space, match it with the preset spatial relationship standard vector, and obtain the relationship matching result of each instance; perform a comprehensive calculation on the relationship matching results of all instances of each candidate view, and select the views with the highest total relationship score as the optimal view set. S5. Input the optimal view set and the original text query into the multimodal large language model to complete the downstream understanding task in the 3D scene.

[0020] In a specific embodiment, step S1 obtains a preliminary view set containing all query-related instances through the instance complete view filtering module. Specifically, let the input multi-view 2D image set be... ,in, Indicates the first A 2D image, This represents the total number of images. The set of query-related instance concepts obtained after text query parsing is... , This represents the total number of instance concepts. For each image... Using the GroundingDINO instance detection model, output the detected instances in this image. The set of 2D bounding boxes corresponding to each instance and the detection confidence of each instance , where each bounding box From coordinates Defined as the x-coordinate of the top left corner, y-coordinate of the top left corner, x-coordinate of the bottom right corner, and y-coordinate of the bottom right corner of the instance in the image.

[0021] Furthermore, based on the bounding box coordinates, a clipping function is used. The formula for extracting local visual information from an instance is shown below: ; in, Indicates the first In the 2D image, the first Local visual information of an instance, Representing an image median coordinate The pixel value at that location.

[0022] Furthermore, the BLIP vision-language model is used to compute local visual information. Corresponding instance concept Instance-level matching score This idea is based on cosine similarity, and the formula is shown below: ; in, yes The feature transpose, and They are respectively and The characteristic norm.

[0023] Furthermore, combined with the detection confidence level The instance-level matching scores are weighted and averaged to obtain the first... Image-level matching results of the images The formula is shown below: ; Furthermore, by setting the image-level matching threshold to 0.6, the following were filtered out. And includes all query-related instances (i.e.) There exists a corresponding Images that meet the matching criteria are used to form a preliminary view set. , This represents the number of views.

[0024] In one specific embodiment, S2 builds upon the initial view set constructed in S1. The following is a process for optimizing perspective diversity: Extract the preliminary view set Each view The corresponding camera parameters, including the shooting direction vector. (Eulerian angles) and spatial location (Three-dimensional coordinate representation). Calculate any two viewpoint images. as well as ( The similarity of perspectives between the two is divided into two parts: similarity of shooting direction and spatial distance.

[0025] 1) Calculate the similarity of shooting directions using standardized cosine similarity. The formula is shown below: ; in, The dot product of the shooting direction vectors, These represent the norms of the camera's shooting direction vector, respectively.

[0026] 2) Calculate spatial distance using the exponential distance decay function. The formula is shown below: ; in, The square of the Euclidean distance in space. The attenuation coefficient is set to 1.0 in this embodiment.

[0027] Furthermore, by fusing the similarity of shooting directions with spatial distance, the viewpoint similarity between views is obtained, and the redundancy score of each view is further calculated. The formula is shown below: ; in, This represents the maximum value of all views except the current view, representing the redundancy score. The larger the value, the more unique the perspective of the view.

[0028] For the preliminary view set Image-level matching results Sort in descending order to obtain the sorted sequence of image-level matching results. By combining the sorted image-level matching results with the redundancy score, the final screening score for each view is calculated. The formula is as follows: ; Based on final screening score Sort the views in descending order and select the top-P views (representing the first P views) to form a subset of candidate views. .

[0029] In a specific embodiment, step S3 uses the subset of candidate views based on the output of S2 to establish the correspondence between 2D instances and 3D objects. The specific process is as follows: First, define a subset of candidate views. The Middle Zhang Viewpoint The set of instance bounding boxes obtained by S1 detection of this view is , This represents the number of instances in the view.

[0030] Furthermore, the SAM2 image segmentation model was used to... Perform instance segmentation and generate a 2D mask for each instance. ;in, For the first A binary mask for each instance (foreground pixel value is 255, background pixel value is 0).

[0031] Further, read the view depth information ( For pixel coordinates, (representing the depth value at that pixel), camera intrinsic parameter matrix and extrinsic parameters (rotation matrix) Translation vector The intrinsic parameter matrix is ​​as follows. The definition is as follows: ; in, as well as For camera focal length, as well as These are the coordinates of the camera's principal point.

[0032] Furthermore, the 2D mask The corresponding pixels are back-projected into 3D space to obtain the 3D point cloud of the instance. The back projection formula is as follows: ; ; in, For pixels 3D coordinates of the camera coordinate system The inverse of the intrinsic parameter matrix. 3D coordinates in the world coordinate system These are the homogeneous coordinates of the pixels.

[0033] Furthermore, computing 3D point clouds The center of mass, to obtain the first 3D center location of an instance The formula is as follows: ; in, 3D point cloud The number of points in the middle.

[0034] Furthermore, 3D object layout information is obtained through an open-source 3D semantic segmentation model. , For the first in the scene The center coordinates of a 3D object; find the nearest neighbor with... To establish a mapping relationship between the 2D instance and the 3D object, find the nearest 3D object, using the following formula: ; in, Index the matched 3D objects. The distance is Euclidean.

[0035] In a specific embodiment, step S4, based on the 3D mapping relationship established in S3, filters the optimal view with consistent spatial relationships. The specific process is as follows: Let the set of spatial relation descriptions obtained from text query parsing be . , Describe the total number of relations, each relation Decomposed into a triple of "subject-relation type-object" (e.g., "chair - on the right - table"). Preset spatial relationship standard vector library. It includes 3D vector representations of common relationships such as "left and right", "front and back", and "adjacent", with the "left and right" relationship vector being the most prominent. "Before and After" Relationship Vector "Nearest neighbor" relationship vector The vectors are obtained through statistical training on large-scale 3D scene data.

[0036] Furthermore, for the subset of candidate views The Middle Zhang Viewpoint Based on the mapping relationship established by S3, obtain the set of 3D instance center locations. and for each relation triple Match the corresponding 3D instance center (Main body) and (object).

[0037] Furthermore, the relative position vector between the subject and the object is calculated using the formula shown below; ; ; in, This is the normalized relative position vector.

[0038] Furthermore, the cosine similarity between the normalized relative position vector and the corresponding relation standard vector is calculated to obtain the matching score of the relation triple in the current view. The formula is shown below: ; Furthermore, if a single instance Participating in multiple relation triples ( For example (The set of relational triples involved), take the maximum matching score of each triple as the score of the instance in the corresponding triple, and then average the scores of all involved triples to obtain the instance. Final relation score The formula is shown below: ; in, Representing relation triples The collection of instances contained For example The number of relational triples involved.

[0039] Furthermore, regarding the current view The total view relation score is obtained by averaging the final relation scores of all instances in the view. The formula is shown below: ; Traverse candidate view subsets Calculate the total relation score for each of the Top-P view sheets and select them. The highest set of views is the optimal set of views. .

[0040] In a specific embodiment, step S5 inputs the optimal view output from S4 along with the original text query into the multimodal large language model to complete downstream understanding tasks such as target localization and spatial relationship reasoning in the 3D scene: ; in, For the optimal set of views, For the current text query, For the current downstream task types, This represents the inference results for a multimodal large model.

[0041] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0042] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A zero-shot 3D scene understanding method based on optimal viewpoint selection, characterized in that, Includes the following steps: S1. The input multi-view 2D images are processed one by one using an instance detection model to obtain the 2D bounding boxes and detection confidence of the query-related instances, and the local visual information of the instances is obtained by cropping according to the bounding boxes; the matching degree between the local visual information of each instance and the text instance concept is calculated by a vision-language model, and the image-level matching result is obtained by combining the detection confidence, and a preliminary view set containing all query-related instances is selected. S2. Extract the camera shooting direction and position information corresponding to each view in the preliminary view set, calculate the view similarity between views, and calculate the redundancy score of each view in one step. Based on the image-level matching results and the redundancy score of each view, the final screening score of each view is calculated, and redundant images with similar viewing angles are sorted and filtered to obtain a subset of candidate views with diverse viewing angles. S3. For each view in the candidate view subset, a 2D mask corresponding to the target instance is generated using the SAM2 image segmentation model; the pixels corresponding to the 2D mask are back-projected to 3D space using a back-projection formula based on image depth information and camera intrinsic and extrinsic parameters, and the 3D center position of the instance is calculated; the 3D center position is matched with the preset 3D object layout information through nearest neighbor search to establish a mapping relationship between 2D instances and 3D objects. S4. Decompose the spatial relationship description in the text query into simple relationship combinations, calculate the relative position vector between instances in 3D space, match it with the preset spatial relationship standard vector, and obtain the relationship matching result of each instance. The relationship matching results of all instances for each candidate view are comprehensively calculated, and the views with the highest total relationship scores are selected as the optimal view set. S5. Input the optimal view set and the original text query into the multimodal large language model to complete the downstream understanding task in the 3D scene.

2. The zero-shot 3D scene understanding method based on optimal viewpoint selection according to claim 1, characterized in that, The formula for calculating the matching degree between the local visual information of each instance and the text instance concept is as follows: ； in, Indicates the first In the 2D image, the first Local visual information of an instance, Indicates the first The concept of a text instance. express and Instance-level matching score; and They are respectively and Characteristic norm, for The feature transpose; The formula for the image-level matching result is: ； in, Indicates the first Image-level matching results for a 2D image; Indicates the first In the 2D image, the first The detection confidence of each instance. For the first Total number of instances in a 2D image.

3. The zero-shot 3D scene understanding method based on optimal viewpoint selection according to claim 1, characterized in that, The calculation of viewpoint similarity between views includes calculating the similarity of shooting directions and spatial distance, and the specific formula is as follows: ； ； Among them, it means and Representing views respectively and view The shooting direction vector, for and Similarity of shooting direction; The dot product of the shooting direction vectors, These represent the norms of the camera's shooting direction vector, respectively. and Representing views respectively and view Spatial location, express and Spatial distance between them; The square of the Euclidean distance in space; The attenuation coefficient; The formula for the redundancy score is: ； in, Represents a view Redundancy fraction; This indicates retrieving the maximum value from all views except the current view; The formula for the final screening score is: ； in, Represents a view The final screening score; Represents the set of preliminary views Image-level matching results Sort in descending order to obtain a view of the sorted sequence of image-level matching results. Image-level matching results.

4. The zero-shot 3D scene understanding method based on optimal viewpoint selection according to claim 1, characterized in that, The back projection formula is: ； ； in, Represents pixels 3D coordinates of the camera coordinate system; Represents the depth information of views within the candidate view subset; The inverse matrix of the intrinsic parameter matrix; Represents the homogeneous coordinates of a pixel; Represents pixels in the world coordinate system 3D coordinates; and Here, represents the camera's extrinsic parameters, and represents the rotation matrix and translation vector, respectively; the intrinsic parameter matrix... The formula is: ； in, and For camera focal length, and The coordinates of the camera's principal point; The formula for calculating the center position of the 3D instance is as follows: ; in, Indicates the first The 3D center location of an instance. Indicates the first 3D point cloud after back projection of an instance express The number of points in the middle.

5. The zero-shot 3D scene understanding method based on optimal viewpoint selection according to claim 1, characterized in that, The formula for calculating the relative position vectors between instances in 3D space is: ； ； in, Representing the subject and object The relative position vectors between them This is the normalized relative position vector; The formula for the relationship matching result is: ； in, Indicates the first A pre-defined spatial relationship standard vector, express and The matching score; The formula for comprehensively calculating the instance relationship matching results for each candidate view is as follows: ； ； in, Representation of instances The final relation score, Representation of instances The set of relational triples involved, Representation of instances The number of relational triples involved. Representing relation triples The collection of instances contained therein; This represents the total relation score of the view. Indicates the current view Total number of instances.

6. The zero-shot 3D scene understanding method based on optimal viewpoint selection according to claim 1, characterized in that, S5 is represented as: ； in, For the optimal set of views, For the current text query, For the current downstream task types, This represents the inference results for a multimodal large model.

Images