Multi-scale three-dimensional scene understanding method based on hierarchical superpoints

Abstract

The invention discloses a multi-scale three-dimensional scene understanding method based on hierarchical super-points, which comprises the following steps: extracting original point cloud features through a scene encoder, generating initial first-level super-points through rapid super-point segmentation, and keeping geometric and semantic details while controlling the scale of the point cloud; then combining a text instruction and a learnable token by an instruction perception superpoint aggregation module, constructing a multi-level superpoint structure layer by layer through an instruction perception aggregation mechanism and a hierarchical feature interaction mechanism, and inputting a large language model LLM by taking a fixed-length top superpoint as a visual token; a super-point query module is embedded in an LLM intermediate layer, a key area is positioned based on attention weight, and fine-grained super-point features are retrieved and injected into a reasoning process; and finally, processing by an LLM decoder to generate an understanding result. The method realizes tight coupling of three-dimensional representation and understanding, has excellent adaptability and performance in various complex scenes, and significantly improves generalization ability.

Description

A Multi-Scale 3D Scene Understanding Method Based on Hierarchical Superpoints Technical Field

[0001] This invention belongs to the technical field of 3D scene understanding, specifically relating to a multi-scale 3D scene understanding method based on hierarchical superpoints. Background Technology

[0002] 3D scene understanding is a core research area in computer vision and artificial intelligence. It aims to accurately interpret real-world scenes by perceiving and analyzing the geometric structure, object distribution, and semantic relationships in 3D space. It is widely used in key areas such as autonomous driving, robot navigation, virtual reality, and smart city construction. With the continuous expansion of related application scenarios, higher demands are being placed on the versatility of 3D scene understanding technology.

[0003] Traditional 3D scene understanding technologies employ a modular paradigm, extracting corresponding modal features through visual encoders and text encoders respectively. These features are then processed by a fusion module and output by a specific classifier or generator. While this type of technology is designed for a single task and has made some progress in its specific domain, it lacks cross-task adaptability and cannot meet the needs of multi-task parallel processing in complex scenarios, severely limiting its versatility.

[0004] In recent years, the rapid development of Large Language Models (LLM) and Multimodal Large Language Models (MLLM) has provided new technical paths for 3D scene understanding, promoting the formation of a more general semantic reasoning paradigm. However, existing LLM-based 3D scene understanding technologies still have significant drawbacks: First, the use of variable-length object-level features to construct scene representations results in a linear increase in the number of features as the scene scope and complexity expand, leading to a surge in memory overhead and a significant decrease in inference efficiency in large-scale complex scenes, making it difficult to adapt to a wider range of application needs. Simultaneously, object feature extraction relies on pre-trained detection models, resulting in weak generalization ability when transferring between scenes. Second, the decoupled architecture between "scene representation" and "scene understanding"—where the model independently constructs scene features and then concatenates them with text features before inputting them into the LLM to complete the understanding task—makes it impossible for the LLM to perform adaptive inference at different spatial scales according to task requirements. Efficiency and accuracy are difficult to balance; while compressing the number of features can improve speed, it loses scene details, and increasing the feature size can supplement information, but it leads to a surge in computational overhead.

[0005] Furthermore, existing technologies suffer from several minor issues: the point cloud preprocessing stage often employs fixed-point downsampling, resulting in irreversible information loss and impacting the expressive power of scene representation; constructing scene representations only at a single object scale leads to limited representation granularity and incomplete semantic coverage, failing to form a unified representation with fixed length, complete information, and cross-scale expressive capabilities; and the lack of fine-grained information selection and efficient supplementation mechanisms based on task requirements easily incurs additional computational overhead when supplementing key details, further exacerbating the conflict between efficiency and accuracy. These problems collectively restrict the application effectiveness of 3D scene understanding technology in large-scale complex scenes, necessitating a technical solution that can overcome these limitations. Summary of the Invention

[0006] The main objective of this invention is to overcome the shortcomings and deficiencies of existing technologies. Addressing the problems of irreversible information loss, poor scalability and generalization, and difficulty in balancing efficiency and accuracy in existing methods, this solution innovatively utilizes a hierarchical superpoint structure to simultaneously achieve compact and information-complete scene representation, as well as embedded fine-grained information query with tight coupling between representation and understanding. This achieves a technological breakthrough and effectively enhances the scalability and inference accuracy of the model in various complex indoor and outdoor scenarios.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a multi-scale 3D scene understanding method based on hierarchical superpoints, comprising the following steps:

[0009] The point features of the original point cloud are extracted by the scene encoder, and the initial level 1 superpoints are generated by the fast superpoint segmentation algorithm, which reduces the size of the point cloud while preserving geometric and semantic details.

[0010] Using the instruction-aware superpoint aggregation module, the initial level 1 superpoint, text instruction, and learnable token are received. Through the instruction-aware aggregation mechanism and the hierarchical feature interaction mechanism, a multi-level superpoint structure is generated layer by layer. The top L-level superpoint of the multi-level superpoint structure is used as a fixed-length visual token, and the lower-level superpoints retain fine-grained geometric and semantic information, where L≥2.

[0011] An embedded superpoint query module is inserted into the intermediate layer of the large language model LLM to extract the attention weight of the text token to the visual token, select the K top-level superpoints with the highest scores as regions of interest, retrieve the low-level superpoint features corresponding to the K top-level superpoints, reuse the learnable token to extract instruction perception detail features through cross attention and interaction with text instructions and low-level superpoint features, and inject the detail features into the learnable token through residual update.

[0012] Learnable tokens with injected detailed features are input into the remaining decoder layer of the LLM, and after processing, 3D scene understanding results are generated.

[0013] As a preferred technical solution, the implementation process of the instruction-aware aggregation mechanism is as follows: using learnable tokens to extract instruction-aware features from text instructions through cross-attention, calculating the similarity between low-level superpoint features and the instruction-aware features to obtain an importance score, and performing weighted aggregation of low-level superpoints based on the importance score to generate high-level superpoints.

[0014] As a preferred technical solution, the hierarchical feature interaction mechanism is implemented as follows:

[0015] After each layer of superpoint aggregation is completed, a sparse attention mechanism is used to enable neighboring superpoints to exchange contextual information and fuse spatial neighborhood information and relative spatial relationships.

[0016] As a preferred technical solution, the K value is adaptively adjusted through model training or a fixed threshold is preset according to the complexity of the scene.

[0017] As a preferred technical solution, the number of levels L of the multi-level superpoint structure is dynamically adjusted according to the scale of the scene.

[0018] As a preferred technical solution, the scene encoder is a dedicated encoder for point cloud feature extraction, which can retain the three-dimensional geometric structure information and semantic association features of the point cloud.

[0019] As a preferred technical solution, the calculation of cross-attention is based on the semantic correlation between text instruction features, learnable token features, and superpoint features.

[0020] Secondly, the present invention provides a multi-scale 3D scene understanding system based on hierarchical superpoints, which is applied to the multi-scale 3D scene understanding method based on hierarchical superpoints, including a scene encoding module, an instruction-aware superpoint aggregation module, an embedded superpoint query module, and a result generation module.

[0021] The scene encoding module is used to extract point features from the original point cloud through the scene encoder, and generate initial level 1 superpoints using a fast superpoint segmentation algorithm, thereby reducing the size of the point cloud while preserving geometric and semantic details.

[0022] The instruction-aware superpoint aggregation module is used to receive the initial level 1 superpoint, text instruction and learnable token, and generate a multi-level superpoint structure layer by layer through the instruction-aware aggregation mechanism and the hierarchical feature interaction mechanism. The top L level superpoint of the multi-level superpoint structure is used as a fixed-length visual token, and the lower level superpoints retain fine-grained geometric and semantic information, where L≥2.

[0023] The embedded superpoint query module is used to insert an embedded superpoint query module into the intermediate layer of the large language model LLM, extract the attention weight of the text token to the visual token, select the K top-level superpoints with the highest scores as regions of interest, retrieve the low-level superpoint features corresponding to the K top-level superpoints, reuse the learnable token to extract instruction perception detail features through cross-attention and interaction with text instructions and low-level superpoint features, and inject the detail features into the learnable token through residual update;

[0024] The result generation module is used to input the learnable tokens with injected detailed features into the LLM residual decoder layer, and generate 3D scene understanding results after processing.

[0025] Thirdly, the present invention provides an electronic device, the electronic device comprising:

[0026] At least one processor; and,

[0027] A memory communicatively connected to the at least one processor; wherein,

[0028] The memory stores computer program instructions that can be executed by the at least one processor, which enable the at least one processor to execute the hierarchical superpoint-based multi-scale 3D scene understanding method.

[0029] Fourthly, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the multi-scale three-dimensional scene understanding method based on hierarchical superpoints.

[0030] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0031] 1. This invention utilizes an innovatively designed instruction-aware superpoint aggregation module to achieve multi-scale accurate modeling of 3D scenes based on a hierarchical superpoint structure, demonstrating significant technical advantages and excellent results. This module uses top-level superpoints of fixed length as visual tokens to input into the large language model, while simultaneously preserving fine-grained geometric and semantic information through lower-level superpoints. This successfully addresses the technical challenge of token length increasing with scene complexity in traditional technologies, achieving an organic unity of compact representation and information completeness. During the construction of multi-layer superpoints, the introduced instruction-aware aggregation mechanism, through dynamic feature aggregation guided by text instructions, accurately aligns instruction semantics with superpoint features, effectively ensuring the semantic relevance and compactness of scene representation. Meanwhile, the hierarchical feature interaction mechanism, through layer-by-layer spatial context interaction, strengthens the association and information transmission between superpoints, significantly improving the robustness and completeness of scene representation, providing a high-quality foundation for subsequent scene understanding.

[0032] 2. The innovative embedded superpoint query module of this invention, by combining a hierarchical superpoint structure with an embedded query mechanism, brings significant technical improvements. This module is embedded in the middle layer of the LLM inference process. Leveraging the attention weights of LLM, it can accurately locate key areas of the scene and dynamically extract core detail information from hierarchical superpoints. Simultaneously, through the embedded query mechanism, learnable tokens are reused to efficiently inject fine-grained information into the LLM inference process, achieving dynamic refinement of the 3D scene representation. This design eliminates the need to increase the number of visual tokens, adaptively supplementing key details during the inference stage. It effectively avoids the inherent contradiction between the number of tokens and computational efficiency, enabling the model to maintain high inference performance while precisely improving scene understanding accuracy, ultimately achieving an optimized balance between efficiency and accuracy. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 is a flowchart of the multi-scale 3D scene understanding method based on hierarchical superpoints according to an embodiment of the present invention;

[0035] Figure 2 is a flowchart of the instruction-aware super-point aggregation module in an embodiment of the present invention;

[0036] Figure 3 is a flowchart of the embedded super-point query module in an embodiment of the present invention;

[0037] Figure 4 is a block diagram of the multi-scale 3D scene understanding system based on hierarchical superpoints according to an embodiment of the present invention.

[0038] Figure 5 is a structural diagram of the electronic device according to an embodiment of the present invention. Detailed Implementation

[0039] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0040] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0041] Explanation of relevant terms in this patent:

[0042] LLM: Large Language Model

[0043] MLLM: Multimodal Large Language Model

[0044] CIDEr: Consensus-based Image Description Evaluation

[0045] METEOR: Metric for Evaluation of Translation with Explicit Ordering

[0046] ROUGE: Recall-Oriented Understudy for Gisting Evaluation

[0047] n-gram: n-gram Language Model

[0048] TF-IDF: Term Frequency-Inverse Document Frequency

[0049] As shown in Figure 1, this embodiment of a multi-scale 3D scene understanding method based on hierarchical superpoints includes the following steps:

[0050] S1. Scene Encoding and Initial Hyperpoint Generation:

[0051] Point features are extracted from the original point cloud using a scene encoder, and initial level 1 superpoints are generated using a fast superpoint segmentation algorithm. This process significantly reduces the size of the point cloud while fully preserving the geometric structure and semantic details of the original point cloud, avoiding the irreversible information loss caused by traditional fixed-point downsampling methods, and laying a solid foundation for subsequent scene representation construction. The scene encoder is a dedicated encoder for point cloud feature extraction, effectively preserving the 3D geometric structure information and semantic association features of the point cloud.

[0052] The generation process of the Level 1 superpoint is as follows:

[0053] Based on the geometric and semantic features of point clouds, a global energy minimization problem is defined. The L0 cut tracing algorithm is used to quickly obtain an approximate solution to divide the point cloud into geometrically and semantically consistent parts, thereby generating level 1 superpoints.

[0054] S2. Unlike traditional methods that construct scene representations only at a single object scale, this invention achieves multi-scale modeling based on the natural hierarchical structure of 3D scene point clouds. A multi-level superpoint structure is generated layer by layer through an instruction-aware superpoint aggregation module, as shown in Figure 2. The top-level L-level superpoints serve as fixed-length visual tokens input to the Large Language Model (LLM), while lower-level superpoints retain fine-grained information. This solves the problem of token length increasing with scene complexity while maintaining information completeness.

[0055] Furthermore, the instruction-aware superpoint aggregation module receives initial level 1 superpoints, text instructions, and learnable tokens. Through instruction-aware feature aggregation and hierarchical feature interaction, it constructs hierarchical superpoints that are both compact and highly expressive. In the process of constructing multi-level superpoint aggregation, the module introduces an instruction-aware aggregation mechanism, where the aggregation weights are dynamically guided by the user-input natural language instructions. Specifically, the module first uses learnable tokens to extract instruction-aware features from the text instructions through cross-attention. Then, it calculates the similarity between the low-level superpoint features and these instruction-aware features, using this as an importance score to weighted aggregate the low-level superpoints, thereby generating high-level superpoints. This instruction-guided dynamic feature aggregation ensures that the generated fixed-length superpoint features are highly condensed and optimized, guaranteeing the semantic relevance and compactness of the scene representation. Furthermore, this module also introduces a hierarchical feature interaction mechanism. After each layer of superpoint aggregation is completed, a sparse attention mechanism enables neighboring superpoints to exchange contextual information. This process expands the semantic awareness range of individual superpoints, allowing each superpoint to integrate spatial neighborhood information and relative spatial relationships, improving the robustness and completeness of the scene representation.

[0056] In this embodiment, the number of levels of the multi-level superpoint structure is L≥2, and can be dynamically adjusted according to the scene scale. The top-level L-level superpoints are used as fixed-length visual tokens input to the Large Language Model (LLM), while the lower-level superpoints retain fine-grained geometric and semantic information, which solves the problem that the token length increases with the scene complexity in traditional technologies.

[0057] S3. Unlike traditional solutions that decouple representation and understanding, this invention utilizes hierarchical superpoints, introducing an embedded superpoint query module in the LLM middle layer to achieve tight coupling between three-dimensional representation and understanding. This mechanism allows large language models to perform adaptive inference at different spatial scales, overcoming the limitation of decoupled designs that cannot dynamically adjust the inference granularity, thus achieving a better balance between model efficiency and accuracy.

[0058] Furthermore, an embedded superpoint query module is inserted into the LLM intermediate layer. Combining the hierarchical superpoint structure with the embedded query mechanism achieves deep coupling between 3D representation and language understanding, enabling the model to perform adaptive inference across different spatial scales of the scene and optimizing the inference process. Specifically, as shown in Figure 3, during the LLM inference process, the module first extracts the attention weights of text tokens on visual tokens and selects the K highest-scoring top-level superpoints as regions of interest, thereby locating the visual regions most relevant to the LLM. Next, for the selected Top-K top-level superpoints, the module retrieves their corresponding low-level (fine-grained) superpoint features from the hierarchical structure to obtain more refined geometric and semantic information from the original point cloud. Subsequently, to efficiently incorporate supplementary fine-grained features into the LLM inference process, the module reuses learnable tokens. Through cross-attention interaction with text instructions and low-level superpoint features, it extracts detailed features of instruction perception and injects the extracted information into the learnable tokens through residual updates. Learnable tokens infused with fine-grained information then participate in subsequent inference in LLM, enabling LLM to dynamically refine the 3D scene representation during inference without increasing the length of additional visual tokens, thus ensuring an effective balance between efficiency and accuracy.

[0059] S4. Result Generation: The learnable tokens, infused with fine-grained detail features, are input into the remaining decoder layers of the LLM. After further processing by the model, the final 3D scene understanding result is generated. The Large Language Model (LLM) includes a single-modal large language model or a multimodal large language model (MLLM), which can be flexibly selected according to the needs of the actual application scenario.

[0060] To comprehensively evaluate the effectiveness and scalability of the proposed multi-scale 3D scene understanding method, this invention underwent systematic validation on a series of 3D scene understanding benchmarks, covering indoor single-room, indoor multi-room, outdoor street-level, and city-level scenes. Validation employed three industry-standard evaluation metrics:

[0061] (1) Consensus-based Image Description Evaluation (CIDEr): Based on n-gram semantic consensus and combined with TF-IDF weighting, the semantic overlap between the generated text and the reference text set is calculated to evaluate the core semantic association of the instruction response.

[0062] (2) Metric for Evaluation of Translation with Explicit Ordering (METEOR): Combining precision, recall, stemming, and synonym matching, METEOR comprehensively evaluates the semantic accuracy and syntactic fluency of instruction response.

[0063] (3) Recall-Oriented Understudy for Gisting Evaluation (ROUGE): The completeness and relevance of the generated text are evaluated by calculating the longest common subsequence and other overlapping features between the generated text and the reference text.

[0064] The above multi-dimensional evaluation metrics fully validated the adaptability and superiority of this invention in different types of 3D scenes. Test results show that, compared with state-of-the-art methods such as LEO, ChatScene, LL3DA, and LSceneLLM, this invention achieves performance breakthroughs in all-scene benchmark tests.

[0065] In indoor multi-room scenarios, this invention achieved a CIDEr score of 122.59 in the multi-room question-and-answer task, an improvement of 5.38 over the second place; a CIDEr score of 6.16 in the multi-room caption task, an improvement of 1.57; and a CIDEr score of 65.15 in the multi-room planning task, an improvement of 2.07. The ROUGE and METEOR indicators were simultaneously optimized, as shown in Table 1.

[0066] Table 1

[0067]

[0068] In a single-room indoor scenario, this invention achieves a CIDEr of 89.12 for question-answering tasks, 232.02 for embodied planning tasks, and 107.31 for embodied question-answering tasks, all of which surpass existing methods, as shown in Table 2.

[0069] Table 2

[0070]

[0071] In outdoor street-level scenarios, the invention achieves an overall inference accuracy of 57.0% on the NuScenes-QA benchmark, reaching the highest level of current general multimodal models and adapting to the complex requirements of sparse point cloud driving scenarios, as shown in Table 3.

[0072] Table 3

[0073]

[0074] In city-level scenarios, this invention achieves an accuracy of 66.63% on the City3DQA benchmark, which is 2.6% higher than the existing best LLM-based model LSceneLLM. It can achieve accurate understanding of large-scale city scenarios without scene graph input, and its scalability is significantly better than traditional methods, as shown in Table 4.

[0075] Table 4

[0076]

[0077] Meanwhile, efficiency tests show that, compared with traditional methods for constructing single-object-level scene representations, this invention significantly reduces computational overhead while maintaining superior performance, with training and inference times reduced by 31.0% and 60.8% respectively, achieving an effective balance between efficiency and accuracy.

[0078] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0079] Based on the same idea as the hierarchical superpoint-based multi-scale 3D scene understanding method in the above embodiments, the present invention also provides a hierarchical superpoint-based multi-scale 3D scene understanding system, which can be used to execute the above-described hierarchical superpoint-based multi-scale 3D scene understanding method. For ease of explanation, the structural diagram of the embodiment of the hierarchical superpoint-based multi-scale 3D scene understanding system only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0080] Please refer to Figure 4. In another embodiment of this application, a multi-scale three-dimensional scene understanding system 100 based on hierarchical superpoints is provided. The system includes a scene encoding module 101, an instruction-aware superpoint aggregation module 102, an embedded superpoint query module 103, and a result generation module 104.

[0081] The scene encoding module 101 is used to extract point features from the original point cloud through the scene encoder, and generate initial level 1 superpoints using a fast superpoint segmentation algorithm, thereby reducing the size of the point cloud while preserving geometric and semantic details.

[0082] The instruction-aware superpoint aggregation module 102 is used to receive the initial level 1 superpoint, text instruction and learnable token, and generate a multi-level superpoint structure layer by layer through the instruction-aware aggregation mechanism and the hierarchical feature interaction mechanism. The top L level superpoint of the multi-level superpoint structure is used as a fixed-length visual token, and the lower level superpoints retain fine-grained geometric and semantic information, where L≥2.

[0083] The embedded superpoint query module 103 is used to insert an embedded superpoint query module into the intermediate layer of the large language model LLM, extract the attention weight of the text token to the visual token, select the K top-level superpoints with the highest scores as regions of interest, retrieve the low-level superpoint features corresponding to the K top-level superpoints, reuse the learnable token to extract instruction perception detail features through cross-attention and interaction with text instructions and low-level superpoint features, and inject the detail features into the learnable token through residual update;

[0084] The result generation module 104 is used to input the learnable tokens with injected detailed features into the LLM residual decoder layer, and generate a 3D scene understanding result after processing.

[0085] It should be noted that the multi-scale 3D scene understanding system based on hierarchical superpoints of the present invention corresponds one-to-one with the multi-scale 3D scene understanding method based on hierarchical superpoints of the present invention. The technical features and beneficial effects described in the embodiments of the multi-scale 3D scene understanding method based on hierarchical superpoints described above are applicable to the embodiments of multi-scale 3D scene understanding based on hierarchical superpoints. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0086] Furthermore, in the implementation of the hierarchical superpoint-based multi-scale 3D scene understanding system in the above embodiments, the logical division of each program module is only an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the configuration requirements of the corresponding hardware or for the convenience of software implementation. That is, the internal structure of the hierarchical superpoint-based multi-scale 3D scene understanding system can be divided into different program modules to complete all or part of the functions described above.

[0087] Referring to Figure 5, in one embodiment, an electronic device is provided for implementing a multi-scale 3D scene understanding method based on hierarchical superpoints. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a multi-scale 3D scene understanding program 203 based on hierarchical superpoints.

[0088] The first memory 202 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as the portable hard drive of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in portable hard drive, SmartMediaCard (SMC), SecureDigital (SD) card, FlashCard, etc., equipped on the electronic device 200. Furthermore, the first memory 202 can include both internal storage units and external storage devices of the electronic device 200. The first memory 202 can be used not only to store application software and various types of data installed on the electronic device 200, such as the code of the multi-scale 3D scene understanding program 203 based on hierarchical superpoints, but also to temporarily store data that has been output or will be output.

[0089] In some embodiments, the first processor 201 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory 202 and calls data stored in the first memory 202 to perform various functions of the electronic device 200 and process data.

[0090] Figure 5 only shows an electronic device with components. Those skilled in the art will understand that the structure shown in Figure 5 does not constitute a limitation on the electronic device 200, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0091] The first memory 202 in the electronic device 200 stores a multi-scale 3D scene understanding program 203 based on hierarchical superpoints, which is a combination of multiple instructions. When run in the first processor 201, it can achieve the following:

[0092] The point features of the original point cloud are extracted by the scene encoder, and the initial level 1 superpoints are generated by the fast superpoint segmentation algorithm, which reduces the size of the point cloud while preserving geometric and semantic details.

[0093] Using the instruction-aware superpoint aggregation module, the initial level 1 superpoint, text instruction, and learnable token are received. Through the instruction-aware aggregation mechanism and the hierarchical feature interaction mechanism, a multi-level superpoint structure is generated layer by layer. The top L-level superpoint of the multi-level superpoint structure is used as a fixed-length visual token, and the lower-level superpoints retain fine-grained geometric and semantic information, where L≥2.

[0094] An embedded superpoint query module is inserted into the intermediate layer of the large language model LLM to extract the attention weight of the text token to the visual token, select the K top-level superpoints with the highest scores as regions of interest, retrieve the low-level superpoint features corresponding to the K top-level superpoints, reuse the learnable token to extract instruction perception detail features through cross attention and interaction with text instructions and low-level superpoint features, and inject the detail features into the learnable token through residual update.

[0095] Learnable tokens with injected detailed features are input into the remaining decoder layer of the LLM, and after processing, 3D scene understanding results are generated.

[0096] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0097] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0098] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0099] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A multi-scale 3D scene understanding method based on hierarchical superpoints, characterized in that, The process includes the following steps: First, point features are extracted from the original point cloud using a scene encoder. An initial level 1 superpoint is generated using a fast superpoint segmentation algorithm, reducing the point cloud size while preserving geometric and semantic details. Second, an instruction-aware superpoint aggregation module receives the initial level 1 superpoints, text instructions, and learnable tokens. Through an instruction-aware aggregation mechanism and a hierarchical feature interaction mechanism, a multi-level superpoint structure is generated layer by layer. The top L-level superpoints of this multi-level structure serve as fixed-length visual tokens, while lower-level superpoints retain fine-grained geometric and semantic information, where L≥2. Third, an embedded superpoint query module is inserted into the intermediate layer of the Large Language Model (LLM). This module extracts the attention weights of the text tokens on the visual tokens, selects the K highest-scoring top-level superpoints as regions of interest, retrieves the corresponding low-level superpoint features, reuses the learnable tokens, and extracts instruction-aware detail features through cross-attention interaction with text instructions and low-level superpoint features. These detail features are then injected into the learnable tokens via residual updates. Finally, the learnable tokens with injected detail features are input into the remaining LLM decoder layer, and after processing, a 3D scene understanding result is generated.

2. The multi-scale 3D scene understanding method based on hierarchical superpoints according to claim 1, characterized in that, The implementation process of the instruction-aware aggregation mechanism is as follows: using learnable tokens to extract instruction-aware features from text instructions through cross-attention, calculating the similarity between low-level superpoint features and the instruction-aware features to obtain an importance score, and performing weighted aggregation of low-level superpoints based on the importance score to generate high-level superpoints.

3. The multi-scale 3D scene understanding method based on hierarchical superpoints according to claim 1, characterized in that, The hierarchical feature interaction mechanism is implemented as follows: after each layer of superpoint aggregation is completed, the sparse attention mechanism enables neighboring superpoints to exchange contextual information and fuse spatial neighborhood information and relative spatial relationships.

4. The multi-scale 3D scene understanding method based on hierarchical superpoints according to claim 1, characterized in that, The K value is adaptively adjusted through model training or a fixed threshold is preset according to the complexity of the scene.

5. The multi-scale 3D scene understanding method based on hierarchical superpoints according to claim 1, characterized in that, The number of levels L in the multi-level superpoint structure is dynamically adjusted according to the scale of the scene.

6. The multi-scale 3D scene understanding method based on hierarchical superpoints according to claim 1, characterized in that, The scene encoder is a dedicated encoder for point cloud feature extraction, which can preserve the three-dimensional geometric structure information and semantic association features of the point cloud.

7. The multi-scale 3D scene understanding method based on hierarchical superpoints according to claim 1, characterized in that, The calculation of cross-attention is based on the semantic correlation between text instruction features, learnable token features, and superpoint features.

8. A multi-scale 3D scene understanding system based on hierarchical superpoints, characterized in that, The multi-scale 3D scene understanding method based on hierarchical superpoints, applied to any one of claims 1-7, includes a scene encoding module, an instruction-aware superpoint aggregation module, an embedded superpoint query module, and a result generation module. The scene encoding module is used to extract point features from the original point cloud using a scene encoder, and generate initial level 1 superpoints using a fast superpoint segmentation algorithm, reducing the point cloud size while preserving geometric and semantic details. The instruction-aware superpoint aggregation module is used to receive the initial level 1 superpoints, text instructions, and learnable tokens, and generate a multi-level superpoint structure layer by layer through an instruction-aware aggregation mechanism and a hierarchical feature interaction mechanism. The top-level L-level superpoints of the multi-level superpoint structure serve as fixed-length visual tokens, while lower-level superpoints retain fine-grained geometric and semantic information, where L≥2. The embedded superpoint query module is used to insert an embedded superpoint query module into the intermediate layer of the Large Language Model (LLM). It extracts the attention weight of the text token to the visual token, selects the K top-level superpoints with the highest scores as regions of interest, retrieves the low-level superpoint features corresponding to the K top-level superpoints, reuses the learnable token to extract instruction perception detail features through cross-attention interaction with text instructions and low-level superpoint features, and injects the detail features into the learnable token through residual update. The result generation module is used to input the learnable token with injected detail features into the remaining decoder layer of the LLM, and generate a 3D scene understanding result after processing.

9. An electronic device, characterized in that, The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores computer program instructions executable by the at least one processor, the computer program instructions being executed by the at least one processor to enable the at least one processor to perform the hierarchical superpoint-based multi-scale 3D scene understanding method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the multi-scale three-dimensional scene understanding method based on hierarchical superpoints as described in any one of claims 1-7.

Images