A three-dimensional semantic scene data construction method, device and medium

Abstract

Embodiments of the present application disclose a three-dimensional semantic scene data construction method, device and medium; the embodiments of the present application divide the macro-to-micro multi-scale voxels, simultaneously capture the global layout, sub-region structure and object details of the scene, and use the semantic landmarks and correlation scores generated by the large language model as semantic references, associate the multi-modal features with understandable semantic concepts, and improve the explainability and accuracy of the semantic labels. Subsequently, only the scene type description needs to be adjusted to adapt to different target scenes, without the need to redesign the landmark generation rules for each target scene, thereby automatically adapting to scene differences and having strong generalization. Finally, the constructed voxel semantic mapping table provides a direct mapping relationship of "coordinate-semantic" under multi-scale, which can be used as an efficient input for downstream tasks such as target detection and path planning, and supports fast three-dimensional scene data query.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional reconstruction, in particular to a three-dimensional semantic scene data construction method, device and medium. BACKGROUND

[0002] At present, the mainstream 3D scene reconstruction general technology is mainly realized based on a traditional 3D modeling method, and can be specifically divided into typical technical paths such as a neural radiation field (NeRF), point cloud reconstruction and mesh representation. These technical solutions usually take multi-view images and sensor data (for example, data collected by an RGB-D camera) as core input data, and complete the geometric reconstruction of a scene through processing and analysis of the above data.

[0003] In order to give the reconstruction result richer semantic information to support subsequent intelligent analysis (such as object recognition, scene understanding, etc.), the prior art further introduces a semantic labeling link. It usually relies on a pre-trained model (such as a convolutional neural network, an instance segmentation model, etc.) to post-process the reconstructed three-dimensional model, so as to attach semantic labels such as object categories, or manually assign semantic labels to objects or regions in the scene.

[0004] However, in the actual application process, its semantic integration method is mostly post-processing, which has obvious limitations: it lacks the ability to automatically perceive and understand complex semantic context from multi-scale observations, resulting in semantic labels that are usually limited to a limited number of predefined categories, and when generalized to new scenes and new objects, it needs to rely on artificial labeling or large-scale specific dataset training again, with high artificial intervention cost. SUMMARY

[0005] The embodiments of the present application provide a three-dimensional semantic scene data construction method, device and medium, which realizes automatic semantic labeling of three-dimensional voxels and provides a structured semantic index for subsequent robot interaction or data screening.

[0006] The embodiments of the present application provide a three-dimensional semantic scene data construction method, which comprises:

[0007] Obtaining a three-dimensional point cloud of a target scene, and performing grid division on the three-dimensional point cloud based on different scales to obtain voxel grid division results at multiple scales, wherein the voxel grid division result at any scale comprises a plurality of voxel blocks;

[0008] At each scale, extracting multi-modal features of each voxel block, and assigning a unique coordinate code to each voxel block;

[0009] Based on the scene type description of the target scene, a large language model is used to generate multiple semantic landmarks contained in the target scene at each scale. Each semantic landmark carries its corresponding relevance score, which is used to characterize the relevance of the semantic landmark at that scale.

[0010] At each scale, for any voxel block, the semantic label of the voxel block is determined based on the multimodal features of the voxel block and the relevance scores of each semantic landmark.

[0011] A voxel semantic mapping table for the target scene at multiple scales is constructed to obtain the three-dimensional semantic scene data of the target scene. The voxel semantic mapping table includes the coordinate encoding and semantic label of the voxel block at any scale.

[0012] Optionally, at each scale, multimodal features of each voxel block are extracted, including:

[0013] At each scale, the color image block corresponding to the voxel block is obtained;

[0014] Visual features are extracted from the color image blocks to obtain the first feature of the voxel block;

[0015] Semantic instance detection is performed on the color image blocks to obtain the second feature of the voxel blocks;

[0016] Spatial attribute features are extracted from the three-dimensional point cloud contained in the voxel block to obtain the third feature of the voxel block;

[0017] Based on the first feature, the second feature, and the third feature, the multimodal features of the voxel block are obtained by fusing them.

[0018] Optionally, at each scale, for any voxel block, based on the multimodal features of the voxel block and the relevance scores of each semantic landmark, the semantic label of the voxel block is determined, including:

[0019] At each scale, for any voxel block, the similarity score between the multimodal features of the voxel block and each semantic landmark is determined;

[0020] Based on the similarity score and the relevance score of each semantic landmark, the matching score between each semantic landmark and the voxel block is determined;

[0021] The semantic labels of the voxel blocks are determined based on the matching scores between each semantic landmark and the voxel block.

[0022] Optionally, based on the matching score between each semantic landmark and the voxel block, the semantic label of the voxel block is determined, including:

[0023] The semantic landmark with the highest matching score is identified as the candidate tag;

[0024] If the matching score of the candidate label is greater than or equal to the score threshold under the scale, then the candidate label is determined as the semantic label of the voxel block;

[0025] If the matching score of the candidate label is less than the score threshold under the scale, then the semantic label of the voxel block is determined to be an undefined label.

[0026] Optionally, the method further includes:

[0027] At each scale, for any voxel block, the corresponding color image block is obtained;

[0028] Based on the observation confidence level corresponding to the color image block, the matching score of the candidate label is corrected to obtain the corrected score;

[0029] If the corrected score of the candidate label is greater than or equal to the score threshold under the scale, then the candidate label is determined as the semantic label of the voxel block;

[0030] If the corrected score of the candidate label is less than the score threshold under the scale, then the semantic label of the voxel block is determined to be an undefined label.

[0031] Optionally, the method further includes:

[0032] The newly added 3D point cloud of the target scene is obtained, and the newly added 3D point cloud is divided into meshes based on different scales to obtain newly added bounding box blocks at each scale.

[0033] At each scale, the newly added bounding box block is compared with each voxel block to determine the voxel block to be updated from multiple voxel blocks.

[0034] Extract the multimodal features of the voxel block to be updated;

[0035] Based on the multimodal features of the voxel block to be updated and the relevance scores of each semantic landmark, the semantic label of the voxel block to be updated is determined.

[0036] The voxel semantic mapping table is updated based on the semantic tags of the voxel blocks to be updated.

[0037] Optionally, at each scale, the newly added bounding box block is compared with each voxel block to determine the voxel block to be updated from multiple voxel blocks, including:

[0038] At each scale, if the spatial overlap rate between the newly added bounding box block and the voxel block is greater than the overlap threshold, then the voxel block is determined as the voxel block to be updated; or,

[0039] At each scale, if the rate of change of the number of three-dimensional point clouds within the voxel block is greater than the change threshold, then the voxel block is determined as the voxel block to be updated.

[0040] Optionally, the method further includes:

[0041] Obtain user requirements and parse the user requirements to obtain data filtering conditions that include target scale, target scene and target semantics;

[0042] Based on the target scale, target scene, and target semantics, data is retrieved from the voxel semantic mapping table to obtain target voxel blocks that meet the data filtering conditions.

[0043] The coordinate encoding, multimodal features, and semantic labels of the target voxel block are converted into a multidimensional tensor format to obtain a multidimensional array, which is used to train or fine-tune the machine learning model.

[0044] This application embodiment also provides a three-dimensional semantic scene data construction device, including:

[0045] The multi-scale voxel block partitioning module is used to acquire the three-dimensional point cloud of the target scene and perform mesh partitioning on the three-dimensional point cloud based on different scales to obtain voxel mesh partitioning results at multiple scales. The voxel mesh partitioning result at any scale includes multiple voxel blocks.

[0046] The voxel block feature extraction module is used to extract the multimodal features of each voxel block at each scale and assign a unique coordinate code to each voxel block.

[0047] The semantic landmark generation module is used to generate multiple semantic landmarks contained in the target scene at each scale based on the scene type description of the target scene through a large language model. Each semantic landmark carries its corresponding relevance score, and the relevance score is used to characterize the relevance of the semantic landmark at that scale.

[0048] The semantic label generation module is used to determine the semantic label of any voxel block at each scale, based on the multimodal features of the voxel block and the relevance scores of each semantic landmark.

[0049] A voxel semantic mapping table construction module is used to construct a voxel semantic mapping table of the target scene at multiple scales to obtain the three-dimensional semantic scene data of the target scene. The voxel semantic mapping table includes the coordinate encoding and semantic label of the voxel block at any scale.

[0050] This application also provides an electronic device, including a memory storing multiple instructions; the processor loads the instructions from the memory to execute the steps in any of the three-dimensional semantic scene data construction methods provided in this application.

[0051] This application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in any of the three-dimensional semantic scene data construction methods provided in this application.

[0052] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the three-dimensional semantic scene data construction methods provided in this application.

[0053] This application embodiment can acquire a 3D point cloud of a target scene and perform meshing on the 3D point cloud based on different scales to obtain voxel meshing results at multiple scales. The voxel meshing result at any scale includes multiple voxel blocks. At each scale, the multimodal features of each voxel block are extracted, and a unique coordinate code is assigned to each voxel block. Based on the scene type description of the target scene, multiple semantic landmarks contained in the target scene at each scale are generated through a large language model. Each semantic landmark carries its corresponding relevance score, which is used to characterize the relevance of the semantic landmark at that scale. At each scale, for any voxel block, the semantic label of the voxel block is determined based on the multimodal features of the voxel block and the relevance scores of each semantic landmark. A voxel semantic mapping table of the target scene at multiple scales is constructed to obtain the 3D semantic scene data of the target scene. The voxel semantic mapping table includes the coordinate code and semantic label of the voxel block at any scale.

[0054] Thus, by segmenting scenes into multi-scale voxels from macro to micro levels, the system simultaneously captures the global layout, sub-region structure, and object details of a scene. It then utilizes semantic landmarks and relevance scores generated by a large language model as semantic references, linking multimodal features with understandable semantic concepts, thereby improving the interpretability and accuracy of semantic labels. Subsequent adjustments to the scene type description are sufficient to adapt to different target scenes, eliminating the need to redesign landmark generation rules for each scene. This allows the system to automatically adapt to scene differences and demonstrates strong generalization capabilities. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart illustrating a method for constructing three-dimensional semantic scene data according to an embodiment of this application;

[0057] Figure 2 This is a schematic diagram of the voxel mesh generation and multimodal feature extraction process in an embodiment of this application;

[0058] Figure 3 This is a flowchart illustrating the process of determining the semantic tags of voxel blocks in an embodiment of this application;

[0059] Figure 4 This is a schematic diagram of the process of local updating of voxel blocks in an embodiment of this application;

[0060] Figure 5 This is a schematic diagram of the structure of a three-dimensional semantic scene data construction device provided in an embodiment of this application;

[0061] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0062] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] Although existing 3D scene reconstruction technologies have certain advantages in geometric modeling, their semantic integration methods are mostly post-processing in practical applications, often requiring manual intervention or relying on specific datasets for training. This results in many shortcomings that still need to be addressed, as follows:

[0064] First, semantic information is scarce and relies on manual annotation. Existing 3D modeling technologies generally lack effective automated semantic injection capabilities. The acquisition of semantic labels typically depends on two methods: one is a predefined label library, where semantic labels are manually assigned to objects or regions in the scene; the other is a recognition model trained on a specific dataset, which is used to perform semantic recognition and annotation of the scene. However, both of these methods have significant limitations. They cannot achieve the automatic generation and understanding of multi-scale semantic context, nor can they be generalized to the reconstruction tasks of new scenes and new objects. When faced with objects not covered in the predefined label library or scenes not present in the training dataset, the semantic annotation accuracy of existing technologies drops significantly, and they may even fail to complete the semantic annotation task.

[0065] Secondly, the dynamic update capability is insufficient. In practical applications, the reconstructed 3D scene often undergoes local changes, such as the addition of new objects, the relocation of existing objects, or changes in their shape. It may also require supplementary observation data to improve the completeness of the scene modeling. However, existing belief graph models based on global consistency typically require a complete scene reconstruction based on all the data when dealing with these local changes or new observation data. This inability to perform local updates targeting only changed areas or new data leads to a sharp increase in computational and time costs when adding new data, severely impacting the practicality and efficiency of the technical solution, and is particularly unsuitable for scenarios with high real-time requirements.

[0066] Third, the reusability of datasets is low. Existing 3D datasets often exhibit a "black box" state or a simple structure, lacking fine-grained internal indexing structures associated with semantic and scale information. In practical research and applications, it is often necessary to extract relevant training data from existing 3D datasets for specific tasks (such as refined tasks like "identifying all sofa armrests"). However, due to the lack of an effective indexing mechanism, researchers find it difficult to quickly and accurately select target data matching the task. This results in a large number of generated 3D datasets not being fully reused, increasing the repetitive costs of data collection and reconstruction, and hindering the improvement of training efficiency for related task models.

[0067] The shortcomings of existing 3D scene reconstruction technologies in terms of semantic automation integration, dynamic update efficiency, and dataset reusability have become key bottlenecks restricting their further promotion and application. Therefore, there is an urgent need for a 3D reconstruction scheme that can achieve deep fusion of semantics and geometry, support efficient local updates, and has structured data indexing, in order to improve the intelligence level and application adaptability of 3D scene modeling.

[0068] In view of this, this application proposes a method for constructing three-dimensional semantic scene data to solve the above-mentioned problems existing in the current 3D scene reconstruction process.

[0069] For details, please refer to Figure 1 The specific process of this three-dimensional semantic scene data construction method can be as follows:

[0070] S110. Obtain the 3D point cloud of the target scene, and perform meshing on the 3D point cloud based on different scales to obtain voxel meshing results at multiple scales. The voxel meshing result at any scale includes multiple voxel blocks.

[0071] In this embodiment, the target scene refers to the specific physical space that needs to be modeled in three dimensions, such as a residential living room, an industrial warehouse, or a hospital ward.

[0072] Mesh generation of 3D point clouds based on different scales refers to dividing the 3D space of the target scene into voxel meshes of different resolutions, such as macro, meso, and micro scales. Each scale corresponds to a different voxel resolution. For example, the voxel resolution at the macro scale is 0.5m×0.5m×0.5m, at the meso scale it is 0.25m×0.25m×0.25m, and at the micro scale it is 0.1m×0.1m×0.1m. This is used to capture multi-level information in the target scene, from global layout to object details.

[0073] A voxel block is the basic unit for multi-scale voxel mesh generation. Each voxel block occupies a fixed spatial range at a specific scale. For example, at a macroscopic scale, a voxel block is a 0.5m × 0.5m × 0.5m cube, containing all the 3D point clouds within that spatial range.

[0074] For details, please refer to Figure 2 The system can acquire RGB image data (i.e., color image sequences, used to provide visual appearance information of the scene) and depth image data (i.e., depth maps registered with the RGB images, used to reflect the spatial distances of each pixel in the scene, thus generating a 3D point cloud) of the target scene using an RGB-D sensor. Then, after temporal alignment and noise filtering of the depth image data, multiple sets of point cloud collections with different levels of detail are generated through back projection and adaptive downsampling. These can be categorized into three scales: macroscopic (low density), mesoscopic (medium density), and microscopic (high density). The macroscopic scale has the lowest point cloud density to capture the global layout (e.g., 10 points per cubic meter), the microscopic scale has the highest point cloud density to preserve object details (e.g., 1000 points per cubic meter), and the mesoscopic scale has a density between the two (e.g., 100 points per cubic meter).

[0075] Then, following the principle of "point cloud density matching voxel resolution" (higher density, lower voxel resolution), the 3D space of the target scene is divided into voxel grids of multiple scales, forming a multi-resolution voxel pyramid. For example, assuming the target scene's spatial size is 10m × 10m × 3m, at the macro scale, it is divided along the length, width, and height directions at intervals of 0.5m, resulting in 20 × 20 × 6 = 2400 macro-scale voxel blocks. At the meso scale, it is divided along the length, width, and height directions at intervals of 0.25m. That is, one macro-scale voxel block contains 2 × 2 × 2 = 8 meso-scale voxel blocks.

[0076] In practical applications, assuming the target scene is a modern residential living room, one macro-scale voxel block may be the entire sofa area, one meso-scale voxel block may be the sofa, coffee table, etc., and one micro-scale voxel block may be the sofa armrest, water glass on the coffee table, etc.

[0077] S120. At each scale, extract the multimodal features of each voxel block and assign a unique coordinate code to each voxel block.

[0078] In this embodiment, the multimodal features of a voxel block refer to composite features extracted from multi-source data of the target scene, used to comprehensively describe the spatial, appearance, and semantic attributes of the voxel block. For example, the multimodal features include: visual features, i.e., the visual semantic information of the RGB image region corresponding to the voxel block, reflecting the appearance features such as color and texture of the spatial region; semantic instance features, i.e., the specific object instances contained within the voxel block and their quantity information (clearly defining "what objects are there" and "how many there are"), used to characterize the discrete semantic units of objects in the scene; and geometric features, i.e., the quantitative description of the spatial structure attributes of the point cloud subset within the voxel block, reflecting the underlying physical characteristics such as the geometric shape and distribution pattern of the region.

[0079] Coordinate encoding refers to a unique spatial identifier assigned to each voxel block, used to uniquely identify the scale level to which each voxel block belongs and its specific location in 3D space. For example, coordinate encoding can be the grid coordinate index of the voxel block in a 3D spatial grid. For instance, the coordinate encoding of a voxel block in the i-th row, j-th column, and k-th layer at a macroscopic scale can be represented as... .

[0080] Further, in step S120 above, at each scale, the multimodal features of each voxel block are extracted, including: at each scale, obtaining the color image block corresponding to the voxel block; performing visual feature extraction on the color image block to obtain the first feature of the voxel block; performing semantic instance detection on the color image block to obtain the second feature of the voxel block; performing spatial attribute feature extraction on the 3D point cloud contained in the voxel block to obtain the third feature of the voxel block; and fusing the first feature, the second feature and the third feature to obtain the multimodal features of the voxel block.

[0081] like Figure 2 As shown, the color image block corresponding to the voxel block refers to the RGB image region corresponding to that voxel block in the RGB image. At each scale, the extraction of multimodal features includes the following three aspects:

[0082] Visual feature extraction: First, the RGB image region corresponding to the voxel block is divided into several image patches. For example, it is divided into 2 patches at a macro scale, 8 patches at a meso scale, and 32 patches at a micro scale. Then, the pre-trained CLIP model (contrastive language-image pre-trained model) is used to extract features from each patch to obtain the visual feature vector of the voxel block, i.e., the first feature.

[0083] Semantic instance feature extraction: The pre-trained SAM model (Segment Anything Model, which can accurately segment any object instance in an image) is used to process the associated RGB image region, and the number and category of independent object instances segmented in the RGB image region corresponding to the current voxel block are counted. For example, "1 table + 2 chairs" is identified. Finally, the statistical result is used as the semantic instance feature of the voxel block, i.e., the second feature.

[0084] Geometric feature extraction includes spatial attribute features such as bounding box volume, point density, and normal vector distribution. Bounding box volume refers to the volume of the smallest circumscribed cuboid of all point clouds within the voxel block, reflecting the overall space occupied by the object. Point density refers to the number of point clouds per unit volume, such as the number of points per cubic meter, reflecting the sampling density of the object's surface. Normal vector distribution refers to the mean, variance, and other statistical measures of the surface normal vectors (direction vectors perpendicular to the surface) of each point cloud, reflecting the orientation changes and smoothness of the object's surface. Based on these spatial attribute features, the geometric features of the voxel block, i.e., the third feature, are obtained.

[0085] Then, by fusing the three types of features, multimodal features of the voxel block are obtained. These features include appearance semantics (visual), object category information (instance), and geometric spatial attributes (point cloud). This allows for a more comprehensive description of the true attributes of the voxel block and avoids semantic misjudgments caused by the limitations of a single modality. For example, relying solely on color can misjudge a black metal block as a "stone," while combining geometric features can identify it as a "table leg."

[0086] In this embodiment, voxel blocks of different scales correspond to the needs of different levels of detail in the scene (capturing the global layout macroscopically and preserving object details microscopically), and the extraction and fusion of multimodal features are adapted to this scale difference. For example, at the microscale: the corresponding color image blocks focus on local details (such as the texture of a sofa armrest), visual features capture subtle color changes (such as the leather reflection of the armrest); semantic instance features count small objects (such as "armrest" and "decoration"), and spatial attribute features accurately describe the surface shape (such as the curvature of the armrest) through high-density point clouds. The fused multimodal features can accurately identify object details (such as "sofa armrest" and "tea cup"), avoiding semantic loss caused by blurred details at the macroscale. This multi-scale adapted multimodal fusion mechanism enables the semantic representation of voxel blocks to reflect both the global scene logic and preserve key local details, thereby supporting multi-level semantic understanding from "overall scene function" to "specific object category".

[0087] S130. Based on the scene type description of the target scene, multiple semantic landmarks contained in the target scene at each scale are generated through a large language model. Each semantic landmark carries its corresponding relevance score, which is used to characterize the relevance of the semantic landmark at that scale.

[0088] In this embodiment, the scene type description is a textual description of the semantic category or functional attributes of the target scene, such as "This is a modern-style residential living room, mainly furnished with a sofa, coffee table, and television." By inputting the scene type description in natural language, the Large Language Model (LLM) is guided to generate semantic landmarks that conform to the scene logic, ensuring that the generated landmarks match the actual function of the scene.

[0089] Semantic landmarks refer to iconic objects or areas within a target scene that possess significant semantic meaning. They serve as semantic reference points connecting the physical space of a scene with human cognition. For example, based on different scales, they can be categorized as follows: Macro-scale landmarks: Describe the overall function or core area of ​​a scene (e.g., "living room area," "nursery area"), covering a large space and used to capture the overall layout of the scene. Meso-scale landmarks: Describe sub-areas or functional modules within a scene (e.g., "living room sofa area," "living room TV area," "walls"), focusing on the intermediate structure of the scene and used to distinguish different functional zones within the scene. Micro-scale landmarks: Describe specific small targets or objects within a sub-area (e.g., "sofa armrest," "television," "tea cup"), focusing on object details and used to accurately locate key interactive objects.

[0090] Each semantic landmark carries a relevance score (the sum of the relevance scores of all semantic landmarks at the same scale is 1), which is used to quantify the relevance of the semantic landmark at the current scale. The higher the relevance score, the greater the contribution of the semantic landmark to the semantics of the scene. For example, at the mesoscale, the relevance score of "living room sofa area" is 0.9, indicating that "living room sofa area" is the core sub-region of the living room.

[0091] Specifically, the commonsense reasoning capabilities of large language models (such as GPT-4o) can be leveraged to design prompt words, generating a structured, multi-scale list of semantic landmarks related to the target scene and their relevance scores. For example, the constructed prompt words could be:

[0092] "Generate three levels of semantic landmarks based on '{scene type}': one macro-scale landmark (describing the overall scene function, such as 'residential living room area'), three meso-scale landmarks (describing sub-regions within the scene, which must include the scene prefix, such as 'living room sofa area'), and three micro-scale landmarks (describing small targets within the sub-regions, such as 'sofa armrest'). Assign a relevance score to each landmark (the sum of relevance scores at the same scale is 1). The output format is [[macro-scale landmark, meso-scale landmark 1, meso-scale landmark 2, meso-scale landmark 3, micro-scale landmark 1, micro-scale landmark 2, micro-scale landmark 3], [macro-scale score, meso-scale score 1, meso-scale score 2, meso-scale score 3, micro-scale score 1, micro-scale score 2, micro-scale score 3]]."

[0093] This embodiment automatically generates multi-scale landmarks using the common-sense reasoning capabilities of LLM, significantly reducing manual costs while ensuring consistency between landmarks and scene logic (e.g., LLM knows that a "living room" typically includes sub-areas such as a "sofa area" and a "TV wall area"). Subsequent adjustments to the scene type description are sufficient to adapt to different scenes (e.g., a "hospital ward" or an "industrial warehouse"), without requiring redesigning landmark generation rules for each scene. This allows for automatic adaptation to scene differences (e.g., the semantic landmarks for a ward include "bed area" and "IV stand"), demonstrating strong generalization capabilities.

[0094] S140. At each scale, for any voxel block, determine the semantic label of the voxel block based on the multimodal features of the voxel block and the relevance scores of each semantic landmark.

[0095] In this embodiment, a semantic label refers to the specific semantic category name assigned to each voxel block (such as "sofa armrest," "carpet," or "chandelier"), used to describe the function or physical attributes of the voxel block in the scene. That is, the semantic label of a voxel block is one of the semantic landmarks; the semantic label of a voxel block is determined by matching semantic landmarks with the voxel block. Furthermore, when a voxel block does not match a suitable semantic landmark, it is assigned the semantic label "undefined."

[0096] For details, please refer to Figure 3 The process of determining the semantic labels of voxel blocks mainly includes:

[0097] S140-1. At each scale, for any voxel block, determine the similarity score between the voxel block's multimodal features and each semantic landmark.

[0098] In this embodiment, after concatenating the visual features, semantic instance features, and geometric features of the voxel block into a comprehensive feature vector (i.e., multimodal features), the semantic description and scale information of the semantic landmarks are encoded into a comprehensive reference vector (e.g., by using cue word engineering to allow LLM to output the semantic vector representation of the landmarks). Then, the cosine similarity or Euclidean distance between the two is calculated (the smaller the distance, the higher the similarity), and the similarity score between the multimodal features of the voxel block and each semantic landmark is obtained.

[0099] For example, if the comprehensive features of a microscale voxel block (including the visual description of "metallic edge + soft adjacent material" and the geometric attribute of "slender strip") have a high similarity to the comprehensive reference vector of the semantic landmark "sofa armrest", then a high similarity score, such as 0.8, will be obtained.

[0100] S140-2. Based on the similarity score and relevance score of each semantic landmark, determine the matching score between each semantic landmark and the voxel block.

[0101] In this embodiment, the matching score between each semantic landmark and the voxel block can be calculated using the following formula:

[0102]

[0103] in, Indicates the first One scale, Represents a voxel block. Indicates semantic landmarks, Indicates the first voxel blocks at various scales and the Matching scores between semantic landmarks Indicates the first The first scale The relevance score of each semantic landmark Indicates the first voxel blocks at various scales and the Similarity scores between semantic landmarks.

[0104] For example, at the micro scale, the relevance score of the semantic landmark "sofa armrest" is 0.9, and the similarity score of a certain voxel block with it is 0.8. Then, at this micro scale, the matching score between the voxel block and the semantic landmark "sofa armrest" is 0.8 × 0.9 = 0.72.

[0105] S140-3. Based on the matching score between each semantic landmark and the voxel block, determine the semantic label of the voxel block.

[0106] Specifically, the semantic landmark with the highest matching score is identified as a candidate label; if the matching score of the candidate label is greater than or equal to the score threshold under the scale, the candidate label is identified as the semantic label of the voxel block; if the matching score of the candidate label is less than the score threshold under the scale, the semantic label of the voxel block is identified as an undefined label.

[0107] The system allows setting score thresholds for different scales based on the needs of various application scenarios, with higher score thresholds set for scales requiring higher accuracy to reduce recognition errors. For example, the score threshold is 0.4 for the macroscopic scale, 0.5 for the mesoscopic scale, and 0.6 for the microscopic scale.

[0108] For a given voxel block, if the matching score of the semantic landmark with the highest matching score is greater than or equal to the score threshold at that scale, then that semantic landmark is assigned to the voxel block as its semantic label. If the matching score of the semantic landmark with the highest matching score is less than the score threshold at that scale, then the voxel block is assigned the semantic label "undefined".

[0109] This embodiment quantifies the feature association between voxel blocks and semantic landmarks through similarity scores, and strengthens the contribution of important semantic landmarks by combining the relevance scores of semantic landmarks. Finally, the most matching semantic landmark is selected as the label, ensuring the accuracy and interpretability of voxel block semantic annotation. At the same time, matching landmarks of different scales with voxel blocks of corresponding scales forms a hierarchical semantic system from global to local. For example, the macroscopic "living room area" includes the mesoscopic "sofa area", and the mesoscopic "sofa area" includes the microscopic "sofa armrest", which is more in line with the scene logic.

[0110] Furthermore, after obtaining the matching scores between each semantic landmark and the voxel block, the above method also includes:

[0111] At each scale, for any voxel block, the corresponding color image block is obtained; based on the observation confidence corresponding to the color image block, the matching score of the candidate label is corrected to obtain the corrected score; if the corrected score of the candidate label is greater than or equal to the score threshold at the scale, the candidate label is determined as the semantic label of the voxel block; if the corrected score of the candidate label is less than the score threshold at the scale, the semantic label of the voxel block is determined as an undefined label.

[0112] Among them, observation confidence is an indicator of the reliability of the accurate observation and characterization of the color image block corresponding to the voxel block (i.e., the corresponding region of the voxel block in the RGB image). The value range is usually [0,1]. The higher the value, the better the observation quality of the image block and the more reliable the information. Conversely, it indicates that there are defects in the observation (such as occlusion, blurring, lighting interference, etc.). Its core reflects whether "the visual information provided by the current image block can truly reflect the actual attributes of the voxel block".

[0113] At each scale, for any voxel block, the observation confidence level can be calculated using the following formula:

[0114]

[0115] in, This indicates the observation confidence level of the voxel block. The horizontal view confidence score represents the visibility of the voxel block relative to the camera's horizontal viewpoint. The closer the voxel block is to the image center, the higher its horizontal view confidence score. The larger. Vertical view confidence indicates the visibility of the voxel block relative to the camera's vertical view. The closer the voxel block is to the image center, the higher its vertical view confidence. The larger. Distance confidence score represents the impact of the voxel's distance relative to the camera on the observation confidence score. When When within the preset "optimal detection range", such as 0.5m-5m, the distance confidence level is... Set to 1.

[0116] In other words, if the voxel block is within the camera's field of view and at a suitable distance, its observation confidence is high, indicating that its observation effect is good.

[0117] After obtaining the observation confidence score of the voxel block, the matching score of the semantic landmark with the highest matching score corresponding to that voxel block is used to correct it, resulting in the corrected matching score. For example, the observation confidence score can be multiplied by the matching score to obtain the corrected score. Then, a threshold judgment is performed based on the corrected matching score to see if the semantic landmark can be assigned to the voxel block as its semantic label, ensuring the accuracy of semantic label matching.

[0118] This embodiment introduces a correction mechanism based on observation confidence to achieve dual verification of "feature semantic relevance × observation reliability", providing a more reliable label foundation for semantic understanding of 3D scenes. It is especially suitable for intelligent applications in dynamic environments, complex lighting or occlusion scenarios (such as robot navigation and virtual reality interaction).

[0119] S150. Construct a voxel semantic mapping table for the target scene at multiple scales to obtain the three-dimensional semantic scene data of the target scene. The voxel semantic mapping table includes the coordinate encoding and semantic label of the voxel block at any scale.

[0120] In this embodiment, the voxel semantic mapping table is a table that records the mapping relationship between the coordinate codes of all voxel blocks in the target scene at each scale and their corresponding semantic labels. For example... Figure 3 As shown, the structure of the voxel semantic mapping table is: macroscale - voxel block (1,2,3) - semantic label 'living room floor'. Simultaneously, the corrected matching score for this semantic label can also be stored together: macroscale - voxel block (1,2,3) - semantic label 'living room floor' - matching score '0.62'.

[0121] Furthermore, in application: user needs can be obtained and analyzed to obtain data filtering conditions including target scale, target scene, and target semantics; based on the target scale, target scene, and target semantics, data can be retrieved in the voxel semantic mapping table to obtain target voxel blocks that meet the data filtering conditions; the coordinate encoding, multimodal features, and semantic labels of the target voxel blocks can be converted into a multidimensional tensor format to obtain a multidimensional array, which is used to train or fine-tune the machine learning model.

[0122] In this embodiment, the user's needs are typically described in natural language or business-oriented terms, such as: "Extract voxel blocks related to 'sofa armrests' at the microscale of a residential living room for training." Therefore, this needs to be parsed into structured conditions with three core dimensions: target scale, specifying the voxel resolution level to be filtered; target scene, limiting the scene range of the data source (e.g., "residential living room," "industrial warehouse"). The set of voxel blocks for a specific scene can be located through the scene name or spatial coordinate range (e.g., the physical space corresponding to the coordinate codes recorded in the voxel semantic mapping table); and target semantics, specifying the objects or semantic tags to be focused on (e.g., "sofa armrests," "coffee table," "chandelier"). This is the most direct semantic filtering condition, ensuring that the filtered voxel blocks have the functions or physical attributes that the user cares about.

[0123] Specifically, user input can be parsed using Natural Language Processing (NLP) technology to generate standardized triplet filtering criteria. For example: target scale = micro, target scene = residential living room, target semantics = sofa armrest.

[0124] Then, a subset of voxel blocks corresponding to the target scale (e.g., microscopic scale) is selected from the voxel semantic mapping table. From this subset, voxel blocks whose coordinate codes fall within the target scene space are selected. For example, by comparing coordinate ranges, it is confirmed that the voxel block belongs to the physical area of ​​a "residential living room". Further selection is made of voxel blocks whose semantic labels are completely consistent with or highly related to the target semantics (e.g., "sofa armrest"). Thus, a set of voxel blocks that meets the conditions of "target scale + target scene + target semantics" is obtained.

[0125] Finally, the coordinate encoding, multimodal features, and semantic labels of the target voxel block are converted into a multidimensional tensor format compatible with machine learning frameworks such as TensorFlow / PyTorch, resulting in a multidimensional array that can be used to train or fine-tune machine learning models.

[0126] This embodiment generates multidimensional tensor data adapted to model tasks by deeply binding the semantic information of the multi-scale voxel semantic mapping table with user needs. This provides structured semantic data support for the customized training of 3D scene understanding models, which can improve data utilization, model targeting and practical application effects.

[0127] In another embodiment of this application, the above method further includes:

[0128] S210. Obtain the newly added 3D point cloud of the target scene, and divide the newly added 3D point cloud into meshes based on different scales to obtain the newly added bounding box blocks at each scale.

[0129] A bounding box is a geometric structure in three-dimensional space used to approximate the extent of a set of points on an object. In this embodiment, the bounding box refers to the smallest axis-aligned cube that contains all newly added 3D point clouds. Please refer to... Figure 4 By discretizing the newly added 3D point cloud of the target scene into structured voxel meshes at different scales, that is, by dividing the bounding box into multiple scales, new bounding box blocks at the corresponding scales can be generated.

[0130] S220. At each scale, the newly added bounding box block is compared with each voxel block to determine the voxel block to be updated from multiple voxel blocks.

[0131] In this embodiment, at each scale, voxel blocks that need to be updated are selected from existing voxel blocks using a dual criterion of spatial overlap rate and point cloud quantity change rate. Specifically, at each scale, if the spatial overlap rate between the newly added bounding box block and the voxel block is greater than an overlap threshold, the voxel block is determined to be a voxel block to be updated; or, at each scale, if the change rate of the 3D point cloud quantity within the voxel block is greater than a change threshold, the voxel block is determined to be a voxel block to be updated.

[0132] For example, if the spatial overlap rate between a voxel block and a newly added bounding box block at the same scale exceeds a threshold (e.g., 3%), it indicates a significant spatial correlation between the voxel block and the newly added data; therefore, the voxel block is identified as a voxel block to be updated. Alternatively, for each existing voxel block, the difference between the current point cloud count and the historical point cloud count is compared, and the rate of change in the point cloud count is calculated. If the rate of change exceeds a threshold (e.g., 5%), it indicates a significant change in the distribution of the point cloud within the voxel block (e.g., the addition of new objects or the removal of existing objects); therefore, the voxel block is identified as a voxel block to be updated.

[0133] Among them, the overlap (spatial range association) and the point number change rate (actual point density change) complement each other to achieve dual-condition redundancy guarantee, so as to accurately trigger the subsequent voxel semantic mapping table update.

[0134] S230. Extract the multimodal features of the voxel block to be updated.

[0135] S240. Based on the multimodal features of the voxel block to be updated and the relevance scores of each semantic landmark, determine the semantic label of the voxel block to be updated.

[0136] S250. Update the voxel semantic mapping table based on the semantic labels of the voxel blocks to be updated.

[0137] like Figure 4As shown, in this embodiment, after determining the voxel block to be updated, the following operations are performed on it again: extract the multimodal features of the newly added point cloud after it falls into the voxel block; based on the updated features, recalculate its similarity score and matching score with the semantic landmark to determine the semantic label of the voxel block to be updated. Finally, update the semantic label of the voxel block in the original voxel semantic mapping table to obtain the updated voxel semantic mapping table. The specific process can be referred to in steps S110-S150 above, and will not be repeated here.

[0138] In addition, if the addition of point cloud results in the generation of new voxel blocks, such as blank areas that were originally not covered by voxels, then voxel blocks are added and the voxel semantic mapping table is updated according to the above steps S110-S150.

[0139] When local changes occur in the scene (such as the addition of new objects, object movement, or viewpoint expansion) or new observation data is acquired, traditional methods typically require reprocessing the entire scene data, resulting in high computational costs and low efficiency. This embodiment, however, employs a "voxel-level dynamic update" mechanism to accurately locate local voxel blocks affected by the new data. Only these voxel blocks undergo local feature recalculation, matching score updates, and index maintenance, ultimately achieving efficient "incremental updates" and ensuring the real-time performance and accuracy of the 3D scene model.

[0140] To better implement the above methods, this application also provides a three-dimensional semantic scene data construction device. This device can be integrated into an electronic device, such as a terminal or server. The terminal can be a tablet computer, a smart Bluetooth device, a laptop computer, or a personal computer; the server can be a single server or a server cluster composed of multiple servers.

[0141] For example, in this embodiment, the method of this application embodiment will be described in detail by taking the three-dimensional semantic scene data construction device specifically integrated into the server as an example.

[0142] For example, such as Figure 5 As shown, the 3D semantic scene data construction device may include a multi-scale voxel block partitioning module 510, a voxel block feature extraction module 520, a semantic landmark generation module 530, a semantic label generation module 540, and a voxel semantic mapping table construction module 550, as follows:

[0143] The multi-scale voxel block partitioning module 510 is used to acquire the three-dimensional point cloud of the target scene and perform mesh partitioning on the three-dimensional point cloud based on different scales to obtain voxel mesh partitioning results at multiple scales. The voxel mesh partitioning result at any scale includes multiple voxel blocks.

[0144] The voxel block feature extraction module 520 is used to extract the multimodal features of each voxel block at each scale and assign a unique coordinate code to each voxel block.

[0145] The semantic landmark generation module 530 is used to describe the scene type based on the target scene. It generates multiple semantic landmarks contained in the target scene at each scale through a large language model. Each semantic landmark carries its corresponding relevance score, which is used to characterize the relevance of the semantic landmark at that scale.

[0146] The semantic label generation module 540 is used to determine the semantic label of a voxel block at each scale for any voxel block based on the multimodal features of the voxel block and the relevance scores of each semantic landmark.

[0147] The voxel semantic mapping table construction module 550 is used to construct voxel semantic mapping tables of the target scene at multiple scales to obtain the three-dimensional semantic scene data of the target scene. The voxel semantic mapping table includes the coordinate encoding and semantic label of the voxel block at any scale.

[0148] Optionally, the voxel block feature extraction module 520 includes:

[0149] The image patch acquisition submodule is used to acquire the color image patch corresponding to the voxel patch at each scale;

[0150] The visual feature extraction submodule is used to extract visual features from color image blocks to obtain the first feature of the voxel block.

[0151] The semantic feature extraction submodule is used to perform semantic instance detection on color image blocks to obtain the second feature of the voxel block;

[0152] The geometric feature extraction submodule is used to extract spatial attribute features from the 3D point cloud contained in the voxel block to obtain the third feature of the voxel block.

[0153] The feature fusion submodule is used to fuse the first feature, the second feature, and the third feature to obtain the multimodal features of the voxel block.

[0154] Optionally, the semantic tag generation module 540 includes:

[0155] The similarity calculation submodule is used to determine the similarity score between the multimodal features of a voxel block and each semantic landmark at each scale for any voxel block.

[0156] The matching score calculation submodule is used to determine the matching score between each semantic landmark and the voxel block based on the similarity score and relevance score of each semantic landmark.

[0157] The semantic tag matching submodule is used to determine the semantic tags of voxel blocks based on the matching scores between each semantic landmark and the voxel block.

[0158] Optionally, the semantic tag matching submodule includes:

[0159] The candidate label determination unit is used to determine the semantic landmark with the highest matching score as the candidate label;

[0160] The first judgment unit is used to determine the candidate label as the semantic label of the voxel block if the matching score of the candidate label is greater than or equal to the score threshold under the scale.

[0161] The second judgment unit is used to determine that the semantic label of the voxel block is an undefined label if the matching score of the candidate label is less than the score threshold under the scale.

[0162] Optionally, the device further includes:

[0163] The image patch acquisition module is used to acquire the color image patch corresponding to any voxel block at each scale.

[0164] The matching score correction module is used to correct the matching score of candidate labels based on the observation confidence corresponding to the color image blocks, and obtain the corrected score.

[0165] The first judgment unit is also used to determine the candidate label as the semantic label of the voxel block if the corrected score of the candidate label is greater than or equal to the score threshold under the scale.

[0166] The first judgment unit is also used to determine the semantic label of the voxel block as an undefined label if the corrected score of the candidate label is less than the score threshold under the scale.

[0167] Optionally, the device further includes:

[0168] A new point cloud acquisition module has been added to acquire new 3D point clouds of the target scene and to divide the new 3D point clouds into meshes based on different scales to obtain new bounding box blocks at each scale.

[0169] The module for determining voxel blocks to be updated is used to compare the correlation between the newly added bounding box block and each voxel block at each scale, and to determine the voxel block to be updated from multiple voxel blocks.

[0170] The voxel block feature extraction module 520 is also used to extract multimodal features of the voxel block to be updated;

[0171] The semantic label generation module 540 is also used to determine the semantic label of the voxel block to be updated based on the multimodal features of the voxel block to be updated and the relevance scores of each semantic landmark.

[0172] The voxel semantic mapping table update module is used to update the voxel semantic mapping table based on the semantic tags of the voxel blocks to be updated.

[0173] Optionally, the voxel block determination module to be updated includes:

[0174] The overlap rate determination submodule is used to determine, at each scale, if the spatial overlap rate between the newly added bounding box block and the voxel block is greater than the overlap threshold, that voxel block is designated as a voxel block to be updated; or...

[0175] The rate of change judgment submodule is used to determine the voxel block as a voxel block to be updated if the rate of change of the number of 3D point clouds in the voxel block is greater than the change threshold at each scale.

[0176] Optionally, the device further includes:

[0177] The user requirements module is used to acquire user requirements and parse them to obtain data filtering conditions that include target scale, target scenario and target semantics.

[0178] The data filtering module is used to retrieve data from the voxel semantic mapping table based on the target scale, target scene and target semantics, and obtain the target voxel blocks that meet the data filtering conditions.

[0179] The data extraction module is used to convert the coordinate encoding, multimodal features, and semantic labels of the target voxel block into a multidimensional tensor format to obtain a multidimensional array. The multidimensional array is used to train or fine-tune the machine learning model.

[0180] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0181] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0182] This application also provides an electronic device, which can be a terminal, a server, or other similar device. The terminal can be a tablet computer, a smart Bluetooth device, a laptop computer, a personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0183] In some embodiments, the three-dimensional semantic scene data construction device can also be integrated into multiple electronic devices. For example, the three-dimensional semantic scene data construction device can be integrated into multiple servers, and the three-dimensional semantic scene data construction method of this application can be implemented by multiple servers.

[0184] In this embodiment, as Figure 6 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0185] The electronic device may include components such as a processor 610 with one or more processing cores, a memory 620 with one or more computer-readable storage media, a power supply 630, an input module 640, and a communication module 650. Those skilled in the art will understand that... Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0186] The processor 610 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 620, and by calling data stored in the memory 620, thereby performing overall detection of the electronic device. In some embodiments, the processor 610 may include one or more processing cores; in some embodiments, the processor 610 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 610.

[0187] The memory 620 can be used to store software programs and modules. The processor 610 executes various functional applications and data processing by running the software programs and modules stored in the memory 620. The memory 620 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 620 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 620 may also include a memory controller to provide the processor 610 with access to the memory 620.

[0188] The electronic device also includes a power supply 630 that supplies power to the various components. In some embodiments, the power supply 630 can be logically connected to the processor 610 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. The power supply 630 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0189] The electronic device may also include an input module 640, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0190] The electronic device may also include a communication module 650. In some embodiments, the communication module 650 may include a wireless module, through which the electronic device can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 650 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0191] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 610 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 620 according to the following instructions, and the processor 610 runs the applications stored in the memory 620, thereby realizing the various functions in the above-mentioned three-dimensional semantic scene data construction method.

[0192] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0193] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute the steps in any of the three-dimensional semantic scene data construction methods provided in embodiments of this application.

[0194] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0195] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the three-dimensional semantic scene data construction method provided in the above embodiments.

[0196] Since the instructions stored in the storage medium can execute the steps in any of the three-dimensional semantic scene data construction methods provided in the embodiments of this application, the beneficial effects that any of the three-dimensional semantic scene data construction methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0197] The foregoing has provided a detailed description of a three-dimensional semantic scene data construction method, device, and medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for constructing three-dimensional semantic scene data, characterized in that, The method includes: A 3D point cloud of the target scene is acquired, and the 3D point cloud is meshed based on different scales to obtain voxel meshing results at multiple scales. The voxel meshing result at any scale includes multiple voxel blocks. At each scale, multimodal features of each voxel block are extracted, and a unique coordinate code is assigned to each voxel block; Based on the scene type description of the target scene, a large language model is used to generate multiple semantic landmarks contained in the target scene at each scale. The semantic landmarks refer to the iconic objects or regions with semantic significance in the target scene. They are semantic reference points that connect the physical space of the scene with human cognition. Each semantic landmark carries its corresponding relevance score. The relevance score is used to characterize the semantic contribution of the semantic landmark to the scene semantics of the target scene at that scale. At each scale, for any voxel block, the semantic label of the voxel block is determined based on the multimodal features of the voxel block and the relevance scores of each semantic landmark. A voxel semantic mapping table for the target scene at multiple scales is constructed to obtain the three-dimensional semantic scene data of the target scene. The voxel semantic mapping table includes the coordinate encoding and semantic label of the voxel block at any scale.

2. The method for constructing three-dimensional semantic scene data as described in claim 1, characterized in that, At each scale, multimodal features of each voxel block are extracted, including: At each scale, the color image block corresponding to the voxel block is obtained; Visual features are extracted from the color image blocks to obtain the first feature of the voxel block; Semantic instance detection is performed on the color image blocks to obtain the second feature of the voxel blocks; Spatial attribute features are extracted from the three-dimensional point cloud contained in the voxel block to obtain the third feature of the voxel block; Based on the first feature, the second feature, and the third feature, the multimodal features of the voxel block are obtained by fusing them.

3. The method for constructing three-dimensional semantic scene data as described in claim 1, characterized in that, At each scale, for any voxel block, based on the multimodal features of the voxel block and the relevance scores of each semantic landmark, the semantic label of the voxel block is determined, including: At each scale, for any voxel block, the similarity score between the multimodal features of the voxel block and each semantic landmark is determined; Based on the similarity score and the relevance score of each semantic landmark, the matching score between each semantic landmark and the voxel block is determined; The semantic labels of the voxel blocks are determined based on the matching scores between each semantic landmark and the voxel block.

4. The method for constructing three-dimensional semantic scene data as described in claim 3, characterized in that, Based on the matching score between each semantic landmark and the voxel block, the semantic label of the voxel block is determined, including: The semantic landmark with the highest matching score is identified as the candidate tag; If the matching score of the candidate label is greater than or equal to the score threshold under the scale, then the candidate label is determined as the semantic label of the voxel block; If the matching score of the candidate label is less than the score threshold under the scale, then the semantic label of the voxel block is determined to be an undefined label.

5. The method for constructing three-dimensional semantic scene data as described in claim 4, characterized in that, The method further includes: At each scale, for any voxel block, the corresponding color image block is obtained; Based on the observation confidence level corresponding to the color image block, the matching score of the candidate label is corrected to obtain the corrected score; If the corrected score of the candidate label is greater than or equal to the score threshold under the scale, then the candidate label is determined as the semantic label of the voxel block; If the corrected score of the candidate label is less than the score threshold under the scale, then the semantic label of the voxel block is determined to be an undefined label.

6. The method for constructing three-dimensional semantic scene data as described in claim 1, characterized in that, The method further includes: The newly added 3D point cloud of the target scene is obtained, and the newly added 3D point cloud is divided into meshes based on different scales to obtain newly added bounding box blocks at each scale. At each scale, the newly added bounding box block is compared with each voxel block to determine the voxel block to be updated from multiple voxel blocks. Extract the multimodal features of the voxel block to be updated; Based on the multimodal features of the voxel block to be updated and the relevance scores of each semantic landmark, the semantic label of the voxel block to be updated is determined. The voxel semantic mapping table is updated based on the semantic tags of the voxel blocks to be updated.

7. The method for constructing three-dimensional semantic scene data as described in claim 6, characterized in that, At each scale, the newly added bounding box block is compared with each voxel block to determine the voxel block to be updated from multiple voxel blocks, including: At each scale, if the spatial overlap rate between the newly added bounding box block and the voxel block is greater than the overlap threshold, then the voxel block is determined as the voxel block to be updated; or, At each scale, if the rate of change of the number of three-dimensional point clouds within the voxel block is greater than the change threshold, then the voxel block is determined as the voxel block to be updated.

8. A method for constructing three-dimensional semantic scene data as described in any one of claims 1-7, characterized in that, The method further includes: Obtain user requirements and parse the user requirements to obtain data filtering conditions that include target scale, target scene and target semantics; Based on the target scale, target scene, and target semantics, data is retrieved from the voxel semantic mapping table to obtain target voxel blocks that meet the data filtering conditions. The coordinate encoding, multimodal features, and semantic labels of the target voxel block are converted into a multidimensional tensor format to obtain a multidimensional array, which is used to train or fine-tune the machine learning model.

9. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to execute the steps in a three-dimensional semantic scene data construction method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps in a three-dimensional semantic scene data construction method as described in any one of claims 1 to 8.

Images