Construction method and system of three-dimensional multi-modal scene knowledge base, terminal and storage medium

Abstract

The invention belongs to the field of data processing, and discloses a three-dimensional multi-modal scene knowledge base construction method and system, a terminal and a storage medium, and the method comprises the steps: obtaining a continuous RGB-D image and a pose sequence in a target scene, and carrying out the standardized preprocessing, and obtaining the preprocessing data; performing target detection processing and instance segmentation processing on the preprocessed data to obtain a pixel-level mask; obtaining image depth information, and performing object point cloud generation processing and cross-frame point cloud fusion processing according to the image depth information and the pixel-level mask to obtain a three-dimensional point cloud fusion feature; obtaining object semantic attributes, and performing spatial relationship calculation according to the three-dimensional point cloud fusion features to obtain a target spatial relationship; and performing knowledge graph construction processing according to the object semantic attributes and the target space relationship to obtain a three-dimensional scene knowledge base. By constructing the three-dimensional scene knowledge base, the cognition and decision-making ability of the robot in the dynamic unstructured environment is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a three-dimensional multi-modal scene knowledge base construction method, system, terminal and computer readable storage medium. BACKGROUND

[0002] With the increasing complexity of service robots in indoor scenes such as homes, offices and warehouses, the demand for "deep cognition" of the environment has exceeded the basic real-time perception. Robots need to understand complex instructions such as "take out the black cover notebook from the left drawer of the study table", which integrates spatial orientation, object attributes and long-term memory. This requires the robot to build a scene knowledge base that can accurately express three-dimensional geometric structures, carry rich semantic information, and support persistent storage and logical reasoning.

[0003] However, in the prior art, when constructing such a knowledge base, the perception range is narrow in unknown or dynamic environments, there is a lack of deep semantic description of objects themselves, and it is easy to be affected by noise in the scene, resulting in difficulties in constructing the scene knowledge base, and failing to meet the normal execution of service robot tasks.

[0004] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0005] The main purpose of the present application is to provide a three-dimensional multi-modal scene knowledge base construction method, system, terminal and computer readable storage medium, which aims to solve the problem that in the prior art, when constructing a scene knowledge base, the perception range is narrow in unknown or dynamic environments, there is a lack of deep semantic description of objects themselves, and it is easy to be affected by noise in the scene, resulting in difficulties in constructing the scene knowledge base, and failing to meet the normal execution of service robot tasks.

[0006] To achieve the above purpose, the present application provides a three-dimensional multi-modal scene knowledge base construction method, which comprises the following steps: Obtain continuous RGB-D images and pose sequences in a target scene, and perform standardization preprocessing on the continuous RGB-D images and the pose sequences to obtain preprocessed data; Perform target detection processing and instance segmentation processing on the preprocessed data to obtain a pixel-level mask; Obtain image depth information, and perform object point cloud generation processing and cross-frame point cloud fusion processing according to the image depth information and the pixel-level mask to obtain three-dimensional point cloud fusion features; Obtain object semantic attributes, and perform spatial relationship calculation according to the three-dimensional point cloud fusion features to obtain target spatial relationships; According to the object semantic attribute and the target spatial relationship, a knowledge graph construction process is performed to obtain a three-dimensional scene knowledge base.

[0007] Optionally, the method for constructing the three-dimensional multi-modal scene knowledge base, wherein the continuous RGB-D images and the pose sequence in the target scene are obtained, and the continuous RGB-D images and the pose sequence are standardized and preprocessed to obtain preprocessed data, specifically including: The continuous RGB-D images and the pose sequence collected by the mobile robot in the target scene through the RGB-D camera are obtained, wherein the continuous RGB-D images include an RGB image and a depth map of each frame, and the pose sequence includes a pose matrix, a timestamp, and camera parameters. The continuous RGB-D images and the pose sequence are standardized and preprocessed to obtain preprocessed data, wherein the standardized preprocessing includes depth unit conversion processing, image size unification processing, and image deblurring processing.

[0008] Optionally, the method for constructing the three-dimensional multi-modal scene knowledge base, wherein the preprocessed data is subjected to target detection processing and instance segmentation processing to obtain a pixel-level mask, specifically including: The preprocessed data is subjected to target detection processing by using a YOLO-World model to obtain a target bounding box. The continuous RGB-D images are subjected to encoding processing by using a preset segmentation model to obtain a high-dimensional image embedding. The target bounding box is input into a prompt encoder of the preset segmentation model, and the prompt encoder is used to perform coordinate frame conversion processing on the target bounding box to obtain a prompt vector. The prompt vector is mapped to the high-dimensional image embedding according to a cross-attention mechanism by using a decoder of the preset segmentation model to obtain a pixel-level mask.

[0009] Optionally, the method for constructing the three-dimensional multi-modal scene knowledge base, wherein the preprocessed data is subjected to target detection processing by using a YOLO-World model to obtain a target bounding box, specifically including: A preset indoor scene object category text is obtained, and a pre-trained CLIP text encoder is used to convert the preset indoor scene object category text into a text vector. The preprocessed data is subjected to feature extraction by using a backbone network in the YOLO-World model to obtain multi-scale visual features. The text vector and the multi-scale visual features are matched by using a RepVL-PAN network to obtain a target bounding box.

[0010] Optionally, the method for constructing the three-dimensional multi-modal scene knowledge base, wherein the image depth information is obtained, and object point cloud generation processing and cross-frame point cloud fusion processing are performed according to the image depth information and the pixel-level mask to obtain three-dimensional point cloud fusion features, specifically comprising: obtaining foreground pixels in the pixel-level mask, and obtaining depth values of the foreground pixels; obtaining camera intrinsic parameters, and performing back-projection of the depth values and the camera intrinsic parameters to a camera coordinate system to obtain a three-dimensional point set; converting the three-dimensional point set to a unified world coordinate system according to the pose matrix to obtain an object-level three-dimensional point cloud; extracting feature vectors corresponding to each target object in the target bounding box through a visual-language pre-training model, and calculating cosine similarity between the feature vectors to obtain appearance similarity; converting the object-level three-dimensional point cloud into a bounding box, and calculating a three-dimensional intersection over union of the bounding box in the unified world coordinate system to obtain a spatial overlap degree; if the appearance similarity and the spatial overlap degree both meet preset requirements, adopting a momentum update strategy to fuse the target objects to obtain three-dimensional point cloud fusion features.

[0011] Optionally, the method for constructing the three-dimensional multi-modal scene knowledge base, wherein the object semantic attributes are obtained, and spatial relationship calculation is performed according to the three-dimensional point cloud fusion features to obtain target spatial relationships, specifically comprising: obtaining a multi-view picture pool of the target objects, and performing screening processing on the multi-view picture pool by adopting a Laplacian gradient variance algorithm to obtain multi-view images; inputting the multi-view images and preset structured prompt words into a multi-modal large model to output object semantic attributes; performing geometric comparison processing and double threshold filtering processing on any two target objects according to the three-dimensional point cloud fusion features to obtain target spatial relationships.

[0012] Optionally, the method for constructing the three-dimensional multi-modal scene knowledge base, wherein the knowledge graph construction processing is performed according to the object semantic attributes and the target spatial relationships to obtain a three-dimensional scene knowledge base, specifically comprising: creating a spatial topological relationship edge according to the target spatial relationships, and creating an attribute association relationship edge according to the object semantic attributes; adopting a graph database as a storage carrier, taking objects in the target scene as entity nodes, and performing knowledge graph construction processing according to the entity nodes, the spatial topological relationship edge, and the attribute association relationship edge to obtain a three-dimensional scene knowledge base.

[0013] In addition, to achieve the above object, the present application also provides a three-dimensional multi-modal scene knowledge base construction system, wherein the three-dimensional multi-modal scene knowledge base construction system comprises: A standardization preprocessing module is configured to acquire continuous RGB-D images and pose sequences in a target scene, and perform standardization preprocessing on the continuous RGB-D images and the pose sequences to obtain preprocessed data. A pixel-level mask generation module is configured to perform target detection processing and instance segmentation processing on the preprocessed data to obtain a pixel-level mask. A three-dimensional point cloud feature generation module is configured to acquire image depth information, and perform object point cloud generation processing and cross-frame point cloud fusion processing according to the image depth information and the pixel-level mask to obtain three-dimensional point cloud fusion features. A spatial relationship calculation module is configured to acquire object semantic attributes, and perform spatial relationship calculation according to the three-dimensional point cloud fusion features to obtain target spatial relationships. A knowledge graph construction module is configured to perform knowledge graph construction processing according to the object semantic attributes and the target spatial relationships to obtain a three-dimensional scene knowledge base.

[0014] In addition, to achieve the above object, the present application also provides a terminal, wherein the terminal comprises a memory, a processor, and a three-dimensional multi-modal scene knowledge base construction program stored in the memory and executable on the processor, and the three-dimensional multi-modal scene knowledge base construction program implements the steps of the three-dimensional multi-modal scene knowledge base construction method when executed by the processor.

[0015] In addition, to achieve the above object, the present application also provides a computer readable storage medium, wherein the computer readable storage medium stores a three-dimensional multi-modal scene knowledge base construction program, and the three-dimensional multi-modal scene knowledge base construction program implements the steps of the three-dimensional multi-modal scene knowledge base construction method when executed by a processor.

[0016] In this invention, continuous RGB-D images and pose sequences from a target scene are acquired, and the continuous RGB-D images and pose sequences are standardized and preprocessed to obtain preprocessed data. The preprocessed data is then subjected to target detection and instance segmentation to obtain pixel-level masks. Image depth information is acquired, and object point cloud generation and cross-frame point cloud fusion are performed based on the image depth information and the pixel-level masks to obtain 3D point cloud fusion features. Object semantic attributes are acquired, and spatial relationship calculations are performed based on the 3D point cloud fusion features to obtain target spatial relationships. Finally, knowledge graph construction is performed based on the object semantic attributes and target spatial relationships to obtain a 3D scene knowledge base. This invention uses a robot to collect continuous RGB-D images and pose sequences from a target scene, and then performs target detection, instance segmentation, object point cloud generation, cross-frame point cloud fusion, and image depth information and target spatial relationship calculations to construct a 3D scene knowledge base. This not only effectively improves the construction accuracy of the 3D scene knowledge base but also significantly enhances the robot's cognitive and decision-making capabilities in dynamic unstructured environments. Attached Figure Description

[0017] Figure 1 This is a flowchart of a preferred embodiment of the method for constructing a three-dimensional multimodal scene knowledge base according to the present invention; Figure 2 This is a schematic diagram of the overall process of a preferred embodiment of the method for constructing a three-dimensional multimodal scene knowledge base of the present invention; Figure 3 This is a schematic diagram of the three-dimensional scene knowledge base construction process of a preferred embodiment of the construction method of the three-dimensional multimodal scene knowledge base of the present invention; Figure 4 This is a schematic diagram of the semantic analysis results of a preferred embodiment of the method for constructing a three-dimensional multimodal scene knowledge base of the present invention; Figure 5 This is a structural diagram of a preferred embodiment of the system for constructing a three-dimensional multimodal scene knowledge base according to the present invention; Figure 6 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] As service robots perform increasingly complex tasks in indoor settings such as homes, offices, and warehouses, their need for "deep environmental cognition" has surpassed basic, immediate perception. Robots need to understand complex commands, such as "take the black-covered notebook from the left drawer of the desk in the study," which integrates spatial orientation, object attributes, and long-term memory. This requires robots to build a scene knowledge base that can accurately represent three-dimensional geometric structures, carry rich semantic information, and support persistent storage and logical reasoning.

[0020] However, existing technologies face three major bottlenecks when constructing such knowledge bases: insufficient open-domain awareness, inadequate semantic information depth, and poor robustness of temporal fusion. These limitations are specifically manifested in the following three technical paths: 1. 3D detection and reconstruction path based on closed set model: Core principle: Relying on a model pre-trained on a fixed-class dataset (such as ScanNet) to perform 3D object detection and scene reconstruction.

[0021] Inherent limitations: Its "closed set" recognition paradigm cannot handle the large number of long-tail objects (such as specific tools, personalized decorations, etc.) that exist in the real open world and are not defined in the training set. This results in a narrow perception range for the system in actual deployment, severely limiting its practicality and generalization ability in unknown or dynamic environments.

[0022] 2. Scene graph construction path based on geometric topology: Core principle: Construct a graph structure representation of the scene by calculating the spatial relationships (such as adjacency and containment) between objects.

[0023] Inherent limitations: These methods typically focus on organizing geometric relationships, while severely lacking in-depth semantic descriptions of the objects themselves. Scene graph nodes are mostly just category labels or abstract feature vectors, failing to effectively utilize prior knowledge to mine the fine-grained attributes (such as color, material, and state) and functional semantics of objects. The constructed knowledge base is "semantically sparse," making it difficult to support complex high-level reasoning tasks that require understanding object attributes and states.

[0024] 3. Temporal feature fusion path based on simple aggregation: Core principle: Direct averaging or weighted averaging of features of the same object in multiple frames of observation to achieve feature fusion.

[0025] Inherent limitations: This type of simple aggregation strategy is highly sensitive to noise in single-frame observations (such as motion blur, occlusion, and sudden changes in illumination), which can easily lead to drift or degradation of object feature representations. In long-term, multi-view observation sequences, it is difficult to maintain the consistency and discriminative power of features, thus affecting the stability of cross-frame object recognition and the accuracy of knowledge base updates.

[0026] In summary, existing technologies lack a complete solution that can simultaneously achieve open-domain object perception, deep semantic information extraction, and robust temporal fusion, ultimately forming a structured, reasonable 3D scene knowledge base.

[0027] This invention addresses the problems of perception closure, semantic sparsity, and information fragmentation in existing indoor robot environmental understanding schemes, and proposes an automatic construction method for a three-dimensional multimodal scene knowledge base oriented towards real-world open domains.

[0028] The core idea of ​​this invention is to construct a complete pipeline from open-domain perception to structured cognition by using continuous RGB-D images and pose sequences collected by a mobile robot as input: First, an open vocabulary model is used to identify and segment arbitrary objects, and their pixel masks are fused with depth information to directly construct object-level 3D point cloud entities; Second, a dual-criteria strategy that integrates appearance semantics and geometric spatial information is designed to achieve robust fusion and continuous optimization of cross-frame observations of the same object; Then, the geometric and semantic attributes of the object are inferred based on multi-view information, and its precise spatial relationship with other objects in the scene is calculated; Finally, all information is constructed into a 3D scene knowledge graph that supports efficient querying and reasoning in the form of "entity-relationship-attribute" triples.

[0029] Through the above-described scheme, this invention achieves end-to-end automatic transformation from raw perceptual data to interpretable and operable scene knowledge. Compared with existing technologies, this invention overcomes the limitations of closed-set perception, enabling the identification and understanding of long-tailed objects in open environments; it generates compact 3D representations rich in semantics and geometry, greatly improving the efficiency of scene information utilization; and it establishes a structured knowledge base that supports incremental updates, providing a reliable knowledge foundation for the robot's long-term interaction and complex task planning, significantly enhancing its cognitive and decision-making capabilities in dynamic unstructured environments.

[0030] The preferred embodiment of the present invention describes a method for constructing a three-dimensional multimodal scene knowledge base, such as... Figure 1 As shown, the method for constructing the three-dimensional multimodal scene knowledge base includes the following steps: Step S10: Obtain continuous RGB-D images and pose sequences in the target scene, and perform standardized preprocessing on the continuous RGB-D images and pose sequences to obtain preprocessed data.

[0031] The core of this invention lies in constructing a high-precision, multimodal, and compact 3D scene representation for indoor service robots, and automatically generating a structured scene knowledge base based on this representation. For example... Figure 2As shown, this invention starts from the RGB-D images and pose sequences collected by the robot, and sequentially completes object-level target detection, instance segmentation, point cloud construction, cross-frame fusion, geometric and semantic attribute extraction, and spatial relationship calculation. Finally, the results are stored in the form of a graph database as queryable and reasonable 3D scene knowledge.

[0032] The overall architecture and main process of this invention are as follows: This invention adopts a modular architecture, taking continuous RGB-D images and pose sequences acquired by the robot as input, and sequentially processing them through six core stages to finally output a structured 3D scene knowledge base. Each stage is based on a unified world coordinate system and unique object identifiers (such as IDs) for data association and transmission, forming a complete closed loop from initial perception to semantic cognition.

[0033] like Figure 3 and Figure 4 As shown, the process is as follows: First, the input data is collected and standardized preprocessed. Then, the open vocabulary model YOLO-World is used for object detection, and pixel-level masks are generated by the segmentation model SAM2 based on its output prompts. Next, the masks are combined with depth information, and object-level point clouds are directly constructed in the world coordinate system through 3D projection. A dual criterion strategy of fusing appearance (CLIP) and geometry (3D IoU) information is adopted to achieve robust matching and model fusion of objects across frames. Subsequently, semantic attributes such as color and material are extracted from multi-view images for each fused 3D object, and its spatial orientation and distance relationship with other objects are calculated. Finally, all entities, attributes, and relationships are integrated into a graph database to construct a queryable and reasonable 3D scene knowledge graph. This architecture supports online incremental updates and offline batch construction, and can flexibly adapt to the deployment needs of indoor service robots in different environments.

[0034] Specifically, continuous RGB-D images and pose sequences are acquired by the mobile robot using an RGB-D camera in the target scene. The continuous RGB-D images include an RGB image and a depth map for each frame, and the pose sequence includes a pose matrix, a timestamp, and camera parameters. The continuous RGB-D images and the pose sequence are then subjected to standardization preprocessing to obtain preprocessed data. The standardization preprocessing includes depth unit conversion, image size unification, and image deblurring.

[0035] This invention includes an RGB-D and pose data acquisition and preprocessing module, responsible for acquiring and preprocessing RGB-D data. Using an RGB-D camera mounted on a mobile robot, RGB images and depth maps are simultaneously acquired for each frame during movement, and the corresponding pose (i.e., the pose matrix in this invention), timestamp, and camera parameters are recorded.

[0036] After data collection is complete, the system executes a standardized preprocessing procedure, including: 1. Depth unit conversion: Converts the pixel values ​​of the depth image into physical depth units in millimeters (mm); 2. Unify image size: Adjust the resolution of the color image and depth image to 1024×1024 to ensure pixel-level alignment; 3. Image Deblurring: The Laplacian operator is applied to calculate the gradient variance of the image, and blurry frames with variances below the threshold are automatically removed to ensure input quality.

[0037] Step S20: Perform target detection and instance segmentation processing on the preprocessed data to obtain a pixel-level mask.

[0038] Specifically, the text of preset indoor scene object categories is obtained, and the preset indoor scene object category text is converted into text vectors using a pre-trained CLIP text encoder; the preprocessed data is used to extract features through the backbone network of the YOLO-World model to obtain multi-scale visual features; the text vectors are matched with the multi-scale visual features through the RepVL-PAN network to obtain the target bounding box.

[0039] This invention includes an object detection and instance segmentation module: employing an open-domain object detection model (YOLO-World model) and a segmentation model (the pre-defined segmentation model Segment Anything Model 2, or SAM2). First, the YOLO-World model is used to identify objects in the input RGB image (the pre-processed data in this invention). The core of YOLO-World lies in open-vocabulary detection, which is no longer limited to predefined categories (such as the 80 categories in the COCO dataset) but achieves recognition through visual-language fusion. The system first receives predefined indoor scene object category text (i.e., pre-defined indoor scene object category text, consisting of 180 manually selected indoor scene object labels). Using a pre-trained CLIP text encoder, this text is converted into computer-understandable vectors (i.e., text vectors), mapping human language to a semantic feature space. Simultaneously, the YOLO backbone network processes the input RGB image (the pre-processed data in this invention) to extract multi-scale visual features. Then, the model uses RepVL-PAN (a reparameterized visual-language path aggregation network) to deeply interact with the image features and text vectors. The model calculates the similarity between features of different regions in the image and text vectors. For example, if the visual features of a certain region in the image highly match the text vector of "chair", the model will generate a bounding box in that region and label the object as a chair (this part is the recognition process of the YOLO-World model, which is the internal execution process of the model and is encapsulated into an interface).

[0040] The continuous RGB-D image is encoded using a preset segmentation model to obtain a high-dimensional image embedding. The target bounding box is input into the cue encoder of the preset segmentation model, and the cue encoder performs coordinate transformation on the target bounding box to obtain a cue vector. The cue vector is mapped to the high-dimensional image embedding by the decoder of the preset segmentation model according to a cross-attention mechanism to obtain a pixel-level mask.

[0041] After the YOLO-World model provides a coarse bounding box (i.e., the target bounding box), SAM 2's task is to precisely extract the image within the box. SAM 2 first encodes the image, processing the entire RGB image into a high-dimensional image embedding in one go. Then, the bounding box output by YOLO-World (i.e., the target bounding box) is used as a spatial cue and input into SAM 2's cue encoder. SAM 2's cue encoder converts the coordinates of the top-left and bottom-right corners of the box into position vectors, telling the model to focus on this rectangular region. Next, SAM 2's decoder receives the image embedding and the cue vector, using a cross-attention mechanism to map the cue vector onto the image embedding. It focuses on analyzing pixel features within the bounding box, automatically identifying the differences between the foreground (object) and the background (environment). Even with highly irregular object edges (such as hair or cutouts), the model can perform pixel-level classification based on feature continuity. Finally, the model outputs a matrix corresponding to the size of the original image, where pixels containing objects are marked as 1 (foreground) and the rest as 0 (background), thus forming a precise pixel-level mask. (This invention ultimately identifies which pixel values ​​on an image belong to which indoor scene object, facilitating the projection of the object's pixel values ​​into 3D space.) Verification has shown that this "open domain detection + cue-based segmentation" strategy significantly improves the system's recognition coverage and segmentation accuracy for diverse objects in complex real-world scenes compared to closed domain models trained on fixed categories.

[0042] Step S30: Obtain image depth information, and perform object point cloud generation processing and cross-frame point cloud fusion processing based on the image depth information and the pixel-level mask to obtain three-dimensional point cloud fusion features.

[0043] This invention incorporates a scene 3D point cloud generation and cross-frame object fusion module, aiming to directly generate and continuously optimize the 3D model of each object in the scene from continuous 2D image observations. Its core process consists of three steps: 1. From mask to 3D; 2. Cross-frame object matching; 3. Feature and model fusion and update.

[0044] Specifically, the foreground pixels in the pixel-level mask are obtained, and the depth values ​​of the foreground pixels are obtained; the camera intrinsic parameters are obtained, and the depth values ​​and the camera intrinsic parameters are back-projected onto the camera coordinate system to obtain a three-dimensional point set; the three-dimensional point set is transformed into a unified world coordinate system according to the pose matrix to obtain an object-level three-dimensional point cloud.

[0045] The process from mask to 3D is an efficient object-level point cloud generation. The scene 3D point cloud generation and cross-frame object fusion module adopts an efficient and accurate object-level 3D transformation workflow. For the object binary mask (i.e., the pixel-level mask in this invention) obtained by the instance segmentation module in each frame, the system only processes the pixels (i.e., the foreground pixels in this invention) within the mask area (with a value of 1). First, using the depth values ​​and camera intrinsic parameters of these pixels, they are back-projected to the camera coordinate system to form a 3D point set of the object in the current camera view. Then, the pose matrix of the camera relative to the world coordinate system in this frame is immediately applied to transform these points into a unified world coordinate system. This method can directly output object-level 3D point cloud fragments (transforming the 3D point set to the world coordinate system yields the 3D point cloud fragments, which can obtain the 3D points of the segmented object (pixels) in this frame in the world coordinate system), abandoning the traditional redundant process of first reconstructing a global dense map and then segmenting in the existing technology, achieving a high degree of focus on computing resources and a highly efficient and concise workflow.

[0046] The feature vectors corresponding to each target object in the target bounding box are extracted by a vision-language pre-trained model, and the cosine similarity between the feature vectors is calculated to obtain the appearance similarity. The object-level 3D point cloud is converted into a bounding box, and the 3D intersection-union ratio of the bounding box in a unified world coordinate system is calculated to obtain the spatial overlap. If both the appearance similarity and the spatial overlap meet the preset requirements, the target object is fused using a momentum update strategy to obtain the 3D point cloud fusion feature.

[0047] The dual-criteria matching process is a robust cross-frame object association: to accurately determine whether observations from different frames belong to the same physical entity, the scene 3D point cloud generation and cross-frame object fusion module innovatively designs and adopts a dual-verification association strategy that integrates appearance semantics and 3D geometric information. The specific implementation of this strategy includes two parallel and indispensable decision branches: First, based on a vision-language pre-trained model (such as CLIP), feature vectors of object image patches are extracted, and the cosine similarity between them is calculated to quantify the semantic consistency of objects at the appearance level.

[0048] The specific process of extracting feature vectors of object image patches based on vision-language pre-trained models (such as CLIP) is as follows: Using the bounding box obtained by 2D object detection (i.e., the target bounding box), the rectangular region corresponding to the object is cropped from the corresponding original RGB image. This rectangular image patch contains only the target object (such as a chair). The cropped image patch is adjusted to the standard input size (224*224 pixel value) required by the CLIP model. The processed image patch is input into the CLIP visual encoder (ViT-B / 32 version). The global feature vector of the encoder output layer is extracted. This vector (512 dimensions) highly condenses the visual semantic information of the object.

[0049] Assume the feature vector of the object in the previous frame is A, and the feature vector of the object in the current frame is B. First, normalize the two feature vectors (ensuring the vector length is 1), then directly calculate the dot product of the two vectors, as shown in the following formula: Similarity = cos(θ) = A * B / (||A|| * ||B||), where cos(θ) is the cosine similarity calculation, and θ is the angle between two feature vectors, with the result ranging from -1 to 1. A value closer to 1 indicates greater semantic similarity in the appearance of the two objects (above 0.7 may indicate they are definitely the same object); a lower value indicates greater difference, thus quantifying the semantic consistency of objects at the appearance level. Next, the corresponding 3D point cloud of the object is converted into a bounding box, and its 3D Intersection over Union (IoU) in a unified world coordinate system is calculated to quantify the geometric overlap of the objects at the spatial level. The final system decision follows a strict "AND" logic: only when two objects simultaneously satisfy both the appearance similarity threshold and the spatial overlap threshold are they determined to be the same instance and fused.

[0050] The specific process of converting the 3D point cloud of an object into a bounding box is as follows: Because the 3D point cloud of an object may be an irregular geometric shape in the world coordinate system, this invention encloses these points in a bounding box for easier mathematical calculations. Specifically, it iterates through all points and finds the maximum and minimum values ​​along the X, Y, and Z axes. These six values ​​directly define a cuboid space, and the bounding box is composed of these eight coordinates.

[0051] The 3D Intersection over Union (3D IoU) calculation process is as follows: The geometric overlap between the object in the previous frame (Box1) and the object in the current frame (Box2) in 3D space is quantified as a key geometric criterion for determining whether they are the same entity. First, the system needs to define the overlap range of Box1 and Box2 along the X, Y, and Z axes. For each coordinate axis, the larger of the starting coordinates of the two bounding boxes is taken as the starting point of the intersection, and the smaller of the ending coordinates is taken as the ending point. If, in any axis, the calculated starting coordinate is greater than the ending coordinate, it indicates that the two bounding boxes are separated in that dimension, i.e., there is no physical overlap. In this case, 3D IoU = 0 is directly determined, and the process ends. If there is an overlapping area along all three axes, the product of these three overlapping dimensions is the volume of the intersection cuboid. Intersection volume = (X-axis overlap length) × (Y-axis overlap length) × (Z-axis overlap length). The union volume represents the total space occupied by the two bounding boxes. It is equal to the sum of the individual volumes of the two bounding boxes minus the overlapping volume of the intersection. Specifically, the union volume = (volume of Box1) + (volume of Box2) - (intersection volume). Finally, dividing the intersection volume by the union volume yields the normalized 3DIoU value. This value ranges from 0 to 1; a higher value indicates a greater degree of spatial overlap between the two objects, and a higher probability that they belong to the same entity.

[0052] This design aims to systematically address the inherent limitations of single-criteria association methods. Its direct technical effects are: by leveraging semantic similarity, it effectively associates the same object whose appearance features fluctuate due to changes in viewpoint, lighting differences, or partial occlusion, overcoming the shortcomings of pure geometric matching in feature description invariance; simultaneously, through 3D spatial overlap constraints, it can accurately distinguish different objects in different spatial locations but potentially similar in appearance (e.g., two identical chairs), eliminating misjudgments in such scenarios caused by pure appearance matching. This synergistic effect fundamentally solves the failure problem of single-criteria methods in complex dynamic scenes, systematically improving the accuracy and robustness of cross-frame object association beyond that of single-criteria methods.

[0053] The momentum fusion update process involves constructing a stable and evolving object model: after determining that they are the same object, in order to construct a robust object model that evolves over time, the scene 3D point cloud generation and cross-frame object fusion module adopts the momentum update strategy to fuse and optimize the appearance features of the object (this is the object features extracted from each frame by the clip, because the same object may have appeared in different frames and the angles taken are different. For robustness, this invention performs a weighted fusion of the features of the same object in different frames). Specifically, let the accumulated global features of an object be F_g (this feature is the global object feature obtained by fusing features from all frames), and the new feature observed in the current frame be F_c (for example, in a scene with 99 frames, the current processing is at frame 34, the chair that appeared in the previous 33 frames (appeared in frames 1, 3, and 6) has its features extracted and then fused to obtain the accumulated features of the previous 33 frames. The chair appears again in the current frame 34, so the chair features of this frame are extracted and then fused with the chair features accumulated in the previous 33 frames. The new feature here is the chair feature extracted in frame 34). Then the updated feature F_g' is calculated by the following formula: F_g'=β·F_g + (1-β)·F_c, where β is the momentum coefficient (usually set to a value close to 1, such as 0.9).

[0054] This design aims to address the problems of traditional feature update methods (such as direct replacement or simple averaging) being sensitive to instantaneous observation noise and prone to feature degradation or drastic fluctuations. Its significant technical advantage lies in the fact that this update mechanism is equivalent to a low-pass filter, making the object's feature representation a smoothly changing "moving average" over time. This not only progressively integrates the object's appearance information from multiple viewpoints and time points to form a more comprehensive and essential semantic expression, but also effectively suppresses anomalous feature noise introduced by single-frame image blur, extreme lighting, or brief occlusion. Therefore, in long-term sequential observations, the system can maintain a more discriminative and stable object feature, greatly enhancing the continuity of cross-frame tracking and the reliability of subsequent retrieval and recognition.

[0055] Technical Advantages Summary: The scene 3D point cloud generation and cross-frame object fusion module achieves the goal of automatically constructing high-quality object-level 3D models from dynamic video streams through a closed-loop process of "direct object-level 3D generation → semantic-geometric dual matching → momentum fusion optimization". This method has systematic advantages in efficiency (i.e., directly processing the target and avoiding global reconstruction), accuracy (i.e., the dual criteria significantly improve the accuracy of association), and robustness (i.e., momentum updates smooth noise and obtain stable features), providing a reliable 3D perception foundation for building a dynamic scene knowledge base that can be used for high-level reasoning.

[0056] Step S40: Obtain the semantic attributes of the object and perform spatial relationship calculation based on the three-dimensional point cloud fusion features to obtain the target spatial relationship.

[0057] Specifically, a multi-view image pool of the target object is obtained, and the Laplacian gradient variance algorithm is used to filter the multi-view image pool to obtain multi-view images; the multi-view images and preset structured prompts are input into a multimodal large model to output the semantic attributes of the object.

[0058] This invention includes an object semantic attribute extraction module for extracting high-level semantic attributes of objects. It fully utilizes the multi-view observation images accumulated for each object during cross-frame fusion. Before attribute extraction, the system first uses the Laplacian gradient variance algorithm to quantify and select the three most accurate representative images (i.e., the multi-view images in this invention) from the object's multi-view image pool. This algorithm measures sharpness by calculating the variance value of the image after processing with the Laplacian operator, effectively filtering out observation frames with rich texture details and clear edge information (i.e., the multi-view images in this invention). Subsequently, these high-quality multi-view images, along with structured prompts, are input into a multimodal large model (such as Qwen2.5-VL-72B). By guiding the model to perform comprehensive reasoning based on multi-angle, high-definition visual evidence, it can more accurately and robustly output standardized attribute descriptions of objects such as color, material, and fine shape (i.e., object semantic attributes). This strategy effectively overcomes the limitations of single-view images, such as occlusion, motion blur, or uneven lighting, thus providing more reliable and higher-quality semantic information for knowledge base construction.

[0059] The specific process of measuring image sharpness by calculating the variance of the image after processing with the Laplacian operator is as follows: First, the input RGB image is converted to grayscale. Second, the grayscale image is convolved using the Laplacian operator to calculate the second derivative of each pixel, generating an edge response map that highlights image edges and texture details. Finally, the statistical variance of all pixel values ​​in this edge response map is calculated. The variance value is directly used as the sharpness score; a larger variance means more sharp edges and higher sharpness, while a smaller variance indicates a blurrier image. A threshold is set to sort the images, selecting the top three frames with the largest variances. This effectively eliminates low-quality images caused by motion blur or focus failure, ensuring that subsequent input materials for large models possess optimal texture and detail quality.

[0060] Based on the 3D point cloud fusion features, geometric comparison and dual threshold filtering are performed on any two target objects to obtain the target spatial relationship.

[0061] This invention also includes a module for calculating spatial relationships between objects: based on the generated object-level 3D point cloud (referring to the fusion features of the 3D point cloud), it quantifies and calculates the directional and distance relationships between objects. This invention directly uses the coordinates of the center point of the 3D bounding box of the object in the world coordinate system for calculation, achieving efficient, stable, and viewpoint-independent spatial relationship reasoning.

[0062] For any two objects A and B, obtain their center point coordinates P_A = (X_A, Y_A, Z_A) and P_B = (X_B, Y_B, Z_B), where P_A is the center point coordinate of object A, X_A is the x-coordinate of the center point coordinate of object A, Y_A is the y-coordinate of the center point coordinate of object A, and Z_A is the y-coordinate of the center point coordinate of object A. Similarly, P_B is the center point coordinate of object B, where X_B is the x-coordinate of the center point coordinate of object B, Y_B is the y-coordinate of the center point coordinate of object B, and Z_B is the y-coordinate of the center point coordinate of object B. Then, calculate the relative position vector D = P_A - P_B = (ΔX, ΔY, ΔZ), where ΔX is the x-coordinate of the relative position vector, ΔY is the y-coordinate of the relative position vector, and ΔZ is the y-coordinate of the relative position vector. First, distance filtering is performed: the Euclidean distance d = ||D||² is calculated, and orientation is determined only for object pairs where d ≤ T_dist (T_dist is 3.0 meters). Then, the orientation relationship is discretized based on the vector components and a preset threshold T_orient (T_orient is set to 0.5 meters): if ΔX < -T_orient, then A is to the left of B (left_of); if ΔX > T_orient, then A is to the right (right_of). Similarly, the above / below relationship is determined based on ΔY, and the in / front_of / behind relationship is determined based on ΔZ. This method, through direct geometric comparison in the world coordinate system and double threshold filtering, ensures the absoluteness, stability, and local relevance of relationship determination, providing crucial spatial context information for the scene knowledge base.

[0063] Step S50: Perform knowledge graph construction processing based on the semantic attributes of the object and the spatial relationship of the target to obtain a three-dimensional scene knowledge base.

[0064] This invention establishes a scene knowledge structured storage module based on a graph database: providing persistent storage and efficient query support for the final 3D scene knowledge output by the system. This module innovatively uses a graph database (Neo4j) as the carrier for knowledge representation and storage, unifying discrete object instances, their attributes, and complex spatial relationships into "entity-relationship-attribute" triples, constructing a reasonable and queryable scene knowledge graph. Compared to traditional relational databases, the node and edge structure of graph databases can more intuitively and efficiently express the networked spatial topology and semantic relationships between objects, and is particularly adept at handling complex "multi-hop" relationship queries, providing a direct data foundation for high-level applications such as robot semantic navigation and scene question answering.

[0065] Specifically, spatial topological relationship edges are created based on the target spatial relationships, and attribute association relationship edges are created based on the semantic attributes of the objects. A graph database is used as the storage medium, and the objects in the target scene are used as entity nodes. Knowledge graph construction is performed based on the entity nodes, the spatial topological relationship edges, and the attribute association relationship edges to obtain a three-dimensional scene knowledge base.

[0066] The process of knowledge graph modeling and storage is as follows: The core design of the scene knowledge structured storage module based on graph database lies in abstracting the 3D scene into an attribute graph with "scene" as the root and "object instances" as entity nodes. Each object node dynamically generates a label based on its semantic category (such as "chair" or "table") and carries a complete set of attributes, including: geometric attributes (3D center coordinates, bounding box size), semantic attributes (color, material, original category label), and system identifiers (globally unique ID, scene ID, observation frame list). All data writing is ensured to have atomicity and security through parameterized querying and transaction management.

[0067] The explicit construction process for spatial and semantic relationships is as follows: The construction of a knowledge graph not only stores entities, but more importantly, explicitly constructs the relationships between entities. This module creates two core relationship edges: 1. Spatial Topological Relationship Edges: Based on the aforementioned spatial relationship calculation results, directed edges are created between object nodes, such as LEFT_OF, ABOVE, IN_FRONT_OF, etc. The creation process is as follows: The creation process directly reads the list of orientation target indices recorded in the object data (such as the object number corresponding to left_of), maps them to standard relationship labels (such as LEFT_OF), and then uses a batch query statement in the database to find the source object and the target object at once, and establishes a directional connection between them. This invention directly uses the template processing and storage provided by neo4j packages. For example, if A is to the left of B, neo4j will record that A executes B, and the relationship of the edge is left of.

[0068] 2. Attribute Association Edges: To facilitate knowledge sharing and standardized queries, the system extracts common attributes (such as color and material) into independent nodes and connects them to object nodes through relationship edges such as HAS_COLOR and MADE_OF. This design avoids redundant storage of attribute values ​​and supports fast response to queries such as "all red objects".

[0069] This invention also establishes an idempotent data import and update mechanism: To ensure system robustness, this invention implements an idempotent data import process. Before importing new scene data, the system checks whether the scene ID exists. If configured for update mode, the old graph data for the scene is automatically cleared and rebuilt; if it does not exist, it is created directly. This mechanism ensures the consistency of the knowledge base version and supports dynamic refreshing of scene data.

[0070] The technical benefits of this invention lie in the fact that by introducing a graph database, the massive and heterogeneous object and relationship data output by the 3D perception system are organized into a structured and semantic knowledge network. This not only achieves efficient storage and management of scene information, but also, due to its inherent advantage in relational querying, makes complex spatial semantic queries such as "finding all wooden objects located to the left of a certain table" direct and efficient, providing a powerful knowledge reasoning foundation for upper-level intelligent applications.

[0071] The automatic construction method for a 3D scene knowledge base proposed in this invention achieves the following significant technical effects through the collaborative innovation of the aforementioned modules: 1. Zero-shot scene understanding capability with open vocabulary: Overcoming the limitation of traditional closed-set models in recognizing objects outside the training set. This invention integrates an open-vocabulary detection model with visual-language pre-trained features, enabling automatic object recognition, segmentation, and knowledge storage in new environments without requiring model fine-tuning for specific indoor scenes or unknown object categories. This allows the robot system to quickly adapt to diverse unstructured environments, significantly reducing reliance on scene-customized data annotation.

[0072] 2. Achieved high-fidelity 3D data compression and efficient storage: This effect is mainly attributed to cross-frame object fusion and point cloud optimization techniques, which transform massive, sparse raw RGB-D point cloud streams into compact 3D models and lightweight knowledge graphs based on independent objects. This method effectively filters out background redundancy and sensor noise, while maximizing data compression and fully preserving the key geometric features of objects (such as size, shape, and orientation), providing an efficient storage foundation for long-term memory and real-time retrieval of large-scale scenes.

[0073] 3. Improved accuracy and robustness of complex spatial semantic reasoning: This effect is achieved through direct geometric calculation and thresholding judgment methods based on the world coordinate system. It explicitly and accurately calculates and solidifies the orientation (left / right, front / back, up / down) and Euclidean distance relationships between objects based on object-level 3D representation under a unified world coordinate system. When processing complex commands involving spatial navigation and object search, the system can directly query deterministic geometric knowledge instead of making inferences, thereby significantly improving the positioning accuracy and success rate of task execution.

[0074] 4. Possesses complete interpretability and traceability: Unlike "black box" end-to-end visual language models, this invention ultimately outputs a structured scene knowledge graph. Every decision or question-and-answer given by the robot (such as "retrieving an object") can be clearly traced back to specific entity nodes, attributes, and relational edges in the graph. This not only endows the system with high interpretability but also makes it possible to correct and dynamically update erroneous knowledge, greatly enhancing the reliability of human-computer interaction and the overall maintainability of the system.

[0075] Compared with the prior art, the innovation of this invention and its resulting technical effects are mainly reflected in the following three aspects. Based on these differences, the technical solutions for which this invention seeks protection include, but are not limited to, the following combinations: Key difference 1: An efficient and highly generalized 3D object construction method based on open vocabulary awareness and mask projection.

[0076] Technical solution: An open vocabulary detection model (such as YOLO-World) is used to perform object recognition on RGB images, and the recognition results are used as prompts to input an instance segmentation model (such as SAM 2) to generate pixel-level object masks; then, only the pixels within the object mask area are directly back-projected to a unified world coordinate system, combined with their corresponding depth information and camera pose, to generate object-level 3D point cloud fragments.

[0077] Technical Effects and Advantages: This invention constructs a direct pathway from open-world perception to 3D modeling. Compared to closed-domain models that rely on training with a limited number of categories, this invention can identify and reconstruct massive amounts of unseen "long-tail" objects without retraining, significantly improving the detection coverage and 3D integrity of objects in real-world complex environments. Simultaneously, through targeted projection from "mask to 3D," it avoids the extensive redundant computation of non-target regions in the traditional "global reconstruction first, then segmentation" process, achieving object-level 3D information generation with both efficiency and accuracy.

[0078] Key difference point two: A robust cross-frame object association and fusion method that integrates semantic appearance and three-dimensional geometry as dual criteria.

[0079] Technical solution: In cross-frame object matching, two metrics are calculated and evaluated simultaneously: a) cosine similarity between object image patch features extracted based on a vision-language model (such as CLIP); b) 3D intersection-union ratio (3D IoU) of the object's 3D point cloud in a unified world coordinate system. Only when both metrics meet preset threshold conditions are the two objects determined to be the same instance, and subsequent feature and geometric fusion is performed.

[0080] Technical Effects and Advantages: This invention constructs a more robust association logic than a single geometric or appearance matching strategy by combining and requiring both semantic similarity (to address changes in viewpoint and lighting) and spatial consistency (to exclude objects that are similar in appearance but different in location). Verification has shown that this dual-criteria strategy can significantly reduce the false matching and false matching rates in complex scenes, effectively combat occlusion and appearance changes, thereby generating more stable, complete, and accurate object-level 3D models and continuously evolving appearance feature representations.

[0081] Key difference three: an online scenario knowledge base building framework that supports incremental updates.

[0082] Technical solution: During robot movement, RGB-D image streams are processed in real time, and the aforementioned object detection, 3D modeling, cross-frame fusion, attribute extraction, and spatial relationship calculation steps are executed sequentially. The results (object entities, semantic attributes, and spatial relationships) are stored in the graph database in real time and incrementally in the form of "entity-relationship-attribute" triples, continuously updating the scene knowledge graph without the need for offline batch post-processing.

[0083] Technical effects and advantages: This invention achieves an online closed loop from raw perception to structured knowledge. Compared to the traditional "collect first, then process offline" paradigm, this framework significantly shortens the delay from environmental observation to knowledge availability, enabling the system to respond instantly to dynamic changes in the environment (such as object movement, addition, or removal), and continuously maintain an up-to-date scene cognition model. This provides timely and accurate knowledge support for the robot's online real-time task planning and interactive decision-making.

[0084] In summary, this invention specifically embodies a system and method for automatically, accurately, and efficiently constructing and continuously maintaining a structured scene knowledge base that covers a wide range of object categories, contains accurate 3D models, rich semantic attributes, and spatial relationships from dynamic RGB-D image streams.

[0085] Furthermore, the core innovation of this invention lies in the systematic methodology and framework design of "open vocabulary perception → object-level 3D visualization → dual-criteria fusion → multimodal knowledge construction," rather than relying on a specific model or algorithm. When implementing this framework, those skilled in the art can make various reasonable substitutions and modifications to the modular technical components based on the same design concept; all such modifications fall within the scope of protection of this invention.

[0086] Among them, the replacement and modification solutions may include: 1. Replaceability of the Perception and Segmentation Modules: This embodiment of the invention uses a combination of YOLO-World and SAM 2 to achieve open vocabulary detection and instance segmentation. Essentially, it utilizes the open vocabulary detection model to provide object category and location cues, and uses a cue-driven segmentation model to generate pixel-level masks. Therefore, any combination of technologies capable of achieving equivalent functionality can be substituted. For example, the detection model can be replaced with OWL-ViT, GLIP, or Grounding DINO; the segmentation model can be replaced with Mask R-CNN, SAM 1.0, or other cue-based segmentation architectures. As long as the open category labels and accurate masks of objects can be output from a single frame of RGB image, it can be seamlessly integrated into the subsequent 3D projection process.

[0087] 2. Substitutability of Feature Extraction and Similarity Measurement: This invention employs the CLIP model to extract appearance features and utilizes cosine similarity for matching. Its core is the introduction of visual features with strong semantic representation capabilities as one of the criteria for cross-frame association. Therefore, any visual encoder capable of generating robust, highly discriminative feature vectors for object images (such as self-supervised or supervised models like DINOv2 and ResNet) can be used as a substitute. Similarity measurement can also be replaced by other metrics such as Euclidean distance and Mahalanobis distance. The key to this invention lies not in a specific feature extractor, but in the design concept of making "semantic appearance similarity" an indispensable fusion criterion alongside "three-dimensional geometric consistency."

[0088] 3. Equivalent Variation of Cross-Frame Fusion Criteria and Strategies: This invention employs "CLIP feature similarity + 3D IoU" as dual criteria. Its design essence lies in the need to simultaneously utilize both appearance semantics and spatial geometry information for joint decision-making to overcome the limitations of single information. Under this principle, the appearance similarity index can be transformed into other feature distance metrics; the geometric overlap index can also be replaced by any metric that reflects spatial overlap or proximity, such as 3D bounding box IoU, chamfer distance, or point cloud center distance. Any fusion strategy that simultaneously incorporates semantic and geometric constraints falls within the core design scope of this invention. 4. Scalability of the Attribute Extraction and Knowledge Storage Backend: Multimodal large models (such as Qwen-VL) are used to extract semantic attributes, demonstrating the ability to leverage general visual-linguistic prior knowledge. This module can be expanded to use other general or specialized models (such as GPT-4V, specialized attribute classifier networks, etc.). Similarly, the knowledge base storage is designed to support graph-structured queries of "entity-relationship-attribute," therefore any storage system that can efficiently support such operations (such as other graph databases, RDF storage, or relational databases with extended graph functionality) can serve as an alternative implementation to Neo4j.

[0089] In summary, any technical solution that follows the core process of "open vocabulary perception → object-level 3D projection → fusion of semantic and geometric criteria → construction of structured knowledge graph" and reasonably replaces or adjusts the parameters of the modular components is an equivalent implementation of the design concept of this invention and should be included within the protection scope of this invention.

[0090] Furthermore, such as Figure 5 As shown, based on the above-described method for constructing a three-dimensional multimodal scene knowledge base, this invention also provides a system for constructing a three-dimensional multimodal scene knowledge base, wherein the system for constructing a three-dimensional multimodal scene knowledge base includes: The standardization preprocessing module 51 is used to acquire continuous RGB-D images and pose sequences in the target scene, and to perform standardization preprocessing on the continuous RGB-D images and pose sequences to obtain preprocessed data; The pixel-level mask generation module 52 is used to perform target detection processing and instance segmentation processing on the preprocessed data to obtain a pixel-level mask; The 3D point cloud feature generation module 53 is used to acquire image depth information, and perform object point cloud generation processing and cross-frame point cloud fusion processing based on the image depth information and the pixel-level mask to obtain 3D point cloud fusion features. The spatial relationship calculation module 54 is used to obtain the semantic attributes of the object and perform spatial relationship calculation based on the three-dimensional point cloud fusion features to obtain the target spatial relationship; The knowledge graph construction module 55 is used to perform knowledge graph construction processing based on the semantic attributes of the object and the target spatial relationship to obtain a three-dimensional scene knowledge base.

[0091] Furthermore, such as Figure 6 As shown, based on the above-mentioned method and system for constructing a three-dimensional multimodal scene knowledge base, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 6 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0092] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a three-dimensional multimodal scene knowledge base construction program 40, which can be executed by the processor 10 to implement the three-dimensional multimodal scene knowledge base construction method of this application.

[0093] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the construction method of the three-dimensional multimodal scene knowledge base.

[0094] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display 30 is used to display information on the terminal and to display a visual user interface.

[0095] In one embodiment, when the processor 10 executes the construction program 40 of the three-dimensional multimodal scene knowledge base in the memory 20, it implements the steps of the three-dimensional multimodal scene knowledge base construction method as described above.

[0096] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a construction program for a three-dimensional multimodal scene knowledge base, and the construction program for the three-dimensional multimodal scene knowledge base implements the steps of the construction method for the three-dimensional multimodal scene knowledge base as described above when executed by a processor.

[0097] In summary, this invention provides a method, system, terminal, and storage medium for constructing a three-dimensional multimodal scene knowledge base. The method includes: acquiring continuous RGB-D images and pose sequences in a target scene, and performing standardized preprocessing on the continuous RGB-D images and pose sequences to obtain preprocessed data; performing target detection and instance segmentation on the preprocessed data to obtain pixel-level masks; acquiring image depth information, and performing object point cloud generation and cross-frame point cloud fusion processing based on the image depth information and the pixel-level masks to obtain three-dimensional point cloud fusion features; acquiring object semantic attributes, and performing spatial relationship calculation based on the three-dimensional point cloud fusion features to obtain target spatial relationships; and performing knowledge graph construction processing based on the object semantic attributes and the target spatial relationships to obtain a three-dimensional scene knowledge base. This invention uses a robot to collect continuous RGB-D images and pose sequences in a target scene, and then performs target detection processing, instance segmentation processing, object point cloud generation processing, cross-frame point cloud fusion processing, and calculation of image depth information and target spatial relationships to construct a three-dimensional scene knowledge base. This not only effectively improves the construction accuracy of the three-dimensional scene knowledge base, but also significantly enhances the robot's cognitive and decision-making capabilities in dynamic unstructured environments.

[0098] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0099] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0100] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for constructing a three-dimensional multimodal scene knowledge base, characterized in that, The method for constructing the three-dimensional multimodal scene knowledge base includes: Acquire continuous RGB-D images and pose sequences in the target scene, and perform normalization preprocessing on the continuous RGB-D images and pose sequences to obtain preprocessed data; The preprocessed data is subjected to target detection and instance segmentation to obtain a pixel-level mask; Image depth information is acquired, and object point cloud generation and cross-frame point cloud fusion are performed based on the image depth information and the pixel-level mask to obtain three-dimensional point cloud fusion features. Obtain the semantic attributes of the object and perform spatial relationship calculation based on the 3D point cloud fusion features to obtain the target spatial relationship; A knowledge graph is constructed based on the semantic attributes of the object and the spatial relationships of the target to obtain a three-dimensional scene knowledge base.

2. The method for constructing a three-dimensional multimodal scene knowledge base according to claim 1, characterized in that, The process of acquiring continuous RGB-D images and pose sequences from the target scene, and performing standardized preprocessing on the continuous RGB-D images and pose sequences to obtain preprocessed data, specifically includes: The mobile robot acquires continuous RGB-D images and pose sequences in a target scene using an RGB-D camera. The continuous RGB-D images include an RGB image and a depth map for each frame, and the pose sequence includes a pose matrix, a timestamp, and camera parameters. The continuous RGB-D image and the pose sequence are subjected to normalization preprocessing to obtain preprocessed data. The normalization preprocessing includes depth unit conversion, image size unification, and image deblurring.

3. The method for constructing a three-dimensional multimodal scene knowledge base according to claim 2, characterized in that, The step of performing target detection and instance segmentation processing on the preprocessed data to obtain a pixel-level mask specifically includes: The YOLO-World model is used to perform target detection processing on the preprocessed data to obtain target bounding boxes; The continuous RGB-D image is encoded using a preset segmentation model to obtain a high-dimensional image embedding. The target bounding box is input into the prompt encoder of the preset segmentation model, and the prompt encoder performs coordinate frame transformation on the target bounding box to obtain the prompt vector. The decoder of the preset segmentation model maps the cue vector to the high-dimensional image embedding using a cross-attention mechanism to obtain a pixel-level mask.

4. The method for constructing a three-dimensional multimodal scene knowledge base according to claim 3, characterized in that, The step of using the YOLO-World model to perform target detection processing on the preprocessed data to obtain target bounding boxes specifically includes: Obtain the preset indoor scene object category text, and use a pre-trained CLIP text encoder to convert the preset indoor scene object category text into a text vector; The preprocessed data is used to extract features through the backbone network of the YOLO-World model to obtain multi-scale visual features. The text vector is matched with the multi-scale visual features using the RepVL-PAN network to obtain the target bounding box.

5. The method for constructing a three-dimensional multimodal scene knowledge base according to claim 4, characterized in that, The process of acquiring image depth information and performing object point cloud generation and cross-frame point cloud fusion processing based on the image depth information and the pixel-level mask to obtain 3D point cloud fusion features specifically includes: Obtain the foreground pixels in the pixel-level mask, and obtain the depth value of the foreground pixels; Obtain the camera intrinsic parameters, and back-project the depth value and the camera intrinsic parameters onto the camera coordinate system to obtain a three-dimensional point set; Based on the pose matrix, the 3D point set is transformed to a unified world coordinate system to obtain an object-level 3D point cloud. The feature vectors corresponding to each target object in the target bounding box are extracted by a vision-language pre-trained model, and the cosine similarity between the feature vectors is calculated to obtain the appearance similarity. The object-level 3D point cloud is converted into a bounding box, and the 3D intersection-over-union ratio of the bounding box in the unified world coordinate system is calculated to obtain the spatial overlap. If both the appearance similarity and the spatial overlap meet the preset requirements, then the momentum update strategy is used to fuse the target object to obtain the three-dimensional point cloud fusion feature.

6. The method for constructing a three-dimensional multimodal scene knowledge base according to claim 5, characterized in that, The process of acquiring the semantic attributes of the object and calculating the spatial relationships based on the 3D point cloud fusion features to obtain the target spatial relationships specifically includes: A multi-view image pool of the target object is obtained, and the Laplacian gradient variance algorithm is used to filter the multi-view image pool to obtain multi-view images. The multi-view images and preset structured prompts are input into a multimodal large model, and the semantic attributes of the objects are output. Based on the 3D point cloud fusion features, geometric comparison and dual threshold filtering are performed on any two target objects to obtain the target spatial relationship.

7. The method for constructing a three-dimensional multimodal scene knowledge base according to claim 1, characterized in that, The step of constructing a knowledge graph based on the semantic attributes of the object and the spatial relationships of the target to obtain a three-dimensional scene knowledge base specifically includes: Spatial topological relationship edges are created based on the target spatial relationship, and attribute association relationship edges are created based on the object semantic attributes; A graph database is used as the storage medium, and the objects in the target scene are used as entity nodes. A knowledge graph is constructed based on the entity nodes, the spatial topological relationship edges, and the attribute association relationship edges to obtain a three-dimensional scene knowledge base.

8. A system for constructing a three-dimensional multimodal scene knowledge base, characterized in that, The system for constructing the three-dimensional multimodal scene knowledge base includes: A standardized preprocessing module is used to acquire continuous RGB-D images and pose sequences in the target scene, and to perform standardized preprocessing on the continuous RGB-D images and pose sequences to obtain preprocessed data; A pixel-level mask generation module is used to perform target detection processing and instance segmentation processing on the preprocessed data to obtain a pixel-level mask; The 3D point cloud feature generation module is used to acquire image depth information, and perform object point cloud generation processing and cross-frame point cloud fusion processing based on the image depth information and the pixel-level mask to obtain 3D point cloud fusion features. The spatial relationship calculation module is used to obtain the semantic attributes of the object and perform spatial relationship calculation based on the three-dimensional point cloud fusion features to obtain the target spatial relationship; The knowledge graph construction module is used to construct a knowledge graph based on the semantic attributes of the object and the spatial relationships of the target, thereby obtaining a three-dimensional scene knowledge base.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a construction program for a three-dimensional multimodal scene knowledge base stored in the memory and executable on the processor. When the construction program for the three-dimensional multimodal scene knowledge base is executed by the processor, it implements the steps of the construction method for the three-dimensional multimodal scene knowledge base as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program for constructing a three-dimensional multimodal scene knowledge base. When the program for constructing the three-dimensional multimodal scene knowledge base is executed by a processor, it implements the steps of the method for constructing a three-dimensional multimodal scene knowledge base as described in any one of claims 1-7.

Images