Method, device, storage medium and equipment for indoor scene reconstruction

Abstract

The present invention provides a method, apparatus, storage medium and equipment for indoor scene reconstruction, including: obtaining laser radar point cloud information and a panoramic RGB image; preprocessing the panoramic RGB image to obtain a six-sided cube image; projecting the laser radar point cloud information onto the six-sided cube image in combination with internal and external parameters of a panoramic camera, extracting the color information of each point in the point cloud information, and generating color point cloud data; generating an RGB-D data sequence based on the distance between each point in the color point cloud data and the panoramic camera; inputting the six-sided cube image into an object segmentation model to divide the indoor scene into a plurality of independent object areas; inputting each independent object area into a visual-language model to obtain a semantic label for each independent object; projecting and aligning each independent object containing the semantic label with the RGB-D sequence to obtain aligned point cloud data; and inputting the aligned point cloud input data into a neural core surface reconstruction model to obtain a reconstructed indoor scene.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method, device, storage medium and equipment for indoor scene reconstruction. Background Art

[0002] In recent years, driven by the collaborative efforts of artificial intelligence, robotics, and spatial intelligence, indoor perception and 3D reconstruction have become hot areas of academic research and engineering practice. The rise of multimodal sensor data fusion has provided new solutions for the perception and reconstruction of indoor scenes. Traditional 3D reconstruction techniques, based on sensors such as lidar and cameras, typically generate the scene's geometric structure by processing point cloud data or images.

[0003] However, as the application scenarios of 3D reconstruction technology continue to expand, existing 3D reconstruction technology has the following problems: First, point cloud information based only on collective information is difficult to meet the needs of high-level applications such as intelligent robot navigation and task planning; second, the processing of large-scale point cloud data is computationally complex and lacks real-time performance; finally, the use of a single sensor limits the detailed expression of the scene and the diversity of applicable scenarios. Summary of the Invention

[0004] Based on this, the present invention provides a method, device, storage medium and equipment for indoor scene reconstruction, which fuses point cloud data and panoramic RGB images to construct image information with depth information and color information, and uses an object segmentation model and a vision-language model based on the fused image information to perform semantic annotation of objects in the scene, thereby constructing an indoor scene containing semantic information, which can provide accurate and rich data support in three-dimensional modeling or path planning scenarios.

[0005] In a first aspect, the present invention provides a method for reconstructing an indoor scene, comprising:

[0006] Obtain lidar point cloud information and panoramic RGB images;

[0007] Preprocessing the panoramic RGB image to obtain a six-sided cube image;

[0008] Combining the intrinsic and extrinsic parameters of the panoramic camera, projecting the laser radar point cloud information onto a six-sided cube image, extracting the color information of each point in the point cloud information, and generating color point cloud data;

[0009] Generate an RGB-D data sequence based on the distance between each point in the color point cloud data and the panoramic camera;

[0010] Inputting the six-sided cube image into an object segmentation model to divide the indoor scene into a number of independent object areas;

[0011] Inputting each of the independent object regions into a vision-language model to obtain a semantic label for each independent object;

[0012] Project each independent object containing semantic labels onto the RGB-D sequence to obtain aligned point cloud data;

[0013] The aligned point cloud input data is input into the neural kernel surface reconstruction model to obtain the reconstructed indoor scene.

[0014] Furthermore, the indoor scene reconstruction method further includes:

[0015] Searching for corresponding independent object results in the reconstructed indoor scene according to the received user instructions;

[0016] Send the searched independent object results to the user end;

[0017] If the independent object result searched out by the user terminal is inconsistent with the user instruction, a supplementary semantic label is added to the opposite object area indicated by the user instruction.

[0018] Furthermore, the object segmentation model is an instance segmentation model.

[0019] Furthermore, the vision-language model is a contrastive language-image pre-training model or a Grounding DINO model.

[0020] Furthermore, the panoramic RGB image is preprocessed to obtain a six-sided cube image, specifically:

[0021] The panoramic RGB image is converted into a six-sided cube image using equirectangular projection.

[0022] Furthermore, the method combines the intrinsic and extrinsic parameters of the panoramic camera, projects the laser radar point cloud information onto a six-sided cube image, extracts the color information of each point in the point cloud information, and generates color point cloud data, including the following steps:

[0023] For any target point in the laser point cloud information, perform the following steps:

[0024] Step S201, correcting the coordinates of the target point according to the external parameters of the panoramic camera to obtain corrected target point coordinates;

[0025] Step S202: rotating the calibration target point coordinates according to the internal parameters of the panoramic camera to obtain the target point mapping coordinates;

[0026] Step S203, combining the width and height of the panoramic RGB image to obtain the projection point coordinates of the target point on the six-sided cube;

[0027] Step S204, recording the color data of the projection point as the color data of the target point;

[0028] Step S205 , repeating steps S201 - S204 until the color data of each target point in the laser point cloud information is determined, and color point cloud data is generated.

[0029] Furthermore, generating an RGB-D data sequence according to the distance between each point in the color point cloud data and the panoramic camera includes the following steps:

[0030] Generate a depth map based on the distance between each point in the color point cloud data and the panoramic camera;

[0031] The depth map is combined with the color point cloud data to obtain an RGB-D data sequence.

[0032] Furthermore, the combining of the depth map with the color point cloud data to obtain an RGB-D data sequence further includes:

[0033] For each target point in the color point cloud data, if the depth value of the target point is less than the historical depth value, the color feature of the target point is updated.

[0034] In the second room, the present invention further provides a device for indoor scene reconstruction, comprising:

[0035] Image acquisition module, used to obtain lidar point cloud information and panoramic RGB images;

[0036] A panoramic to hexahedron conversion module is used to pre-process the panoramic RGB image to obtain a hexahedron image;

[0037] A point cloud data color extraction module is used to combine the intrinsic and extrinsic parameters of the panoramic camera, project the lidar point cloud information onto a six-sided cube image, extract the color information of each point in the point cloud information, and generate color point cloud data;

[0038] The depth parameter combination module is used to generate an RGB-D data sequence based on the distance between each point in the color point cloud data and the panoramic camera;

[0039] An object segmentation module, configured to input the six-sided cube image into an object segmentation model to divide the indoor scene into a plurality of independent object regions;

[0040] A semantic labeling module, configured to input each of the independent object regions into a vision-language model to obtain a semantic label for each independent object;

[0041] The data fusion module is used to project and align each independent object containing semantic labels with the RGB-D sequence to obtain the aligned point cloud data;

[0042] The scene reconstruction module is used to input the aligned point cloud input data into the neural kernel surface reconstruction model to obtain the reconstructed indoor scene.

[0043] In a third aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any one of the indoor scene reconstruction methods in the first aspect.

[0044] In a fourth aspect, the present invention further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it executes any one of the indoor scene reconstruction methods in the first aspect.

[0045] The beneficial effects of adopting the above technical solution are as follows: this embodiment integrates the lidar point cloud information and the panoramic RGB image, realizes the precise alignment of the handheld lidar and the panoramic camera data, and constructs a three-dimensional scene graph with an open vocabulary through the algorithm; it can recognize unseen objects and generate semantically rich labels, breaking through the limitations of predefined categories and laying a technical foundation for open semantic segmentation. In terms of scene reconstruction, a neural kernel surface reconstruction model is used to perform high-precision geometry and texture reconstruction of the object instance point cloud, effectively capturing the complex structure and details of the scene, while truly restoring the color and texture information. Through object-level instance segmentation and real-time updated scene graph construction technology, the present invention can adapt to changes in dynamic scenes, such as the movement, addition or removal of objects, significantly improving the understanding and modeling capabilities of dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments or the prior art.

[0047] Figure 1 This is a schematic diagram of a method for indoor scene reconstruction in one embodiment of the present application;

[0048] Figure 2 This is a schematic diagram of data collection in an indoor scene reconstruction method in one embodiment of the present application;

[0049] Figure 3 This is a schematic diagram of combining a panoramic RGB image with LiDAR point cloud information in one embodiment of the present application;

[0050] Figure 4 This is a schematic diagram of the scene reconstruction process in the method for indoor scene reconstruction in one embodiment of the present application;

[0051] Figure 5 This is a schematic diagram of the semantic labeling process in the method for indoor scene reconstruction in one embodiment of the present application;

[0052] Figure 6 This is a schematic diagram of object segmentation in a method for indoor scene reconstruction in one embodiment of the present application;

[0053] Figure 7 Schematic diagram of an apparatus for indoor scene reconstruction in one embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In order to explain the present invention in more detail, the method, device, storage medium and equipment for indoor scene reconstruction provided by the present invention are specifically described below in conjunction with the drawings.

[0055] Unless otherwise defined, the technical or scientific terms used in this application should have the usual meanings understood by people with ordinary skills in the field to which the invention belongs. The words "first", "second" and similar terms used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one", "an" or "the" do not indicate a quantity limitation, but rather indicate the presence of at least one. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0056] In recent years, the rise of multimodal sensor data fusion technology has provided new solutions for the perception and reconstruction of indoor scenes. By combining high-precision point cloud data from lidar with rich visual information from cameras, researchers can introduce semantic understanding while preserving geometric information. However, how to efficiently fuse multimodal data and achieve end-to-end high accuracy and efficiency in scene segmentation, semantic annotation, and 3D reconstruction remains a challenging research topic.

[0057] Deep learning semantic segmentation techniques based on LiDAR point clouds, such as the Random Sampled PointNet (RandLA-Net), the Kernel Point Convolution (KPConv), and the Submanifold Sparse Convolutional Network (SparseConvNet), achieve object recognition and semantic segmentation in scenes by modeling the geometric properties of point cloud data. These methods perform well on sparse point clouds, capturing local geometric features and generating 3D semantic information. However, these methods have significant limitations. Their reliance on geometric features and lack of semantic understanding make them incapable of segmenting or recognizing new objects of unknown types or in complex scenes. Furthermore, these methods struggle to effectively handle object changes in dynamic scenes and cannot update semantic information in real time. Furthermore, due to the massive volume of LiDAR point cloud data, directly processing large-scale point cloud data consumes high computational resources and memory, making it difficult to meet the requirements of real-time applications. Point cloud-based segmentation methods are limited in tasks involving open scenes and multimodal requirements.

[0058] Semantic segmentation techniques based on RGB images (such as Mask R-CNN, DeepLab, and Vision-Transformer) rely on convolutional neural networks or visual transformers and have achieved remarkable results in extracting image features and instance segmentation. They can recognize a variety of object types and generate high-resolution segmentation results. However, the implementation of this method relies solely on two-dimensional images, has limited modeling capabilities for three-dimensional geometric information, and has difficulty processing spatial structures and depth information in point cloud data. In addition, the above-mentioned technologies rely heavily on labeled data, cannot recognize new categories outside of closed vocabulary, and have weak generalization capabilities. In dynamic scenes, they lack the ability to dynamically update semantic information and cannot adapt to scene changes in real time, making it difficult to support complex scene planning tasks, such as robot navigation and grasping tasks that require spatial semantic relationships.

[0059] Furthermore, existing technologies include multimodal semantic segmentation technologies (such as BEVFusion, FusionMLP, and MaskFusion), which achieve semantic segmentation and understanding of three-dimensional scenes by fusing the advantages of LiDAR point cloud information with RGB image data. Leveraging the semantic information in image data and the geometric information in LiDAR point cloud data enables a more comprehensive description of the scene and significantly improves the accuracy of semantic segmentation. However, this multimodal semantic segmentation technology lacks an efficient mechanism for updating semantic information, making it difficult to meet real-time requirements. Furthermore, multimodal data fusion increases computational complexity and places high demands on hardware resources, limiting its application in real-time tasks.

[0060] Based on the consideration of the above-mentioned prior art, the present invention provides a method for indoor scene reconstruction, which obtains an RGB-D image by fusing point cloud data and panoramic RGB images, and combines the instance segmentation model and the vision-language model to perform object segmentation and semantic annotation to obtain an indoor scene with semantic labels.

[0061] The embodiment of the present application provides an application scenario of the method for indoor scene reconstruction, which includes the terminal device provided in the embodiment, wherein the terminal device includes but is not limited to a smart phone and a computer device, wherein the computer device can be at least one of a desktop computer, a portable computer, a laptop computer, a mainframe computer, a tablet computer, etc. The terminal device receives the indoor point cloud data sent by the laser radar and the panoramic RGB image sent by the panoramic camera, constructs a three-dimensional scene containing semantic information, and combines the attached Figure 1 The diagram shows a schematic diagram of the indoor scene reconstruction method. For the specific process, please refer to the embodiment of the indoor scene reconstruction method.

[0062] Step S101: Obtain lidar point cloud information and panoramic RGB images.

[0063] Among them, combined with Figure 2 The data acquisition diagram for the indoor scene reconstruction method shown in the figure below illustrates lidar point cloud data, which refers to a set of vectors in a three-dimensional coordinate system of an indoor scene acquired using lidar. This data provides information about the 3D geometric structure of the indoor scene. A panoramic RGB image is a 360-degree panoramic RGB image of an indoor scene captured using a panoramic camera. An RGB image is an image generated from varying intensities of the three basic colors red (red), green (green), and blue (blue). In an RGB image, each pixel typically consists of three components, representing the brightness values of the three colors red, green, and blue, respectively. The value of each component in an RGB image ranges from 0 to 255.

[0064] In this embodiment, as shown in the attached Figure 2As shown, a handheld laser radar is used to obtain laser radar point cloud information to provide 3D geometric structure information of the indoor scene; a panoramic camera is used to collect 360-degree panoramic RGB images of the indoor scene to supplement semantic information and texture details. In this embodiment, the handheld laser radar can be a Lingguang L1 laser radar, and the panoramic camera can be an Insta360 panoramic camera. In this embodiment, the handheld laser radar and the panoramic camera are integrated into one design.

[0065] Step S102 : pre-processing the panoramic RGB image to obtain a six-sided cube image.

[0066] Specifically, considering that the panoramic RGB image will be severely stretched and distorted in the polar regions (such as the top and bottom of the panoramic RGB image) due to the projection method, resulting in blurred or deformed image details, which is not conducive to the subsequent indoor scene reconstruction process, the panoramic RGB image needs to be converted into a six-sided cube image that is closer to a conventional plane image and has almost no distortion. It should be noted that in this embodiment, the equidistant cylindrical projection method can be used to convert the panoramic RGB image into a six-sided cube image. Specifically, for each pixel point in the panoramic RGB image, spherical coordinates can be used. The pixel points of the six-sided cube image are converted according to the following expression The expression is as follows:

[0067] , ,

[0068] Spherical coordinates Indicates the longitude of the panoramic RGB image pixel on the spherical surface, , spherical coordinates Indicates the latitude of the panoramic RGB image pixel on the spherical surface, .

[0069] Equirectangular projection involves projecting the various components of a stereoscopic image onto a plane at equal distances, ensuring a one-to-one correspondence between the size of the image on the projection plane and the size of the stereoscopic image. The goal of equirectangular projection is to display a larger stereoscopic figure on a smaller projection plane while preserving the shape and proportions of the figure. After equirectangular projection, each face of the six-sided cube image more closely resembles the perspective of a conventional two-dimensional image, with virtually no distortion in the center region, making it easier to perceive objects in indoor scenes.

[0070] In step S103 , the lidar point cloud information is projected onto a six-sided cube image in combination with the internal and external parameters of the panoramic camera, and the color information of each point in the point cloud information is extracted to generate color point cloud data.

[0071] Specifically, combined with Figure 3The diagram below shows a schematic diagram of the combination of a panoramic RGB image and LiDAR point cloud information. In this embodiment, the panoramic camera and LiDAR are integrated. The camera's intrinsic parameters include its focal length and principal point coordinates, while the camera's extrinsic parameters are the relative pose between the LiDAR and the camera. The point cloud data is precisely projected onto a six-sided cube image, and the color information of each point in the point cloud is extracted to generate color point cloud data. This color point cloud data includes not only the 3D structure of the point cloud data but also its color information.

[0072] In this embodiment, the laser radar point cloud information is projected onto a six-sided cube image, and the color information of each point in the point cloud information is extracted to generate color point cloud data using the following method:

[0073] For any target point in the laser point cloud information, perform the following operations:

[0074] Step S201 , correcting the coordinates of the target point according to the external parameters of the panoramic camera to obtain corrected target point coordinates.

[0075] Specifically, the expression for correcting the coordinates of the target point according to the external parameters of the panoramic camera is:

[0076] ,

[0077] To correct the target point coordinates, is the coordinate of the target point, is the external parameter of the panoramic camera, which represents the relative position between the lidar and the panoramic camera.

[0078] Step S202 : rotating the calibrated target point coordinates according to the internal parameters of the panoramic camera to obtain the target point mapping coordinates.

[0079] Specifically, the coordinates of the calibrated target point are rotated according to the internal parameters of the panoramic camera, and the expression of the target point mapping coordinates is obtained as follows:

[0080] ,

[0081] Map coordinates for the target point, is the internal parameter of the panoramic camera, where is the horizontal focal length of the panoramic camera, is the vertical focal length of the panoramic camera, is the horizontal coordinate of the principal point of the image, is the vertical coordinate of the principal point of the image.

[0082] Step S203 , obtaining the projection point coordinates of the target point on the six-sided cube by combining the width and height of the panoramic RGB image.

[0083] Specifically, the expression for the projection coordinates of the target point on the six-sided cube is obtained by combining the width and height of the panoramic RGB image:

[0084] ,

[0085] ,

[0086] is the coordinate of the projection point of the six-sided cube, is the width of the panoramic RGB image, is the height of the panoramic RGB image, is the spherical coordinate of the panoramic RGB image corresponding to the target mapping coordinate, , .

[0087] Step S204: Record the color data of the projection point as the color data of the target point.

[0088] Specifically, the color data of the projection point is recorded as the color data of the target point as follows:

[0089] ,

[0090] is the color data of the target point, The color data of the projection point.

[0091] Step S205 , repeating steps S201 - S204 until the color data of each target point in the laser point cloud information is determined, and color point cloud data is generated.

[0092] Step S104: Generate an RGB-D data sequence based on the distance between each point in the color point cloud data and the panoramic camera.

[0093] Specifically, generating an RGB-D data sequence based on the distance between each point in the color point cloud data and the panoramic camera includes the following steps:

[0094] Step S301, generating a depth map based on the distance between each point in the color point cloud data and the panoramic camera;

[0095] Step S302: combining the depth map with the color point cloud data to obtain an RGB-D data sequence.

[0096] The above step S302 combines the depth map with the color point cloud data to obtain an RGB-D data sequence, which includes the following specific steps:

[0097] For any depth image pixel coordinate in the depth map , use the panoramic camera internal parameters to back-project it into spatial coordinates , according to the spatial coordinates Find the nearest neighbor in the color point cloud data , the nearest neighbor The RGB value is matched to the depth map pixel coordinates ; Thus, the depth image pixel coordinates combined with the RGB values can generate a quad ,in The value is the RGB value of the nearest neighbor point. is the distance value of the depth image pixel. When each depth image pixel coordinate in the depth map is assigned the RGB value of the nearest neighbor in the color point cloud data and the corresponding quadruple is generated, the quadruple of all depth image pixels in the depth map is Construct a four-channel image or sequence array and sort it by frame number to get the RGB-D data sequence.

[0098] It should be noted that when generating a depth map based on the distance between the color point cloud data and the panoramic camera, the lack of sufficient density and resolution of the coefficient point cloud leads to incorrect projection, the sparsity of the point cloud map leads to "leakage" problems, the incorrect projection of occluded points, and the lack of point cloud support for complete surface data of occluded objects, making it impossible to correctly judge occlusion relationships. To address this issue, the depth buffer is used to compare the depth value of the current point with the value in the depth buffer. The color or features of the pixel are only updated when the depth of the current point is less than the recorded depth value. To address the occlusion problem, a 5*5 minimum filter is used for screening to process errors or noise introduced by projecting the sparse point cloud onto the image frame, thereby achieving the effect of removing outliers or incorrectly projected points.

[0099] This data processing process not only achieves precise alignment of point cloud information and image data, but also lays the foundation for subsequent semantic segmentation and 3D reconstruction.

[0100] Step S105: input the six-sided cube image into an object segmentation model to divide the indoor scene into a number of independent object areas.

[0101] Specifically, the object segmentation model in this embodiment can be a Segment Anything Model (SAM model), which divides indoor scenes into several independent object regions by generating a binary semantic mask from a six-sided cube image. It should be noted that the SAM model can segment objects in any image using a variety of interactive cues (such as points, boxes, text, or masks), achieving object segmentation without the need for task-specific fine-tuning.

[0102] Furthermore, the above-mentioned binary semantic mask is then projected into the three-dimensional point cloud space through the camera's intrinsic and extrinsic parameters, where the intrinsic parameters are used to calculate the projection coordinates and the extrinsic parameters are used to describe the posture relationship between the lidar and the panoramic camera, thereby realizing the alignment of multimodal data and the generation of object-level instance point clouds.

[0103] Step S106 : Inputting each of the independent object regions into a vision-language model to obtain a semantic label for each independent object.

[0104] Specifically, in this embodiment, combined with the attached Figure 6 The figure shows a schematic diagram of object segmentation in the indoor scene reconstruction method. The vision-language model is a contrastive language-image pre-training model (CLIP model), which can extract features from the RGB image area of each independent object area and generate open-vocabulary semantic labels.

[0105] Furthermore, before step S106 , a closest point priority strategy may be provided to eliminate projection errors in each independent object region to ensure the geometric accuracy of the object point cloud.

[0106] Specifically, for each semantic instance, a corresponding 3D point cloud subset is extracted based on the projection relationship or preliminary segmentation result as the input area for subsequent processing; considering various possible depth deviations in image back-projection or multi-view synthesis, an error estimate between the observed distance and the ideal position of each point is established; for candidate points in the same pixel area or spatial neighborhood, a "nearest point first" strategy is adopted, and only the point with the smallest distance from the panoramic camera is retained as the valid observation result; by excluding redundant projection points that are far away or obstructed, the geometric artifacts and ghosting interference caused by multi-view reprojection can be effectively reduced; the screened point cloud geometry is used as a high-precision geometric representation and input into the vision-language model for semantic label prediction to improve recognition accuracy and robustness.

[0107] Furthermore, considering the complexity and strong combination of semantic objects, as well as the lack of clarity and ambiguity of individual semantic labels, the open-vocabulary semantic labels in this embodiment can not only mark the object's category (such as "chair" or "table"), but also supplement attribute descriptions (such as "red round object"), providing multiple label candidates for complex objects. The specific process for supplementing attribute descriptions is as follows:

[0108] Step S1061 : searching for corresponding independent object results in the reconstructed indoor scene according to the received user instruction.

[0109] Step S1062: Send the searched independent object results to the user terminal.

[0110] In step S1063, if the independent object search results reported by the user terminal are inconsistent with the user instruction, a supplementary semantic label is added to the independent object area indicated by the user instruction. It should be noted that the user instruction is a natural language description of the object to be searched. The natural language recognition model can extract a specific object description from the user instruction. Based on this object description, the semantic labels of each independent object in the reconstructed indoor scene can be traversed to search for the corresponding independent object results.

[0111] The supplementary semantic tags include at least one of a color attribute tag, a shape attribute tag, and a position attribute tag. A color attribute tag indicates the color attribute of an independent object, such as red, green, or yellow; a shape attribute tag indicates the shape attribute of an independent object, such as circle, rectangle, or heart; and a position attribute tag indicates the specific location of an independent object in the reconstructed scene, such as the upper left, northeast corner, or between object A and object B. The supplementary semantic tags, formed by at least one of the color, shape, and position attribute tags, can be used to adjust for situations in which the original label in the reconstructed indoor scene is ambiguous or not sufficiently indicative. For example, if the original label is "chair," but chairs exist at various locations or in various colors and shapes in the reconstructed indoor scene, the independent object results for all chairs in the reconstructed indoor scene will be fed back to the user end. If the independent object results fed back by the user end are inconsistent with the user's instruction (the user instruction is to find a chair with a red heart-shaped backrest), a supplementary semantic tag (such as "red, heart-shaped backrest") is added based on the independent object area indicated by the user instruction.

[0112] In addition, for the semantic tags of the open vocabulary mentioned above, a specific confidence assessment can be generated in combination with the semantic tags when users search for specific objects in the subsequent stage, thereby improving the accuracy of semantic annotation. The confidence assessment refers to the accuracy of the results when users search for specific objects in the subsequent stage.

[0113] In step S107 , each independent object containing a semantic label is projected and aligned with the RGB-D sequence to obtain aligned point cloud data.

[0114] Step S108: input the aligned point cloud input data into the neural core surface reconstruction model to obtain a reconstructed indoor scene.

[0115] Specifically, as attached Figure 4The scene reconstruction process diagram shown in the figure shows that during the scene reconstruction phase, the Navigation Knowledge Situation and Reasoning (NKSR) model is used to perform high-precision 3D geometric reconstruction of the segmented object point cloud. The NKSR model uses a neural network to learn local geometric features in point cloud data and predict the object's 3D surface based on these features. It should be noted that the NKSR model combines convolutional neural networks (CNNs) and kernel methods. By learning the local geometric information of point cloud information, it can efficiently and accurately reconstruct object surfaces, providing high-quality basic data for subsequent applications such as virtual reality, indoor navigation, and scene simulation.

[0116] Based on the above solution, a handheld lidar is combined with a panoramic camera. By accurately aligning the three-dimensional geometric information of the lidar point cloud information with the texture and color information of the panoramic camera, the limitations of a single modality are broken through, and standardized RGB-D data is generated. This multimodal fusion method effectively improves the comprehensiveness and detail expression capabilities of scene perception, and provides rich data support for subsequent semantic segmentation, three-dimensional modeling, and path planning. In addition, this embodiment combines the SAM model and the CLIP model to achieve semantic segmentation of open vocabulary, and can automatically generate semantic labels and instance segmentation results without additional data annotation or model training. Compared with traditional predefined category segmentation technology, it breaks through category restrictions, has strong versatility and adaptability, can identify unknown objects in complex scenes and assign semantic labels, and greatly reduces the cost of model development and the difficulty of annotation.

[0117] Furthermore, in this embodiment, the SAM model and CLIP model are combined to detect objects and generate specific labels, which are not restricted by predefined categories. However, due to the ambiguity of label generation, the detection results may sometimes be unstable or inconsistent. Figure 5 As shown in the semantic labeling process diagram, this embodiment can also use the Grounding DINO (Grounding Distillation with NO labels, unlabeled object detection model) model to annotate semantic labels for each independent object area, and introduce category files as a predefined category label set for the object detection model to ensure the consistency and repeatability of the segmentation results. It can define category files according to its actual needs to improve the stability of the segmentation results.

[0118] In order to better illustrate the advantages of the above-mentioned indoor scene reconstruction method in practical applications, the above-mentioned method is applied to indoor gas pipeline wiring and indoor robot navigation for illustration.

[0119] (1) Example of indoor gas pipeline wiring:

[0120] Using a handheld LiDAR and panoramic camera, the system collects point cloud data and 360° panoramas of the kitchen space. The panoramas are converted into six-sided stereo images, and a depth map is generated based on the distance between the point cloud and the panoramic camera. This generates RGB-D data, and combines SAM and CLIP to achieve open-vocabulary semantic segmentation. This automatically identifies and annotates the position and shape of objects such as walls, ceilings, floors, and furniture and appliances that may obstruct wiring. CLIP generates semantic labels, adding detailed attribute descriptions to segmented objects. The NKSR algorithm is then used to perform high-precision 3D reconstruction of the point cloud, producing a 3D kitchen scene model with rich geometric details and realistic textures.

[0121] Based on this, and incorporating pipeline routing rules, the optimal gas pipeline route is automatically planned, bracket fixing points are designed, and hazardous areas are marked to ensure safe and rational routing. This indoor scene reconstruction method improves the efficiency of gas wiring design while significantly reducing data acquisition and modeling costs. Its high convenience and intelligent advantages provide strong technical support for gas wiring construction, maintenance, and safety assessment.

[0122] (2) Indoor robot navigation scenario:

[0123] Using LiDAR and panoramic cameras to collect point cloud data and 360° panoramas of indoor scenes, the panoramas are converted into six-sided stereo images. A depth map is generated based on the distance between the point cloud and the panoramic camera, generating RGB-D data. By segmenting the large model (SAM) and generating semantic labels using CLIP, the system accurately perceives and analyzes the environment and constructs a semantically-informed 3D map. Based on this semantically-informed 3D map, the robot can avoid obstacles and adjust its navigation path, using algorithms such as A* / Dijkstra to plan the optimal path. This system has been effectively applied to fields such as home services, commercial logistics, and intelligent inspection, demonstrating its technological advantages and social value through intelligence, flexibility, and efficiency.

[0124] It should be understood that although the Figure 1 The steps in the flowchart are shown in the order indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Figure 1 At least part of the steps may include multiple sub-steps or sub-stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0125] The embodiment disclosed in the above invention describes in detail the method for indoor scene reconstruction. The method disclosed in the present invention can be implemented using various devices. Therefore, the present invention also discloses an apparatus for indoor scene reconstruction. Figure 7 , specific embodiments are given below to explain in detail.

[0126] Image acquisition module 501, used to acquire lidar point cloud information and panoramic RGB images;

[0127] A panorama-to-hexahedron conversion module 502 is configured to pre-process the panoramic RGB image to obtain a hexahedron image;

[0128] The point cloud data color extraction module 503 is used to combine the internal and external parameters of the panoramic camera, project the laser radar point cloud information onto a six-sided cube image, extract the color information of each point in the point cloud information, and generate color point cloud data;

[0129] a depth parameter combining module 504 for generating an RGB-D data sequence based on the distance between each point in the color point cloud data and the panoramic camera;

[0130] An object segmentation module 505 is configured to input the six-sided cube image into an object segmentation model to divide the indoor scene into a plurality of independent object regions;

[0131] A semantic labeling module 506 is configured to input each of the independent object regions into a vision-language model to obtain a semantic label for each independent object;

[0132] The data fusion module 507 is used to perform projective alignment between each independent object containing a semantic label and the RGB-D sequence to obtain aligned point cloud data;

[0133] The scene reconstruction module 508 is used to input the aligned point cloud input data into the neural core surface reconstruction model to obtain a reconstructed indoor scene.

[0134] Regarding the apparatus for indoor scene reconstruction, please refer to the above definition of the method and will not be repeated here. Each module in the above apparatus can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above modules can be embedded in or independent of the processor of the terminal device in the form of hardware, or can be stored in the memory of the terminal device in the form of software so that the processor can call and execute the operations corresponding to each of the above modules.

[0135] In one embodiment, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above-mentioned method for indoor scene reconstruction when executed by a processor.

[0136] The computer-readable storage medium may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), a hard disk, or ROM. Optionally, the computer-readable storage medium includes a non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program code for executing any of the steps of the above-described method. This program code can be read from or written to one or more computer program products, and the program code may be compressed in a suitable form.

[0137] In one embodiment, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor performs the above-mentioned indoor scene reconstruction method when executing the computer program.

[0138] The computer device includes a memory, a processor, and one or more computer programs, wherein the one or more computer programs can be stored in the memory and configured to be executed by one or more processors, and the one or more application programs are configured to execute the above-mentioned indoor scene reconstruction method.

[0139] A processor may include one or more processing cores. The processor utilizes various interfaces and circuits to connect the various components within the entire computer device. It executes instructions, programs, code sets, or instruction sets stored in memory, and accesses data stored in memory to perform various functions of the computer device and process data. Optionally, the processor may be implemented in the form of at least one of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor and may be implemented separately via a communications chip.

[0140] The memory may include random access memory (RAM) or read-only memory (ROM). The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may also store data created by the terminal device during use, etc.

[0141] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for indoor scene reconstruction, characterized in that: include: Obtain lidar point cloud information and panoramic RGB images; Preprocessing the panoramic RGB image to obtain a six-sided cube image; Combining the intrinsic and extrinsic parameters of the panoramic camera, projecting the laser radar point cloud information onto a six-sided cube image, extracting the color information of each point in the point cloud information, and generating color point cloud data; Generate an RGB-D data sequence based on the distance between each point in the color point cloud data and the panoramic camera; Inputting the six-sided cube image into an object segmentation model to divide the indoor scene into a number of independent object areas; Inputting each of the independent object regions into a vision-language model to obtain a semantic label for each independent object; Project each independent object containing semantic labels onto the RGB-D sequence to obtain aligned point cloud data; Input the aligned point cloud input data into the neural kernel surface reconstruction model to obtain the reconstructed indoor scene; Searching for corresponding independent object results in the reconstructed indoor scene according to the received user instructions; Send the searched independent object results to the user end; If the independent object result searched out by the user terminal is inconsistent with the user instruction, a supplementary semantic label is added to the opposite object area indicated by the user instruction; The supplementary semantic tag includes at least one of a color attribute tag, a shape attribute tag, and a position attribute tag.

2. The method for indoor scene reconstruction according to claim 1, wherein: The object segmentation model is an instance segmentation model.

3. The method for indoor scene reconstruction according to claim 2, wherein: The vision-language model is a contrastive language-image pre-trained model or a Grounding DINO model.

4. The method for indoor scene reconstruction according to claim 3, wherein: The panoramic RGB image is preprocessed to obtain a six-sided cube image, specifically: The panoramic RGB image is converted into a six-sided cube image using equirectangular projection.

5. The method for indoor scene reconstruction according to claim 4, characterized in that: The method combines the internal and external parameters of the panoramic camera, projects the laser radar point cloud information onto a six-sided cube image, extracts the color information of each point in the point cloud information, and generates color point cloud data, including the following steps: For any target point in the laser point cloud information, perform the following steps: Step S201, correcting the coordinates of the target point according to the external parameters of the panoramic camera to obtain corrected target point coordinates; Step S202: rotating the calibration target point coordinates according to the internal parameters of the panoramic camera to obtain the target point mapping coordinates; Step S203, combining the width and height of the panoramic RGB image to obtain the projection point coordinates of the target point on the six-sided cube; Step S204, recording the color data of the projection point as the color data of the target point; Step S205 , repeating steps S201 - S204 until the color data of each target point in the laser point cloud information is determined, and color point cloud data is generated.

6. The method for indoor scene reconstruction according to claim 5, characterized in that: Generating an RGB-D data sequence according to the distance between each point in the color point cloud data and the panoramic camera includes the following steps: Generate a depth map based on the distance between each point in the color point cloud data and the panoramic camera; The depth map is combined with the color point cloud data to obtain an RGB-D data sequence.

7. The method for indoor scene reconstruction according to claim 6, wherein: The combining of the depth map with the color point cloud data to obtain an RGB-D data sequence further includes: For each target point in the color point cloud data, if the depth value of the target point is less than the historical depth value, the color feature of the target point is updated.

8. A device for indoor scene reconstruction, characterized in that: include: Image acquisition module, used to obtain lidar point cloud information and panoramic RGB images; A panoramic to hexahedron conversion module is used to pre-process the panoramic RGB image to obtain a hexahedron image; A point cloud data color extraction module is used to combine the intrinsic and extrinsic parameters of the panoramic camera, project the lidar point cloud information onto a six-sided cube image, extract the color information of each point in the point cloud information, and generate color point cloud data; The depth parameter combination module is used to generate an RGB-D data sequence based on the distance between each point in the color point cloud data and the panoramic camera; An object segmentation module, configured to input the six-sided cube image into an object segmentation model to divide the indoor scene into a plurality of independent object regions; A semantic labeling module, configured to input each of the independent object regions into a vision-language model to obtain a semantic label for each independent object; The data fusion module is used to project and align each independent object containing semantic labels with the RGB-D sequence to obtain the aligned point cloud data; The scene reconstruction module is used to input the aligned point cloud input data into the neural core surface reconstruction model to obtain the reconstructed indoor scene; Searching for corresponding independent object results in the reconstructed indoor scene according to the received user instructions; Send the searched independent object results to the user end; If the independent object result searched out by the user terminal is inconsistent with the user instruction, a supplementary semantic label is added to the opposite object area indicated by the user instruction; The supplementary semantic tag includes at least one of a color attribute tag, a shape attribute tag, and a position attribute tag.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for indoor scene reconstruction according to any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, it performs the method for indoor scene reconstruction according to any one of claims 1 to 7.

Images