Semantic-based indoor three-dimensional layout construction method and system, terminal and storage medium

Abstract

The invention relates to the technical field of indoor space modeling, and discloses a semantic-based indoor three-dimensional layout construction method and system, a terminal and a storage medium. The method comprises the following steps: acquiring a plurality of indoor pictures and position information and direction information of corresponding shooting points; a layout estimation model is adopted to carry out layout estimation on the multiple indoor pictures, multiple layout estimation results are obtained, and each layout estimation result comprises multiple layout elements; according to the position information and the direction information, semantic tags are added to the layout elements; and by taking the same semantic tag as a splicing reference, splicing the plurality of layout elements in the same region to obtain an overall three-dimensional layout. The invention provides an indoor three-dimensional layout construction scheme which is low in cost, high in efficiency, high in automation degree and rich in semantic information in result, and effectively overcomes the defects of expensive equipment, tedious process, semantic lack, difficulty in extension and the like in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of indoor space modeling, and in particular to a semantic-based indoor three-dimensional layout construction method, system, terminal and storage medium. BACKGROUND

[0002] In the prior art, the construction of indoor three-dimensional layout mainly relies on sensor-based methods (such as laser scanning or structured light imaging) or image-based methods (such as multi-view stereo vision). These methods have the following limitations: Sensor-based methods: require expensive professional equipment (such as laser radar or depth camera), high cost and complex operation. At the same time, these methods focus on the collection of geometric data, lack of understanding of scene semantics, resulting in the generated layout model being only a point cloud or a grid, which cannot automatically identify the semantic type of layout elements (such as distinguishing between walls and windows), limiting its practicality in smart home, augmented reality (AR) or virtual reality (VR) and other applications.

[0003] Image-based methods: although the hardware cost is reduced, they usually rely on feature point matching and bundle adjustment techniques. For indoor scenes, layout elements often lack distinctive textures (such as plain walls), making feature extraction difficult, matching fails, causing reconstruction holes or distortion. In addition, when multiple pictures are spliced, due to changes in viewing angle and inconsistent lighting, cumulative errors are easily generated, requiring a lot of manual correction, resulting in slow construction speed and inability to guarantee the overall consistency of the layout.

[0004] Insufficient semantic integration: existing methods rarely integrate semantic information into the reconstruction process, resulting in low accuracy of layout element recognition and matching. For example, when splicing pictures from different angles, without semantic labels as constraints, relying only on geometric information can easily cause incorrect alignment, affecting the accuracy of the three-dimensional layout.

[0005] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0006] The main purpose of the present application is to provide a semantic-based indoor three-dimensional layout construction method, system, terminal and computer readable storage medium, which aims to solve the problems of low efficiency, accuracy and automation in the prior art of semantic-based indoor three-dimensional layout construction.

[0007] To achieve the above purpose, the present application provides a semantic-based indoor three-dimensional layout construction method, which comprises the following steps: Obtain multiple indoor pictures and corresponding position information and direction information of the shooting points; perform layout estimation on multiple indoor pictures by using a layout estimation model to obtain multiple layout estimation results, each of which comprises multiple layout elements; add a semantic label to the layout element according to the position information and the direction information; splice multiple layout elements in the same region by using the same semantic label as a splicing reference to obtain an overall three-dimensional layout.

[0008] Further, the indoor pictures are obtained by shooting with a smart phone, the position information is obtained by a WiFi fingerprint positioning method, the direction information is calculated by a built-in magnetometer and accelerometer of the smart phone, and is divided into multiple directions, including east, south, west, north, northeast, southeast, southwest, and northwest directions.

[0009] Further, the layout estimation model uses a neural network architecture based on an encoder-decoder; The encoder comprises multiple sequentially connected down-sampling modules, each of which is composed of a lightweight convolution unit and is used for multi-scale feature extraction on an input indoor picture to output feature maps with different levels of abstraction. The lightweight convolution unit is constructed based on a depth separable convolution and is used for semantic segmentation of floor, wall, and ceiling layout elements. The decoder is cascaded with the encoder and is configured to up-sample deep features output by the encoder and fuse shallow features from corresponding levels of the encoder through skip connection to recover spatial details of the feature maps and finally output a pixel-level semantic segmentation result.

[0010] Further, the semantic label has a format of position-direction-attribute, wherein the position is a three-dimensional coordinate, the direction is a shooting orientation, and the attribute is a layout element category.

[0011] Further, the splicing of multiple layout elements in the same region by using the same semantic label as a splicing reference to obtain an overall three-dimensional layout comprises: Based on the longitude and latitude of the shooting point of each indoor picture and the floor height, a spatial consistency threshold is set, indoor pictures with a Z coordinate difference within the range of a single floor height are screened out, the XY plane Euclidean distance of the shooting points between the screened indoor pictures is calculated, and indoor pictures with an XY plane Euclidean distance less than the spatial consistency threshold are clustered into the same region group. For indoor pictures belonging to the same region group, semantic layout elements contained in each indoor picture are identified, the identification results of all indoor pictures are traversed and compared, layout elements that appear in at least two indoor pictures and have the same semantic label are found out, and the found layout elements are established as reference elements for subsequent splicing. selecting one element from the reference elements to define a local world coordinate system, projecting the shooting point coordinates of each indoor picture onto the plane where the reference element is located, and estimating the transformation parameters between the local coordinate systems of each indoor picture by calculating the relative spatial relationship between the projected positions of each shooting point; generating an overall three-dimensional layout according to the transformation parameters.

[0012] Further, the generating of the overall three-dimensional layout according to the transformation parameters comprises: unifying the layout elements identified in all indoor pictures into the local world coordinate system according to the transformation parameters; aligning based on the reference elements, fusing and deduplicating the elements in the overlapping areas, and finally generating the overall three-dimensional layout.

[0013] Further, the splicing of multiple layout elements in the same area with the same semantic label as the splicing reference to obtain an overall three-dimensional layout further comprises: integrating new layout estimation results; dynamically updating and optimizing the overall three-dimensional layout according to the new layout estimation results.

[0014] In addition, to achieve the above-mentioned purposes, the present application also provides a semantic-based indoor three-dimensional layout construction system, wherein the semantic-based indoor three-dimensional layout construction system comprises: a data acquisition module configured to acquire multiple indoor pictures and corresponding position information and direction information of shooting points; a layout estimation module configured to perform layout estimation on the multiple indoor pictures by using a layout estimation model to obtain multiple layout estimation results, each of which comprises multiple layout elements; a semantic adding module configured to add semantic labels to the layout elements according to the position information and the direction information; a layout splicing module configured to splice multiple layout elements in the same area with the same semantic label as the splicing reference to obtain an overall three-dimensional layout.

[0015] In addition, to achieve the above-mentioned purposes, the present application also provides a terminal, wherein the terminal comprises a memory, a processor, and a semantic-based indoor three-dimensional layout construction program stored on the memory and executable on the processor, and the semantic-based indoor three-dimensional layout construction program, when executed by the processor, implements the steps of the above-mentioned semantic-based indoor three-dimensional layout construction method.

[0016] In addition, to achieve the above object, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a semantic-based indoor three-dimensional layout construction program, and the semantic-based indoor three-dimensional layout construction program realizes the steps of the semantic-based indoor three-dimensional layout construction method when executed by a processor.

[0017] The application has the following advantages: by acquiring indoor pictures and their corresponding spatial context (position and direction) at the acquisition end at one time, it fundamentally ensures that each picture has uniform spatial coordinates and observation angles that can be understood by a computer, provides an indispensable prerequisite for subsequent semantic addition and multi-picture fusion, and avoids matching confusion that can be caused by asynchronous data. Semantic segmentation is performed by using a layout estimation model to realize automatic and intelligent conversion from a two-dimensional image to a three-dimensional layout element, and to directly output a pixel-level result with category information. This replaces the process of relying on manual annotation or complex geometric reasoning in traditional methods, greatly improves the efficiency and automation degree of layout extraction, and the semantic segmentation result output itself provides a structured layout element that can be further processed for subsequent steps. By using information combination and semantic labeling, the layout information is upgraded from a purely geometric level to a semantic level with rich meaning, and an intelligent layout that can be understood and processed by a machine is generated. By binding the layout element with a specific spatial position and direction, a high-level semantic description such as “(east) wall surface” is formed. This makes the generated layout not just a collection of points, lines and surfaces, but a digital spatial model containing rich information such as “what”, “where” and “which way”, greatly expanding its application value. Using the same semantics as the basis for matching and splicing replaces the complex geometric feature matching algorithm in traditional methods, significantly reduces the computational complexity and the probability of matching errors, and makes the splicing process simpler and more reliable. In summary, the application provides an indoor three-dimensional layout construction scheme with low cost, high efficiency, high automation degree and rich semantic information, effectively overcoming the defects of expensive equipment, complicated process and lack of semantics in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a flowchart of a preferred embodiment of the semantic-based indoor three-dimensional layout construction method of the application; Figure 2 is a specific flowchart of a preferred embodiment of the semantic-based indoor three-dimensional layout construction method of the application; Figure 3 is a schematic diagram of layout estimation and semantic addition of indoor pictures in the semantic-based indoor three-dimensional layout construction method of the application; Figure 4 is a schematic diagram of indoor picture layout and semantic estimation for splicing in the semantic-based indoor three-dimensional layout construction method of the application; Figure 5 is a schematic diagram of two indoor picture layout splicing in the semantic-based indoor three-dimensional layout construction method of the present application; Figure 6 is a structure diagram of a preferred embodiment of the semantic-based indoor three-dimensional layout construction system of the present application; Figure 7 is a structure diagram of a preferred embodiment of the terminal of the present application. DETAILED DESCRIPTION

[0019] The present application provides a semantic-based indoor three-dimensional layout construction method, system, terminal and storage medium. In order to make the purpose, technical scheme and effect of the present application more clear and explicit, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0020] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood as having meanings consistent with those in the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such.

[0021] In addition, if the present application embodiments involve descriptions such as "first", "second", etc., the descriptions of "first", "second", etc. are only for the purpose of description, and should not be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can be explicitly or implicitly included at least one of the features. In addition, the technical solutions of various embodiments can be combined with each other, but it must be based on the realization of ordinary skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, and is not within the scope of protection required by the present application.

[0022] The semantic-based indoor three-dimensional layout construction method of the preferred embodiment of the present application, as shown in Figure 1 and 2 The semantic-based indoor three-dimensional layout construction method comprises the following steps: S10, acquiring multiple indoor pictures and corresponding position information and direction information of the shooting points.

[0023] It should be noted that step S10 aims to synchronously collect multi-element data of indoor environment through a smart phone.

[0024] Further, the indoor picture is obtained by shooting with a smart phone; specifically: The user uses the camera of the smart phone to shoot an RGB image of the indoor scene. When shooting, make sure that the image clearly covers the main layout elements (such as walls, floors, and ceilings), and the resolution is recommended to be 256*256 pixels or higher to facilitate subsequent layout estimation.

[0025] Further, the position information is obtained by a WiFi fingerprint positioning method, which is divided into two stages of offline collection and online matching; specifically: Offline collection stage: A fingerprint database is established in advance. The target indoor area is divided into uniform grids (the grid spacing is recommended to be 0.5-2 meters), and sampling points are marked at each grid intersection, and their three-dimensional coordinates (x, y, z) are recorded. The WiFi module of the smart phone is used to stay at each sampling point for 3-10 seconds, and the MAC address and RSSI (Received Signal Strength Indicator) value of the surrounding WiFi hotspots are collected multiple times. Finally, it is stored as a fingerprint record, which includes the following fields: sampling point ID, physical coordinates, WiFi MAC address, RSSI average, and collection time. The establishment of the fingerprint database needs to consider environmental interference, and the positioning accuracy of 1-3 meters can usually be achieved.

[0026] Online matching stage: The user is at any position in the indoor, and the smart phone scans the WiFi signal strength vector (format {(MAC1, RSSI1), (MAC2, RSSI2),...}) in real time. The K-Nearest Neighbor (KNN) matching algorithm is adopted: the Euclidean distance between the real-time vector and all sampling points in the database is calculated, and the K closest sampling points (K=3-5) are selected, and their coordinates are weighted and averaged with the distance reciprocal as the weight, to obtain the current position (x, y, z). At the same time, the floor is determined by the barometric pressure sensor in the phone (each floor height is set to 3 meters, for example, Z=0 is the first floor, and Z=3 is the second floor), to ensure that the coordinates contain floor information.

[0027] Further, the direction information is calculated by the built-in magnetometer and accelerometer of the smart phone, and is divided into multiple directions. Among them, the directions include east, south, west, north, northeast, southeast, southwest, and northwest directions.

[0028] In this embodiment, the direction information is calculated by the built-in magnetometer and accelerometer of the smart phone. The specific process is as follows: 1. Initialize the sensor service, register the listeners of the magnetometer and accelerometer, and obtain the X, Y, Z axis original data in real time.

[0029] 2. Filter and preprocess the original data (such as low-pass filtering), to remove noise.

[0030] 3. Calculate the pitch and roll of the device using accelerometer data to determine the device tilt state; then tilt compensate the magnetometer data to eliminate bias.

[0031] 4. Generate a 3x3 rotation matrix based on the compensated data to describe the mapping relationship between the device coordinate system and the geographic coordinate system (east, north, and sky directions).

[0032] 5. Extract the azimuth angle (0°~360°, 0° for north) from the rotation matrix and divide the direction according to the angle: east (67.5°~112.5°), south (157.5°~202.5°), west (247.5°~292.5°), north (337.5°~22.5°), northeast (22.5°~67.5°), southeast (112.5°~157.5°), southwest (202.5°~247.5°), northwest (292.5°~337.5°). The direction accuracy in a static scene can reach ±5°~10°.

[0033] Through this step, each indoor picture is associated with a position coordinate (such as (10, 10, 9)) and a direction (such as east), laying the foundation for subsequent semantic addition.

[0034] S20, a layout estimation model is used to estimate the layout of multiple indoor pictures, and multiple layout estimation results are obtained, each of which includes multiple layout elements.

[0035] Further, the layout estimation model uses an encoder-decoder based neural network architecture, wherein the encoder comprises a plurality of sequentially connected down-sampling modules, each of which is composed of a lightweight convolution unit, for multi-scale feature extraction of the input indoor picture to output feature maps with different levels of abstraction; The lightweight convolution unit is constructed based on depth separable convolution, which is used for semantic segmentation of floor, wall, and ceiling layout elements; The decoder is cascaded with the encoder and is configured to up-sample the deep features output by the encoder and fuse shallow features from the corresponding levels of the encoder through skip connection to restore the spatial details of the feature maps, and finally output pixel-level semantic segmentation results.

[0036] In this embodiment, the specific implementation of each part of the layout estimation model is as follows: Encoder part: input RGB image (e.g. 3x256x256), extract multi-scale features by multiple down-sampling modules (DSBlock). Each DSBlock adopts a dual-branch structure: one branch uses 2x2 max pooling to preserve detail features (e.g. corner edges), another branch uses 3x3 convolution with stride 2 for semantic down-sampling. The two branches' outputs are fused by 1x1 convolution, realizing channel number doubling (e.g. 64→128) and resolution halving (e.g. 128x128→64x64). Alternately stack lightweight convolution units (LFBlock) and DSBlock, LFBlock is based on depthwise separable convolution, containing 3x3 dilation convolution (dilation=2 / 4) to expand receptive field, followed by 1x1 convolution, BN, ReLU activation and residual connection, finally reducing feature map resolution to 16x16 and channel number to 128 to capture global layout semantics.

[0037] Decoder part: up-sample the deep features output by encoder, realized by up-sampling modules (USBlock). USBlock uses 3x3 transpose convolution with stride 2 to restore low-resolution feature maps to original size 2 times (e.g. 32x32→64x64), and cooperates with LFBlock to refine edge details. Decoder adopts a dual-branch design: branch 1 outputs 1-channel depth map (auxiliary geometric constraint), branch 2 outputs 15-channel feature map, processed by fusion module (Fusion_block). Fusion_block compresses features to layout class number (e.g. 5 classes: ground, ceiling, left wall, right wall and back wall) by 1x1 convolution, then outputs pixel-level semantic segmentation results by SoftMax classifier.

[0038] Model training: use public dataset (LSUN) for training. Loss function is multi-task loss: semantic segmentation branch uses multi-class cross-entropy loss, edge heat map branch uses binary cross-entropy loss, total loss expression is: ; where, is the total loss function, which is the overall objective function of model training, by minimizing this value to optimize network parameters, so that the model learns accurate semantic segmentation and clear boundary details at the same time; is the semantic segmentation loss function, which uses multi-class cross-entropy loss, it calculates the difference between the model's predicted pixel-level class (e.g. ground, wall, ceiling) and the true label, ensuring that the main area of layout elements is correctly classified, which is the main part of the loss function, directly determines the global accuracy of layout estimation; where, is the edge loss function, which adopts binary cross-entropy loss and focuses on the boundaries of the predicted layout elements (such as the junction of the wall and the ground), and improves the boundary accuracy and continuity of the segmentation result by comparing the edge heat map output by the model with the real edge label; the constant 0.5 is a weight coefficient for balancing the relative importance of the two loss terms.

[0039] It should be noted that the public dataset is a large-scale real-world scene image dataset containing multiple daily scene categories and widely used in research and model training of computer vision tasks such as scene understanding, image generation and target detection.

[0040] The optimizer adopts Adam (lr=1e-4, β1=0.9), the learning rate is polynomially decayed (power=0.9), and the batch size is 32. The training is performed for 120 cycles, the bottom layer of the encoder is frozen for the first 30 cycles, the whole network is fine-tuned for the last 90 cycles, and the early stopping mechanism is enabled (if the average intersection over union (mIoU) of the model on the validation set does not improve for 15 consecutive training rounds, the training is terminated in advance).

[0041] where, lr is the initial learning rate, the learning rate of the embodiment is gradually reduced by using a polynomial strategy; β1 is a first-order momentum decay coefficient, power is the decay index of the learning rate using the polynomial strategy, and batch size is the number of samples per training batch.

[0042] Through this step, a single indoor picture can output a layout estimation result, as shown in Figure 3 The figure shows an example of the original picture, the layout estimation result and the semantic addition.

[0043] S30, according to the position information and the direction information, adding a semantic label to the layout element.

[0044] Further, the format of the semantic label is position-direction-attribute, wherein the position is a three-dimensional coordinate, the direction is a shooting direction, and the attribute is a layout element category.

[0045] The specific description is as follows: Position: three-dimensional coordinate (x, y, z), taking a point of the building (such as the entrance) as the origin, and the Z coordinate represents the floor.

[0046] Direction: shooting direction, such as east, northeast, etc.

[0047] Attribute: layout element category, such as ground, wall, ceiling.

[0048] For example, a user takes a picture facing northeast at location (20, 15, 6), the layout estimation identifies elements such as the right wall, ceiling, etc. With the direction information resolution (e.g. azimuth 45° corresponds to northeast), add semantic label to the wall: northeast-wall. The label adding process is as follows: 1. Parse the layout segmentation result, extract the pixel area of each layout element.

[0049] 2. According to the shooting direction, map the direction attribute to the layout element (e.g. the wall direction is consistent with the shooting direction).

[0050] 3. Combine the location coordinates to generate a complete semantic label, such as (20, 15, 6)-north-wall.

[0051] This step enables the layout element to have semantic context, facilitating subsequent matching, as shown in Figure 3 . Among them, the layout element after adding semantics is marked as a directional attribute.

[0052] S40, based on the semantic label, match the layout elements of other indoor pictures, and splice to construct the overall three-dimensional layout.

[0053] Further, the same semantic label is used as the splicing reference to splice multiple layout elements in the same area, and the overall three-dimensional layout is obtained, including the following steps: S41, based on the latitude and longitude (XY coordinates) of the shooting point of each indoor picture and the floor height (Z coordinate), set a spatial consistency threshold, select indoor pictures with a Z coordinate difference within a single floor height range, calculate the XY plane Euclidean distance between the shooting points of the selected indoor pictures, and cluster indoor pictures with an XY plane Euclidean distance less than the spatial consistency threshold to the same area group. Specifically as follows: Based on the Z coordinate to verify the floor consistency (e.g. Z=6 represents the same floor), calculate the Euclidean distance of XY coordinates. Set the distance threshold (e.g. 3 meters), if the distance between two points is less than or equal to the threshold, it is determined as the same area. The threshold setting logic is based on the indoor space scale: the length of an ordinary room is 3-6 meters, and the threshold of 3 meters can cover the same room or functional area; the threshold can be adjusted in actual application to adapt to the scene.

[0054] S42, for indoor pictures belonging to the same area group, identify the semantic layout elements (such as walls, doors and windows, fixed furniture, etc.) contained in each indoor picture, traverse and compare the identification results of all indoor pictures, find out the layout elements that appear in at least two indoor pictures and have the same semantic label, and determine the found layout elements (e.g. a common load-bearing wall or a complete long wall) as the reference elements for subsequent splicing.

[0055] For example, both pictures contain the label: north-wall, then use it as the matching point.

[0056] S43. Select the most stable and longest element (such as the main wall) from the reference elements to define a local world coordinate system. Project the coordinates of the shooting points of each indoor image onto the plane where the reference element is located. By calculating the relative spatial relationship (such as relative distance and angle) between the projected positions of each shooting point, estimate the transformation parameters (rotation and translation) between the local coordinate systems of each indoor image.

[0057] S44. Generate the overall three-dimensional layout based on the transformation parameters. Specifically, this includes: Based on the transformation parameters, all layout elements identified in the indoor images are uniformly transformed to the local world coordinate system; Alignment is performed based on the reference elements, and elements in overlapping areas are merged and deduplicated to finally generate the overall 3D layout.

[0058] For example, Figure 3 and Figure 4 The layout estimation results all include the north wall. The projected distance between the two shooting points on the north wall (e.g., 1.8 meters) is calculated as the splicing reference. After alignment, the layout elements are merged through geometric transformations (e.g., translation and rotation), and the ceiling and floor areas are completed.

[0059] As can be seen, step S40 achieves automatic stitching of multiple image layouts through semantic tags.

[0060] Furthermore, the step of using the same semantic tag as a splicing benchmark to splice multiple layout elements within the same area to obtain an overall three-dimensional layout further includes: Incorporating new layout estimation results, such as Figure 5 As shown; Based on the new layout estimation results, the overall three-dimensional layout is dynamically updated and optimized.

[0061] It should be noted that this invention supports incorporating layout estimation results from new images. When a new image is added, steps S41-S43 are repeated to optimize and supplement the existing layout, such as filling holes or correcting boundaries, to achieve iterative improvement of the layout.

[0062] In summary, the beneficial effects of the embodiments of the present invention are at least as follows: The present application patent realizes multiple significant beneficial effects through fusing intelligent mobile phone multi-source data, deep learning layout estimation and semantic driven stitching. Firstly, the ordinary intelligent mobile phone and WiFi fingerprint positioning technology are used to greatly reduce the hardware cost and improve the universality of the scheme, so that the indoor layout construction can be widely applied to personal or small and medium-sized scenarios. Secondly, the lightweight deep learning model based on encoder-decoder realizes efficient automatic layout estimation, the processing speed is fast and the parameter quantity is reduced by 60%, which overcomes the bottleneck of complex data processing of traditional methods. In addition, through multi-source information fusion (such as direction solving and semantic label adding), the layout element alignment accuracy is high, the direction information solving accuracy is ± 5°~10°, such as shown in Figure 4 , which ensures the robustness and reliability of the layout. More importantly, semantic integration deepens scene understanding in the form of position-direction-attribute label, so that the layout has explainability and supports path planning or smart home and other advanced applications. At the same time, the dynamic updating mechanism allows new pictures to be integrated into the existing layout, such as shown in Figure 5 , which realizes iterative improvement through semantic matching and adapts to space changes. Overall, the present application not only improves the construction efficiency and accuracy, but also expands the application potential in the fields of intelligent buildings, AR / VR and the like.

[0063] Further, as shown in Figure 6 , based on the above-mentioned semantic-based indoor three-dimensional layout construction method, the present application also correspondingly provides a semantic-based indoor three-dimensional layout construction system, wherein the semantic-based indoor three-dimensional layout construction system comprises: a data acquisition module 51 for acquiring multiple indoor pictures and corresponding position information and direction information of shooting points; a layout estimation module 52 for performing layout estimation on the multiple indoor pictures by using a layout estimation model to obtain multiple layout estimation results, each of which comprises multiple layout elements; a semantic adding module 53 for adding semantic labels to the layout elements according to the position information and the direction information; a layout stitching module 54 for matching the layout elements of other indoor pictures based on the semantic labels, and stitching to construct an overall three-dimensional layout.

[0064] Further, as shown in Figure 7 , based on the above-mentioned semantic-based indoor three-dimensional layout construction method and system, the present application also correspondingly provides a terminal, which comprises a processor 10, a memory 20 and a display 30. Figure 7 Only part of the components of the terminal are shown, but it should be understood that all the shown components are not required to be implemented, and more or fewer components can be alternatively implemented.

[0065] The memory 20 can be an internal storage unit of the terminal in some embodiments, such as a hard disk or a memory of the terminal. The memory 20 can also be an external storage device of the terminal in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 20 can include both an internal storage unit and an external storage device of the terminal. The memory 20 is used to store application software installed on the terminal and various types of data, such as program codes of the installed terminal, etc. The memory 20 can also be used to temporarily store data that has been output or will be output. In an embodiment, the memory 20 stores a semantic-based indoor three-dimensional layout construction program 40, which can be executed by the processor 10 to implement the semantic-based indoor three-dimensional layout construction method of the present application.

[0066] The processor 10 can be a Central Processing Unit (CPU), a microprocessor or other data processing chip in some embodiments, which is used to run program codes stored in the memory 20 or process data, such as to execute the semantic-based indoor three-dimensional layout construction method, etc.

[0067] The display 30 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. in some embodiments. The display 30 is used to display information of the terminal and to display a visualized user interface. The components of the terminal communicate with each other through a system bus.

[0068] The present application also provides a computer readable storage medium, wherein the computer readable storage medium stores a semantic-based indoor three-dimensional layout construction program, which, when executed by a processor, implements the steps of the semantic-based indoor three-dimensional layout construction method as described above.

[0069] It should be noted that, as used in this document, the terms "includes" and / or "containing" or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements is not limited to those elements, but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the identified element.

[0070] Of course, those of ordinary skill in the art will recognize that all or portions of the methods of implementing the above-described embodiments can also be embodied in the form of computer executable instructions stored in a non-transitory computer readable storage medium, which instructions, when executed by a computer, implement the methods of the above-described embodiments. As used in the present disclosure, the term "non-transitory computer readable storage medium" encompasses only a computer readable medium that does not store data permanently over time. It is understood that the terms "some" and "at least" are defined with reference to the claims.

[0071] It should be understood that the application is not limited to the examples described above, which can be modified or transformed by those of ordinary skill in the art in light of the above description, and all such modifications and transformations are intended to fall within the scope of the appended claims of the present application.

Claims

1. A semantic-based method for constructing indoor 3D layouts, characterized in that, The semantic-based indoor 3D layout construction method includes the following steps: Acquire multiple indoor images along with the location and orientation information of the corresponding shooting points; A layout estimation model is used to estimate the layout of multiple indoor images to obtain multiple layout estimation results, each of which includes multiple layout elements. Based on the location information and the direction information, semantic tags are added to the layout elements; Using the same semantic tag as the splicing benchmark, multiple layout elements within the same area are spliced ​​together to obtain an overall three-dimensional layout.

2. The semantic-based indoor 3D layout construction method according to claim 1, characterized in that, The indoor images are captured by a smartphone, the location information is obtained through WiFi fingerprint positioning, and the direction information is calculated by the magnetometer and accelerometer built into the smartphone and divided into multiple directions, including east, south, west, north, northeast, southeast, southwest and northwest.

3. The semantic-based indoor 3D layout construction method according to claim 1, characterized in that, The layout estimation model employs an encoder-decoder based neural network architecture; The encoder includes multiple downsampling modules connected in sequence. Each downsampling module is composed of a lightweight convolutional unit, which is used to extract multi-scale features from the input indoor image to output feature maps with different levels of abstraction. The lightweight convolutional unit is built based on depthwise separable convolution and is used to semantically segment the layout elements of the ground, walls and ceiling. The decoder is cascaded with the encoder and configured to upsample the deep features output by the encoder and fuse the shallow features from the corresponding layer of the encoder through jump connections to restore the spatial details of the feature map and finally output pixel-level semantic segmentation results.

4. The semantic-based indoor 3D layout construction method according to claim 1, characterized in that, The semantic tag is formatted as position-direction-attribute, where position is three-dimensional coordinates, direction is the shooting orientation, and attribute is the layout element category.

5. The semantic-based indoor 3D layout construction method according to claim 4, characterized in that, The step of using the same semantic tag as a splicing benchmark to splice multiple layout elements within the same area to obtain an overall three-dimensional layout includes: Based on the latitude, longitude, and floor height of the shooting point of each indoor image, a spatial consistency threshold is set to filter out indoor images whose Z coordinate difference is within the range of a single floor height. The XY plane Euclidean distance between the shooting points of the filtered indoor images is calculated, and indoor images whose XY plane Euclidean distance is less than the spatial consistency threshold are clustered into the same region group. For indoor images belonging to the same area group, the semantic layout elements contained in each indoor image are identified. The identification results of all indoor images are traversed and compared. The layout elements that appear simultaneously in at least two indoor images and have the same semantic label are found. The identified layout elements are established as the reference elements for subsequent splicing. Select one element from the reference elements to define a local world coordinate system, project the coordinates of the shooting points of each indoor image onto the plane where the reference element is located, and estimate the transformation parameters between the local coordinate systems of each indoor image by calculating the relative spatial relationship between the projected positions of each shooting point. The overall three-dimensional layout is generated based on the transformation parameters.

6. The semantic-based indoor 3D layout construction method according to claim 5, characterized in that, The step of generating the overall three-dimensional layout based on the transformation parameters includes: Based on the transformation parameters, all layout elements identified in the indoor images are uniformly transformed to the local world coordinate system; Alignment is performed based on the reference elements, and elements in overlapping areas are merged and deduplicated to finally generate the overall 3D layout.

7. The semantic-based indoor 3D layout construction method according to claim 6, characterized in that, The process involves using the same semantic tag as a splicing benchmark to splice multiple layout elements within the same area to obtain an overall three-dimensional layout, and then further includes: Incorporate new layout estimation results; Based on the new layout estimation results, the overall three-dimensional layout is dynamically updated and optimized.

8. A semantic-based indoor 3D layout construction system, characterized in that, The semantic-based indoor 3D layout construction system includes: The data acquisition module is used to acquire multiple indoor images and the corresponding location and orientation information of the shooting points; The layout estimation module is used to perform layout estimation on multiple indoor images using a layout estimation model to obtain multiple layout estimation results, each of which includes multiple layout elements. A semantic addition module is used to add semantic tags to the layout elements based on the location information and the direction information; The layout splicing module is used to splice multiple layout elements in the same area using the same semantic tag as the splicing reference to obtain an overall three-dimensional layout.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a semantic-based indoor 3D layout construction program stored in the memory and executable on the processor. When the semantic-based indoor 3D layout construction program is executed by the processor, it implements the steps of the semantic-based indoor 3D layout construction method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a semantic-based interior 3D layout construction program, which, when executed by a processor, implements the steps of the semantic-based interior 3D layout construction method as described in any one of claims 1-7.

Images