Indoor scene structure recognition method and device for complex surface

Abstract

The application discloses a kind of indoor scene structure identification method and device for complex surface, the method comprises: extracting structure inclination feature and color texture feature to the point in indoor scene three-dimensional point cloud, three-dimensional point cloud is projected to two-dimensional plane, obtains projection graph;Using horizontal, vertical decoder, structure inclination high-order feature map and color texture high-order feature map are handled, and after fusion, fusion horizontal correlation score and fusion vertical correlation score are obtained;Based on the attribution vector of adjacent pixel of fusion horizontal, vertical correlation score and attribution vector, the attribution vector of pixel is calculated, and fine superpixel attribution probability graph is obtained;Superpixel region is aggregated, and indoor scene structure surface is output.The above technical scheme is adopted, and the attribution of structure inclination and color texture is effectively fused by the decoding of horizontal and vertical directions;The attribution vector transmitted in horizontal direction and vertical direction is calculated for each pixel, the accuracy of superpixel region is improved, and the structure recognition accuracy is improved.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional scene semantic segmentation technology, and in particular to a method and apparatus for structural recognition of indoor scenes with complex surfaces. Background Technology

[0002] Early semantic segmentation relied primarily on handcrafted features and traditional machine learning methods, which achieved some success in simple scenarios but exhibited significant limitations in complex indoor environments.

[0003] With the rise of deep learning and big data technologies, automatic feature learning methods based on neural networks have gradually replaced traditional methods and made significant progress in the field of semantic segmentation, but there are also some shortcomings.

[0004] While image-based recognition methods can effectively extract two-dimensional visual features such as color and texture, they cannot utilize three-dimensional geometric information, leading to misclassification when distinguishing surfaces with similar textures but different spatial orientations. Point cloud-based recognition methods, although capable of directly processing three-dimensional data, suffer from shortcomings in local feature extraction and spatial relationship modeling, making it difficult to accurately capture the detailed features of complex surfaces. Furthermore, existing multimodal fusion methods fail to effectively coordinate the complementary relationships between different features, easily resulting in feature distortion and segmentation ambiguity at complex geometric boundaries such as wall corners and wall-to-floor junctions. In addition, the prevalence of furniture occlusion and the coexistence of multi-scale elements in indoor environments further exacerbates the problems of data gaps and feature imbalances, limiting the performance of existing technologies. Summary of the Invention

[0005] Purpose of the invention: This invention provides a method and apparatus for structural recognition in indoor scenes with complex surfaces, aiming to solve the technical problem of low accuracy in structural recognition in complex scenes due to the lack of effective coordination between color and texture features and geometric features.

[0006] Technical Solution: This invention provides a method for structural recognition in indoor scenes with complex surfaces, comprising: acquiring image sequences of an indoor scene, performing 3D reconstruction to obtain a 3D point cloud model of the indoor scene; for points in the 3D point cloud, extracting the structural tilt features and color texture features corresponding to the points, projecting the 3D point cloud onto a 2D plane to obtain a 2D projection map, wherein the points on the 2D plane have structural tilt features and color texture features based on the projection relationship, thereby obtaining a structural tilt projection map and a color texture projection map; inputting the structural tilt projection map and the color texture projection map into a dual-branch convolutional neural network, and extracting high-order feature maps of structural tilt and color texture through the encoder therein; using the horizontal decoder and vertical decoder in the dual-branch convolutional neural network for extracting horizontal feature correlation and for extracting vertical feature correlation, respectively processing the high-order feature maps of structural tilt... The image and color texture high-order feature map are processed to output the horizontal correlation score of structural tilt, the horizontal correlation score of color texture, the vertical correlation score of structural tilt, and the vertical correlation score of color texture. The horizontal correlation score of structural tilt and the horizontal correlation score of color texture are fused, and the vertical correlation score of structural tilt and the vertical correlation score of color texture are fused to obtain the fused horizontal correlation score and the fused vertical correlation score, respectively. The 2D projection map is sampled to obtain the initial superpixel assignment probability map. For the pixels in the initial superpixel assignment probability map, the pixel assignment vector is calculated based on the fused horizontal correlation score and the assignment vector of the horizontally adjacent pixels, and the pixel assignment vector is calculated based on the fused vertical correlation score and the assignment vector of the vertically adjacent pixels to obtain the refined superpixel assignment probability map. The superpixel regions in the refined superpixel assignment probability map are aggregated to output the indoor scene structure surface.

[0007] Specifically, the structural tilt feature is extracted as follows: the set of neighboring points in the 3D point cloud is extracted, and a microplane is obtained by fitting and calculating the microplane's spatial normal vector; the spatial normal vector is projected onto the horizontal plane to obtain the horizontal plane normal vector, and the angle between the horizontal plane normal vector and the X-axis of the 3D point cloud coordinate system is calculated to obtain the interior structure orientation angle; the angle between the spatial normal vector and the horizontal plane normal vector is calculated to obtain the spatial angle; the interior structure orientation and the spatial angle are mapped to the HSV color space to obtain the structural tilt feature.

[0008] Specifically, hue (H) is calculated based on the ratio between the interior structural orientation angle and 360 degrees; brightness (V) is calculated based on the ratio between the spatial angle and 90 degrees; and saturation (S) is a preset fixed value.

[0009] Specifically, the convolution kernel size of the horizontal decoder is smaller in height than in width, so that the receptive field extends in the horizontal direction; the convolution kernel size of the vertical decoder is larger in height than in width, so that the receptive field extends in the vertical direction.

[0010] Specifically, the convolution kernel of the horizontal decoder has a height of 1; the convolution kernel of the vertical decoder has a width of 1.

[0011] Specifically, the horizontal correlation scores of structural tilt and color texture are input into a learnable orientation weighting module in a two-branch convolutional neural network. The orientation weighting module weights and fuses the horizontal correlation scores of structural tilt and color texture according to the global statistical distribution of the correlation scores, and outputs a fused horizontal correlation score. The vertical correlation scores of structural tilt and color texture are input into a learnable orientation weighting module in a two-branch convolutional neural network. The orientation weighting module weights and fuses the vertical correlation scores of structural tilt and color texture according to the global statistical distribution of the correlation scores, and outputs a fused vertical correlation score.

[0012] Specifically, based on the fusion horizontal association score at the same initial resolution level as the initial superpixel assignment probability map, the fusion horizontal association score of a pixel is used as a weight to weight the assignment vectors of the left and right adjacent pixels of the fused pixel, thus calculating the pixel's assignment vector; based on the fusion vertical association score at the same initial resolution level as the initial superpixel assignment probability map, the fusion vertical association score of a pixel is used as a weight to weight the assignment vectors of the up and down adjacent pixels of the fused pixel, thus calculating the pixel's assignment vector; the assignment vectors of all pixels in the initial superpixel assignment probability map are calculated to obtain the refined superpixel assignment probability map.

[0013] Specifically, the degree of regional heterogeneity of adjacent superpixel regions is determined based on the KL divergence of the joint histogram distribution of local binary patterns and local contrast of adjacent superpixel regions, as well as the length of the common boundary between adjacent superpixel regions; and the superpixel regions are aggregated based on the degree of regional heterogeneity of adjacent superpixel regions.

[0014] Specifically, based on the degree of regional heterogeneity of adjacent superpixel regions and the size of adjacent superpixel regions, a merging cost is calculated, and a decision is made on whether to aggregate the superpixel regions based on the value of the merging cost. The merger cost is calculated using the following formula: Cost(R i ,R j =Heterogeneity(R) i ,R j (|Ri||Rj|) / (|Ri|+|Rj|), Among them, Cost(R) i ,R j ) represents the adjacent superpixel region R i and superpixel region Rj The merging cost between them, |Ri| and |Rj| represent the superpixel region R respectively. i and R j The area, Heterogeneity (R i ,R j ) represents the adjacent superpixel region R i and R j The degree of regional heterogeneity between them; Heterogeneity (R i ,R j )= D KL (P i ||P j )+γ(1-L ij / L min ), Among them, P i ||P j R represents the adjacent superpixel region i and R j The joint histogram distribution of local binary pattern and local contrast, D KL L represents the KL divergence. ij R represents the adjacent superpixel region i and R j The length of the common boundary, L min γ represents the minimum boundary length threshold, and γ represents the adjustment coefficient.

[0015] This invention also provides a structure recognition device for indoor scenes with complex surfaces, comprising: a 3D reconstruction unit, a projection unit, an encoding unit, a decoding unit, a fusion unit, a pixel attribution calculation unit, and an aggregation unit, wherein: the 3D reconstruction unit is used to acquire image sequences of the indoor scene, perform 3D reconstruction, and obtain a 3D point cloud model of the indoor scene; the projection unit is used to extract the structural tilt features and color texture features corresponding to the points in the 3D point cloud, project the 3D point cloud onto a 2D plane, and obtain a 2D projection map, wherein the points on the 2D plane have structural tilt features and color texture features based on the projection relationship, thereby obtaining a structural tilt projection map and a color texture projection map; the encoding unit is used to input the structural tilt projection map and the color texture projection map into a dual-branch convolutional neural network, and extract high-order feature maps of structural tilt and color texture through the encoder therein; the decoding unit is used to utilize the dual-branch convolutional neural network for extracting horizontal feature correlation and for extracting vertical feature correlation. The horizontal decoder and vertical decoder process the high-order feature map of structural tilt and the high-order feature map of color texture, respectively, and output the horizontal correlation score of structural tilt, the horizontal correlation score of color texture, the vertical correlation score of structural tilt, and the vertical correlation score of color texture. The fusion unit is used to fuse the horizontal correlation score of structural tilt and the horizontal correlation score of color texture, and to fuse the vertical correlation score of structural tilt and the vertical correlation score of color texture, respectively, to obtain the fused horizontal correlation score and the fused vertical correlation score. The pixel assignment calculation unit is used to sample the two-dimensional projection map to obtain an initial superpixel assignment probability map. For the pixels in the initial superpixel assignment probability map, the pixel assignment vector is calculated based on the fused horizontal correlation score and the assignment vector of the horizontally adjacent pixels, and the pixel assignment vector is calculated based on the fused vertical correlation score and the assignment vector of the vertically adjacent pixels, to obtain a refined superpixel assignment probability map. The aggregation unit is used to aggregate the superpixel regions in the refined superpixel assignment probability map and output the indoor scene structure surface.

[0016] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: Based on the structural tilt projection map and color texture projection map, the structural tilt and color texture are effectively fused by decoding in the horizontal and vertical directions, which significantly improves the segmentation accuracy of complex geometric boundaries; furthermore, by calculating the horizontal and vertical transmission vectors for each pixel on the assignment probability map one by one, the accuracy of the superpixel region is significantly improved, thereby improving the accuracy of structural recognition. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the indoor scene structure recognition method for complex surfaces provided by the present invention. Figure 2 A schematic diagram illustrating the results of applying the method provided by this invention. Detailed Implementation

[0018] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0019] See Figure 1 This is a flowchart illustrating the indoor scene structure recognition method for complex surfaces provided by the present invention.

[0020] In this embodiment of the invention, an image sequence of an indoor scene is acquired, and three-dimensional reconstruction is performed to obtain a three-dimensional point cloud model of the indoor scene.

[0021] In practice, handheld digital devices, panoramic cameras, or mobile acquisition devices can be used to acquire multi-view image sequences of indoor scenes. Using 3D reconstruction algorithms such as structure-of-motion reconstruction and multi-view stereo, a dense 3D point cloud model with realistic scale is generated. This point cloud contains the 3D coordinates (X, Y, Z) and color information (R, G, B) of each point, laying the foundation for subsequent feature extraction.

[0022] In this embodiment of the invention, for points in a three-dimensional point cloud, the structural tilt features and color texture features corresponding to the points are extracted, and the three-dimensional point cloud is projected onto a two-dimensional plane to obtain a two-dimensional projection map. The points on the two-dimensional plane have structural tilt features and color texture features based on the projection relationship, thereby obtaining a structural tilt projection map and a color texture projection map.

[0023] In this embodiment of the invention, structural tilt features are extracted as follows: A set of neighboring points in a 3D point cloud is extracted, and a microplane is obtained through fitting calculation. The spatial normal vector of the microplane is then obtained. This microplane normal vector is projected onto a horizontal plane to obtain a horizontal plane normal vector. The angle between the horizontal plane normal vector and the X-axis of the 3D point cloud coordinate system is calculated to obtain the indoor structural orientation angle (ranging from 0° to 360°). The angle between the spatial normal vector and the horizontal plane normal vector is calculated to obtain a spatial angle (ranging from 0° to 90°). The indoor structural orientation and spatial angle are mapped to the HSV color space to obtain structural tilt features.

[0024] In practice, the method for determining the microplane is to find a plane such that the sum of the squares of the distances from all neighboring points to that plane is minimized (i.e., the least squares method).

[0025] In practice, the images are projected onto the same imaging viewpoint (two-dimensional plane) using collinearity equations.

[0026] In this embodiment of the invention, the orientation of the interior structure and the included angle of the space are mapped to the HSV color space, with hue H, based on the orientation angle O of the interior structure. i The ratio between H and 360 degrees is calculated (H=O).i / 360°); Brightness V, based on the spatial angle θ i The ratio between V and 90 degrees is calculated (V=1-θ). i / 90°); Saturation S, preset to a fixed value (can be set according to actual application scenarios).

[0027] In practice, the color texture projection map, in which the color texture features are generated by directly projecting the color information (R, G, B) of each point in the point cloud.

[0028] In this embodiment of the invention, the structural tilt projection map and the color texture projection map are input into a dual-branch convolutional neural network, and the encoder therein extracts the high-order feature map of structural tilt and the high-order feature map of color texture.

[0029] In practice, a pre-trained network such as ResNet or VGG is used as the backbone, including two branches, with the two branches respectively taking the color texture projection map I as input. RGB and structural tilt projection diagram I HSV Each branch can be equipped with an encoder, which outputs a high-order feature map F of the structural tilt at the same scale. RGB and color texture high-order feature map F HSV .

[0030] In this embodiment of the invention, the horizontal decoder and vertical decoder in the dual-branch convolutional neural network, which are used to extract the horizontal feature correlation and the vertical feature correlation, are used to process the high-order feature map of structural tilt and the high-order feature map of color texture, respectively, and output the horizontal correlation score of structural tilt, the horizontal correlation score of color texture, the vertical correlation score of structural tilt and the vertical correlation score of color texture.

[0031] In this embodiment of the invention, the convolution kernel size of the horizontal decoder is smaller in height than in width, so that the receptive field extends in the horizontal direction; the convolution kernel size of the vertical decoder is larger in height than in width, so that the receptive field extends in the vertical direction.

[0032] In this embodiment of the invention, the convolution kernel of the horizontal decoder has a height of 1; the convolution kernel of the vertical decoder has a width of 1.

[0033] In practice, based on the setting of the convolution kernel size, the horizontal decoder is a long and flat rectangle that covers multiple pixels (e.g., 3) only in the horizontal direction (left and right) and only in the current row in the vertical direction; the vertical decoder is a long and thin rectangle that covers multiple pixels (e.g., 3) only in the vertical direction (up and down) and only in the current column in the horizontal direction.

[0034] In practical implementation, compared to standard 3×3 convolutions which simultaneously mix horizontal and vertical information, making it difficult for the network to distinguish the dominant direction if a region has both horizontal and vertical textures / structures, using separate horizontal and vertical decoders followed by fusion offers the advantage of allowing the network to clearly identify regions with higher (or lower) horizontal correlation scores but lower (or higher) vertical correlation scores. This allows the network to determine whether a structure is horizontally or vertically dominant, thereby improving the precision of structural segmentation.

[0035] In this embodiment of the invention, the horizontal correlation score of structural tilt and the horizontal correlation score of color texture are fused, and the vertical correlation score of structural tilt and the vertical correlation score of color texture are fused to obtain the fused horizontal correlation score and the fused vertical correlation score, respectively.

[0036] In this embodiment of the invention, the horizontal correlation score of structural tilt and the horizontal correlation score of color texture are input into a learnable directional weighting module in a two-branch convolutional neural network. The directional weighting module weights and fuses the horizontal correlation scores of structural tilt and color texture according to the global statistical distribution of the correlation scores, and outputs a fused horizontal correlation score. The vertical correlation score of structural tilt and the vertical correlation score of color texture are input into a learnable directional weighting module in a two-branch convolutional neural network. The directional weighting module weights and fuses the vertical correlation scores of structural tilt and color texture according to the global statistical distribution of the correlation scores, and outputs a fused vertical correlation score.

[0037] In practice, the learnable orientation weighting module is implemented by a lightweight network (fully connected layer). It is trained through end-to-end backpropagation, determining the weighting between structural tilt (horizontal and vertical) association scores and color texture (horizontal and vertical) association scores by examining the global statistical distribution of horizontal and vertical association scores. For example, during training, for sample pixels with a high mean structural tilt association score and a low mean color texture association score, their true labels in the training set are often dominated by structural tilt features, reflecting their actual structure. The network automatically adjusts the weighting to increase the weight of the structural tilt (horizontal and vertical) association score. In the actual decision-making process, the network assigns appropriate weights to each pixel in each input image based on the parameters determined during training.

[0038] In practice, it is important to note that each pixel has its own independent horizontal correlation score for structural tilt, horizontal correlation score for color and texture, vertical correlation score for structural tilt, and vertical correlation score for color and texture. After weighted fusion, each pixel has its own independent horizontal correlation score and vertical correlation score for fusion.

[0039] In this embodiment of the invention, the two-dimensional projection map is sampled to obtain an initial superpixel assignment probability map; for the pixels in the initial superpixel assignment probability map, the pixel assignment vector is calculated based on the fusion of the horizontal association score and the assignment vector of the horizontally adjacent pixels, and the pixel assignment vector is calculated based on the fusion of the vertical association score and the assignment vector of the vertically adjacent pixels, thus obtaining a refined superpixel assignment probability map.

[0040] In practice, the two-dimensional projection image is sampled with a step size s to generate an initial superpixel assignment probability map S0, which includes the initial superpixels. The step size s determines the approximate size of the superpixels. If the image size is H×W, the expected number of superpixels generated is approximately K=(H / s)×(W / s).

[0041] In practice, the initial superpixel attribution probability map includes multiple initial superpixels. Each pixel in the initial superpixel attribution probability map has a probability of being assigned to one or more initial superpixels.

[0042] In this embodiment of the invention, based on the fusion horizontal association score at the same initial resolution level as the initial superpixel attribution probability map, the fusion horizontal association score of a pixel is used as a weight to weight the attribution vectors of the left and right adjacent pixels of the fused pixel, thereby calculating the pixel's attribution vector; the step of calculating the pixel's attribution vector based on the fusion vertical association score and the attribution vectors of vertically adjacent pixels includes: based on the fusion vertical association score at the same initial resolution level as the initial superpixel attribution probability map, the fusion vertical association score of a pixel is used as a weight to weight the attribution vectors of the up and down adjacent pixels of the fused pixel, thereby calculating the pixel's attribution vector; the attribution vectors of all pixels in the initial superpixel attribution probability map are calculated to obtain a refined superpixel attribution probability map.

[0043] In practice, the outputs of the encoder and decoder can contain multiple feature maps at different scales. The resolution of the feature maps is determined by the scale of these feature maps. The fusion horizontal (vertical) association score, which is at the same initial resolution level as the initial superpixel assignment probability map, indicates that the resolution of the feature map used to calculate the fusion horizontal (vertical) association score is the same as the resolution of the initial superpixel assignment probability map.

[0044] In practice, horizontal (vertical) correlation scores are fused as guides for structural tilt and texture, and the propagation and assignment of pixel attribution vectors within rows and columns are performed in a fully differentiable manner, ultimately outputting a higher resolution attribution map S. l .

[0045] In practice, horizontal in-row propagation is used to calculate the attribution probability. For any pixel (i, j) to be determined in the attribution probability map, the left and right nearest pixels (i, j) in the same row of its corresponding attribution probability map are located.left ) and (i,j right (This can also be multiple). The correlation score A is determined using the current pixel's fusion level. h (i, j) are used as weights, and the attribution propagation weights are dynamically calculated using the sigmoid function σ-normalization: weight left = σ(A h (i, j)), weight right =1 - weight left (The weights of the left and right vectors can also be interchanged). Then, by weighted aggregation of the attribute vectors of the left and right adjacent pixels, the attribute vector of the current pixel is calculated: S1 h (i,j)=weight left ×S0(i, j left ) + weight righ t×S0(i, j right ).

[0046] Among them, S1 h (i,j) represents the attribution vector of pixel (i,j) after horizontal propagation, S0(i,j) left ) and S0(i,j right S0(i, j) represents the attribution vectors of the two pixels to the left and right of the current pixel. In the initial stage, that is, before the pixel has been updated and its attribution vector has been calculated through horizontal or vertical propagation, S0(i, j) represents the attribution vectors of the two pixels to the left and right of the current pixel. left ) and S0(i, j right The pixel fusion level correlation score can be used temporarily. After calculating the attribution vector, S0(i, j) left ) and S0(i, j right The pixel's attribution vector is used in the calculation. Horizontal propagation calculation enables superpixel boundaries to be intelligently extended and merged along the horizontal direction based on the lateral correlation of the image structure.

[0047] In specific implementation, the vertical column propagation is used to calculate the assignment probability. Based on the assignment probability calculated by horizontal propagation (in this embodiment, the calculation is performed first horizontally and then vertically; alternatively, the calculation can be performed first vertically and then horizontally, i.e., the assignment vector of the pixel is calculated based on the assignment vector of the pixel obtained by vertical propagation, and then the assignment vector of horizontal propagation is calculated), the resulting assignment probability map S1 is used. h Perform thinning in the vertical direction. For the same pixel (i, j), in S1 h In the middle, locate the nearest vertical pixel in the same column (i up , j) and (i down (j) (or multiple). Similarly, the vertical fusion correlation score A of the current pixel is used. v(i,j) are used as weights: weight up = σ(A v (i, j)), weight down = 1 - weight up (The weights of the up and down vectors can also be interchanged). Then, the vector belonging to the current pixel is calculated: S1(i,j)=weight up ×S1 h (i up , j) + weight down ×S1 h (i down , j).

[0048] Where S1(i,j) represents the attribution vector of pixel (i,j) after horizontal and vertical propagation, S l S1 represents a higher resolution attribution probability map. h (i up , j) and S1 h (i down , j) represent the attribution vectors (after horizontal propagation calculation) of the two pixels directly above and below the current pixel. Vertical propagation calculation allows the superpixel boundary to be precisely adjusted in the vertical direction based on structural continuity.

[0049] In practice, based on the initial assignment probability map S0 and the association scores of the same level of resolution, a higher resolution assignment probability map S1 is calculated. S1 can be used as a fine superpixel assignment probability map, or the process can be continued to iterate. Based on the assignment probability map S1 and the association scores of the same level of resolution, a higher resolution assignment probability map can be calculated according to the above calculation process.

[0050] In practice, through this cascaded operation of alternating row and column propagation, attribution information, guided by association scores, adaptively flows and merges on the image plane in a data-driven manner. Since all operations (including weighting, summation, and sigmoid) are differentiable molecular operations, the entire interpolation mechanism allows gradients to propagate smoothly from the final segmentation loss back to the front-end feature extraction network, thus achieving end-to-end joint optimization from fusing association scores to the superpixel generation model. This design enables the generated superpixels to closely fit the geometric boundaries of objects such as walls and furniture in indoor scenes, laying a solid foundation for subsequent fine semantic segmentation.

[0051] In practice, after obtaining the detailed superpixel attribution probability map, the superpixel region to which each pixel belongs can be determined.

[0052] In this embodiment of the invention, the superpixel regions in the fine superpixel attribution probability map are aggregated to output the structural surface of the indoor scene.

[0053] See Figure 2 This is a schematic diagram illustrating the result of applying the method provided by the present invention.

[0054] In this embodiment of the invention, the degree of regional heterogeneity of adjacent superpixel regions is determined based on the KL divergence of the joint histogram distribution of local binary patterns and local contrast of adjacent superpixel regions, and the length of the common boundary between adjacent superpixel regions; and the superpixel regions are aggregated based on the degree of regional heterogeneity of adjacent superpixel regions.

[0055] In this embodiment of the invention, the degree of regional heterogeneity can be calculated using the following formula: Heterogeneity (R i ,R j )= D KL (P i ||P j )+γ(1-L ij / L min ), Among them, P i ||P j R represents the adjacent superpixel region i and R j The joint histogram distribution of Local Binary Pattern (LBP) and Local Contrast (LC), D KL L represents the KL divergence (function). ij R represents the adjacent superpixel region i and R j The length of the common boundary, L min γ represents the minimum boundary length threshold, and γ represents the adjustment coefficient.

[0056] In practice, the problem of minimizing the approximation error of adjacent regions can be transformed into finding the region pair with the minimum merging cost.

[0057] In this embodiment of the invention, a merging cost is calculated based on the degree of regional heterogeneity of adjacent superpixel regions and the size of their areas. The decision to merge the superpixel regions is then based on this merging cost. Generally, if the merging cost is less than a preset threshold, adjacent superpixel regions can be merged.

[0058] In this embodiment of the invention, the merger cost is calculated using the following formula: Cost(R i ,R j =Heterogeneity(R) i,R j (|Ri||Rj|) / (|Ri|+|Rj|), Among them, Cost(R) i ,R j ) represents the adjacent superpixel region R i and superpixel region R j The merging cost between them, |Ri| and |Rj| represent the superpixel region R respectively. i and R j The area, Heterogeneity (R i ,R j ) represents the adjacent superpixel region R i and R j The degree of regional heterogeneity between them.

[0059] In practice, a multi-feature hierarchical optimization strategy is adopted. This includes: Primary merging: assigning higher weight to structural tilt features, prioritizing merging to form main structures such as walls and ceilings. Intermediate merging: increasing the weight of color and texture features to separate detailed elements such as doors, windows, and furniture from the main structures. Advanced merging: employing a multi-feature fusion similarity metric to ensure that the final segmentation boundary simultaneously satisfies geometric consistency and texture discriminability.

[0060] In practical implementation, in the formula for calculating the degree of regional heterogeneity, D KL Represents color texture features (joint histogram distribution of Local Binary Pattern (LBP) and Local Contrast (LC), (1-L) ij / L min The value originates from the boundary length, corresponding to the structural tilt feature. Therefore, in the calculation of the above multi-feature hierarchical optimization strategy, in the primary merging stage, the adjustment coefficient γ can be increased, i.e., the weight of the structural tilt feature is increased, the proportion of the common boundary length term increases, and region merging is given more weight, focusing on whether it is on the same plane, whether it is a continuous structure, whether it is a large structure such as a wall / roof / ground, etc. Color and texture differences can be tolerated, prioritizing the consistency of geometric structure; in the intermediate merging stage, the adjustment coefficient can be decreased, i.e., the weight of the color and texture features is increased, and region merging is given more weight, focusing on whether the texture is similar and whether the colors are close, ensuring texture / appearance consistency, while the geometric structure has already been prioritized for merging in the primary merging stage, and has a certain geometric structure basis, which can further ensure the accuracy of color and texture merging; in the advanced merging stage, the structural tilt feature and the color and texture feature can be set to have the same weight.

[0061] In practical implementation, the primary merging prioritizes merging the main structures such as large planes, walls, ceilings, and floors, aiming to form the main interior structure. When superpixels have aggregated into large continuous planes and the boundaries are roughly aligned with the walls / floors / ceilings, the primary merging can be considered complete. In the intermediate merging, doors, windows, furniture, decorative surfaces, etc., are separated from the existing large structure, aiming to separate detailed objects. The main structure remains unchanged, but sub-regions distinguished by texture / color appear inside, indicating that the intermediate merging is complete. In the advanced merging, the boundaries conform to the actual geometric plane without destroying the texture semantics, aiming for a final accurate structural surface. When the segmentation boundaries simultaneously satisfy geometric smoothness, correct texture, and no over-segmentation / under-segmentation, the advanced merging can be considered complete.

[0062] This invention also provides an indoor scene structure recognition device for complex surfaces, comprising: a 3D reconstruction unit, a projection unit, an encoding unit, a decoding unit, a fusion unit, a pixel attribution calculation unit, and an aggregation unit, wherein: the 3D reconstruction unit is used to acquire image sequences of the indoor scene, perform 3D reconstruction, and obtain a 3D point cloud model of the indoor scene; the projection unit is used to extract the structural tilt features and color texture features corresponding to the points in the 3D point cloud, project the 3D point cloud onto a 2D plane, and obtain a 2D projection map, wherein the points on the 2D plane have structural tilt features and color texture features based on the projection relationship, thereby obtaining a structural tilt projection map and a color texture projection map; the encoding unit is used to input the structural tilt projection map and the color texture projection map into a dual-branch convolutional neural network, and extract high-order feature maps of structural tilt and color texture through the encoder therein; the decoding unit is used to utilize the water in the dual-branch convolutional neural network for extracting horizontal feature correlation and for extracting vertical feature correlation. The flat decoder and vertical decoder process the high-order feature maps of structural tilt and color texture, respectively, and output the horizontal correlation score of structural tilt, the horizontal correlation score of color texture, the vertical correlation score of structural tilt, and the vertical correlation score of color texture. The fusion unit is used to fuse the horizontal correlation score of structural tilt and the horizontal correlation score of color texture, and to fuse the vertical correlation score of structural tilt and the vertical correlation score of color texture, respectively, to obtain the fused horizontal correlation score and the fused vertical correlation score. The pixel assignment calculation unit is used to sample the two-dimensional projection map to obtain an initial superpixel assignment probability map. For the pixels in the initial superpixel assignment probability map, the pixel assignment vector is calculated based on the fused horizontal correlation score and the assignment vector of the horizontally adjacent pixels, and the pixel assignment vector is calculated based on the fused vertical correlation score and the assignment vector of the vertically adjacent pixels, to obtain a refined superpixel assignment probability map. The aggregation unit is used to aggregate the superpixel regions in the refined superpixel assignment probability map and output the indoor scene structure surface.

[0063] In specific implementations, the methods, steps, or functions performed by the execution unit of the indoor scene structure recognition device for complex surfaces provided by the present invention can refer to the indoor scene structure recognition method for complex surfaces provided by the present invention.

Claims

1. A method for structural recognition in indoor scenes with complex surfaces, characterized in that, include: The image sequence of the indoor scene is acquired, and 3D reconstruction is performed to obtain a 3D point cloud model of the indoor scene. For points in a 3D point cloud, extract the structural tilt features and color texture features corresponding to the points, project the 3D point cloud onto a 2D plane to obtain a 2D projection map. The points on the 2D plane have structural tilt features and color texture features based on the projection relationship, thus obtaining a structural tilt projection map and a color texture projection map. The structural tilt projection map and the color texture projection map are input into a two-branch convolutional neural network, and the encoder in the network extracts the high-order feature map of structural tilt and the high-order feature map of color texture. By utilizing the horizontal decoder and vertical decoder in the dual-branch convolutional neural network for extracting horizontal feature correlation and for extracting vertical feature correlation, the high-order feature maps of structural tilt and color texture are processed respectively, and the horizontal correlation score of structural tilt, the horizontal correlation score of color texture, the vertical correlation score of structural tilt, and the vertical correlation score of color texture are output. The horizontal correlation score of structural tilt and the horizontal correlation score of color texture are fused together, and the vertical correlation score of structural tilt and the vertical correlation score of color texture are fused together to obtain the fused horizontal correlation score and the fused vertical correlation score, respectively. The two-dimensional projection map is sampled to obtain an initial superpixel assignment probability map. For the pixels in the initial superpixel assignment probability map, the pixel assignment vector is calculated based on the fused horizontal association score and the assignment vector of the horizontally adjacent pixels. The pixel assignment vector is also calculated based on the fused vertical association score and the assignment vector of the vertically adjacent pixels to obtain a refined superpixel assignment probability map. Aggregate the superpixel regions in the fine superpixel attribution probability map to output the structural surface of the indoor scene.

2. The method for indoor scene structure recognition of complex surfaces according to claim 1, characterized in that, For points in a 3D point cloud, the extraction of structural tilt features and color texture features corresponding to those points includes: Among them, the structural tilt feature is extracted: Extract the neighboring point set of points in the 3D point cloud, fit and calculate the microplane, and obtain the spatial normal vector of the microplane; Project the spatial normal vector onto the horizontal plane to obtain the horizontal plane normal vector. Calculate the angle between the horizontal plane normal vector and the X-axis of the 3D point cloud coordinate system to obtain the orientation angle of the interior structure. Calculate the angle between the spatial normal vector and the horizontal plane normal vector to obtain the spatial angle. By mapping the orientation of the interior structure and the spatial angle to the HSV color space, the structural tilt characteristics are obtained.

3. The method for indoor scene structure recognition for complex surfaces according to claim 2, characterized in that, The process of mapping the interior structure orientation and spatial angle to the HSV color space to obtain structural tilt characteristics includes: Hue (H) is calculated based on the ratio between the interior structural orientation angle and 360 degrees; Brightness (V) is calculated based on the ratio between the spatial angle and 90 degrees; Saturation (S) is a fixed value preset.

4. The method for indoor scene structure recognition for complex surfaces according to claim 1, characterized in that, The horizontal decoder has a convolution kernel size where the height is smaller than the width, so that the receptive field extends horizontally; the vertical decoder has a convolution kernel size where the height is greater than the width, so that the receptive field extends vertically.

5. The method for indoor scene structure recognition for complex surfaces according to claim 4, characterized in that, The convolution kernel of the horizontal decoder has a height of 1; the convolution kernel of the vertical decoder has a width of 1.

6. The method for indoor scene structure recognition for complex surfaces according to claim 1, characterized in that, The process of fusing the horizontal correlation score of structural tilt and the horizontal correlation score of color texture, and fusing the vertical correlation score of structural tilt and the vertical correlation score of color texture, includes: The horizontal correlation scores of structural tilt and color texture are input into the learnable orientation weighting module in the dual-branch convolutional neural network. The orientation weighting module weights and fuses the horizontal correlation scores of structural tilt and color texture according to the global statistical distribution of the correlation scores, and outputs the fused horizontal correlation score. The vertical correlation scores of structural tilt and color texture are input into the learnable orientation weighting module in the dual-branch convolutional neural network. The orientation weighting module weights and fuses the vertical correlation scores of structural tilt and color texture according to the global statistical distribution of the correlation scores, and outputs the fused vertical correlation score.

7. The method for indoor scene structure recognition of complex surfaces according to claim 1, characterized in that, The calculation of the pixel's attribution vector based on the fused horizontal correlation score and the attribution vector of horizontally adjacent pixels includes: Based on the fusion level association score at the same initial resolution level as the initial superpixel attribution probability map, the fusion level association score of the pixel is used as a weight to weight the attribution vectors of the left and right adjacent pixels of the fused pixel, and the pixel attribution vector is calculated. The calculation of the pixel's attribution vector based on the fused vertical correlation score and the attribution vector of vertically adjacent pixels includes: Based on the fused vertical correlation score at the same initial resolution level as the initial superpixel attribution probability map, the fused vertical correlation score of the pixel is used as a weight to weight the attribution vectors of the pixels above and below the fused pixel, and the pixel attribution vector is calculated. Calculate the assignment vectors of all pixels in the initial superpixel assignment probability map to obtain the refined superpixel assignment probability map.

8. The method for indoor scene structure recognition for complex surfaces according to claim 1, characterized in that, The aggregation of superpixel regions in the fine superpixel attribution probability map includes: The degree of regional heterogeneity of adjacent superpixel regions is determined by the KL divergence of the joint histogram distribution of local binary patterns and local contrast of adjacent superpixel regions, and the length of the common boundary between adjacent superpixel regions. Superpixel regions are aggregated based on the degree of regional heterogeneity among adjacent superpixel regions.

9. The method for indoor scene structure recognition for complex surfaces according to claim 8, characterized in that, The aggregation of superpixel regions in the fine superpixel attribution probability map includes: The merging cost is calculated based on the degree of regional heterogeneity of adjacent superpixel regions and the size of the adjacent superpixel regions. The value of the merging cost is then used to determine whether to aggregate the superpixel regions. The merger cost is calculated using the following formula: Cost(R i ,R j )=Heterogeneity(R i ,R j )(|Ri||Rj|) / (|Ri|+|Rj|), Among them, Cost(R) i ,R j ) represents the adjacent superpixel region R i and superpixel region R j The merging cost between them, |Ri| and |Rj| represent the superpixel region R respectively. i and R j The area, Heterogeneity (R i ,R j ) represents the adjacent superpixel region R i and R j The degree of regional heterogeneity between them; Heterogeneity(R i ,R j )= D KL (P i ||P j )+γ(1-L ij / L min ), Among them, P i ||P j R represents the adjacent superpixel region i and R j The joint histogram distribution of local binary pattern and local contrast, D KL L represents the KL divergence. ij R represents the adjacent superpixel region i and R j The length of the common boundary, L min γ represents the minimum boundary length threshold, and γ represents the adjustment coefficient.

10. A device for recognizing indoor scene structures on complex surfaces, characterized in that, include: The system comprises a 3D reconstruction unit, a projection unit, an encoding unit, a decoding unit, a fusion unit, a pixel attribution calculation unit, and an aggregation unit, among which: The three-dimensional reconstruction unit is used to acquire image sequences of indoor scenes, perform three-dimensional reconstruction, and obtain a three-dimensional point cloud model of the indoor scene. The projection unit is used to extract the structural tilt features and color texture features corresponding to the points in the three-dimensional point cloud, project the three-dimensional point cloud onto a two-dimensional plane, and obtain a two-dimensional projection map. The points on the two-dimensional plane have structural tilt features and color texture features based on the projection relationship, thereby obtaining a structural tilt projection map and a color texture projection map. The encoding unit is used to input the structural tilt projection map and the color texture projection map into a dual-branch convolutional neural network, and extract the high-order feature map of structural tilt and the high-order feature map of color texture through the encoder therein; The decoding unit is used to process the high-order feature map of structural tilt and the high-order feature map of color texture respectively using the horizontal decoder and the vertical decoder in the dual-branch convolutional neural network for extracting horizontal feature correlation and for extracting vertical feature correlation, and output the horizontal correlation score of structural tilt, the horizontal correlation score of color texture, the vertical correlation score of structural tilt and the vertical correlation score of color texture. The fusion unit is used to fuse the horizontal correlation score of structural tilt and the horizontal correlation score of color texture, and to fuse the vertical correlation score of structural tilt and the vertical correlation score of color texture, to obtain the fused horizontal correlation score and the fused vertical correlation score, respectively. The pixel attribution calculation unit is used to sample the two-dimensional projection map to obtain an initial superpixel attribution probability map; for the pixels in the initial superpixel attribution probability map, the pixel attribution vector is calculated based on the fusion of the horizontal association score and the attribution vector of the horizontally adjacent pixels, and the pixel attribution vector is calculated based on the fusion of the vertical association score and the attribution vector of the vertically adjacent pixels to obtain a refined superpixel attribution probability map. The aggregation unit is used to aggregate the superpixel regions in the fine superpixel attribution probability map and output the indoor scene structure surface.

Images