A grating image-based house layout identification and three-dimensional reconstruction method and system

Abstract

A grating image-based house layout identification and three-dimensional reconstruction method and system belong to the technical field of house layout image identification.The present application comprises: step 1, collecting and screening high-quality layout maps suitable for the national conditions of China to construct a grating layout map vector dataset; step 2, identifying the walls, doors and windows and scale in the layout map based on a key point detection network, generating and screening candidate primitives through axial alignment rules and constraints; step 3, detecting and positioning the scale end points in a Yolov8 and Shi-Tomasi corner detection combined manner, and recognizing the scale digital characters through a pre-trained multi-modal optical character recognition model OFA-OCR; step 4, designing and implementing a layout identification and three-dimensional reconstruction system based on a WebGL framework, supporting the rendering of the walls and doors and windows after reconstruction, the user interaction with the reconstructed layout, the forward and backward functions of the layout operation history record.

Description

TECHNICAL FIELD

[0001] The application relates to a house layout identification and three-dimensional reconstruction method and belongs to the technical field of house layout image identification. BACKGROUND

[0002] House layout identification and three-dimensional reconstruction technology has important research and application value in related fields such as architectural design and interior design. Traditional house layout identification and three-dimensional reconstruction methods usually require manual participation and complex measurement work, or even use a laser radar to scan the entire layout to obtain 3D point cloud data. Such reconstruction work is not only time-consuming and laborious, but also depends on existing buildings in many cases, cannot be reconstructed in advance, and is prone to human error and low accuracy. SUMMARY

[0003] The application adopts the technical scheme for solving the technical problems proposed above:

[0004] A house layout identification and three-dimensional reconstruction method based on a raster image comprises the following steps: 1. Collect and screen high-quality layout pictures suitable for the national conditions in China to construct a raster layout picture vector data set; 2. Identify the walls, doors and windows and scale in the layout picture based on a key point detection network, generate and screen candidate primitives through axial alignment rules and constraint conditions; 3. Detect and locate the scale end points in a Yolov8 and Shi-Tomasi corner detection combined manner, and identify the scale digital characters through a pre-trained multi-modal optical character recognition model OFA-OCR; and 4. Design and implement a layout identification and three-dimensional reconstruction system based on a WebGL framework, support the rendering of the walls and doors and windows after reconstruction, the interaction of the user with the reconstructed layout, the forward and backward functions of the layout operation history record.

[0005] A house layout identification and three-dimensional reconstruction system based on a raster image is realized based on a house layout identification and three-dimensional reconstruction method based on a raster image, and comprises the following modules: a layout identification and reconstruction module for receiving a layout picture file uploaded by a user, identifying and reconstructing the 2D layout picture file uploaded by the user, and generating a 3D layout view; a layout wall and door and window interaction module for adding, deleting and modifying the walls, doors and windows in the 2D and 3D views; and a layout design history record module for recording the design operation history of the user, supporting the undo, redo, save and load design schemes.

[0006] Preferably, the house type identification and reconstruction module comprises: a file uploading unit configured to receive a house type drawing file uploaded by a user, support.png,.jpg,.jpeg formats, and the file size is not more than 2MB, and the resolution is not more than 1920x1080; an image processing unit configured to send the house type drawing file uploaded by the user to a back end for identification and reconstruction, and generate 2D and 3D house type views; a scale identification and adjustment unit configured to identify a scale in the house type drawing, and allow a user to adjust the position and length of the scale; and a view switching unit configured to switch between the 2D and 3D views.

[0007] Preferably, the image processing unit realizes house type identification and reconstruction through the following steps: the house type drawing file uploaded by the user is packaged into a FormData object and sent to the back end; the back end saves the house type drawing file on an OSS, and forwards a picture address and a task ID to a message queue; the front end regularly queries a task state according to the task ID, and returns an identification result and loads the house type drawing in a 2D Canvas after the back end completes the house type identification; and a 3D reconstruction view is generated according to the identification result, and displayed in the upper right corner.

[0008] Preferably, the house type wall and door and window interaction module comprises: a wall editing unit configured to add, delete and merge walls in the 2D view, and support adjusting the length and position of the wall through dragging; a door and window editing unit configured to add, delete and modify doors and windows in the 2D view, and the position of the door and window must be limited to the wall; and a 3D view synchronization unit configured to synchronously update the operation in the 2D view to the 3D view.

[0009] Preferably, the wall editing unit supports the following operations: when a wall is deleted, the doors and windows on the wall are automatically deleted, and adjacent walls are redrawn; when walls are merged, adjacent axial aligned walls are merged into a same wall; and the length and position of the wall are adjusted by dragging a round dot interaction area at both ends of the wall.

[0010] Preferably, the door and window editing unit supports the following operations: a plurality of types of doors and windows are provided for a user to select; a door or window is added on a wall by clicking a mouse, and the length of the door or window is adjusted by dragging; and the geometry and semantic constraints of the door and window limit that the door and window can only be added and modified on the wall.

[0011] Preferably, the house type design history record module comprises: a history operation record unit configured to record the design operation history of a user, support undo and redo operations; a design scheme saving unit configured to serialize the design scheme of the user and save to a back end database; and a design scheme loading unit configured to load the design scheme saved by the user, and support deleting the saved design scheme.

[0012] The application has the beneficial effects that:

[0013] Raster images are two-dimensional plane images obtained by devices such as cameras or laser scanners, which contain a large amount of information about the structure and layout of the house. Due to its low price, convenient dissemination and vivid image characteristics, house plan raster images are widely used in daily life. The recognition and reconstruction technology based on raster images provides a more convenient and automated method for house plan recognition and three-dimensional reconstruction, which meets people's expectations. By analyzing the lines, corners and textures in the raster image, the house plan information such as the location, size and connection relationship of the room can be inferred. At the same time, by using the perspective changes of multiple raster images, three-dimensional reconstruction of the house can be realized, and a geometric structure house model can be generated. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a schematic diagram of wall key points; Figure 2 is a schematic diagram of door key points; Figure 3 is a schematic diagram of window key points; Figure 4 is an example of a house plan raster image; Figure 5 is an example of house plan primitive annotation data; Figure 6 is an example of scale line end point annotation process; Figure 7 is a scale line end point annotation data file; Figure 8 is a wall key point category definition diagram; Figure 9 is a ConvNeXt network block structure diagram; Figure 10 is an FPN structure diagram; Figure 11 is a PANet network structure; Figure 12 is a BiFPN network structure diagram; Figure 13 is a BiFPN network structure diagram based on spatial attention; Figure 14 is a spatial attention module structure diagram; Figure 15 is a schematic diagram of the maximum bounding box of the house area; Figure 16 is a schematic diagram of scale end point area detection; Figure 17 is a schematic diagram of scale end point corner detection; Figure 18 is a schematic diagram of OFA-OCR scale number recognition result; Figure 19 is a schematic diagram of a house plan recognition system architecture; Figure 20 is a schematic diagram of a house plan recognition and reconstruction process; Figure 21 is a schematic diagram of a house plan recognition and reconstruction module timing diagram; Figure 22 is a schematic diagram of a user wall and door and window interaction process; Figure 23 is a schematic diagram of a wall and door and window interaction module timing diagram; Figure 24 is a schematic diagram of a history record saving process; Figure 25 is a schematic diagram of a history record restoring process; Figure 26 is a schematic diagram of a new and old scene tree comparison algorithm process; Figure 27 is a scale setting interface; Figure 28is the reconstruction result of the house type under 2D perspective; Figure 29 is the reconstruction result of the house type under 3D perspective; Figure 30 is the house type background transparency adjustment diagram; Figure 31 is the three-dimensional reconstruction result of the common house type; Figure 32 is the selected target wall schematic diagram; Figure 33 is the wall deletion schematic diagram; Figure 34 is the axial dragging of the wall edge; Figure 35 is the wall dragging process along the cross axis; Figure 36 is the wall dragging end state diagram; Figure 37 is the addition of sliding door schematic diagram; Figure 38 is the historical record operation interface after dragging the wall; Figure 39 is the historical record state before the drag is restored; Figure 40 is the user login page; Figure 41 is the persistent history design record. DETAILED DESCRIPTION

[0015] Embodiment one: the embodiment discloses a house type identification and three-dimensional reconstruction method based on a raster image, including the following steps:

[0016] Step 1. Collect and screen high-quality house type diagrams suitable for the national conditions of China, and construct a raster house type diagram vector dataset;

[0017] The house type plane picture data is derived from Internet pictures. After artificial screening, more than 5000 house type plane pictures are selected as house type data (3600 house type diagrams as a training set, and the remaining 1400 pictures as a test set) by comprehensive consideration. The geometric information of each plane picture image is annotated by manual annotation method, and the lines representing walls and doors and windows are labeled. In the training, the dataset is labeled and the positions of the key points of each type are calculated in real time to construct a Ground-truth heat map representation to train the network, and the annotation information is converted into a connection layer representation, as follows:

[0018] (1) For door and window key point objects, directly read the position points of both ends of the corresponding door and window category in the annotation, and determine the direction information of the position points of both ends to determine the key point category at the position of the door and window end point;

[0019] (2) For wall key points, they are usually calculated from multiple wall elements, so the wall intersection points are determined by calculating the local connection of the wall, that is, whether there are other walls on the upper, lower, left and right sides of the wall is checked, and the results are de-duplicated to accurately calculate the category and position of each wall key point; the results are shown in Figures 1-3 , wherein Figure 1 is the wall key point, Figure 2 is the door key point, Figure 3 is the window key point, and inFigures 1-3 In this context, each key point not only expresses location information but also carries semantic information about its category.

[0020] Floor plan and its labeled data as follows Figure 4 and Figure 5 As shown, each row represents a wall or door / window primitive. The first four columns represent the positions of the two endpoints of the primitive relative to the image dataset. The fifth column represents the type of primitive: wall represents a wall, opening represents a window, and door represents a door. The sixth and seventh columns are reserved categories. The sixth column is the major category of walls and doors / windows, reserving some interior and exterior walls for walls, and some subcategories for doors and windows, such as single doors or double doors. The seventh column is the orientation information of the door.

[0021] The dataset of endpoint regions for floor plan icons was annotated in YOLO format. The endpoints of the scale area for each floor plan icon were manually annotated using the MakeSense annotation tool, with only one category for scale endpoints. The annotation process can be found in [link to annotation process]. Figure 6 Use a bounding box to select the endpoints of the scale lines on the floor plan, and then make fine adjustments and corrections. The data file for the scale line endpoint annotations can be found here. Figure 7 ;exist Figure 7 In the image, the first column represents the target category. In this task, there is only one category: the endpoint region of the scale line. The following four columns represent the x-coordinates and y-coordinates of the center point (with the top-left corner of the image as the origin), the width of the detection box, and the height of the detection box, respectively. The length and width are normalized from the original image, mapping them to 0 and 1.

[0022] Before formal training, the data images undergo preprocessing, including filling the images with squares and adjusting them to a resolution of 512×512. Simultaneously, to prevent overfitting and thus weak generalization ability of the trained neural network, data augmentation is employed to expand the dataset. This application uses two data augmentation methods: noise filling, which adds random Gaussian noise to the floor plan images; and random image rotation, with rotation angles of 0 degrees, 90 degrees, 180 degrees, and 270 degrees.

[0023] Step 2. Based on the key point detection network (CPN-Floor), identify the walls, doors, windows and scale in the floor plan, and generate and filter candidate primitives through axial alignment rules and constraints;

[0024] Step 21. Define the key points of the raster floor plan elements;

[0025] The main elements of the house type structure are walls and doors and windows, and the main elements of the house type structure are coded as a set of connection points with categories, and the wall structure (represented by a set of connection points of walls and wall intersections) has 4 connection types of walls: I type, L type, T type and cross type; there are 8 categories of door connection points and 8 categories of window connection points; the 8 categories of door and window connection points include: two parallel and collinear wall midpoint connection points, two parallel and collinear wall eccentric connection points, two vertical wall angle connection points, two vertical wall non-angle connection points, wall corner and wall midline connection points, wall endpoint connection points, wall intersection connection points, and free connection points;

[0026] In the house type plan, the house type structure main elements walls, doors and windows are composed. First, the main elements are coded as a set of connection points with categories. In the house type identification task, the identification of wall structure is the most common and important task. In this embodiment, the wall structure is defined by a set of connection points of walls and wall intersections, and there are 4 connection types of walls, I type, L type, T type and cross type. A plane rectangular coordinate system is constructed with the wall connection point as the origin. I type considers four directions, corresponding to four categories, 30 to 60 degrees, 120 to 150 degrees, 210 to 240 degrees and 300 to 330 degrees. L type and T type also consider rotation, each category is rotated by 90 degrees in the positive direction from the previous category, each has 4 categories, and cross type has only 1 category. The wall connection point category is defined as Figure 8 .

[0027] The door and window are represented as a line in the house type diagram, with 45 degrees as a step, and there are 8 directions. Each category of door and window has 8 key points, i.e. 8 categories. Since common doors are double sliding doors and single doors, both of which have common widths, such as single doors with widths between 0.8m and 1.2m, and double sliding doors with lengths usually between 1.8m and 3m, the approximate category of the door can be determined by adding length information in the house type diagram. In this task, only the category of ordinary windows is considered. In this embodiment, the position information of doors and windows is focused on. There are 8 categories of door connection points and 8 categories of window connection points.

[0028] After obtaining the key points of each category of elements through the key point detection network, the connection points are encoded into geometric primitives through alignment rules. The wall and door and window are represented as a line, and the effective primitive should be connected according to the direction represented by the key point category. At the same time, due to some prior knowledge of the wall, the wall primitive must form a closed one-dimensional loop, and the door and window must be located on the wall. Through a simple heuristic post-processing method, a high-level structure of the plane vector representation can be obtained, thereby realizing the vectorization of the main elements of the house type.

[0029] Step 22. Use the ConvNeXt-B pre-trained in Image-21K as the basic feature extraction network to extract the multi-scale features of the house type;

[0030] This embodiment uses the ConvNeXt-B pre-trained in Image-21K as the basic feature extraction network, and fine-tunes it in the downstream dataset training of this task.

[0031] VGG proposes that the backbone network is divided into several network block structures, each network block reduces the feature map by a fixed multiple through pooling, and each network block is composed of several different basic layer operations. ConvNeXt proposes stages based on blocks, each stage is composed of several network blocks, and the proportion of blocks in each stage is 1:1:9:1. Each stage is usually composed of 3 blocks, and the final block number and proportion are 3:3:27:3. At the same time, in terms of activation function, ConvNeXt uses GELU activation function, which effectively avoids the possible problems of ReLU activation function. The block structure diagram of ConvNeXt network is as follows Figure 9 In the design of network blocks, the idea of ResNeXt is adopted, and a 7x7 convolution kernel is used for deep convolution, and more groups are used to expand the width of the feature vector. Deep convolution is a special case of group convolution, and the number of groups is equal to the number of input channels. This convolution method has been widely used in MobileNetV2 and Xception. Deep convolution performs convolution operation on each input channel with a separate convolution kernel, and only performs convolution operation on each channel. The information between channels is independent, and this convolution method is commonly used to extract spatial information without considering the correlation between channels. This will have better performance in the task of key point detection of house type drawings which focuses on spatial information. The hidden layer in the block also uses the inverse bottleneck proposed in MobileNetV2, and the hidden bottleneck dimension is four times wider than the input dimension. This design brings better results in ResNet-200 and Swin-B system. This depth separable convolution has lower parameter quantity and operation cost without losing too much performance. At the same time, Layer Norm is used instead of Batch Norm in deep convolution and pointwise convolution. Regularization techniques can improve model convergence and reduce overfitting, but Batch Norm also has some more complex problems that may adversely affect the model. By learning from the simpler LayerNorm used in Transformer, good performance can be achieved in the application scenario of this embodiment. In ConvNeXt, using Layer Norm has better performance than Batch Norm, and the overall structure of ConvNeXt-B network is shown in Table 1:

[0032] Table 1 ConvNeXt-Base network structure diagram

[0033]

[0034] where c represents the input channel, s represents the convolution step, and Conv represents a convolution block composed of a convolution layer, a normalization layer, and a ReLU layer.

[0035] Step 23. Based on the house type multi-scale features extracted in step 22, a bidirectional feature pyramid network (BiFPN) is used for neural network feature fusion. After each feature fusion in BiFPN, a spatial attention weight is calculated and multiplied on the fused feature map. The spatial attention calculation formula is:

[0036] M s (F)=sigmoid(conv 7*7 (concat(AvgPool(F),MaxPool(F))))

[0037] where F is the feature map that needs to calculate spatial attention, Ms is the spatial attention feature map, which has only one channel and the same feature width and height as F, conv7*7 is a convolution layer with a convolution kernel size of 7, concat represents a channel splicing operation, AvgPool is an average pooling in the channel direction, and MaxPool is a maximum pooling in the channel direction.

[0038] Feature Pyramid Networks (FPN) is a neural network feature fusion module that mainly solves the problem of multi-scale changes in target detection. Shallow features have high spatial resolution for positioning and are beneficial for target positioning, but have weak semantic information for recognition. Deep features have more semantic information and are beneficial for classification and recognition, but have low spatial resolution. Therefore, a structure similar to U-Net is usually used to maintain the spatial resolution and semantic information of the feature layer. FPN is a computationally efficient top-down network structure with lateral connections, which further improves the U-shaped structure using deep supervision information. This structure is used to construct feature maps with high-level fused semantic features of different sizes, which can fuse low and high resolution maps with less computational cost. The FPN structure is as follows: Figure 10 : where the lateral connection uses a convolution kernel size of 1 to adjust the number of feature channels, and the up-sampled feature map is added pixel by pixel to obtain the output feature of each layer.

[0039] In this embodiment, four layers of original features of different scales are used for feature fusion. The output features of each stage of the ConvNeXt backbone are {C1, C2, C3, C4}, and the input features for feature fusion in this task are {C1, C2, C3, C4}. In order to improve the efficiency of information transmission and maintain the integrity of the final information, the obtained feature maps are fused by the feature fusion module, and all the pyramid level features are connected as HyperNet to integrate information of different levels, instead of simply using the final up-sampling result at the end of the HourglassNet module. Then, two 1x1 convolution layers are used to adjust the channel number to the prediction channel number, and the heat map is output as the network output.

[0040] PANet is a very good design in semantic segmentation and object detection tasks, which adds an additional path aggregation network to enhance feature fusion and expression based on FPN. PANet is shown in Figure 11 This design has been proven to be effective, but this design also increases the additional computational cost. In order to improve the computational efficiency, BiFPN is proposed, which optimizes the cross-scale connection of PANet as follows:

[0041] (1) If a calculated feature does not fuse with other features, i.e., only one input edge, its contribution in the feature fusion process is low, and it can be deleted and optimized;

[0042] (2) If the original input is at the same level as the output feature, an additional connection will be added, which does not significantly increase the computational cost for fusing more features;

[0043] (3) Unlike PANet, which has only one top-down and one bottom-up path, BiFPN can repeat the feature fusion path multiple times, and more BiFPNs can be connected to enable higher-level feature fusion.

[0044] Therefore, the structure of BiFPN can be obtained, as shown in Figure 12 , only one BiFPN is shown. Figure 12

[0045] ​In addition, when FPN features are fused, the feature maps with smaller sizes are often upsampled and added to the pixel level of the transversely connected feature maps. However, in the BiFPN, different input features have different feature resolutions, and the geometric or semantic information contained also has important degrees, so the contribution of the input feature maps to the output features is not equal. Therefore, each feature map will learn a weight when the features are fused, and then the pixel values are weighted and added. The unbounded weight parameter of the traditional method will cause unstable training, and the weight calculation after softmax normalization will increase the calculation cost, resulting in slow training. Therefore, the embodiment adopts fast standardization fusion, and the fast standardization fusion weight calculation formula is as shown in the following formula:

[0046]

[0047] wherein α i represents the weight of the feature map to be fused, ω i represents the current feature map weight coefficient learned by the network, which is greater than 0 through the ReLU activation function, and ε is a small value greater than 0 to avoid numerical instability, usually 0.0001. The advantage of this calculation is to normalize the real feature map weight, and avoid the high-cost softmax operation. In order to further improve the efficiency, after the features of different sizes are pooled or upsampled, the pixel-level feature-weighted addition is performed, and then a depth separable convolution is performed. The depth separable convolution block is composed of a channel convolution with a convolution kernel size of 3 and a normal convolution with a convolution kernel size of 1, so that Figure 12 take the M3 feature map in the formula as an example, the calculation formula is as shown in the following formula:

[0048]

[0049] wherein dsconv represents the depth separable convolution operation, ω1 and ω2 are the weights of the third layer input feature and the fourth layer input feature respectively, ε is a small value greater than 0, and are calculated by C3 and C4 through transverse connection respectively, and upsample represents the upsample operation. The embodiment is inspired by the BiFPN, and replaces the feature fusion module in the CPN with a four-layer BiFPN.

[0050] The information of features with different sizes after BiFPN has been fully fused, but the spatial information has not been valued and effectively utilized. In the key point detection task of the embodiment, the spatial position accuracy of the house type key point is required to be higher. In the feature pyramid, the features with higher resolution and larger size often contain more location information. The direct connection of BiFPN in the larger feature map can effectively transmit spatial information, but the feature map from the top of the pyramid through upsampling may lose part of the spatial information. In order to enhance the expression of this part of feature information, a spatial attention weight is calculated after each feature fusion in BiFPN, and is multiplied on the fused feature map. The spatial attention calculation formula is as shown in the following formula:

[0051] M s (F)=sigmoid(conv 7*7 (concat(AvgPool(F),MaxPool(F))))

[0052] The final spatial attention module output is as shown in the following formula. In the processing of spatial attention in the embodiment, the feature map directly multiplied with the feature weight is not directly output. In order to make the attention module more stable, the final output is the superposition of the original feature map and the feature map multiplied by the spatial weight;

[0053] F out =F+F*M s (F)

[0054] According to the above description, the BiFPN with spatial attention is adopted to replace the feature pyramid structure in CPN in the embodiment, and the final structure is as shown in Figure 13 , wherein the SAB module structure is as shown in Figure 14 .

[0055] Step 24. According to the house type scale feature after the feature fusion in step S23, the difference between the predicted value and the true value is measured by using binary cross entropy loss, and the binary cross entropy loss function calculation formula is:

[0056]

[0057] Where L BCE is the binary cross entropy objective function, represents the true value of the nth class at pixel position (i, j), is the confidence value at the same position;

[0058] But the detection task in the embodiment, a feature map has preset maximum 100 key point positions, if the same as the human key point detection task, using L2 loss to calculate will cause the loss in the task is difficult to calculate, thereby causing the model is difficult to accurately learn the position of each key point, so the embodiment adopts the way of inflation operation to construct the real position of key point to replace the Gaussian distribution, in the loss function aspect will L2 loss is replaced by binary cross entropy loss is more appropriate, the last output of the network is normalized using Sigmoid, map to 0 and 1 between.L2 loss is calculated by the following formula:

[0059]

[0060] Where N is the total amount of samples, y i is the true label, is the model prediction value corresponding to the sample.

[0061] The key point detection network calculates the heat map of the key point, and each channel contains the position heat map of a type of house key point. The output is a group of detection heat maps containing N types of house key points. The goal is to obtain the key point position probability distribution consistent with the true value. The label of each position is a binary classification problem, indicating whether it contains a key point. For the confidence of the output of the Sigmoid activation, the binary cross entropy loss calculation formula is as follows:

[0062]

[0063] Step 25. Use the non-maximum suppression method to obtain the specific classification points of the model output;

[0064] Non-maximum suppression is a crucial step to obtain the specific classification points of the model output, aiming to select a point closest to the true key point as the key point in a region. In the model prediction, a neighboring region may generate a range score for a target point, and the point with the highest confidence in a region needs to be selected as the representative key point of the region. Then, no key points of the same category are selected in the neighborhood of this point, so that only one key point is generated in a range neighborhood for subsequent primitive construction. For the model result heat map in this task, each heat map will predict a class of key points. After fixing the maximum number of predicted key points, the position with the highest confidence in the heat map is obtained as a candidate key point position by Argmax each time. The Argmax function is used to find the position of the input value that makes the given function take the maximum value. Then, starting from this point, depth-first traversal of the graph is performed, and the selected candidate key point and the pixel with a confidence greater than the specified threshold are not used as candidate key point positions. Then, the above steps are repeatedly executed to select multiple key point positions until the confidence of the newly selected key point position is less than the specified threshold.

[0065] The algorithm flow is shown in Tables 2 and 3, where M is the network prediction heat map, c is the confidence threshold, m is the maximum number of predicted key points, p is the candidate key point, and n is the point that needs non-maximum suppression.

[0066] Table 2: Key point non-maximum suppression algorithm flow steps

[0067]

[0068] Table 3: Non-maximum suppression depth traversal algorithm

[0069]

[0070] Step 26: After obtaining the key points of the wall and the door and window, the wall and the door and window candidate are obtained by axial alignment within a certain threshold, wherein the wall primitive is formed by aligning the key points of two walls, the key points of two doors form a door primitive, and the key points of two windows form a window primitive.

[0071] Step 27: After vectorizing the wall primitive, door primitive and window primitive obtained in step 26, the following constraints are added during the primitive selection process:

[0072] (1) Mutual exclusion constraint: when two primitives are close in space, especially within 10 pixels, they cannot be selected at the same time to ensure that the two primitives are not too close.

[0073] (2) Door and window position constraint: doors and windows must be located on the wall, for each door and window primitive, the wall primitive where the door and window is located must be found, and the door and window primitive must be aligned with the wall primitive, otherwise the door and window primitive is removed;

[0074] (3) Connectivity constraint: for horizontal and vertical wall connection points and all door and window connection points, the degree of the connection point (i.e. the number of connections) must match the number of candidate primitives; the specific formula is as follows:

[0075] J wall (j)=∑P wall (p)

[0076] Where j represents the connection point, p represents the candidate primitive, and P is an indicator variable indicating whether the pth primitive exists. In the wall axis connection process, the nearest principle is followed to form the primitive.

[0077] (4) For the connection point of the inclined wall, similar to the door and window, it is preferentially connected to the axial inclined wall connection point to form a wall primitive. Considering the completeness problem, the inclined wall connection point is allowed to form a horizontal or vertical wall primitive with one other type of wall key point. In the processing of inclined walls, it is encouraged to form wall primitives with more other types of connection points, even if there may be repeated primitives, which will be removed in the final wall primitive deduplication process, thereby to some extent alleviating the problem of missing some wall primitives due to classification errors.

[0078] The final output is close to the vector representation of the actual house type diagram, but there are still some problems: the connection points are not well aligned, because some coordinate errors are allowed when constructing wall and door and window primitives according to the connection points to encourage more primitive candidates. The alignment problem can be well solved by a certain threshold of horizontal and vertical correction. In order to achieve a more beautiful and accurate vectorization result, the middle point between the two pixels close to the horizontal and vertical wall start and end points is connected as the wall midpoint position, and the horizontal and vertical wall is corrected with the coordinates of this point. Secondly, the door and window are not completely located on the wall, and after the position constraint test of the door and window primitive, it is aligned with the nearest wall in the same direction.

[0079] Step 3. Use the combination of Yolov8 and Shi-Tomasi corner detection to detect and locate the scale end points, and use the pre-trained multi-modal optical character recognition model OFA-OCR to recognize the scale digital text;

[0080] The scale of the house type drawing refers to the ratio of the actual size of a certain area of the house type to the pixel width of the raster image of the area, and is an important indicator for the two-dimensional and three-dimensional reconstruction of the house type drawing based on the house type drawing. Therefore, it is very important to quickly and accurately identify and calculate the scale in the area of the house type drawing. The scale area is composed of scale marks and numbers near the marks. The scale of the house type drawing is generally distributed on the outside of the central house area. The background is relatively monotonous, but the scale area style is different, and there are various scale marks and number position areas. The scale calculation process is as follows: first, the scale area is segmented from the house type drawing, then the size number area and the scale mark area are segmented respectively, the scale mark endpoint position and the corresponding mark number are determined respectively, and the scale size is calculated jointly. Unlike the traditional house scale calculation, in this embodiment, not only the value of the scale needs to be calculated, but also the accurate position of the scale needs to be calculated, so that the scale can be automatically identified in the subsequent reconstruction work and the scale size can be manually specified and adjusted. Therefore, this embodiment proposes a method to identify and calculate various styles of scales in the house type drawing. This method can cover most of the scale styles in the house type drawing, has high accuracy, and can obtain accurate calculation of the scale without complex post-processing methods, and has strong anti-interference and generalization. Specifically:

[0081] Step 31. Segmentation of the scale area from the house type drawing. The scale area is segmented from the house type drawing, which is the first step of scale calculation. By observing the house type drawing, it is found that the peripheral walls and doors and windows of the house type area usually form a closed area, and the scale marks are usually not completely closed. Therefore, a simple and efficient scale area segmentation method is proposed: first, find the closed space of the house type area, then remove it, and then the scale area around the house type area is obtained. Taking the upper, lower, left and right four areas, the scale area image around the house type area can be obtained for subsequent accurate calculation. The segmentation method proposed in this embodiment is a simple, fast and accurate scale area segmentation method, which is suitable for house type drawings with only one house type area. In this embodiment, Opencv-python is used as the basic library for image processing. The steps of the whole method are as follows:

[0082] Step 311. Threshold segmentation: for a color house type drawing, first convert it to a grayscale image, then calculate the average grayscale of the image as the threshold, and then perform binaryzation processing on the image;

[0083] Step 312. Contour detection: find the largest outer contour, i.e. the contour of the house type area. Specifically, use the contour finding function findContours provided by opencv to find all contours in the house type drawing, and filter out the one with the largest perimeter as the contour of the house type area;

[0084] Step 313. Find the maximum bounding rectangle: after finding the outline of the house type area, use the boundingRect function of opencv to obtain the maximum bounding rectangle of the house type area contour; remove the area in the maximum bounding rectangle from the original picture.

[0085] By the above algorithm, a picture containing only the scale area can be obtained. Thus, the four surrounding scale parts can be easily obtained by the upper, lower, left and right four regions, as shown in Figure 15

[0086] Step 32. Scale end point area detection and scale corner point positioning

[0087] Yolov8 is used to detect the scale mark end point area to regress the bounding box of the scale end point area, and then Shi-Tomasi corner detection is used to accurately position the scale end point area.

[0088] The two endpoints of the scale mark determine the pixel length of a certain segment of the scale. The calculation formula of the scale is:

[0089]

[0090] Where L p represents the pixel length; L gt represents the real length.

[0091] Due to the diversification of scale mark style, direct corner point detection on the scale area will cause a very large error. Therefore, in this embodiment, the scale end point area detector is used to locate the range of the two end corner points of the scale, and then the detected corner point area is carefully detected to reduce the interference of non-scale end point area. Finally, the corner point detection result is optimized to determine the final scale end point position.

[0092] Yolov8 is a very excellent target detector with wide application in downstream tasks and strong generalization ability. In this embodiment, Yolov8 is selected to detect the scale mark end point area to regress the bounding box of the scale end point area, as shown in Figure 16 ​As shown in the figure; after detecting the end point area, the specific position of the scale end point is still not accurate enough, and if only the center point of the detection box is used, it will cause a larger error in the calculated scale. Therefore, in the detected scale end point area, according to the characteristics of the scale end point, the corner point detection is used for accurate positioning of the scale end point area. This embodiment uses Shi-Tomasi corner detection, and opencv provides an interface function goodFeaturesToTrack for implementing Shi-Tomasi, which can be used to realize corner detection. The parameter src in it is the picture of the segmented scale end point area, maxCorners is the number of corner points returned by each region, which is set to 10 in this task, qualityLevel is the quality level of the detected corner points, and the task requires high-quality corner points. The threshold value is set to 0.8, which is a relatively balanced value, and the minimum distance minDistance between the corner points is set to 3, indicating that only one corner point can be generated within a range of 3 pixels. The average of the obtained corner point coordinates is taken as the scale end point of this region, as shown in Figure 17 The figure shows that the round dots on the scale end point line are the average calculation results after corner detection.

[0093] Step 33. Recognize the numbers in the scale by using the OFA-OCR pre-training model;

[0094] The numbers in the scale are usually located in the middle or upper and lower regions of the scale line segment. The size of a scale can be calculated by the two end points of the scale line segment and the scale numbers in the middle. There are various text and number information in the house type diagram, some of which are very strong interference. Traditional optical character recognition (OCR) may not perform well in complex backgrounds. This embodiment selects OFA-OCR for scale ruler number recognition, which can accurately recognize numbers and Chinese characters in simple and complex backgrounds, and is more suitable for house type diagram understanding tasks in China. With the development of multi-modal pre-training models, visual tasks and natural language processing gradually converge, and multi-modal models can perform cross-modal text and image understanding and generation. The OCR task involves two modalities of visual task and text task. Pre-training a model with a Transformer encoder-decoder architecture on a large-scale multi-modal dataset and fine-tuning it for downstream tasks has become a new direction for unifying vision and natural language.

[0095] OFA-OCR is an advanced Chinese multi-modal optical character recognition model based on the Transformer-based encoder-decoder framework, which is based on the multi-modal pre-training model OFA-Chinese and fine-tuned. Thanks to OFA, which is multi-modal pre-trained on general domain vision and language data, OFA-OCR achieves very high quality OCR performance after fine-tuning on downstream datasets in benchmark tests on Chinese OCR tasks. It achieves better recognition results than other OCR models in more extensive and complex images, reaching the top level. The inference of digital recognition on the OFA-OCR pre-training model is as follows Figure 18 The boxed number is the position and content of the recognized number.

[0096] Step 34. Scale bar and scale number pairing is performed to accurately calculate the scale;

[0097] After scale endpoint corner detection and scale number recognition, scale bar and scale number pairing is needed to accurately calculate the scale, and all region scales are rotated to the horizontal direction. The pairing method is to find the nearest bounding box containing the number in the upper and lower regions of the midpoint of the line segment connected by the two corners. If found, pairing is successful and the scale value can be calculated. If not found, this scale bar is abandoned. The advantage of this is that it can avoid errors caused by scale bar discontinuity due to scale style, which leads to pairing and scale calculation errors.

[0098] Statistical mathematical analysis is performed on the previously calculated scale, which may contain incorrect values. First, calculate the standard deviation and mean of all calculated scales. The distribution of the calculated scale values can be approximately considered as a normal distribution. Filter out the scale values within one standard deviation. These scale values that pass the one standard deviation test can be approximately considered as the scale of the house type drawing, and then perform an average operation on them to obtain the final scale value. The standard deviation and mean calculation formula is as follows:

[0099]

[0100] Where N is the number of scale samples, l i is the calculated scale value.

[0101] Step 4. Design and implement a house type recognition and three-dimensional reconstruction system based on the WebGL framework, support rendering of reconstructed wall and door window views, user interaction with reconstructed house types, house operation history forward and backward functions.

[0102] Specific implementation two, in this embodiment, the design and implementation of the WebGL framework based on the house type recognition and three-dimensional reconstruction system is emphasized, which supports the rendering of the wall and door window view after reconstruction, the interaction of the user on the reconstructed house type, the forward and backward functions of the house type operation history record.

[0103] In the development of large front-end graphics applications, data-driven views can control the flow of data in a more fine-grained manner, and views need to have corresponding data models as support. When the user operates the view, it will trigger the change of its corresponding data model, thereby triggering the re-rendering of the view. There are two mainstream design patterns for this kind of view and data separation implementation: MVC and MVVM. In the MVC design pattern, the Controller controller is responsible for receiving user input and requests, and then dispatching the request to the corresponding Model for processing, and then triggering the update of the corresponding View object, thereby updating the view performance. This update usually needs to determine a certain data to the corresponding view object. MVVM decouples the user-defined data object and the real view data exchange through a virtual view object, and gets the minimum update of the view object through the comparison algorithm of the virtual view object. This design pattern is usually applied to the case where it is difficult to determine which view objects need to be updated when the data object is updated. These two design patterns can fully realize the modularization of functions, independent maintenance of each module, mutual independence between modules, high cohesion within modules, and low coupling between modules.

[0104] The main project is developed using Vue and Element Plus, and uses the Typescript language. The graphics rendering part will be a dependent package of the front-end system display project, and will be installed and imported by the front-end main project after construction. Referring to the front-end MVC design pattern, the entire graphics rendering module is composed of the following packages: VisualModule package, Schema package, Interaction package, History package, and RenderApp package.

[0105] VisualModule package: mainly stores the encapsulation of 2D house scene and 3D scene, divided into 2D view model and 3D view model, including the following core classes: Visual2dModule, used to integrate the application, container, resource loading provided by PixiJS, and receive user interaction on 2D area and event notification from the scene; Visual3dModule, mainly used to integrate Babylon's scene, camera, light, material, map, resource loading and other functions, receive various events from the current scene data object; VisualWall2d and VisualWall3d are respectively encapsulation of wall model 2D and 3D view model, used to calculate wall geometry data in 2D and 3D scene, render wall view; VisualWallAttachment2d and VisualWallAttachement3d are respectively encapsulation of door and window view model, used to calculate the geometric properties of doors and windows in 2D and 3D scene, render door and window view; VisualRoom2d and VisualRoom3d are used for house view rendering.

[0106] Schema package: used to describe the data model structure of the current scene defined by the user in the scene, register data objects in this package, complete the unified serialization and deserialization operation of custom wall, door, window and other data objects, and is a collection of all data objects describing the current house. It contains important classes: SceneSchema, which is used to describe the collection of data objects and the relationship between data objects in the current scene, and is the container of all data models of walls and doors. Unlike Babylon.js Scene, SceneSchema describes the relationship between data object elements in the current house defined by the user, not the relationship between view objects rendered in the scene; EntityWall and EntityWallAttachment are data models of walls and doors, encapsulating inherent properties of walls and doors such as length, thickness, starting position, end position, height above ground, etc.

[0107] Interaction package, mainly provides interaction and operation on 2D scene view, receives and processes user interaction in the scene, and changes the corresponding wall and door data object. Contains core classes: InteractionDrawWall, used for wall drawing; InteractionEdit, used for wall and door property editing; InteractionTransport, used for wall and door dragging; InteractionScale, used for specifying the scale of the current house background.

[0108] History package: used to manage the historical state of the data in the current scene, providing undo and redo functions for the data in the scene, which caches the historical state of the data objects in the current scene based on the changes of the data in the scene, depends on the Schema package for data definition, including the core class HistoryManager, which is mainly used for caching data objects of the current scene tree and comparing new and old scene trees, thereby realizing the forward and backward of the current scene tree history record.

[0109] RenderApp package: used to integrate the contents of the above packages, encapsulated into an integrated class as the entrance of the rendering module for external projects, containing the core class RenderApp, which is used to integrate the above packages and expose to the outside for use.

[0110] BFF backend service is developed using Nest.js, which communicates with the front-end service as the main backend service, usually one front-end project needs one backend, all internal communication is solved by communication between backend servers, mainly including the core class FileService for floor plan upload, FloorPlanService for floor plan design persistence.

[0111] The floor plan recognition module is developed and deployed separately, using FastAPI to provide services externally, which is a separate service of the backend system, called by other modules that need it through network requests, realizing the decoupling between modules, making the deployment of AI-related services more convenient, and transmitting the floor plan recognition and vectorization data to the BFF backend through JSON data format, and then returning it to the front-end after corresponding business encapsulation. The floor plan recognition module contains the core class PredictService, which is mainly responsible for loading and reasoning the scale recognition model and floor plan recognition model. The overall system architecture is shown in Figure 19 .

[0112] Design of floor plan recognition and reconstruction module: the main function of the module is to recognize the raster floor plan image uploaded by the user, and render the recognition result in the front-end, and according to the reconstructed 2D vectorized floor plan, construct and render a 3D floor plan model that can be displayed in real time in the 3D scene, to realize the effect of reconstructing a 3D floor plan from a 2D raster floor plan image.

[0113] Floor plan recognition and reconstruction flowchart Figure 20As shown, the user first needs to upload the house type drawing to be identified, wrap the picture into FormData and send it to the backend for house type drawing scale identification and wall, door and window primitive reconstruction and vectorization, then load the picture into the 2D canvas element, use the AssetsLoader provided by PixiJS to load the house type background picture into the middle of the window, and center align and scale according to the width and height of the picture, so that the user uploaded house type drawing can be located in the center of the user's visual area. The server calls the method of embodiment one to perform vectorization reconstruction and scale calculation of the house type wall and door and window, when the BFF receives the front-end request, it will upload the user picture to the OSS, and forward the request and picture OSS address to the processing module of the house type identification service. Then encapsulate the identified scale and house type primitive information in JSON data format. After the front-end receives the response, it will first pop up the position and number of the scale ruler for the user to confirm, to facilitate fine-tuning the scale area, after confirmation, the graph rendering module will create wall and door and window data objects and view objects according to the vectorization result, and display them to the canvas element, if the user uploaded house type drawing does not have scale information originally, a default scale ruler appears in the center of the 2D canvas, the user manually specifies the scale, and then the real size calculation of the primitive is performed.

[0114] The house type identification and reconstruction module timing diagram is shown in Figure 21 As shown. The house type reconstruction main application and the rendering application are located in the web front end, the house type reconstruction main application is responsible for the page UI, and the rendering application is responsible for the graph rendering of the house type 2D and 3D area. The house type drawing upload service and the house type identification service are located in the backend, wherein the house type drawing upload service is located in the backend BFF, which undertakes general requests of the main application, and the house type identification service is an AI house type identification separately deployed, which is called through other backend services.

[0115] This embodiment realizes the asynchronous communication of BFF service and AI server through the message queue mode, decouples the message queue between the message producer and the consumer, multiple consumers compete to process the information of the same queue, which plays a load balancing effect and achieves the effect of traffic peak shaving. The information consumer processes the information through the message queue and notifies the message producer in an asynchronous network request mode. This method fully uses asynchronous message notification and increases the system concurrency capability, which is a relatively mature solution. Traditional deployment and small-scale containerized deployment can be well compatible, and this method has low implementation cost. Considering the mainstream message queue middleware on the market and the adaptation degree of Nest.js and FastAPI, RabbitMQ is used as the message queue of the system, and the Work Queue working mode is adopted. This working mode is suitable for distributing time-consuming tasks among multiple workers. After receiving the user request, the BFF returns a unique TaskId to the front end, the front end uses the TaskId to poll the BFF to check whether the task is completed, and then the BFF sends the TaskId and the picture address to the message queue. Multiple AI servers will listen to the bound message queue and compete to process the house type identification message. The prefetch is set to 1, and after processing a task, the worker takes out a new message from the queue for reasoning. After reasoning, in order to ensure the accuracy and reachability of the message, the manual message confirmation method is used to ensure that the message has been consumed by the worker. In order to increase the concurrency of the AI service, the asynchronous message to the message queue is confirmed, and the AI service sends a network request to notify the BFF service of the TaskId and the identification result and caches it after completing the reasoning of a task. When the front end polling arrives, the result is returned to the front end, and the TaskId and the result are deleted from the cache.

[0116] After the house type identification and reconstruction, the user may make some changes to the house type, such as changing the walls and doors and windows in the house type diagram. Therefore, the generated house type diagram is designed for interactive design in the design tool, so that the user can change the position and style of the wall and window elements in the house type. By registering the mouse click and drag events in the canvas, the selected wall or door and window primitive object is determined according to the coordinate position, and then the user mouse coordinate position is obtained in real time in the mouse drag event. The data object related attributes of the user operation are uniformly processed through the corresponding interactive object, the node change event is triggered to change the view object corresponding to the data object, so as to realize a series of interactive effects of the user on the wall, door and window and other house type primitives.

[0117] The user wall and door and window interaction flowchart is as follows Figure 22The current interaction object and interaction category are first determined, and then the relevant attributes of the data object are changed through the corresponding interaction logic, and then a data node update event is published to update the view. The system includes 2D and 3D view objects, and the change of wall and door and window data objects triggers a notification to make both the 2D and 3D view objects receive the notification of needing to be re-rendered, which is a specific implementation of the publish-subscribe mode. VisualModule2d and VisualModule3d both listen to the data node change event from SceneSchema. The wall and door and window interaction module timing diagram is shown in Figure 23 When processing the interaction of walls and doors and windows, the rendering application in the Web front end completely processes the mouse events of the user in the Canvas area. InteractionManager implements the callback of PixiJS for 2D view interaction, determines the view object being interacted in the current user mouse event according to the id of the registered user event callback information. Then InteractionManager judges the operation of the current user on the house primitive view according to the user interaction behavior, and takes different data object update logic. Some primitive data objects may have geometric position association with other data objects, such as the connection of walls, the existence of doors and windows on the wall, and the like, which also need to be found out as data objects that need to be updated at the same time. After changing the geometric attributes of the related data objects through InteractionManager, the data objects that need to be updated are dispatched through SceneSchema to publish node data update events, and through receiving the changed nodes in VisualModel, multiple view objects are updated at one step to improve the rendering performance.

[0118] The house type design history record module is designed. When the user designs the house type, it often needs to be redone or the previous operation needs to be undone due to different inspirations or drawing errors. Serializing all data objects in the database or file in the backend, and then updating the entire canvas scene data through front-end and back-end network communication requests to realize the re-rendering of the house type reconstruction view is a very time-consuming work. When the scene is complex and the house type is large, it will cause the page to be stuck to a certain extent, and network communication is time-consuming. The system relies on the data structure of the nodes in the scene to realize an incremental update method without persistence, which can update the entire front-end without the time-consuming network request to obtain the persistent data. When the user changes the position of the wall or the door and window, or adds some nodes, the scene tree nodes are first serialized, and then the historical nodes are saved to the node pool, and the Hash tree of the serialized scene tree nodes is calculated and cached. The Hash value in the Hash tree identifies the unique state of the corresponding node object in the scene tree. When the user clicks the forward and back buttons of the history progress, the difference between the two Hash trees is compared, and it is calculated which data node object needs to be updated, added, or deleted. Then the node is taken out from the historical data object node pool, and the history record state is restored. The history record saving flowchart is shown in Figure 24 When the user performs view interaction on the house type node, as shown in the first half of the interaction module, the view object and data object of the wall or door and window node that the user is currently operating and the associated data object need to be found. After the user interaction is completed, the SceneSchema dispatches the house element node data object change event, and the HistoryManager in the history record module receives the node data object change event from the SceneSchema. The current scene tree is serialized, and the serialized scene tree under the current scene is cached.

[0119] The flowchart of restoring the history record is shown in Figure 25As shown, when the user clicks the history forward or backward button, the history scene cache queue takes out the Hash tree of the target scene, and compares the differences between the two Hash scene trees through a comparison algorithm, and concludes that the data object node changes have the following four types: added nodes, deleted nodes, nodes with changed properties, and nodes with changed parent nodes. The changed nodes are sorted according to the following rules: nodes that need to be added in advance due to parent node changes, nodes with parent node changes, nodes that need to be deleted, normally added nodes, and nodes with changed properties. Add the data nodes that need to be updated to the current scene, recreate their view objects and re-render them. Since there is a serialization operation of the model object when saving the history, some properties of the model object are values calculated based on the inherent properties of the model when the scene is loaded. These calculated properties do not need to be serialized. The embodiment uses the Typescript decorator to define the Serializable decorator as a serializer and deserializer on each property that needs to be serialized and deserialized using the decorator pattern. The decorated property metadata is recorded, and the serialization and deserialization process of the data objects such as house walls and doors and windows is manually controlled. Each time the data objects of the walls and doors and windows are serialized, the properties of the decorated property metadata are serialized. Deserialization is the same, and the property depth cloning is performed according to the decorated property metadata. The serialization and deserialization of the data object nodes in the scene and the comparison operation of the scene tree are CPU-intensive tasks. Since the web front-end language used is single-threaded JavaScript, it is easy to cause page lag and lose user interaction response. Therefore, the operation of saving the history is placed in the JavaScript micro-task queue to be executed asynchronously to avoid blocking the JavaScript main thread and causing the browser page to lag or lose response. At the same time, in order to simplify the serialization operation of the tree structure object, the data nodes in the scene tree are flattened, and the tree structure of the scene tree is converted into a flattened array. All nodes in the array maintain the index of the parent-child node relationship in the original scene tree through the parent and children properties. After the data nodes in the scene tree are serialized, they are still relatively large. If the scene tree is directly cached, the full replacement update will delete all the old data object nodes of the old scene tree and reload the new scene tree to render the 2D and 3D views, which is a very time-consuming operation. When the scene tree is large, high-frequency undo and redo operations can easily cause the browser to lag. From a macro perspective, full update of all nodes in the scene tree is not scientific. During an operation process, most of the nodes in the scene tree are identical to the old node scene tree. Therefore, a Hash-Diff algorithm is used to compare the differences between the two serialized scene trees to achieve incremental update of the scene tree. The algorithm flow is as follows:

[0120] (1) Calculate the Hash value of all data object nodes in the serialized scene tree, get a scene Hash tree, which is used to represent the unique state of the data nodes contained in the original scene tree;

[0121] (2) Save the calculated scene Hash tree using a queue, and add the Hash tree to the queue;

[0122] (3) All data object nodes in the scene tree are indexed by node Hash value, and the data object node is the value, to construct a historical node cache pool Map, so that the data object node can be found through the node Hash value. This node pool stores all data object nodes that have appeared in the history record.

[0123] Through the above Hash algorithm, the bloated scene tree can be simplified to a Hash tree with the same structure, and all nodes in the Hash tree can be indexed in the node pool through the data object node Hash value.

[0124] When the undo and redo operations are triggered, the pointer indicating the current scene tree will move forward or backward to find the Hash tree under the previous scene, and perform the Hash tree Diff algorithm between the new and old scenes to compare which nodes in the two scene trees have changed. The Diff algorithm is as follows Figure 26 The time complexity and space complexity of the Diff algorithm are both O(N), where N is the number of data object nodes in the scene tree. After finding the data nodes in the history record, these nodes contain all the data to create their view objects, which are re-added to the current scene tree, and their view objects are recreated and rendered on the screen.

[0125] Table 4 User Information Table

[0126]

[0127] Table 5 House Design Record Table

[0128]

[0129] Through the above description, the fast operation history record requirement of short frequency has been solved, mainly solving the problem that the designer may need to quickly return to the previous house type design state due to operation errors and the like when designing the current house type. When the user designs the house type or pauses the work, it is desired to save the current house type state, which is different from the history record state forward and backward. The state forward and backward of the design history record is completely based on the operation of the front-end browser memory. Once the user refreshes the browser, the design history record and the house type design will completely disappear, so a house type design scheme remote persistent storage is needed to persistently store the house type design that the user wants to save. This persistence is based on the user's active desire to save the current design house type history record state. In this embodiment, a MySQL database is selected as the house type design persistent data storage to save the user's house type design scheme. The current user's login Token is used for verification, and the current house type serialized data is stored as a string in the database.

[0130] The house type design persistent database design involves two tables, namely a user table and a house type design record table. The detailed design of the two tables is shown in Table 4 and Table 5. The key data of the user is the username and the password, and the username uniquely identifies a user. The user ID is a unique ID assigned by the server to each user. The house type history record table takes the username as the primary key and records the design scheme data of each user. The scheme name and the scheme design time also need to be recorded. The active field is a record soft deletion control field. When the user deletes the record, the row record is not really deleted, but the field is set to 0 to indicate the unusable state, thereby maintaining the persistence and integrity of the design data.

[0131] According to the above description, the house type reconstruction system is implemented. The system development environment is shown in Table 6.

[0132] Table 6 System development environment table

[0133]

[0134]

[0135] After the system is built, the system mainly has three function modules, namely a house type identification and reconstruction module, a house type wall and door and window interaction module, and a house type design history record module.

[0136] House type identification and reconstruction module: this module is a house type identification and reconstruction module. The user first needs to upload a house type drawing file and click the Select File button under the Background bar on the right. The module supports.png,.jpg,.jpeg format picture files, and the file size cannot exceed 2MB. The resolution size of the house type drawing uploaded by the user cannot exceed 1920x1080.

[0137] The user selects the house plan to be identified and uploads it. After being packaged into a FormData object and sent to the BFF backend, the BFF backend receives the upload picture request, saves the picture on the OSS, and forwards the picture address and the TaskId of the current task to the message queue. The frontend returns the TaskId and picture address to the frontend. The frontend needs to use the TaskId to query the BFF service every 5 seconds to determine whether the current task is complete. After the backend AI house plan recognition is completed and a response network request is returned, the BFF service returns the recognition result to the frontend in the next frontend polling. The house plan selected by the user is loaded into the central 2D canvas using AssetsLoader, and a scale ruler is rendered. If there is a correctly identified scale result in the current house plan, the scale ruler area appears at the identified scale position, as shown in FIG. 8. Figure 27

[0138] If there is no scale that can be identified in the current house plan, the ruler appears in the center of the house plan. The user can drag the position of the scale ruler and click the circle buttons on both sides of the ruler to adjust the length of the scale. After the user confirms that the position is correct, the 2D plane house is drawn in the 2D canvas area by using the keyboard to click the Enter key and release, as shown in FIG. 9. Figure 28 Figure 29 Figure 29 After the identification is completed, the right panel appears to adjust the transparency and display scale of the house plan background. By reducing the transparency of the background image, the user can reduce the interference of the background house plan after completing the identification and drawing of the house plan, or delete the scale background image to focus on the house design. Figure 30

[0139] More three-dimensional reconstruction results of house plans are shown in FIGS. 10(a), 10(b), 10(c), and 10(d). Figure 31 Figure 31 FIG. 10(a) is a house reconstruction result with an inclined wall, Figure 31 FIG. 10(b) is a three-room-one-hall-one-bathroom house reconstruction result, Figure 31 FIG. 10(c) is a one-room house reconstruction result, Figure 31 and FIG. 10(d) is a two-room-one-hall-one-bathroom house reconstruction result. Figure 31 The four figures basically include common house plans in current urban areas in China. Figure 31 As can be seen from the four figures, the embodiment performs 3D reconstruction on the two-dimensional vectorization result of the house plan, and the effect is relatively good. Complex house plans with inclined doors and windows can also be accurately restored.

[0140] ​​​House type wall and door window interaction module: after the user rebuilds the house type, he or she may need to modify the structure of the house type, change the wall type structure, or change the size or type of the door and window. Some wall drawing and commonly used door and window adding buttons are provided on the right side of the page. Click the button to activate the corresponding adding function and draw it in the 2D canvas area. The added door and window model will also be displayed in the 3D view. Click the wall in the 2D canvas, and it will become a highlighted blue part. A black box interaction prompt will pop up above it. This interaction prompt can be dragged to the left to move the position. For walls, the delete operation and wall merging operation can be performed, such as Figure 32 .

[0141] The delete operation can delete the wall, such as Figure 33 , and the door and window on the wall will be deleted, and the wall view at both ends of the wall will be redrawn. Merging walls will merge the axially aligned walls adjacent to the wall into the same wall. At this time, the user can select the green dot interactive area at both ends of the wall with the mouse, axially drag the edge of the wall to change the length of the wall, such as Figure 34 , or drag the wall along the cross axis of the wall to change the position of the wall. The dragging process is shown in Figure 35 . When the user drags the wall along the cross axis to the specified position, releases the left mouse button, and the wall position is changed to the mouse cursor area position, and the two walls adjacent to the wall will also change the length to the current wall position, such as Figure 36 .

[0142] The system provides a function of drawing walls and doors and windows, and limits the position of doors and windows to be limited to walls. Three doors and three windows and opening interactive buttons are provided on the right side. The user can add doors and windows to the house type. For example, click the button to display the 2D picture of the sliding door drawn by PixiJS at the mouse pointer. Move the mouse to the position where you want to add the door, and click the left mouse button to add the corresponding door to the mouse pointer position. At the same time, the door will enter the blue highlighted selected state, as shown in Figure 37 . By controlling the 2D view, the 3D structure of the house type will also be loaded and updated, and the outline of the applied door model will also enter the blue highlighted selected state. Like the axial interaction of the wall, the edge can be dragged to change the length, but it does not provide cross-axis dragging like the wall. Doors and windows have more strict geometric constraints and semantic constraints.

[0143] The house type design history record module: in this module, the user can restore his historical operation, the default forward and backward times are 20 times, the user changes the attribute value of any primitive node in the current scene each time, which triggers the operation of saving the history record, when the user wants to restore the last operation, click the right side Undo button in the right side History Functions option bar to restore the last addition, deletion or modification of the node state in the current scene, such as clicking the wall body in the lower left, dragging to the right, and Figure 38 .

[0144] If you want to restore the previous position state, click the Undo button to restore the state before dragging as shown in Figure 39 . Similarly, if you want to redo the operation just now, click the Redo button to redo the user's operation, that is, restore the state shown in 38. If the user wants to completely abandon the current house type design and start again, click the Clear Floor in the upper left corner of the 2D house type rendering area or the Clear in the right side History Functions option bar to completely clear all wall and door and window data model objects and view objects in the current scene house type area, and save a history record.

[0145] When the user needs to persistently save the history record, click the Save Plan button. If the current user is not logged in, you need to enter the account and password in the dialog box to log in to obtain the user login state for saving the record, as shown in Figure 40 . After the back-end user table verification, the Token is issued to the user through JWT, and the front-end attaches the Token information in the request header when making a request to the back-end in the future, so that the back-end does not need to save the user login state to reduce the server storage pressure. When the user logs in successfully, click the Save Plan button again, a dialog box will pop up for the user to input the name of the current plan, and after the user confirms, the user's current design plan is serialized in the form of a string and sent to the back-end, and stored in the house type design record table.

[0146] When the user logs in successfully, the house type design persistent history record appears below the right side toolbar, which contains the current user's house type design plan name and design time, and the records are sorted from new to old according to time, as shown in Figure 41Lower right corner. After the user saves the scheme, all design schemes will be reacquired to keep the latest data display. When the user selects each record, a dialog box will pop up, indicating that the user can apply this design scheme record or delete this design record. When the user needs to apply the design scheme record, because all data in the current house design area will be cleared, the user will be prompted to save the scheme in advance when the house design is not empty. When the user needs to delete the scheme, the database performs soft deletion on the corresponding record line to facilitate administrator query or direct management of design data, maintain data persistence and integrity.

Claims

1. A grating image-based house layout identification and three-dimensional reconstruction method, characterized in that, Comprise the following steps: Step 1. Collect and screen high-quality house type drawings suitable for the national conditions of China, and construct raster house type drawing vector data set; Step 2. Identify the walls, doors and windows, and scales in the house type drawing based on a key point detection network, generate and screen candidate primitives through axial alignment rules and constraint conditions, and the specific method is: Step 21. Define key points of raster house type drawing elements; The main elements constituting the house type structure are walls and doors and windows, and the main elements constituting the house type structure are coded as a set of connected points with categories, and there are four types of wall connection types for wall structure: I type, L type, T type and cross type; there are 8 categories of door connection points, and there are also 8 categories of window connection points; Step 22. Use ConvNeXt-B pre-trained in Image-21K as the basic feature extraction network to extract house type multi-scale features; Step 23. According to the house type multi-scale features extracted in step 22, use a bidirectional feature pyramid network to perform feature fusion of the neural network, and after each feature fusion in BiFPN, a spatial attention weight is calculated and multiplied on the fused feature map, and the spatial attention calculation formula is: ； Wherein F is a feature map requiring calculation of spatial attention, Ms is a spatial attention feature map, which has only one channel and the same feature width and height as F, is a convolutional layer with a kernel size of 7, concat represents a channel concatenation operation, AvgPool is an average pooling operation in the channel direction, and MaxPool represents a maximum pooling operation in the channel direction; Step 24. According to the house type scale features after feature fusion in step S23, use binary cross entropy loss to measure the difference between the predicted value and the true value, and the binary cross entropy loss function calculation formula is: ； wherein is a binary cross-entropy objective function, denotes a pixel position at which n the true value of the class, N is the total number of samples, and denotes the detection heat map of N types of house key points; Step 25. Use the non-maximum suppression method to obtain the specific classification points of the model output; Step 26. After obtaining the key points of the walls and doors and windows, the wall and door and window candidates are obtained through axial alignment within a certain threshold, wherein the wall primitive is formed by aligning the key points of two walls, the key points of two doors form a door primitive, and the key points of two windows form a window primitive; Step 3. Use Yolov8 and Shi-Tomasi corner detection combined to detect and locate the scale endpoints, and use the pre-trained multi-modal optical character recognition model OFA-OCR to recognize the scale digital characters; Step 4. Design and implement a house type recognition and three-dimensional reconstruction system based on the WebGL framework, support rendering of the wall and door and window view after reconstruction, user interaction on the reconstructed house type, house type operation history record forward and backward function.

2. The raster image-based house layout identification and 3D reconstruction method according to claim 1, characterized in that: In step 2, the method of identifying walls, doors and windows, and scales in the house type drawing based on a key point detection network, generating and screening candidate primitives through axial alignment rules and constraint conditions, also includes: Step 27. After vectorization post-processing of the wall primitive, door primitive and window primitive obtained in step 26, the specific way is: during the primitive selection process, the following constraints are added: (1) Mutual exclusion constraint: when two primitives are less than or equal to 10 pixels in space, they are not selected at the same time; (2) Door and window position constraint: doors and windows must be located on walls, for each door and window primitive, the wall primitive where the door and window is located must be found, and the door and window primitive is forced to align with the wall primitive, otherwise the door and window primitive is removed; (3) Connectivity constraints: for horizontal and vertical wall connection points and all door and window connection points, the degree of the connection point must match the number of candidate primitives; (4) For the connection point of the inclined wall, consistent with the door and window, select the connection point connected with the axial inclined wall to form a wall primitive.

3. The method for house layout recognition and 3D reconstruction based on raster images according to claim 1, characterized in that: In step 3, the specific process of scale end point detection and positioning using Yolov8 and Shi-Tomasi corner detection combination, and scale number recognition by pre-trained multi-modal optical character recognition model OFA-OCR includes: Step 31. Segmentation of scale area from house plan; Step 32. Scale end point area detection and scale corner positioning; Yolov8 is used to detect the scale mark end point area to regress the bounding box of the scale end point area, and then Shi-Tomasi corner detection is used for accurate positioning of the scale end point area position; Step 33. Use OFA-OCR pre-trained model to recognize the numbers in the scale; Step 34. Scale mark and scale number pairing to accurately calculate the scale.

4. The raster image-based house layout identification and 3D reconstruction method according to claim 3, characterized in that: The specific method of step 31 to segment the scale area from the house plan is: Step 311. Binary threshold segmentation: for color house plan, first convert it to grayscale image, then calculate the average gray level of the image as the threshold, and perform binaryzation on the image; Step 312. Contour detection: find the largest contour, which is the contour of the house plan area, specifically use the contour finding function findContours provided by opencv to find all contours in the house plan, and filter out the one with the largest perimeter as the contour of the house plan area; Step 313. Find the largest bounding rectangle: after finding the contour of the house area, use the boundingRect function of opencv to get the largest bounding rectangle of the house area contour.

5. A grating image-based house layout identification and 3D reconstruction system, which is implemented based on the grating image-based house layout identification and 3D reconstruction method according to any one of claims 1-4, characterized in that, It includes: House type recognition and reconstruction module, used to receive user uploaded house plan file, recognize and reconstruct user uploaded 2D house plan file, and generate 3D house view; House type wall and door and window interaction module, used for adding, deleting and modifying wall, door and window in 2D and 3D view; House type design history record module, used to record user's design operation history, support undo, redo, save and load design scheme. 6.The raster image-based house layout identification and 3D reconstruction system according to claim 5, characterized in that: The house type recognition and reconstruction module includes: File upload unit, used to receive user uploaded house plan file, supporting.png,.jpg,.jpeg format, and file size not exceeding 2MB, resolution not exceeding 1920x1080; Image processing unit, used to send user uploaded house plan file to backend for recognition and reconstruction, generating 3D house view; Scale recognition and adjustment unit, used to recognize the scale in the house plan and allow the user to adjust the position and length of the scale; View switching unit, used to switch between 2D and 3D view.

7. The raster image-based house layout identification and 3D reconstruction system according to claim 6, characterized in that: The image processing unit realizes house type recognition and reconstruction through the following steps: Packaging the user uploaded house type drawing file as a FormData object and sending it to the backend; The backend saves the house type drawing file on the OSS, and forwards the picture address and task ID to the message queue; The front end regularly queries the task status according to the task ID, and returns the recognition result and loads the house type drawing in the 2D canvas after the backend completes the house type recognition; According to the recognition result, a 3D reconstruction view is generated, and the right upper corner is displayed.

8. The raster image-based house layout identification and 3D reconstruction system according to claim 5, characterized in that: The house type wall and door and window interaction module comprises: A wall editing unit for adding, deleting, and merging walls in the 2D view, and supporting length and position adjustment of the walls through dragging; A door and window editing unit for adding, deleting, and modifying doors and windows in the 2D view, and the position of the doors and windows must be limited to the walls; A 3D view synchronization unit for synchronously updating the 2D view to the 3D view. 9.The raster image-based house layout identification and 3D reconstruction system according to claim 5, characterized in that: The house type design history record module comprises: A history operation record unit for recording the design operation history of the user, supporting undo and redo operations; A design scheme saving unit for serializing and saving the design scheme of the user to the backend database; A design scheme loading unit for loading the saved design scheme of the user, and supporting deletion of the saved design scheme.

Images