Picture-based Automatic House Type Construction Method, Device and Computer Equipment

Abstract

The present invention discloses a method, apparatus and computer device for automatically constructing a house type based on pictures, including: classifying house type elements of a house type picture by using a semantic segmentation model; extracting candidate key points in the house type element classification result by using a key point detection model, and after screening the candidate key points, interrupting and clustering the walls based on the determined key points to construct wall elements; detecting image connected components according to the house type element classification result to construct door elements, window elements and column elements, and associating these three types of elements with the wall elements; and correcting the constructed wall elements, door elements, window elements and column elements in combination with the association result to obtain the constructed house type diagram. This method can improve the accuracy and efficiency of automatic house type construction.

Description

Technical Field

[0001] The present invention belongs to the field of house type design, and particularly relates to a method, a device and a computer device for automatically constructing a house type based on pictures. Background Art

[0002] Home decoration design is a time-consuming and laborious task. A complete design plan usually takes a designer a lot of time to design and modify. With the continuous progress of computer technology, intelligent design tools are also constantly innovating and optimizing to help designers improve design efficiency.

[0003] Common intelligent design capabilities include, given a house type and some furniture, automatically realizing the reasonable placement of furniture through algorithms, or given a house type, intelligently identifying the house type structure through algorithms to realize the automatic layout of water and electricity points. However, whether it is the intelligent placement of home decoration or the automatic layout of water and electricity points, they all rely on the description of basic house type elements. Users first need to directly operate the visual house type plan (2D or 3D scene) in the design tool to carry out subsequent design links, and all of these are inseparable from the extraction and construction of house type elements.

[0004] Traditional house type construction usually restores the house type information described in the drawing stroke by stroke in the design tool by a large amount of manpower based on the given house type drawing. In addition, based on industry specifications, house type drawings usually have a certain similarity in structure. For example, the thickness of building walls is usually 120mm or 240mm. For a large number of similar house type drawings, practitioners need to spend almost the same amount of time mechanically repeating the entire drawing process, and the design efficiency is extremely low.

[0005] In order to improve design efficiency, traditional house type automatic construction methods usually identify house type elements by performing some fixed image operations on the house type drawing based on fixed rules. For example, a method for recognizing and generating a three-dimensional house type from a photographed house type drawing disclosed in patent document CN105279787A, and another method for automatically extracting house type elements based on machine vision disclosed in patent document CN108763606A. However, due to the wide sources and diverse styles of house type drawings, it is difficult for traditional rule-based recognition to achieve satisfactory results.

[0006] In summary, neither manually drawing the house type by a large amount of manpower nor identifying the house type by rules can well meet the requirements of high-quality and efficient house type plan construction. Summary of the Invention

[0007] In view of the above technical problems, the purpose of the present invention is to provide a method and a device for automatically constructing a house type based on pictures, so as to improve the accuracy and efficiency of house type automatic construction.

[0008] To achieve the above invention object, an embodiment of the first aspect provides a method for automatically constructing a house type based on pictures, including the following steps:

[0009] Classify house type elements in the house type picture using a semantic segmentation model;

[0010] Extract candidate key points from the house type element classification result using a key point detection model. After screening the candidate key points, break and cluster the walls based on the determined key points to construct wall elements;

[0011] Perform image connected component detection based on the house type element classification result to construct door elements, window elements, and column elements, and associate these three elements with the wall elements;

[0012] Correct the constructed wall elements, door elements, window elements, and column elements in combination with the association result to obtain the constructed house type diagram.

[0013] In one embodiment, before classifying house type elements, perform normalization operations on the house type picture, including resetting the picture size of the house type picture and performing pixel standardization on the house type picture according to the standard value and variance;

[0014] Input the normalized house type picture into the semantic segmentation model for house type element classification.

[0015] Before extracting candidate key points, perform binarization and normalization operations on the house type element classification result. Among them, the normalization operation includes resetting the picture size of the house type element classification result and performing pixel standardization on the house type element classification result according to the standard value and variance;

[0016] Input the normalized house type element classification result into the key point detection model for extracting candidate key points.

[0017] In one embodiment, the semantic segmentation model is constructed based on the PSPNet model and optimized by parameters; the key point detection model is constructed based on the CenterNet model and optimized by parameters.

[0018] In one embodiment, screen the candidate key points according to a set threshold to determine the key points. After breaking the wall marks in the house type element classification result based on the key points, cluster the pixels belonging to the same wall, and calculate the starting pixel, ending pixel, and thickness of the pixel area of each clustering cluster to construct wall elements.

[0019] In one embodiment, after constructing the wall elements, extract the load-bearing wall area in the house type element classification result, judge the overlapping area between the wall elements and the load-bearing wall area, and use the wall elements with an overlapping area greater than the set first overlapping area threshold as load-bearing wall elements.

[0020] In one embodiment, the image connected component detection based on the classification result of housing type elements includes:

[0021] According to the classification result of housing type elements, adjacent pixels with the same pixel value are clustered into a pixel connected region, and the pixel region is dilated. The dilation result of the image is screened according to the set threshold, and the obtained pixel connected regions are combined with the corresponding pixel classifications to construct door elements, window elements, and column elements.

[0022] In one embodiment, associating door elements, window elements, and column elements with wall elements includes:

[0023] After extracting the contour of each pixel connected region, calculate the overlapping area between the wall element and the door element, window element, and column element respectively. If the overlapping area exceeds the second overlapping area threshold, then associate the wall element with the door element, window element, and column element.

[0024] In one embodiment, the correction of the constructed wall elements, door elements, window elements, and column elements in combination with the association result includes:

[0025] For the wall element, calculate the closed loop for the wall element. Each closed loop forms a room element, and perform collinearity judgment on all wall elements, merge the same type of walls, and use the longest wall thickness in the collinear walls as the thickness of the merged wall;

[0026] For the door element and window element, according to the association result between the door element, window element and the wall element, calculate the door center line and window center line according to the wall to which the door and window belong, and perform normalization processing, including: the length of the door element and window element cannot exceed the wall to which they belong, merge the collinear doors and windows, and set the door type and window type of the merged structure according to the door element and window element with the largest area, remove the door element or window element where the illegal middle lines of the door and window intersect, and retain the door element or window element with the larger area among the intersecting elements;

[0027] For the column element, when there is an intersection between the column element and the wall, calculate the circumscribed rectangle of the contour formed after the wall cuts the column as the outer contour of the column; when there is no intersection between the column element and the wall, calculate the circumscribed rectangle of the column element as the outer contour.

[0028] To achieve the above object of the invention, a second aspect of the embodiment provides a housing type automatic construction device based on pictures, including:

[0029] A housing type element classification module for classifying housing type pictures using a semantic segmentation model;

[0030] The housing type element construction module is used to extract candidate key points in the housing type element classification result by using a key point detection model. After screening the candidate key points, it breaks and clusters the walls based on the determined key points to construct wall elements. It is also used to perform image connected component detection according to the housing type element classification result to construct door elements, window elements, and column elements, and associate these three elements with the wall elements.

[0031] The housing type correction module corrects the constructed wall elements, door elements, window elements, and column elements in combination with the association result to obtain the constructed housing type diagram.

[0032] To achieve the above-mentioned invention purpose, an embodiment of the third aspect provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the picture-based housing type automatic construction method described in the first aspect.

[0033] Compared with the prior art, the beneficial effects of the present invention at least include:

[0034] Adopting a machine learning method, learning the housing type element features from a large amount of housing type picture data, enabling the semantic segmentation model and the key point detection model to have stronger generalization performance, and being able to achieve the housing type element classification and key point detection of various style housing type pictures.

[0035] Using the key point detection model to achieve the instance-level segmentation of wall elements, constructing individual wall elements based on the instance-level segmentation result, thus avoiding a large amount of ineffective and complex post-processing logic and improving the import efficiency and accuracy.

[0036] Combining the housing type element classification by the semantic segmentation model, the construction of wall elements by the key point detection model, the construction of door elements, window elements, and column elements by the connected component detection, and the correction processing of elements, realizing the automatic construction of the housing type, saving the user's time for drawing the housing type, and improving the housing type drawing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0038] Figure 1 It is a flowchart of the picture-based housing type automatic construction method provided by the embodiment.

[0039] Figure 2It is the classification result diagram of the house type elements in the house type pictures provided by the embodiment;

[0040] Figure 3 It is the structure result of the wall elements provided by the embodiment;

[0041] Figure 4 It is the structure schematic diagram of the automatic house type construction device based on pictures provided by the embodiment. Detailed implementation manners

[0042] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific implementation manners described herein are only used to explain the present invention and do not limit the protection scope of the present invention.

[0043] To solve the problems that designers cannot search for house types and need to spend a lot of time manually drawing house types, and the traditional house type construction based on rule import has poor effects, the embodiment provides an end-to-end automatic house type construction method and device based on pictures. Based on deep learning, by learning a large number of pictures and solutions, a semantic segmentation model and a key point detection model that can automatically classify house type elements with high precision and speed are constructed. Based on these two models and combined with image processing methods, when a house type picture is given, the house type elements can be automatically extracted and constructed, improving the house type design efficiency.

[0044] Figure 1 It is the flowchart of the automatic house type construction method based on pictures provided by the embodiment. As Figure 1 shown, the automatic house type construction method based on pictures provided by the embodiment includes the following steps:

[0045] Step 1, house type element classification, that is, using the semantic segmentation model to classify the house type elements in the house type picture.

[0046] House type element classification refers to pixel classification of any input house type picture, and determining house type elements according to the pixel classification result. Among them, house type elements include wall elements, door elements, window elements, column elements, etc., and the areas corresponding to these elements are wall areas, door areas, window areas, column areas, etc.

[0047] Specifically, the house type element classification process includes:

[0048] First, perform a normalization operation on the house type picture, specifically including resetting the picture size of the house type picture and performing pixel standardization on the house type picture according to the standard value and variance;

[0049] Then, input the normalized house type pictures of the normalization operation into the semantic segmentation model. After calculation, the probability that each pixel belongs to different semantic categories is obtained. For each pixel point, compare the probabilities of different semantic categories, and select the maximum probability as the house type element category of the current pixel point, thus realizing the classification of house type elements. Figure 2 It is a visualization diagram of an exemplary classification result of house type elements.

[0050] In the embodiment, the semantic segmentation model is constructed based on the PSPNet model. The specific construction process includes: using the house type pictures with the true labels of house type elements as sample data, inputting the sample data into the PSPNet model, and taking the minimum of the cross-entropy loss function between the predicted label and the true label of the house type elements output by the model as the optimization goal to optimize the parameters of the PSPNet model. After the optimization is completed, extract the PSPNet model with the determined parameters as the semantic segmentation model.

[0051] Since the semantic segmentation model is constructed through learning a large number of pictures and schemes, it has strong generalization ability, can realize the classification of house type elements for house type pictures of different styles, and can not only ensure the classification accuracy but also improve the classification efficiency.

[0052] Step 2, construction of house type elements, that is, use the key point detection model and connected component detection to construct wall elements, door elements, window elements, and column elements, and associate the door elements, window elements, and column elements with the wall elements.

[0053] In the embodiment, use the key point detection model to extract the candidate key points in the classification result of house type elements. After screening the candidate key points, based on the determined key points, break and cluster the walls to construct wall elements. The specific process includes:

[0054] First, perform binarization and normalization operations on the classification result of house type elements. Among them, the normalization operation includes resetting the picture size of the classification result of house type elements and performing pixel standardization on the classification result of house type elements according to the standard value and variance.

[0055] Then, input the picture of the classification result of house type elements after the normalization operation into the key point detection model. After calculation, a number of candidate key points and their confidence levels are obtained, and a number of qualified key points are selected from the candidate key points based on a certain threshold.

[0056] Next, after breaking the wall markings in the picture of the classification result of house type elements based on the key points, cluster the pixels belonging to the same wall, and calculate the starting pixel, ending pixel, and thickness of the pixel area of each cluster to construct wall elements. In the embodiment, the starting pixel, ending pixel, and thickness of the pixel area can be determined according to the positional relationship of the pixel points in the cluster.

[0057] Finally, extract the load-bearing wall areas in the classification results of housing type elements, judge the overlapping area between the wall elements and the load-bearing wall areas, and regard the wall elements with an overlapping area greater than the set first overlapping area threshold as load-bearing wall elements.

[0058] In the embodiment, the first overlapping area threshold is set according to the actual application situation and is not specifically limited. The key point detection model is constructed based on the CenterNet model. The specific construction process includes: using the housing type pictures marked with the key points of the housing type area as sample data, inputting the sample data into the CenterNet model, and optimizing the parameters of the CenterNet model by integrating the original loss function of the CenterNet model. After the optimization is completed, extract the CenterNet model determined by the parameters as the key point detection model. Figure 3 Exemplarily, the candidate key points obtained by performing key point detection using this key point detection model are given. The visualization results of the wall key points determined after screening are as Figure 3 shown.

[0059] In the embodiment, image connected component detection is performed according to the classification results of housing type elements to construct door elements, window elements, and column elements, and these three types of elements are associated with the wall elements. The specific process includes:

[0060] First, according to the detailed categories of housing type elements, the detailed categories include sliding doors, single-leaf doors, double-leaf doors, ordinary windows, bay windows, floor-to-ceiling windows, flue ducts, and ordinary columns, etc., and screen out the image areas in the classification results of housing type elements whose corresponding pixel categories are the above-mentioned detailed categories.

[0061] Then, cluster the adjacent pixels with the same pixel value into a pixel connected area, perform image dilation on the pixel area, screen the image dilation result according to the set threshold, and construct door elements, window elements, and column elements by combining the screened pixel connected areas with the corresponding pixel classifications.

[0062] Finally, after extracting the contours of each pixel connected area, calculate the overlapping areas between the wall elements and the door elements, window elements, and column elements respectively. If the overlapping area exceeds the second overlapping area threshold, then associate the wall elements with the door elements, window elements, and column elements, thus realizing the associated binding of the door elements, window elements, and column elements with the wall instances.

[0063] Step 3, housing type correction, that is, correct the constructed wall elements, door elements, window elements, and column elements in combination with the association results to obtain the constructed housing type drawing.

[0064] The housing type elements constructed in Step 2 are relatively rough, so these housing type elements need to be corrected again. The specific correction process includes:

[0065] For wall elements, a search loop algorithm can be used to calculate the closed loops of wall elements. Each closed loop forms a room element, and collinearity judgment is performed on all wall elements to merge the same type of walls, and the thickness of the longest wall in the collinear walls is used as the thickness of the merged wall;

[0066] For door elements and window elements, based on the association results between door elements, window elements and wall elements, the center lines of doors and windows are calculated according to the walls to which the doors and windows are attached and normalized processing is performed, including: the lengths of door elements and window elements cannot exceed the walls to which they belong, collinear doors and windows are merged, and the door type and window type of the merged structure are set according to the door elements and window elements with the largest area, illegal door elements or window elements where the center lines of the doors intersect are removed, and the door elements or window elements with the larger area among the intersecting elements are retained;

[0067] For column elements, when there is an intersection between the column element and the wall, the contour of the wall will cut the contour of the column, and the circumscribed rectangle of the contour formed after the wall cuts the column is calculated as the outer contour of the column; when there is no intersection between the column element and the wall, the circumscribed rectangle of the column element is calculated as the outer contour.

[0068] So far, all the house type elements in the given input house type picture have been extracted, including walls, doors, windows, columns and rooms. The two-dimensional vector information of the obtained house type elements can be visualized in the design tool to obtain a complete house type plan for users to perform layout operations.

[0069] Compared with the traditional method of identifying house type elements in pictures through rules, the above-mentioned embodiment provides a method for automatically constructing house types based on pictures, which uses a machine learning method to learn the characteristics of house type elements through a large number of house type picture data, so that the semantic segmentation model and the key point detection model have stronger generalization performance, and can realize the classification of house type elements and the detection of key points for various style house type pictures.

[0070] Compared with other recognition methods that combine semantic segmentation with post-processing rules, the above-mentioned embodiment provides a method for automatically constructing house types based on pictures, which uses a key point detection model to realize the instance-level segmentation of wall elements, and constructs individual wall elements based on the instance-level segmentation results, thus avoiding a large amount of ineffective and complex post-processing logic and improving the import efficiency and accuracy.

[0071] The method for automatically constructing house types based on pictures provided by the above-mentioned embodiment combines a semantic segmentation model to realize house type element classification, a key point detection model to realize wall element construction, a connected component detection to realize the construction of door elements, window elements and column elements, and the correction processing of elements, realizing the automatic construction of house types, saving the user's time for drawing house types and improving the house type drawing efficiency.

[0072] Figure 4It is a schematic structural diagram of a device for automatically constructing house types based on pictures provided by an embodiment. As Figure 4 shown, the device for automatically constructing house types provided by the embodiment includes:

[0073] A house type element classification module, configured to classify house type elements in a house type picture by using a semantic segmentation model;

[0074] A house type element construction module, configured to extract candidate key points in the house type element classification result by using a key point detection model, and after screening the candidate key points, interrupt and cluster the walls based on the determined key points to construct wall elements; it is also configured to perform image connected component detection according to the house type element classification result to construct door elements, window elements, and column elements, and associate these three types of elements with the wall elements;

[0075] A house type correction module, which corrects the constructed wall elements, door elements, window elements, and column elements in combination with the association result to obtain the constructed house type diagram.

[0076] It should be noted that when the device for automatically constructing house types provided in the above embodiment performs automatic house type construction, the above example should be given according to the division of the above functional modules. The above functions can be completed by different functional modules as needed, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the device for automatically constructing house types provided in the above embodiment and the embodiment of the method for automatically constructing house types belong to the same concept. For the specific implementation process, please refer to the embodiment of the method for automatically constructing house types, which will not be elaborated here.

[0077] The embodiment also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above method for automatically constructing house types based on pictures, including the following steps:

[0078] Step 1, house type element classification, that is, classifying house type elements in a house type picture by using a semantic segmentation model.

[0079] Step 2, house type element construction, that is, constructing wall elements, door elements, window elements, and column elements by using a key point detection model and connected component detection, and associating the door elements, window elements, and column elements with the wall elements.

[0080] Step 3, house type correction, that is, correcting the constructed wall elements, door elements, window elements, and column elements in combination with the association result to obtain the constructed house type diagram.

[0081] It should be noted that the computer memory can be a volatile memory proximal to the computer, such as RAM, or a non-volatile memory, such as ROM, FLASH, floppy disk, mechanical hard disk, etc., or a remote storage cloud. The computer processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), that is, the steps of automatic floor plan construction based on pictures can be implemented through these processors.

[0082] The above specific embodiments have elaborated in detail on the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not used to limit the present invention. Any modifications, supplements, equivalent replacements, etc. made within the principle scope of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for automatically constructing a house type based on pictures, characterized in that, it includes the following steps: Using a semantic segmentation model to classify house type elements in the house type picture; Using a key point detection model to extract candidate key points in the house type element classification result, and after screening the candidate key points, interrupting and clustering the walls based on the determined key points to construct wall elements; Performing image connected component detection based on the house type element classification result to construct door elements, window elements, and column elements, including: according to the house type element classification result, clustering adjacent pixels with the same pixel value into a pixel connected region, and performing image dilation on the pixel region, screening the image dilation result according to the set threshold, and combining the screened pixel connected regions with the corresponding pixel classifications to construct door elements, window elements, and column elements; and associating these three elements with the wall elements, including: after extracting the contour of each pixel connected region, calculating the overlapping area between the wall elements and the door elements, window elements, and column elements respectively, if the overlapping area exceeds the second overlapping area threshold, then associating the wall elements with the door elements, window elements, and column elements; Combining the association results to correct the constructed wall elements, door elements, window elements, and column elements to obtain the constructed house type diagram, and the correction process includes: For the wall elements, calculating closed loops for the wall elements, each closed loop forms a room element, and performing collinearity judgment on all wall elements, merging the same type of walls, and taking the thickness of the longest wall in the collinear walls as the thickness of the merged wall; For the door elements and window elements, according to the association results between the door elements, window elements and the wall elements, calculating the door center line and window center line according to the walls to which the doors and windows are attached, and performing normalization processing, including: the lengths of the door elements and window elements cannot exceed the walls to which they belong, merging the collinear doors and windows, and setting the door type and window type of the merged structure according to the door elements and window elements with the largest area, removing the door elements or window elements with illegal intersections of the door center line and window center line, and retaining the door elements or window elements with the larger area among the intersecting elements; For the column elements, when there is an intersection between the column element and the wall, calculating the circumscribed rectangle of the contour formed after the wall cuts the column as the outer contour of the column; when there is no intersection between the column element and the wall, then calculating the circumscribed rectangle of the column element as the outer contour.

2. The method for automatically constructing a house type based on pictures according to claim 1, characterized in that, before classifying the house type elements, performing a normalization operation on the house type picture, including resetting the picture size of the house type picture and performing pixel standardization on the house type picture according to the standard value and variance; Inputting the normalized house type picture into the semantic segmentation model for classifying the house type elements; before extracting the candidate key points, performing binarization and normalization operations on the house type element classification result, wherein the normalization operation includes resetting the picture size of the house type element classification result and performing pixel standardization on the house type element classification result according to the standard value and variance; Inputting the normalized house type element classification result into the key point detection model for extracting candidate key points.

3. The method for automatically constructing a house type based on pictures according to claim 1, characterized in that, The semantic segmentation sub-model is constructed based on the PSPNet model with optimized parameters; The key point detection model is constructed based on the CenterNet model with optimized parameters.

4. The method for automatically constructing a house type based on a picture according to claim 1, characterized in that, Candidate key points are screened according to a set threshold to determine key points. After interrupting the wall markings in the classification result of house type elements based on the key points, pixels belonging to the same wall are clustered, and the starting pixel, ending pixel, and thickness of the pixel area of each cluster are calculated to construct wall elements.

5. The method for automatically constructing a house type based on a picture according to claim 1, characterized in that, After constructing the wall elements, the load-bearing wall area in the classification result of house type elements is extracted, the overlapping area between the wall elements and the load-bearing wall area is judged, and the wall elements with an overlapping area greater than the set first overlapping area threshold are used as load-bearing wall elements.

6. A device for automatically constructing a house type based on a picture, implemented by using the method for automatically constructing a house type based on a picture according to any one of claims 1-5, characterized in that, comprising: A house type element classification module for classifying house type pictures by using a semantic segmentation model; A house type element construction module for extracting candidate key points in the classification result of house type elements by using a key point detection model, and after screening the candidate key points, interrupting and clustering the walls based on the determined key points to construct wall elements; It is also used to detect image connected components according to the classification result of house type elements to construct door elements, window elements, and column elements, and associate these three elements with the wall elements; A house type correction module for correcting the constructed wall elements, door elements, window elements, and column elements in combination with the association result to obtain the constructed house type diagram.

7. A computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the steps of the method for automatically constructing a house type based on a picture according to any one of claims 1-5 are implemented.

Images