Method and equipment for segmenting house type image subspace and extracting structural data

Abstract

The invention discloses a method and equipment for segmenting a house type image subspace and extracting structural data, and relates to the technical field of computers.The method comprises the steps that the category of the house type image subspace is defined, a house type image is marked through category definition to obtain a marked file, and a training data set is constructed; key area detection is carried out on the house type image in the training data set, and key areas of the house type image and the annotation file are intercepted to form a final house type image training data set; training an AI model according to the house type image training data set to obtain segmentation results of all categories in the house type image; and carrying out optimization processing on the segmentation result, obtaining the contour of each subspace and door and window structure information through an image processing method, and generating a structure schematic grey-scale map of each subspace. Compared with an existing method, a subspace extraction method is added to house type image recognition, and a soft decoration designer can be helped to obtain house type image space data more conveniently.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device that can help home designers to automatically disassemble and analyze, segment subspaces of floor plans, and extract structural data. Background Art

[0002] Soft furnishing design is a very important step in house decoration. It refers to designing the layout of furniture for each room in the house, selecting furniture styles and sizes, etc. Generally, the designer refers to the residential floor plan to decompose the walls, doors, windows and each sub-space structure (bedroom, living room, dining room, etc.) to draw CAD drawings, create 3D models, render furniture in them, generate renderings and show them to customers.

[0003] Whether the designer's judgment on the structure and size of the subspace in the floor plan is accurate directly affects the design effect. Therefore, the accuracy of the structural data of the subspace in the floor plan is the basis of all work.

[0004] In this process, the designer will disassemble the doors, windows, walls and other structures in the floor plan step by step, and then draw them step by step, which is inefficient. An existing floor plan recognition method can identify the walls, doors, windows and subspaces of the floor plan and generate a vector map recognition result. However, this method does not perform separate disassembly and structural information analysis on the subspace. In some application scenarios, designers also need to obtain the structure and size information of each subspace before they can perform soft decoration design work on the space. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method and device for segmenting a floor plan subspace and extracting structural data, so as to help soft furnishing designers quickly obtain floor plan data.

[0006] An aspect of an embodiment of the present invention provides a method for segmenting a subspace of a floor plan and extracting structural data, comprising: Define the subspace category of the floor plan, annotate the floor plan through the category definition to obtain the annotation file, and build a training data set; Performing key area detection on the floor plan in the training data set, and intercepting key areas of the floor plan and the annotation file to form a final floor plan training data set; Train the AI ​​model according to the floor plan training data set to obtain segmentation results of all categories in the floor plan; The segmentation results are optimized, and the outline of each subspace and the door and window structure information are obtained through an image processing method, and a schematic grayscale image of the structure of each subspace is generated.

[0007] Optionally, the method further includes defining a category of the floor plan subspace, annotating the floor plan by the category definition to obtain an annotation file, and constructing a training data set, which step includes: Analyze a large number of floor plans and define the subspace categories that should be included in the floor plan, including: living room, dining room, kitchen, balcony, master bedroom, bedroom, master bathroom, toilet, other rooms, and walls, doors, and windows; The floor plan is annotated according to the subspace category, specifically in the form of annotating different subspaces at the pixel level with different colors to obtain an annotation file of the floor plan.

[0008] Optionally, the method further comprises the step of detecting key areas of the floor plan in the training data set, and intercepting key areas of the floor plan and the annotation file to form a final training data set, the step comprising: Use a pre-trained target detection algorithm to detect key areas of the floor plan; Intercepting the floor plan and the corresponding marked file area according to the detected key area; The intercepted floor plan and the annotation file are sorted to obtain a data set for AI model training.

[0009] Optionally, the method further comprises using a pre-trained target detection algorithm to perform key area detection on the floor plan, the step comprising: Mark key areas on the floor plan and generate a key area dataset for the floor plan; Training a target detection model based on the key area dataset of the floor plan; The target detection model is used to detect key areas of the floor plan.

[0010] Optionally, the method further includes training an AI model according to the floor plan training data set to obtain segmentation results of all categories in the floor plan, and the step includes: DeepLabv3+ is used as the backbone network, and the training on the floor plan dataset is completed by setting the input preprocessing process and the multi-task loss function; The loss functions for multiple tasks are defined as: cross entropy loss function, LovaszSoftmaxLoss function, affinity field loss function; A fixed weight is set for each of the loss functions, and their sum is used as the final loss function; The trained AI model is used to identify the floor plan and obtain semantic segmentation prediction results.

[0011] Optionally, the method further includes optimizing the segmentation result, obtaining the contour and door and window structure information of each subspace by an image processing method, and generating a schematic grayscale image of the structure of each subspace. This step includes: The blank areas in the semantic segmentation prediction result are filled according to the local adjacent pixel categories, and the door, window, and wall categories are linearized to obtain an optimized semantic segmentation result; Extracting the contour, door and window structure information of each subspace according to the optimized semantic segmentation result; A grayscale image representing the structure of each subspace is drawn according to the contour, door and window structure information of each subspace.

[0012] Optionally, the method further comprises extracting the outline, door and window structure information of each subspace according to the optimized semantic segmentation result, the step comprising: For each subspace category, the subspace corresponding to the color is extracted by using the HSV color, that is, only the binary mask image of the color block is extracted from the optimized semantic segmentation result; Perform edge detection on the binary mask image by using the edge detection Canny algorithm; find the starting point of the contour according to the edge information, traverse the entire contour according to the preset rules, and store the points on the contour; connect adjacent contour points into a complete contour; Optimize each contour; Find the starting points of doors and windows and record their locations.

[0013] Optionally, the method further comprises optimizing each contour, the step comprising: Deleting two adjacent points in the contour whose distance is no more than 20 cm in sequence, and adding the middle point between the two points to the contour; Sequentially straighten two connected line segments formed by three adjacent points in the contour so that the two line segments are connected horizontally or vertically.

[0014] Optionally, the method further comprises the step of finding the starting point of the door and window and recording the position, which step comprises: Traversing the points in the optimized subspace contour, determining the category of the points within a radius of 5 pixels in the optimized semantic segmentation result; If the category is a door in the subspace category, then the point coordinates are added to the array representing the door; if the category is a window in the subspace category, then the point coordinates are added to the array representing the door; Select the coordinates of the first and last points in the array representing the door as the starting coordinate points of the door in the subspace; The coordinates of the first and last points in the array representing the window are selected as the starting coordinate points of the window in the subspace.

[0015] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method described above.

[0016] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0017] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0018] The embodiments of the present invention define categories of subspaces of floor plans, and construct an algorithm training data set by annotating the data set according to the definition; use the data set to train a semantic segmentation model, which can use floor plans as input and output semantic segmentation results of floor plans; based on the semantic segmentation results of floor plans, extract each subspace structure in the floor plan through a variety of image processing methods, including the outline of each subspace and door and window position information, and save them in the form of grayscale images. Compared with existing methods for identifying floor plans, the present invention adds a subspace extraction method, which is a method obtained based on the process processing of all steps of the present invention, and can help soft furnishing designers obtain floor plan space data more conveniently. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a flowchart of the overall steps of an embodiment of the present invention; Figure 2 It is an illustration of the key areas in the floor plan; Figure 3a-3c is the segmentation result image; Figure 4a-4b It is a grayscale visualization of the structural information of each subspace in the floor plan. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0022] In order to train a deep learning model for identifying floor plans, we collected common floor plans on the market, defined subspace categories, and labeled floor plans to construct a data set. We also optimized the original data set using a key area detection algorithm to obtain the data set used by the training model.

[0023] After training the deep learning model, the model is used to obtain the segmentation result of the floor plan by taking the floor plan as input. The subspace is extracted on the segmentation result map and the spatial contour and door and window structure information are obtained. First, the grayscale image of the subspace is extracted based on the rule of different coloring of different spaces in the segmentation result. Then the subspace contour is calculated by iterative adaptive point algorithm. Then, the number of subspace contour vertices is deleted through iterative optimization to obtain a better contour. Then, by scanning each point of each edge in the contour, the element category near the point is determined to obtain the door and window position in the space. Finally, the grayscale representation of the subspace is drawn using the corresponding position information of the contour, door, and window.

[0024] Specifically, this example describes a method for segmenting a subspace of a floor plan and extracting structural data. The method flow chart is as follows: Figure 1 As shown. First, collect common floor plans on the market, define subspace categories, and annotate floor plans to construct a data set. Then optimize the original data set through a key area detection algorithm to obtain a data set used by the training model. Then train the deep learning model to obtain the floor plan segmentation result map. Then process the segmentation result map to obtain the subspace contour and door and window structure information, and draw a grayscale representation map. Specifically, it includes the following steps: 1. Define subspace categories and create a floor plan recognition dataset; I. Define the subspace categories in the floor plan as balcony, living room, dining room, kitchen, second bedroom, master bedroom, second bathroom, master bathroom, cloakroom, entrance, corridor, other areas, door, window and wall; II. Label each floor plan based on the categories by using rectangles or polygons to label each category subspace with different colors.

[0025] 2. Use the key area detection model to process the original data set to obtain the training data set; I. Create the data set required for the key area detection model; a. If Figure 2As shown in the figure, the key area of ​​the floor plan refers to the main area of ​​the floor plan after removing advertisements and additional promotional information from the original picture; b. Select 500 original floor plans and corresponding annotated pictures from the above data set to intercept the same area, and obtain the floor plans and annotated pictures of key areas; c. Arrange the above key area floor plans and key area annotation maps to obtain the key area data set; II. Using the above key area dataset, the key area detection model is trained using the target detection YOLO algorithm; III. Use the key area detection model trained in step II to process the floor plan recognition dataset, obtain the key areas of all original floor plans, and perform the same interception processing on the floor plans and annotated drawings in the floor plan recognition dataset based on the key area range. Finally, the floor plan recognition dataset used to train the floor plan recognition model is obtained.

[0026] 3. Train the deep learning model to segment and identify the floor plan; DeepLabv3+ is used as the backbone network, and the training on the floor plan dataset is completed by setting the input preprocessing process and the multi-task loss function; I. Use multiple loss functions, including cross entropy loss function, LovaszSoftmaxLoss function, affinity field loss function: d. Cross Entropy Loss Function

[0027] in is the label of the element in the ith floor plan, is the probability that the i-th pixel belongs to the k-th category in the floor plan. K represents the total number of categories in the floor plan; e. LovaszSoftmaxLoss function

[0028] in is the Jaccard index of the kth category of the floor plan; f. Affinity field loss function

[0029]

[0030]

[0031]

[0032] in represents the neighborhood, is the KL divergence between the two distributions, and the loss is for all categories Loss and The sum of losses; II. Define the weights of each loss function as 0.4, 0.3, and 0.3 respectively, and calculate the final loss function as the weighted sum of the above d, e, and f loss functions; III. Train the floor plan recognition model through the loss function in II, and use the floor plan as input to obtain the floor plan segmentation result map, such as Figure 3a-3c , different shades of colors represent different categories.

[0033] 4. Process the segmentation result image to obtain the contour of each subspace and the door and window structure information; I. Fill the blank areas in the semantic segmentation prediction results according to the local adjacent pixel categories, and perform linear processing on the door, window, and wall categories to obtain an optimized semantic segmentation result; g. Fill in the interrupted doors, windows and walls through the closing operation in morphological operations; h. Except for the living room, dining room, corridor, and entrance, a single area surrounded by boundaries (doors, windows, walls) is treated as one type of space. The processing logic is: if there are more than two types of element points in the area, the category with the largest number of element points is selected as the final category of the area; II. Extracting the contour, door and window structure information of each subspace according to the optimized semantic segmentation results; i. on the segmentation result image, extract different subspace regions by HSV color and convert them into grayscale images; j. Firstly, the grayscale image is processed by an edge detection algorithm to extract the edge to obtain an initial contour, and then the minimum polygon of the contour, i.e. the optimized contour, is obtained by an iterative adaptive point algorithm; k. Further iterative optimization of the contour; a) When the distance between two vertices of a polygon is less than a threshold, the two points are merged into one point, and the coordinates are the average of the two vertex coordinates. The above threshold is half the thickness of the wall; b) When the vector formed by two adjacent points is offset within 20° in the horizontal or vertical direction, the vector is straightened. The specific operation is to align the coordinates of the point with the smaller horizontal or vertical coordinate with the coordinates of the other point, so that the two points are aligned in the horizontal or vertical direction; l. Find the position of doors and windows on the outline; a) Scan along the contour edge to obtain the point category around each pixel. If a door category pixel is found, record it; the first and last points finally recorded are the starting and ending points of the door; b) Scan along the contour edge to obtain the point category around each pixel. If a window category pixel is found, record it; the first and last points finally recorded are the starting point and middle point of the window; III. Draw a grayscale image representing the structure of each subspace according to the contour, door and window structure information of each subspace. Figure 4a-4b , where black represents the background, white lines represent the outline of the subspace (walls), light white dashed lines represent windows, and openings represent doors; In summary, the method of the present invention has the following characteristics: After applying this method, designers in the field of soft decoration can directly obtain the structure of each subspace in the floor plan, including the shape outline and door and window information of the subspace, which helps designers to further complete the design and analysis of the subspace. The operation is simple, which saves designers the work of analyzing floor plans and greatly improves the work efficiency of designers in residential soft decoration design.

[0034] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0035] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0036] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0037] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0038] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0039] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0040] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0041] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0042] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for segmenting a subspace of a floor plan and extracting structural data, characterized in that: include: Define the subspace category of the floor plan, annotate the floor plan through the category definition to obtain the annotation file, and build a training data set; Performing key area detection on the floor plan in the training data set, and intercepting key areas of the floor plan and the annotation file to form a final floor plan training data set; Train the AI ​​model according to the floor plan training data set to obtain segmentation results of all categories in the floor plan; Optimizing the segmentation results, obtaining the outline of each subspace and door and window structure information through an image processing method, and generating a grayscale diagram of the structure of each subspace; The blank areas in the semantic segmentation prediction results are filled according to the local adjacent pixel categories, and the door, window, and wall categories are linearized to obtain the optimized semantic segmentation results; According to the optimized semantic segmentation result, the contour, door and window structure information of each subspace is extracted, which includes extracting the subspace corresponding to the color through HSV color for each subspace category, that is, only extracting the binary mask image of the color block in the optimized semantic segmentation result; performing edge detection on the binary mask image through the edge detection Canny algorithm; finding the starting point of the contour according to the edge information, and traversing the entire contour according to the preset rules, and storing the points on the contour; connecting adjacent contour points into a complete contour; optimizing each contour; Find the starting point of doors and windows and record the location; A grayscale image representing the structure of each subspace is drawn according to the contour, door and window structure information of each subspace.

2. The method for segmenting a floor plan subspace and extracting structural data according to claim 1, characterized in that: The method of defining the subspace category of the floor plan, labeling the floor plan by the category definition to obtain a labeling file, and constructing a training data set includes: Analyze a large number of floor plans and define the subspace categories that should be included in the floor plans, including: living room, dining room, kitchen, balcony, master bedroom, second bedroom, master bathroom, second bathroom, entrance hall, corridor, cloakroom, other areas, and walls, doors, and windows; The floor plan is annotated according to the subspace category, specifically in the form of annotating different subspaces at the pixel level with different colors to obtain an annotation file of the floor plan.

3. The method for segmenting a floor plan subspace and extracting structural data according to claim 1, characterized in that: The detecting of key areas of the floor plan in the training data set and intercepting key areas of the floor plan and the annotation file to form a final training data set includes: Use a pre-trained target detection algorithm to detect key areas of the floor plan; Intercepting the floor plan and the corresponding marked file area according to the detected key area; The intercepted floor plan and the annotation file are sorted to obtain a data set for AI model training.

4. The method for segmenting a floor plan subspace and extracting structural data according to claim 3, characterized in that: The using a pre-trained target detection algorithm to perform key area detection on the floor plan includes: Mark key areas on the floor plan and generate a key area dataset for the floor plan; Training a target detection model based on the key area dataset of the floor plan; The target detection model is used to detect key areas of the floor plan.

5. The method for segmenting a floor plan subspace and extracting structural data according to claim 1, characterized in that: The AI ​​model is trained according to the floor plan training data set to obtain segmentation results of all categories in the floor plan, including: DeepLabv3+ is used as the backbone network, and the training on the floor plan dataset is completed by setting the input preprocessing process and the multi-task loss function; The loss functions for multiple tasks are defined as: cross entropy loss function, LovaszSoftmaxLoss function, affinity field loss function; A fixed weight is set for each of the loss functions, and their sum is used as the final loss function; Use the trained AI model to identify the floor plan and obtain semantic segmentation prediction results.

6. The method for segmenting a floor plan subspace and extracting structural data according to claim 1, characterized in that: The steps of optimizing each contour include: Deleting two adjacent points in the contour whose distance is no more than 20 cm in sequence, and adding the middle point between the two points to the contour; Sequentially straighten two connected line segments formed by three adjacent points in the contour so that the two line segments are connected horizontally or vertically.

7. The method for segmenting a floor plan subspace and extracting structural data according to claim 1, characterized in that: The step of finding the starting point of the door and window and recording the position includes: Traversing the points in the optimized subspace contour, determining the category of the points within a radius of 5 pixels in the optimized semantic segmentation result; If the category is a door in the subspace category, then the point coordinates are added to the array representing the door; if the category is a window in the subspace category, then the point coordinates are added to the array representing the door; Select the coordinates of the first and last points in the array representing the door as the starting coordinate points of the door in the subspace; The coordinates of the first and last points in the array representing the window are selected as the starting coordinate points of the window in the subspace.

8. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

Images