Method for generating data set based on semantic control, system for generating image

Abstract

The present invention provides a method for generating a data set based on semantic control and a system for generating an image, comprising: filtering out qualified data and partially qualified data from an original data set of a house layout diagram according to a data screening rule, and deleting unqualified data; performing image segmentation and target detection for semi-automatic annotation of qualified data to obtain house layout information; performing image segmentation and target detection for manually annotated partially qualified data to obtain house layout information; the house layout information comprises: room type and room coordinate points; judging and calculating the number of room types, room orientations, and room adjacent relationships according to the house layout information; generating text descriptions according to a text description template according to the number of room types, room orientations, and room adjacent relationships; filtering the text descriptions again, normalizing the data, and generating a final target detection data set; the present invention has the advantages of being able to shorten the design cycle and allowing non-professional house designers to participate in the design.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to a method for generating a data set based on semantic control and a system for generating an image. Background Art

[0002] Layout construction is an important application in the field of image generation research. Currently, layout construction can be performed through traditional methods based on hand-crafted energy minimization, powerful data-driven techniques, and methods for graph-constrained layout generation. In 2010, Merrel et al. learned the properties of rooms from a given dataset through a Bayesian network and synthesized layouts through random methods, but it is not suitable for large-scale generation. In 2014, Peng et al. designed a framework to generate floor plans by tiling the outline of an arbitrarily shaped building with a set of deformable templates; Ma et al. gave a set of polygonal "building blocks" and their connection constraints to generate different game-level layouts. These methods are based on energy minimization to generate house layouts with a small manual cost. In 2019, Wu et al. proposed a data-driven method to drive the generation of residential floor plans by learning thousands of sample data. The method starts with the living room, adds rooms sequentially through an encoder-decoder network, and then performs a final post-processing of vectorization; Li et al. proposed a non-sequential adversarial generation method called LayoutGAN, which has a self-attention mechanism in the generator and a wireframe renderer in the discriminator. In 2020, Nauata et al. proposed a House-GAN model that generates house layouts by inputting bubble maps.

[0003] However, these methods need to generate building layouts by adjusting the inner edges of a given building outline and cannot generate room layouts based on natural language.

[0004] As for the layout design of a house, it is a complex task that usually requires professional designers with professional knowledge. House design is a time-consuming iterative process. Usually, the process of a house design begins with drawing a simple dot-line diagram to illustrate the number, type, orientation and adjacent relationship of the rooms. Then the designer makes the corresponding floor plan, collects feedback from the customer, and returns to the drawing for modification and refinement. Such an iterative process may reduce the quality of the design given a limited budget and time.

[0005] In order to solve the above problems, people have been seeking an ideal technical solution. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies of the prior art. On the one hand, the present invention provides a method for constructing a house layout generation data set based on semantic control, which can shorten the design cycle of professional house designers and allow non-professional house designers to participate in the design.

[0007] On the other hand, the present invention provides a system for generating images based on a semantically controlled house layout adversarial network model.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] A first aspect of the present invention provides a method for constructing a house layout generation dataset based on semantic control, the method comprising:

[0010] Filter out qualified data and partially qualified data from the original data set of the house layout diagram according to a predetermined data screening rule, and delete unqualified data;

[0011] Performing semi-automatic image segmentation and target detection on the qualified data to obtain house layout information;

[0012] Performing manually annotated image segmentation and target detection on the partially qualified data to obtain house layout information;

[0013] The house layout information includes room type and room coordinate points;

[0014] Calculate the number of room types, room orientations, and room adjacency relationships based on the house layout information;

[0015] Generate a text description according to a text description template according to the number of room types, room orientations, and room adjacency relationships;

[0016] The text description is filtered again, and the data is normalized to generate a final target detection data set.

[0017] Based on the above, the room types include: living room LivingRoom, bedroom Bedroom, balcony Balcony, kitchen Kitchen, bathroom BathRoom and study Library.

[0018] Based on the above, the data screening rules are: the rooms in the house layout diagram have clear boundaries; the house layout diagram at least includes bedroom, kitchen, and bathroom room types; the house layout diagram is a whole, and the rooms included are connected to each other.

[0019] Based on the above, the method for generating a text description according to a text description template includes: automatically generating a corresponding text description for the number of each room type, the room orientation, and the room adjacency relationship according to a pre-set text description template, and storing it in an Excel table; the text description template is that each row represents a house, and the data of each row includes the house number, the number of LivingRooms, the number of Bedrooms, the number of Balconies, the number of Kitchens, the number of BathRooms, the number of Libraries, room orientations, and room adjacency relationships.

[0020] Based on the above, the method of calculating the number of room types, room orientations, and room adjacency relationships according to the house layout information includes:

[0021] According to the room type, the number of each room type included in each house layout diagram is counted; according to the room coordinate points, it is determined whether the room is a rectangular or polygonal annotation, and then the center point coordinates of the room are calculated, and according to the center point coordinates, the quadrant in which the room is located is determined, thereby determining the orientation of the room; all rooms in the house layout diagram are combined in pairs, and the Manhattan distance between the two rooms is calculated. If the Manhattan distance is less than half of the sum of the length and width of the outer boundary boxes of the two rooms plus θ, it is determined that the two rooms are adjacent.

[0022] Based on the above, the method of performing image segmentation and target detection on the qualified data for semi-automatic annotation to obtain house layout information includes: using a target detection model to perform image segmentation and target detection on the house layout diagram, and obtaining a target bounding box list and a corresponding confidence score list of the house layout image; performing non-maximum suppression processing on the target bounding box list and the corresponding confidence score list to obtain the room type and room coordinate point of each room in the house layout diagram; and renaming multiple rooms of the same category in the room type list using Arabic numerals.

[0023] Based on the above, the target detection model is a MaskRCNN model.

[0024] Based on the above, the target detection model is the YOLACT model.

[0025] Based on the above, a method for obtaining house layout information by manually annotating the partially qualified data through image segmentation and target detection is as follows: the room types on the house layout diagram in the partially qualified data are annotated using the labelme tool; after the annotation is completed, a json file is generated for each house layout diagram; the json file is parsed to obtain the room type and room coordinate points in each house layout diagram.

[0026] A second aspect of the present invention provides a system for generating images based on a semantically controlled house layout adversarial network model, the system comprising:

[0027] A pre-trained text encoder takes the dataset obtained by the construction method as input and generates corresponding sentence vectors according to the text description in the dataset;

[0028] The model training module establishes a generative adversarial network model including a generator, a discriminator and a loss function; the sentence vector of the text encoder is used as the input of the generative adversarial network model to train the generative adversarial network model to obtain a trained generative adversarial network model;

[0029] The image generation module inputs the text description to be tested into the trained generative adversarial network model to obtain the generated image.

[0030] Compared with the prior art, the present invention has outstanding substantive features and significant progress, specifically:

[0031] On one hand, the present invention obtains house layout information by screening original data, and then uses semi-automatic annotation of house layout diagram for qualified data, performs image segmentation and target detection; manually annotates house layout diagram for some qualified data, performs image segmentation and target detection, and obtains house layout information; then generates text descriptions about the number of room types, room orientations, and room adjacent relationships according to the house layout; finally, the text description template is filtered again, the data set is normalized, and the final data set is generated; thereby providing a method for constructing a house layout generation data set based on semantic control, which can shorten the design cycle of professional house designers and allow non-professional house designers to participate in the design.

[0032] On the other hand, the present invention uses a data set as the input of a generative adversarial network model, conducts a comprehensive analysis of the data characteristics of the data set, and finally outputs a semantically-based house layout image through a trained generative adversarial network model, thereby providing a system for generating images based on a semantically-controlled house layout adversarial network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is an example diagram of a house layout diagram with text description in one embodiment of the present invention.

[0034] Figure 2 It is a schematic diagram of the framework of a method for constructing a house layout generation data set based on semantic control according to the present invention.

[0035] Figure 3 It is a flow chart of generating a data set in one embodiment of the present invention.

[0036] Figure 4 This is an example diagram of manually annotating and generating text descriptions in an embodiment of the present invention.

[0037] Figure 5 This is an example diagram of semi-automatic annotation to generate text description in an embodiment of the present invention.

[0038] Figure 6 It is a network architecture diagram of a system for generating images based on a semantically controlled house layout adversarial network model of the present invention.

[0039] Figure 7 It is a schematic diagram of the AttnGAN model of the present invention stacking multiple generators and discriminators to generate high-resolution images. DETAILED DESCRIPTION

[0040] The technical solution of the present invention is further described in detail below through specific implementation methods.

[0041] Example 1

[0042] like Figure 2 As shown, this embodiment provides a method for constructing a house layout generation dataset based on semantic control, including:

[0043] Step 1, filtering out qualified data and partially qualified data from the original data set of the house layout diagram according to the data screening rules, and deleting unqualified data; wherein the original data set is derived from the open source 3D Future data set;

[0044] The data screening rules are as follows: the boundaries of the rooms in the house layout diagram need to be clear; the house layout diagram at least includes the room types of bedroom, kitchen, and bathroom; the house layout diagram is a whole, and the rooms included are interconnected.

[0045] Step 2, performing semi-automatic image segmentation and target detection on the qualified data to obtain house layout information; performing manually labeled image segmentation and target detection on the partial qualified data to obtain house layout information; the house layout information includes: room type, room coordinate points; the room type includes 6 room types, namely living room LivingRoom, bedroom Bedroom, balcony Balcony, kitchen Kitchen, bathroom BathRoom and study Library.

[0046] Step 3, based on the house layout information, determine and calculate the number of room types, room orientations, and room adjacency relationships; generate a text description based on the number of room types, room orientations, and room adjacency relationships according to a text description template; filter the text description again, normalize the data, and generate a final target detection data set.

[0047] The present invention obtains house layout information by screening the original data, using semi-automatic annotation of house layout diagram for qualified data, performing image segmentation and target detection; manually annotating house layout diagram for part of qualified data, performing image segmentation and target detection, and obtaining house layout information; then generating text descriptions based on the extracted house layout information through calculation and judgment; finally filtering the text description template again, normalizing the data set, and generating the final data set; according to the data set construction principle, after instance annotation and strict review and screening, the final constructed house data set contains 4938 images with a resolution of 640×480; in order to support the application of image generation technology under semantic control in house design, this embodiment constructs a data set with large data volume, rich scenes, and high label quality. At the same time, the data set can provide data support for understanding house layout and identifying room types; thereby providing a method for constructing a house layout generation data set based on semantic control, which can shorten the design cycle of professional house designers and allow non-professional house designers to participate in the design.

[0048] Determination of the number of room types, room orientation, and room adjacency

[0049] According to the house layout information, the number of room types, room orientations, and room adjacency relationships are determined and calculated, the method comprising: according to the room types, counting the number of each room type included in each house layout diagram;

[0050] According to the room coordinate points, determine whether the room is a rectangle or a polygon, then calculate the center point coordinates of the room, and according to the center point coordinates, determine the quadrant where the room is located, thereby determining the orientation of the room;

[0051] All rooms in the house layout are combined in pairs and the Manhattan distance between two rooms is calculated. If the Manhattan distance is less than half the sum of the length and width of the outer bounding boxes of the two rooms plus θ, the two rooms are considered to be adjacent. θ can be any very small number, such as 0.001.

[0052] Generate text description

[0053] like Figure 1 As shown, the text description is generated according to the text description template based on the number of room types, room orientation, and room adjacent relationship; the text description generation includes automatically generating a corresponding text description according to the number of each room type, room orientation, and room adjacent relationship according to a preset text description template, and storing it in an Excel table;

[0054] The text description template has each line representing a house, and the data in each line includes the house number, the number of Living Rooms, the number of Bedrooms, the number of Balconies, the number of Kitchens, the number of Bath Rooms, the number of Libraries, the room orientation, and the room adjacency relationship.

[0055] Qualified Data

[0056] like Figure 5 As shown, the method of performing semi-automatic image segmentation and target detection on the qualified data to obtain house layout information includes:

[0057] S31. Use the target detection model to perform image segmentation and target detection on the house layout image, and obtain a list of target bounding boxes of the house layout image and a list of their corresponding confidence scores;

[0058] Preferably, the target detection model is a MaskRCNN model;

[0059] The method of using the MaskRCNN model to obtain a list of target bounding boxes and their corresponding confidence scores for a house layout image includes:

[0060] S311. Using the house layout diagram in the qualified data as input to the RPN network;

[0061] S312. Use the RPN network to generate a candidate region ROI;

[0062] S313. Extract features from the candidate region ROI using a ResNet-101 residual convolutional network;

[0063] S314. Then use FPN (Feature Pyramid Network) to perform multi-scale target detection, fuse the features of each level, extract the overall features of the image, obtain the feature map of the image, and further obtain the feature map of each ROI area in the image;

[0064] S315. Perform pixel correction on the feature map of each ROI region using ROI Align;

[0065] S316. The feature map of each ROI area after pixel correction is predicted using the branch network FCN to obtain the target category and target box, thereby obtaining a list of target bounding boxes and their corresponding confidence score lists.

[0066] S32. Perform non-maximum suppression processing on the target bounding box list and its corresponding confidence score list, eliminate redundant bounding boxes, and obtain the room type and room coordinate points in each of the house layout diagrams;

[0067] The method for obtaining the room type and room coordinate points in each house layout diagram by non-maximum suppression and elimination of redundant bounding box processing includes:

[0068] S321. Sort according to the confidence scores in the confidence score list;

[0069] S322. Select the bounding box with the highest confidence and add it to the final output list, and delete it from the bounding box list;

[0070] S323. Calculate the area of ​​all bounding boxes;

[0071] S324. Calculate the IoU between the bounding box with the highest confidence and other candidate boxes;

[0072] S325. Delete the bounding boxes whose IoU is greater than the threshold;

[0073] S326. Repeat steps S21 to S25 until the bounding box list is empty;

[0074] Because the house images in this dataset have multiple target objects, considering the overlap of multiple target objects, we do not directly use the bounding box with the highest confidence, but gradually filter out other windows.

[0075] S33. Rename multiple rooms of the same category in the room type list using Arabic numeral numbers;

[0076] Since the room types obtained by the target detection model, such as MaskRCNN, when detecting the house layout diagram do not distinguish between instances, the category labels of the same room types in the same house layout diagram are the same. Therefore, after non-maximum suppression processing, it is necessary to rename different instances of multiple rooms of the same category. Here, Arabic numerals are used for renaming; for example, the list of room types before renaming is ['BathRoom', 'LivingRoom', 'Bedroom', 'Balcony', 'Bedroom', 'Kitchen', 'BathRoom', 'BathRoom', 'Balcony'], and after renaming and distinguishing different instances, the list is ['BathRoom', 'LivingRoom', 'Bedroom', 'Balcony', 'Bedroom1', 'Kitchen', 'Bedroom1', 'Bedroom2', 'Balcony1'] to ensure the consistency of the subsequent text description with the manually labeled part.

[0077] Partially qualified data

[0078] like Figure 4As shown, the method of performing manually annotated image segmentation and target detection on the partially qualified data to obtain the house layout information includes:

[0079] Use labelme tool to label the room type of each room on the house layout in the qualified data; the room types include: LivingRoom, Bedroom, Balcony, Kitchen, BathRoom, Library. Different rooms of the same category are distinguished by Arabic numerals. For example, if there are two Bedrooms in a house, the labels will be marked as Bedroom and Bedroom1 respectively. There are two types of annotation boxes created, namely Polygons and Rectangle, corresponding to polygonal target rooms and rectangular target rooms respectively. After the annotation is completed, a json file is generated for each house layout;

[0080] Parse the json file to obtain the room type and room coordinate points in each house layout diagram.

[0081] Overall process

[0082] like Figure 3 As shown in the figure, the overall process of generating a data set is as follows:

[0083] S1. Collect raw data;

[0084] S2. Raw data screening;

[0085] S3. If the data is qualified, the MaskRCNN model is used for target detection; the room type, number of rooms, and room coordinate points are obtained;

[0086] If some of the data is qualified, the manually labeled target detection is performed; the labels are checked, and if they are unqualified, they are manually labeled again; if the labels are qualified, the JSON file is parsed to obtain the room type, number of rooms, and room coordinate points; and the unqualified data is deleted;

[0087] S4. Determine the room orientation and room adjacent relationship;

[0088] S5. Generate a text description according to the text description template;

[0089] S6. Filter the data again, delete unqualified data, and obtain a data set;

[0090] S7. Normalize the dataset to obtain the final dataset.

[0091] Example 2

[0092] The difference between this embodiment and embodiment 1 is that the target detection model is a YOLACT model;

[0093] The target detection model is used to perform image segmentation and target detection on the house layout image, and obtain a target bounding box list and a corresponding confidence score list of the house layout image; wherein the target detection model is a YOLACT model; the YOLACT model is a one-stage target detection model, and the target recognition speed will be faster.

[0094] Example 3

[0095] The difference between this embodiment and embodiment 1 is that a system for generating images based on a semantically controlled house layout adversarial network model is provided, such as Figure 6 As shown, including:

[0096] The pre-trained text encoder uses the data set obtained by the construction method described in Example 1 as input and generates corresponding sentence vectors according to the text description in the data set;

[0097] The model training module establishes a generative adversarial network model including a generator, a discriminator and a loss function; the sentence vector of the text encoder is used as the input of the generative adversarial network model to train the generative adversarial network model to obtain a trained generative adversarial network model;

[0098] The image generation module inputs the text description to be tested into the trained generative adversarial network model to obtain the generated image.

[0099] In this embodiment, the generative adversarial network model established is a generative adversarial network GAN.

[0100] By constructing a generative adversarial network model that integrates text and image information, designing the objective function and loss function, the effectiveness of the house dataset in the task of generating images from text is verified, and finally high-quality realistic house layout images that match the text are generated.

[0101] Generator

[0102] The generator includes an upsampling layer, a residual block, and a data block; the input of the generator includes a sentence vector generated by the pre-trained text encoder; it also includes a noise vector sampled from a Gaussian distribution to ensure the diversity of the generated image; the noise vector is first fed into a fully connected layer, reshaped into (1, 4, 4) and then input into the upsampling layer for upsampling the image features, and then the text and image features are fused through the residual block and the data block, and finally the image features are converted into an image through the convolution layer.

[0103] Discriminator

[0104] The discriminator includes several downsampling blocks and convolutional layers; first, the discriminator converts the house layout map into a feature map, downsamples it by a series of downsampling blocks, and then copies and connects the sentence vector to the image features. The discriminator promotes the generator to synthesize higher quality images with semantic consistency between text and images by distinguishing the generated images from real samples.

[0105] Model training module

[0106] The generative adversarial network model directly synthesizes high-resolution images through a pair of generators and discriminators. The discriminator D has two training objectives: (1) to evaluate whether the input image is real or fake; (2) to determine whether the image-text conditions match. The input of the discriminator has five data types: real images, generated images, real images and corresponding matching texts, generated images and matching texts, and real images and unmatched texts. During the training process, the loss function parameters are gradually adjusted to make the images generated by the generator more realistic and the discriminator more accurate in judging the authenticity of the image. The loss function of the discriminator is:

[0107]

[0108] The loss function of the generator is:

[0109]

[0110] where z is a noise vector sampled from a Gaussian distribution; e is a sentence vector, is the unmatched sentence vector; P g , P r They represent synthetic data distribution and real data distribution respectively.

[0111] Example 4

[0112] The difference between this embodiment and Embodiment 3 is that the generative adversarial network is a deep attention generative adversarial network AttnGAN.

[0113] AttnGAN can fully utilize the word-level information in the generative adversarial network to generate fine-grained images, such as Figure 7 As shown, the AttnGAN contains multiple generators and discriminators. For example, generating a 256px×256px image requires 3 discriminators corresponding to 3 generation stages.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or some technical features can be replaced by equivalents without departing from the spirit of the technical solution of the present invention, which should be included in the scope of the technical solution for protection of the present invention.

Claims

1. A method for constructing a house layout generation dataset based on semantic control, characterized in that: The method comprises: Filter out qualified data and partially qualified data from the original data set of the house layout diagram according to a predetermined data screening rule, and delete unqualified data; Performing semi-automatic image segmentation and target detection on the qualified data to obtain house layout information of each house layout diagram; Performing manually annotated image segmentation and target detection on the partially qualified data to obtain house layout information of each house layout diagram; The house layout information includes room type and room coordinate points; The method of performing semi-automatic image segmentation and target detection on the qualified data to obtain house layout information includes: Use the target detection model to perform image segmentation and target detection on the house layout image, and obtain a list of target bounding boxes of the house layout image and a list of their corresponding confidence scores; Performing non-maximum suppression processing on the target bounding box list and its corresponding confidence score list to obtain the room type and room coordinate points in each of the house layout diagrams; Rename multiple rooms of the same category in the room type list using Arabic numerals; Method for performing manually annotated image segmentation and target detection on the partially qualified data to obtain house layout information: Use labelme tool to label the room types on the house layout diagram in the partial qualified data; After the marking is completed, a json file is generated for each house layout diagram; Parse the json file to obtain the room type and room coordinate points in each house layout diagram; According to the house layout information of each house layout diagram, the number of room types, room orientations, and room adjacency relationships of each house layout diagram are calculated; Generate a text description according to a text description template according to the number of room types, room orientations, and room adjacency relationships; The text description is filtered again, and the data is normalized to generate a final target detection data set.

2. The method for constructing a house layout generation dataset based on semantic control according to claim 1, characterized in that: The room types include: living room, bedroom, balcony, kitchen, bathroom and library.

3. The method for constructing a house layout generation dataset based on semantic control according to claim 1, characterized in that: The data screening rules are: The rooms in the house layout have clear boundaries; The house layout diagram shall at least include bedroom, kitchen, and bathroom room types; The house layout is a whole, and the rooms it contains are connected to each other.

4. The method for constructing a house layout generation dataset based on semantic control according to claim 1, characterized in that: The method for generating a text description according to a text description template includes: The number, room orientation, and room adjacency of each room type are automatically generated into a corresponding text description based on a pre-set text description template and stored in an Excel table; The text description template has each line representing a house, and the data in each line includes the house number, the number of Living Rooms, the number of Bedrooms, the number of Balconies, the number of Kitchens, the number of Bath Rooms, the number of Libraries, the room orientation, and the room adjacency relationship.

5. The method for constructing a house layout generation dataset based on semantic control according to claim 1, characterized in that: Methods for calculating the number of room types, room orientations, and room adjacency relationships based on the house layout information of each house layout diagram include: Based on the room type, count the number of each room type included in each house layout; According to the room coordinate points, determine whether the room is a rectangle or a polygon, then calculate the center point coordinates of the room, and according to the center point coordinates, determine the quadrant where the room is located, thereby determining the orientation of the room; All rooms in the house layout are combined in pairs and the Manhattan distance between the two rooms is calculated. If the Manhattan distance is less than half of the sum of the length and width of the outer bounding boxes of the two rooms plus θ, the two rooms are considered to be adjacent.

6. The method for constructing a house layout generation dataset based on semantic control according to claim 1, characterized in that: The target detection model is a MaskRCNN model.

7. The method for constructing a house layout generation dataset based on semantic control according to claim 1, characterized in that: The target detection model is the YOLACT model.

8. A system for generating images based on a semantically controlled house layout adversarial network model, characterized in that: The system comprises: A pre-trained text encoder, taking a data set obtained by the construction method described in any one of claims 1 to 7 as input, and generating a corresponding sentence vector according to the text description in the data set; The model training module establishes a generative adversarial network model including a generator, a discriminator and a loss function; the sentence vector of the text encoder is used as the input of the generative adversarial network model to train the generative adversarial network model to obtain a trained generative adversarial network model; The image generation module inputs the text description to be tested into the trained generative adversarial network model to obtain the generated image.

Images