Layout diagram generation method

Abstract

An embodiment of the present invention discloses a method for generating a layout diagram. The generation method includes: determining a semantic segmentation feature map of the house image through a first network based on a basic feature map and an edge line feature map of a house image; determining a wall feature map of the house image through a second network based on the basic feature map and the semantic segmentation feature map of the house image; determining a corner point feature map and a corner line feature map of the house image through a third network based on the basic feature map and the wall feature map of the house image; and generating a layout diagram of the house image based on the corner point feature map and the corner line feature map of the house image. The present invention effectively abstracts the relevant information between multiple tasks through the hierarchical structure of the multi-task joint model to promote the joint learning of multiple tasks, thereby effectively extracting the spatial information of the indoor layout and realizing the accurate generation of the indoor layout diagram.

Description

Technical Field

[0001] An embodiment of the present invention relates to a method for generating a layout diagram. Background Art

[0002] House image layout estimation involves capturing interior environmental data from a monocular camera and then using artificial intelligence to simulate human perception to determine the house's layout. This technology can be applied in the real estate sector, for example, to review the quality of uploaded images of a house site. It can also be widely used in other fields, such as 3D reconstruction, robotic indoor navigation, and VR / AR.

[0003] Currently, house image layout estimation technology primarily relies on traditional image algorithms or neural network learning algorithms. Specifically, these algorithms extract image features (such as local edges, shapes, colors, and textures) and then use post-processing methods to obtain the final house layout. Due to severe occlusion by wall corners and annotation errors, existing house image layout prediction techniques suffer from low accuracy and insufficient robustness. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for generating a layout diagram, which effectively extracts the spatial information of the indoor layout through a layer-by-layer guided joint learning method, thereby realizing the accurate generation of the indoor layout diagram.

[0005] To achieve the above-mentioned objectives, the first aspect of the present invention provides a method for generating a layout diagram, the generating method comprising: determining a semantic segmentation feature map of the house image through a first network in a multi-task joint model based on a basic feature map and an edge line feature map of a house image; determining a wall feature map of the house image through a second network in the multi-task joint model based on the basic feature map and the semantic segmentation feature map of the house image; determining a corner point feature map and a corner line feature map of the house image through a third network in the multi-task joint model based on the basic feature map and the wall feature map of the house image; and generating a layout diagram of the house image based on the corner point feature map and the corner line feature map of the house image.

[0006] In at least one embodiment, determining the wall feature map of the house image through the second network in the multi-task joint model includes: determining the depth feature map of the house image through the first subnetwork in the second network based on the basic feature map and semantic segmentation feature map of the house image; and determining the wall feature map of the house image through the second subnetwork in the second network based on the basic feature map and the depth feature map of the house image. Accordingly, determining the corner point feature map and corner line feature map of the house image through the third network in the multi-task joint model includes: determining the corner point feature map and corner line feature map of the house image through the third network based on the basic feature map, depth feature map and wall feature map of the house image.

[0007] In at least one embodiment, determining the corner point feature map and corner line feature map of the house image through the third network includes: determining the depth information of each wall area in the basic feature map based on the depth feature map and the wall feature map; when the depth information indicates that the distance between the first wall and the second wall is less than a preset distance, determining that the one with the smaller depth between the first wall and the second wall is a non-wall; removing the wall features corresponding to the non-wall from the wall feature map to obtain an updated wall feature map; and determining the corner point feature map and corner line feature map of the house image based on the updated wall feature map.

[0008] In at least one embodiment, the generation method further includes: obtaining multiple house image samples in a virtual reality scene, wherein the semantic segmentation features, wall features, corner point features and corner line features of the multiple house image samples are all labeled; labeling the depth features of the multiple house image samples based on the point cloud data of the multiple house image samples to obtain the multiple labeled house image samples; and using the multiple labeled house image samples to train the multi-task joint model.

[0009] In at least one embodiment, the training of the multi-task joint model using the multiple labeled house image samples includes: obtaining basic features and edge line features of the multiple labeled house image samples; processing the basic features and edge line features of the multiple labeled house image samples through the multi-task joint model to determine the loss of the semantic segmentation features, the loss of the depth features, the loss of the wall features, the loss of the corner point features, and the loss of the corner line features; and adjusting the feature parameters in the multi-task joint model based on the loss of the semantic segmentation features, the loss of the depth features, the loss of the wall features, the loss of the corner point features, and the loss of the corner line features.

[0010] In at least one embodiment, the generation method further includes: extracting an edge line feature map of the house image by a line feature extraction method; and extracting a basic feature map of the house image by a segmentation network based on the edge line feature map of the house image.

[0011] In at least one embodiment, generating a layout diagram of the house image includes: determining the corner points of the house image based on a corner point feature map of the house image; determining the corner lines of the house image based on a corner point and corner line feature map of the house image; and generating the layout diagram based on the corner points and corner lines of the house image.

[0012] Through the above technical solution, the present invention creatively first determines the semantic segmentation feature map of the house image based on the basic feature map and the edge line feature map of the house image through a first network; then, based on the semantic segmentation feature map obtained by the first network, the wall feature map of the house image is guided to be determined through a second network; then, based on the wall feature map obtained by the second network, the corner point feature map and the corner line feature map of the house image are guided to be determined through a third network; finally, based on the corner point feature map and the corner line feature map of the house image, a layout map of the house image is generated. Therefore, the present invention effectively abstracts the relevant information between multiple tasks through the hierarchical structure of the multi-task joint model to promote the joint learning of multiple tasks, thereby effectively extracting the spatial information of the indoor layout and realizing the accurate generation of the indoor layout map.

[0013] A second aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the above-mentioned method for generating a layout diagram.

[0014] A third aspect of the present invention provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; and the processor for reading the executable instructions from the memory and executing the instructions to implement the above-mentioned method for generating a layout diagram.

[0015] A fourth aspect of the present invention provides a computer program product, comprising a computer program, wherein the computer program implements the above-mentioned method for generating a layout diagram when executed by a processor.

[0016] Other features and advantages of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following detailed description, they are used to explain the embodiments of the present invention, but do not constitute a limitation of the embodiments of the present invention. In the accompanying drawings:

[0018] Figure 1 This is a schematic diagram of an applicable scenario of the layout diagram generation method provided by an embodiment of the present invention;

[0019] Figure 2 This is a schematic diagram of an applicable scenario of the layout diagram generation method provided by an embodiment of the present invention;

[0020] Figure 3 is a flow chart of a method for generating a layout diagram provided by one embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of the structure of generating a layout diagram provided by an embodiment of the present invention;

[0022] Figure 5 This is a flowchart of training a multi-task joint model provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following describes the specific embodiments of the present invention in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0024] Many houses in house images are not empty; they are often equipped with furniture, heating systems, and other accessories. These often obscure corners, making it difficult to extract house layout information. Accurately extracting these obscured corners is crucial for various applications, including 3D house reconstruction, robotic indoor navigation, and AR and VR.

[0025] Figure 1 The house image shown is provided with multiple tables and chairs, and two of the tables are respectively provided with two table lamps. Figure 1 It can be seen that the table, chairs, and desk lamps block multiple corners in the house. For example, a vertical corner between walls is blocked, and two horizontal corners between the wall and the floor are blocked.

[0026] This paper uses a multi-task joint model to Figure 1 After the house layout information is extracted from the house image shown in FIG, five corner lines are obtained, such as Figure 2 As shown. Figure 2The five corner lines in the figure can construct the walls, ceiling and floor of two facades, so as to obtain Figure 1 The three-dimensional structure of the house image shown in FIG. Wherein, the multi-task joint model may include a first network, a second network and a third network, such as Figure 4 The multi-task joint model is based on the multi-task learning framework of convolutional neural networks, which guides and jointly learns semantic segmentation, depth, wall surface, corner points, and corner lines layer by layer.

[0027] Furthermore, in the real estate sector, the technology for obtaining house layout information provided by the present disclosure can also be used to conduct compliance audits on house images provided by real estate agents. For example, based on the house layout information obtained by the present disclosure, it can automatically determine whether the house image includes at least two walls, a floor, and a ceiling. If the image does not include at least two walls, a floor, and a ceiling, the image is considered non-compliant. For non-compliant house images (e.g., images that only include a floor and a wall), the real estate agent can be asked to provide a new image.

[0028] Unless otherwise specified, the walls in this disclosure may include the walls, ceilings and floors in the layout drawings.

[0029] Figure 3 FIG. 1 is a flow chart of a method for generating a layout diagram according to an embodiment of the present invention. Figure 1 As shown, the generating method may include the following steps S301-S304.

[0030] Before executing step S301 , the method for generating a layout diagram may further include: extracting an edge line feature diagram of the house image by using a line feature extraction method.

[0031] The house image in the present disclosure may refer to an image obtained by photographing the interior of a house using a camera. The house image in the present disclosure may be a monocular image, and the house image in the present disclosure may be an RGB (Red, Green, Blue) image. That is, the house image in the present disclosure may be an image obtained by photographing the interior of a house using a monocular camera. The monocular camera may include a camera, a mobile phone with a camera function, or a tablet computer with a camera function.

[0032] The line feature extraction method may include: Line Segment Detection (LSD) algorithm or Hough transform. Of course, the line feature extraction method may also be other traditional line feature extraction methods.

[0033] The edge line feature is a feature map composed of edge line features of each object in the house image (e.g., roof, floor, wall, table, curtain, window, wardrobe, cabinet, etc.). The edge line feature can indicate the edge line category to which each pixel in the house image belongs, that is, whether it is an edge line.

[0034] Before executing step S301 , the method for generating a layout diagram may further include: extracting a basic feature map of the house image through a segmentation network based on the edge line feature map of the house image.

[0035] The structure of the segmentation network can adopt an encoder-decoder structure, for example, Unet (Unity network) or deep residual network (DRN) can be selected. Figure 4 As shown, the input of the segmentation network is the house image and its edge map, and the basic feature map of the house image can be obtained through the segmentation network.

[0036] It should be understood that the above-described methods for obtaining edgeline feature maps and basic feature maps are merely exemplary, and the embodiments of the present disclosure are not limited thereto. For example, in some exemplary embodiments, edgeline feature maps and basic feature maps may be pre-stored or obtained using other algorithms. For example, in some exemplary embodiments, a basic feature map may be obtained directly from a house image without requiring an edgeline feature map.

[0037] Step S301 : Based on the basic feature map and the edge line feature map of the house image, a semantic segmentation feature map of the house image is determined by a first network in a multi-task joint model.

[0038] Specifically, the basic feature map and the edge line feature map are used as the input of the first network, and the semantic segmentation feature map of the house image is obtained through the first network, such as Figure 4 shown.

[0039] The semantic segmentation process in this disclosure may refer to performing dense predictions on each pixel in a house image, thereby predicting the category information corresponding to each pixel. This category information may be object category information, region category information, or the like. The category information corresponding to each pixel forms a semantic segmentation feature map for the house image. The spatial resolution of the semantic segmentation feature map for the house image may be the same as the spatial resolution of the house image.

[0040] Step S302: Based on the basic feature map and the semantic segmentation feature map of the house image, determine the wall feature map of the house image through the second network in the multi-task joint model.

[0041] In one embodiment, after obtaining the semantic segmentation feature map through the first network, the basic feature map and the semantic segmentation feature map are used as inputs of the second network, and the wall feature map of the house image is obtained through the second network, such as Figure 4 As shown. Specifically, the pixels of the same semantics are aggregated into the same wall. The results of the wall feature maps thus obtained are inaccurate, and thus the indoor layout structure obtained based on the wall feature map is also not very accurate. Especially in practical applications, errors in semantic segmentation processing may lead to semantic recognition errors. For example, a table may also be identified as a floor. Therefore, the wall feature map corresponding to the floor obtained by the above embodiment is shifted upward as a whole, which will eventually lead to inaccurate corner points and corner lines obtained by the wall feature map, and thus inaccurate indoor layout results.

[0042] In another embodiment, the second network can be combined with depth information to identify wall features to avoid inaccurate wall recognition results caused by semantic segmentation processing. For step S302, determining the wall feature map of the house image by the second network in the multi-task joint model includes: determining the depth feature map of the house image by the first subnetwork in the second network based on the basic feature map and the semantic segmentation feature map of the house image; and determining the wall feature map of the house image by the second subnetwork in the second network based on the basic feature map and the depth feature map of the house image.

[0043] In the present disclosure, depth feature processing may refer to performing dense prediction on each pixel in a house image, thereby predicting the depth information corresponding to each pixel. The depth information corresponding to each pixel forms a depth feature map for the house image. The spatial resolution of the depth feature map for the house image may be the same as the spatial resolution of the house image.

[0044] Specifically, the second network may include a first sub-network and a second sub-network. Figure 4 As shown, after obtaining the semantic segmentation feature map through the first network, the basic feature map and the semantic segmentation feature map are used as the input of the first subnetwork in the second network, and the depth feature map of the house image is obtained through the first subnetwork; then the basic feature map and the depth feature map obtained through the first subnetwork are used as the input of the second subnetwork in the second network, and the wall feature map of the house image is obtained through the second subnetwork.

[0045] For example, first, the depth information of the table and the depth information of the floor can be obtained through the first sub-network. Then, the basic feature map is analyzed through the second sub-network, and the pixels at the same depth are aggregated into the same wall. For example, the pixels in the same depth range are aggregated into the desktop, and the pixels in another depth range are aggregated into the floor. Although the table and the floor are both identified as floors through semantic segmentation processing, the desktop and the floor can be further identified as two different walls based on the depth information, and the pixels on the desktop and the floor will not be aggregated. Therefore, this embodiment can achieve accurate identification of different walls, thereby obtaining an accurate wall feature map, and then based on the accurate wall feature map, an accurate indoor layout can be generated.

[0046] The above step S302 guides the detection of the wall by obtaining the semantic segmentation (or combining semantic segmentation and depth features) through the upper network in the multi-task joint model, so as to very well abstract the relevant information between semantics (or semantics and depth) and the wall, and promote the joint learning and application of multiple tasks.

[0047] Step S303: Based on the basic feature map and the wall feature map of the house image, determine the corner point feature map and the corner line feature map of the house image through the third network in the multi-task joint model.

[0048] The corner point feature map can still be considered as a semantic segmentation feature map, and the semantic segmentation feature map is a semantic segmentation feature map based on the corner point category. In other words, the corner point feature map can respectively indicate the corner point category to which each pixel in the house image belongs.

[0049] The corner line feature map can still be considered as a semantic segmentation feature map, and the semantic segmentation feature map is a semantic segmentation feature map based on the corner line category. In other words, the corner line feature map can indicate the corner line category to which each pixel in the house image belongs.

[0050] In one embodiment, after obtaining the wall feature map through the second network, the basic feature map and the wall feature map are used as inputs of the third network, and the corner point feature map and the corner line feature map of the house image are obtained through the third network, such as Figure 4As shown. Specifically, according to the wall feature map, the corresponding corner points and corner lines on each wall in the basic feature map are determined. The results of each wall feature map obtained in this way are inaccurate, and the indoor layout structure obtained based on the wall feature map is also not very accurate. Especially in practical applications, due to the error in semantic segmentation processing, semantic recognition errors may occur. For example, the table may also be identified as the floor. Therefore, the wall aggregated by the second network may include valid walls and invalid walls (i.e., non-walls). That is, the result of the wall feature map obtained by the above embodiment is not very accurate. For example, the table top close to the floor may be identified as the floor. Ultimately, the corner points and corner lines obtained by the wall feature map are inaccurate, which in turn leads to inaccurate results of the indoor layout.

[0051] In another embodiment, the second network can be combined with depth information to identify wall features to avoid interference from other non-wall features close to the wall, thereby achieving effective recognition of wall features. If the wall feature map is obtained in combination with the depth feature map of the house image, then for step S303, the determining of the wall corner point feature map and the wall corner line feature map of the house image by the third network in the multi-task joint model includes: based on the basic feature map, depth feature map and wall feature map of the house image, determining the wall corner point feature map and the wall corner line feature map of the house image by the third network in the multi-task joint model.

[0052] like Figure 4 As shown, after obtaining the wall feature map through the second network, the basic feature map, the depth feature map and the wall feature map are used as inputs of the third network, and the corner point feature map and the corner line feature map of the house image are obtained through the third network.

[0053] Among them, the determining of the wall corner point feature map and the wall corner line feature map of the house image through the third network includes: determining the depth information of each wall area in the basic feature map based on the depth feature map and the wall feature map; when the depth information indicates that the distance between the first wall and the second wall is less than a preset distance, determining that the one with the smaller depth between the first wall and the second wall is a non-wall; removing the wall features corresponding to the non-wall from the wall feature map to obtain an updated wall feature map; and determining the wall corner point feature map and the wall corner line feature map of the house image based on the updated wall feature map.

[0054] Specifically, after the depth information of the desktop and the depth information of the floor can be obtained through the second network, the basic feature map is analyzed through the third network, and the effective wall is screened in combination with the depth information of the desktop and the floor to obtain a valid wall feature map. For example, if the pixel points on the desktop and the pixel points on the floor are not within the preset depth range (that is, the distance between the desktop and the floor is less than the preset distance), it is determined that the desktop with a smaller depth (that is, indicating that the table is above the floor) is not the floor (that is, the desktop is not a wall), that is, the non-wall feature map in the wall feature map obtained in step S302 is excluded. Then, the wall feature map after excluding the non-wall features (that is, the valid wall feature map) is processed to determine the corner point feature map and the corner line feature map of the house image.

[0055] In the above step S303, the depth features and wall features obtained by the upper network in the multi-task joint model jointly guide the estimation of corner points and lines, thereby being able to very well abstract the relevant information between multiple tasks of depth, wall, corner points and lines, and promote the joint learning and application of multiple tasks.

[0056] Step S304 : generating a layout diagram of the house image based on the corner point feature map and the corner line feature map of the house image.

[0057] For step S304, generating a layout diagram of the house image may include: determining the corner points of the house image based on a corner point feature map of the house image; determining the corner lines of the house image based on a corner point and corner line feature map of the house image; and generating the layout diagram based on the corner points and corner lines of the house image.

[0058] Specifically, first, non-maximum suppression is performed on the prediction results of the corner point feature map. More specifically, the operation is to perform maximum pooling on the prediction results of the corner point feature map; points with a probability greater than a threshold (for example, set to 0.6) are taken as corner points (up to 10), thereby screening valid corner points. Figure 2 The dots shown represent the determined corner points. Next, any two of the corner points are connected, and the resulting line is used as a candidate line. Based on the corner line feature map, it is determined whether the average thermal value of the candidate line area is greater than a threshold (e.g., set to 0.8). If the average thermal value of a particular candidate line is greater than the threshold, the particular candidate line is determined to be a corner line. Figure 2 The line shown is the determined corner line. Next, the enclosed area formed by the corner line and the image edge is determined as the wall surface. This ultimately yields a precise interior layout structure encompassing the corner point, corner line, and wall surface.

[0059] Therefore, the multi-task joint model mainly uses the segmentation semantic results obtained in the upper layer to guide depth estimation, uses the depth estimation obtained in the upper layer to guide wall detection, and uses the depth and wall surface to jointly guide the estimation of wall corner points and lines. The multi-task joint model disclosed herein can be a model obtained through the training process described below. Therefore, the various embodiments of the present disclosure can very well abstract the relevant information between multiple tasks with this hierarchical structure, promoting the joint learning of multiple tasks.

[0060] Before executing step S301, the generation method may further include the following steps S501-S503 to implement the training of the multi-task joint model, such as Figure 5 shown.

[0061] Step S501: Acquire multiple house image samples in a virtual reality scene.

[0062] The semantic segmentation features, wall features, corner point features, and corner line features of the plurality of house image samples are all annotated. Specifically, the semantic segmentation features, wall features, corner point features, and corner line features of the house image samples can be annotated manually.

[0063] Step S502 : annotating the depth features of the plurality of house image samples based on the point cloud data of the plurality of house image samples to obtain the plurality of annotated house image samples.

[0064] For a house image in a virtual realization scene, corresponding depth information may be determined based on the point cloud distance of the house image, and depth features of the house image may be annotated.

[0065] Step S503: Use the multiple labeled house image samples to train the multi-task joint model.

[0066] For the step S503, the training of the multi-task joint model using the multiple labeled house image samples may include: obtaining basic features and edge line features of the multiple labeled house image samples; processing the basic feature maps and edge line feature maps of the multiple labeled house image samples through the multi-task joint model to determine the loss of the semantic segmentation features, the loss of the depth features, the loss of the wall features, the loss of the corner point features, and the loss of the corner line features; and adjusting the feature parameters in the multi-task joint model based on the loss of the semantic segmentation features, the loss of the depth features, the loss of the wall features, the loss of the corner point features, and the loss of the corner line features.

[0067] Specifically, the labeled house image and its edge line feature map are first input into the segmentation network, and the basic features of the house image are extracted through the segmentation network to obtain the corresponding basic feature map; then the basic feature map and the edge line feature map are superimposed and processed by 1*1 convolution to output the semantic segmentation feature map (for example, the semantics of roof, floor, wall, table, curtain, window, wardrobe, cabinet, etc.) and its corresponding loss; then the basic feature map and the semantic segmentation feature map are superimposed and processed by 3*3 convolution to obtain the depth feature map and its corresponding loss; then the basic feature map and the depth feature map are superimposed and processed by 3*3 convolution to obtain the wall feature map and its corresponding loss; finally, the basic feature map, the depth feature map and the wall feature map are superimposed and processed by 3*3 convolution to obtain the wall feature map and its corresponding loss. Figure 3 The channels are superimposed and processed by 3*3 convolution to obtain the corner point feature map and the corner line feature map and their respective losses.

[0068] The following describes the calculation methods of the loss of each of the above features.

[0069] Regarding semantic segmentation features, cross entropy loss can be used for multi-class discrimination. Since the positive and negative categories of the heat maps of walls, corner points, and corner lines are seriously unbalanced, focal loss can also be used. Specifically, the loss of semantic segmentation features is expressed as follows (1): seg :

[0070] loss seg =-(y*log(p)+(1-y)*log(1-p)) (1)

[0071] Among them, y is the label value (which can be represented by 0 and 1); p is the predicted probability.

[0072] Regarding deep features, smoothl1 loss is used to calculate the loss of deep features. epth , as shown in the following formula (2). Compared with the L2 loss function, smoothhl1 loss is not sensitive to outliers and abnormal values, the gradient change is relatively smaller, and the training is more stable.

[0073]

[0074] Among them, y is the label value (which can be represented by 0 and 1); p is the predicted probability.

[0075] Regarding the wall surface, corner points and corner lines, the losses of the wall surface, corner points and corner lines can be calculated using the following formula (3):

[0076] loss heatmap =-(α(1-p) γ ylog(p)+(1-α)pγ (1-y)log(1-p)) (3)

[0077] Where y is the label value (which can be represented by 0 or 1); p is the predicted probability; a is a hyperparameter used to adjust the ratio of positive and negative samples; and r is a hyperparameter used to adjust the difficulty coefficient.

[0078] When the loss of the semantic segmentation feature, the loss of the depth feature, the loss of the wall feature, the loss of the corner point feature and the loss of the corner line feature are obtained in the above manner, the weighted results of the above losses are calculated to adjust the feature parameters in the multi-task joint model based on the weighted results.

[0079] After training the multi-task joint model, based on the trained multi-task joint model and the house image, the indoor layout information can be represented from top to bottom, from easy to difficult, and from local to overall, so as to accurately generate the corresponding indoor layout map.

[0080] In summary, the present invention creatively first determines the semantic segmentation feature map of the house image based on the basic feature map and the edge line feature map of the house image through a first network; then, based on the semantic segmentation feature map obtained by the first network, the wall feature map of the house image is guided to be determined through a second network; then, based on the wall feature map obtained by the second network, the corner point feature map and the corner line feature map of the house image are guided to be determined through a third network; finally, based on the corner point feature map and the corner line feature map of the house image, a layout map of the house image is generated. Therefore, the present invention effectively abstracts the relevant information between multiple tasks through the hierarchical structure of the multi-task joint model to promote the joint learning of multiple tasks, thereby effectively extracting the spatial information of the indoor layout and realizing the accurate generation of the indoor layout map.

[0081] An embodiment of the present invention provides a layout diagram generation system, which includes: a semantic segmentation device for determining a semantic segmentation feature map of a house image based on a basic feature map and an edge line feature map of the house image through a first network in a multi-task joint model; a wall feature extraction device for determining a wall feature map of the house image based on the basic feature map and the semantic segmentation feature map of the house image through a second network in the multi-task joint model; a point and line extraction device for determining a corner point feature map and a corner line feature map of the house image based on the basic feature map and the wall feature map of the house image through a third network in the multi-task joint model; and a generation device for generating a layout diagram of the house image based on the corner point feature map and the corner line feature map of the house image.

[0082] The specific details and benefits of the layout diagram generation system provided by the present invention can be found in the above description of the layout diagram generation method, which will not be repeated here.

[0083] An embodiment of the present invention further provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the computer program implements the above-mentioned method for generating a layout diagram.

[0084] An embodiment of the present invention further provides an electronic device, comprising: a processor; a memory for storing a computer program; the processor is configured to read the computer program from the memory and execute the computer program to implement the above-mentioned method for generating a layout diagram.

[0085] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the computer program implements the above-mentioned method for generating a layout diagram.

[0086] The above describes in detail the optional implementation methods of the embodiments of the present invention in conjunction with the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above implementation methods. Within the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical solutions of the embodiments of the present invention, and these simple modifications all fall within the scope of protection of the embodiments of the present invention.

[0087] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any appropriate manner without contradiction. To avoid unnecessary repetition, the embodiments of the present invention will not further describe various possible combinations.

[0088] Those skilled in the art will understand that all or part of the steps in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a single-chip microcomputer, chip or processor to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program code.

[0089] In addition, various implementations of the embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the embodiments of the present invention, they should also be regarded as the contents disclosed in the embodiments of the present invention.

Claims

1. A method for generating a layout diagram, characterized in that: The generation method comprises: Extract edge line feature maps of house images using line feature extraction method; Based on the edge line feature map of the house image, extracting a basic feature map of the house image through a segmentation network; Determining a semantic segmentation feature map of the house image by a first network in a multi-task joint model based on the basic feature map and the edge line feature map of the house image; Based on the basic feature map and the semantic segmentation feature map of the house image, determining a wall feature map of the house image through the second network in the multi-task joint model; Based on the basic feature map and the wall feature map of the house image, determining a corner point feature map and a corner line feature map of the house image through a third network in the multi-task joint model; and A layout diagram of the house image is generated based on the corner point feature map and the corner line feature map of the house image.

2. The method for generating a layout diagram according to claim 1, wherein: Determining the wall feature map of the house image by the second network in the multi-task joint model includes: Determining a deep feature map of the house image by a first subnetwork in the second network based on the basic feature map and the semantic segmentation feature map of the house image; and Based on the basic feature map and the depth feature map of the house image, a second sub-network in the second network is used to determine a wall feature map of the house image. Determining the corner point feature map and the corner line feature map of the house image by using the third network in the multi-task joint model includes: Based on the basic feature map, the depth feature map and the wall feature map of the house image, the corner point feature map and the corner line feature map of the house image are determined by the third network.

3. The method for generating a layout diagram according to claim 2, wherein: Determining the corner point feature map and the corner line feature map of the house image by the third network includes: Determining depth information of each wall area in the basic feature map based on the depth feature map and the wall feature map; If the depth information indicates that the distance between the first wall and the second wall is less than a preset distance, determining that the one with the smaller depth between the first wall and the second wall is a non-wall; removing the wall features corresponding to the non-wall surface from the wall feature map to obtain an updated wall feature map; and Based on the updated wall surface feature map, a wall corner point feature map and a wall corner line feature map of the house image are determined.

4. The method for generating a layout diagram according to claim 2, wherein: The generating method further comprises: Acquire multiple house image samples in a virtual reality scene, wherein semantic segmentation features, wall features, corner point features, and corner line features of the multiple house image samples are all labeled; Annotating the depth features of the plurality of house image samples based on the point cloud data of the plurality of house image samples to obtain the plurality of annotated house image samples; and The multi-task joint model is trained using the multiple labeled house image samples.

5. The method for generating a layout diagram according to claim 4, wherein: The training of the multi-task joint model using the multiple labeled house image samples includes: Obtaining basic features and edge line features of the plurality of labeled house image samples; Processing the basic features and edge line features of the plurality of labeled house image samples through the multi-task joint model to determine the loss of the semantic segmentation feature, the loss of the depth feature, the loss of the wall feature, the loss of the corner point feature, and the loss of the corner line feature; and Based on the loss of the semantic segmentation feature, the loss of the depth feature, the loss of the wall feature, the loss of the corner point feature and the loss of the corner line feature, the feature parameters in the multi-task joint model are adjusted.

6. The method for generating a layout diagram according to claim 1, wherein: The generating of the layout diagram of the house image comprises: Determining the corner points of the house image based on the corner point feature map of the house image; Determining a corner line of the house image based on a feature map of corner points and corner lines of the house image; and The layout diagram is generated based on the corner points and corner lines of the house image.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method for generating a layout diagram according to any one of claims 1 to 6.

8. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for generating a layout diagram according to any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the method for generating a layout diagram according to any one of claims 1 to 6 is implemented.

Images