Three-dimensional model generation method, electronic device, storage medium and program product

Abstract

The present disclosure provides a method for generating an indoor three-dimensional model, an electronic device, a readable storage medium, and a computer program product. The method for generating an indoor three-dimensional model of the present disclosure includes: obtaining a two-dimensional indoor depth map; performing semantic segmentation processing on the two-dimensional indoor depth map, obtaining and retaining multiple pixel points representing the spatial structure in the two-dimensional indoor depth map, and obtaining a two-dimensional pixel map based on the retained pixel points; projecting the pixels in the two-dimensional pixel map into a one-dimensional point cloud, and obtaining a one-dimensional depth map based on the one-dimensional point cloud; and establishing a three-dimensional model based on the one-dimensional depth map to obtain an indoor three-dimensional model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method for generating an indoor three-dimensional model, an electronic device, a storage medium, and a program product. Background Art

[0002] In the field of home decoration, it is usually necessary to use indoor pictures to build indoor 3D models, so as to edit various decoration effects on the indoor 3D models for preview. At present, the solution of building indoor 3D models based on indoor pictures is usually to estimate the layout of the indoor 2D perspective view based on the deep learning method, obtain the area where the floor, wall, and ceiling are located, and then build the 3D model based on the layout estimation result. This solution is less effective when facing more complex indoor structures. Summary of the invention

[0003] The present disclosure provides a method for generating an indoor three-dimensional model, an electronic device, a storage medium and a program product.

[0004] According to one aspect of the present disclosure, a method for generating an indoor three-dimensional model is provided, comprising:

[0005] Get a 2D indoor depth map;

[0006] Performing semantic segmentation processing on the two-dimensional indoor depth map, retaining multiple pixel points representing the spatial structure in the two-dimensional indoor depth map to obtain a two-dimensional pixel map;

[0007] Projecting pixels in the two-dimensional pixel map into a one-dimensional point cloud, and obtaining a one-dimensional depth map based on the one-dimensional point cloud;

[0008] A three-dimensional model is established based on the one-dimensional depth map to obtain an indoor three-dimensional model.

[0009] According to at least one embodiment of the present disclosure, a method for generating an indoor three-dimensional model, obtaining a two-dimensional indoor depth map, includes:

[0010] Get a 2D interior perspective image;

[0011] Performing depth estimation on pixel points of the two-dimensional indoor perspective image to obtain depth information of each pixel point of the two-dimensional indoor perspective image;

[0012] The depth information of each pixel point of the two-dimensional indoor perspective image is rendered as each pixel point of the two-dimensional indoor depth map to obtain the two-dimensional indoor depth map.

[0013] According to at least one embodiment of the present disclosure, a method for generating an indoor three-dimensional model performs semantic segmentation processing on the two-dimensional indoor depth map, obtains and retains a plurality of pixel points representing a spatial structure in the two-dimensional indoor depth map, and obtains a two-dimensional pixel map based on the retained pixel points, including:

[0014] Performing semantic segmentation processing on the two-dimensional indoor depth map to obtain a furniture segmentation mask representing the furniture;

[0015] In the two-dimensional indoor depth map, the pixels in the area where the furniture segmentation mask is located are filtered out so that the pixels representing the spatial structure are retained, and the two-dimensional pixel map is obtained based on the retained pixels.

[0016] According to at least one embodiment of the present disclosure, the method for generating an indoor three-dimensional model performs semantic segmentation processing on the two-dimensional indoor depth map, obtains and retains a plurality of pixel points representing the spatial structure in the two-dimensional indoor depth map, and obtains a two-dimensional pixel map based on the retained pixel points, and also includes:

[0017] Performing semantic segmentation processing on the two-dimensional indoor depth map to obtain a hole structure segmentation mask representing the door opening and / or window opening;

[0018] The outline of the hole structure segmentation mask is marked in the two-dimensional pixel image to obtain a hole structure area marking box.

[0019] According to at least one embodiment of the present disclosure, a method for generating an indoor three-dimensional model, establishing a three-dimensional model based on the one-dimensional depth map to obtain an indoor three-dimensional model, includes:

[0020] Superimposing the hole structure area annotation frame on the one-dimensional depth map to obtain a one-dimensional synthetic image;

[0021] Generate a three-dimensional geometric surface based on the coordinates and depth information of the pixel points in the one-dimensional synthetic image;

[0022] The three-dimensional geometric surface is subjected to hole opening processing based on the hole structure area annotation frame in the one-dimensional synthetic image to obtain the indoor three-dimensional model.

[0023] According to at least one embodiment of the present disclosure, a method for generating an indoor three-dimensional model projects pixels in the two-dimensional pixel map into a one-dimensional point cloud, and obtains a one-dimensional depth map based on the one-dimensional point cloud, including:

[0024] Projecting pixels in the two-dimensional pixel map into a one-dimensional point cloud along a top-down angle as a one-dimensional wall structure depth map;

[0025] Projecting the pixels in the two-dimensional pixel map into a one-dimensional vertical viewing angle point cloud along the vertical viewing angle as a one-dimensional top-to-ground structure depth map;

[0026] The one-dimensional wall structure depth map and the one-dimensional ceiling structure depth map are synthesized into a dual-channel pixel map as the one-dimensional depth map.

[0027] The method for generating an indoor three-dimensional model according to at least one embodiment of the present disclosure further includes:

[0028] Adding furniture models to the indoor three-dimensional model to obtain a three-dimensional decoration effect model;

[0029] The three-dimensional decoration effect model is rendered into a two-dimensional image to obtain a two-dimensional decoration effect diagram.

[0030] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the method for generating an indoor three-dimensional model of any embodiment of the present disclosure.

[0031] According to another aspect of the present disclosure, a readable storage medium is provided, in which execution instructions are stored. When the execution instructions are executed by a processor, they are used to implement the method for generating an indoor three-dimensional model of any embodiment of the present disclosure.

[0032] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method for generating an indoor three-dimensional model according to any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0034] Figure 1 It is a flowchart of a method for generating an indoor three-dimensional model according to an embodiment of the present invention.

[0035] Figure 2 It is a schematic diagram of an application scenario of a method for generating an indoor three-dimensional model according to an embodiment of the present invention.

[0036] Figure 3 It is a flowchart of a method for generating an indoor three-dimensional model according to an embodiment of the present invention.

[0037] Figure 4 It is a flowchart of a method for generating an indoor three-dimensional model according to an embodiment of the present invention.

[0038] Figure 5 It is a flowchart of a method for generating an indoor three-dimensional model according to an embodiment of the present invention.

[0039] Figure 6 It is a flowchart of a method for generating an indoor three-dimensional model according to an embodiment of the present invention.

[0040] Figure 7 It is a flowchart of a method for generating an indoor three-dimensional model according to an embodiment of the present invention.

[0041] Figure 8 It is a schematic diagram of the process of obtaining a one-dimensional depth map in a method for generating an indoor three-dimensional model according to an embodiment of the present invention.

[0042] Fig. 9 It is a flowchart of a method for generating an indoor three-dimensional model according to an embodiment of the present invention.

[0043] Fig.10 It is a schematic block diagram of the structure of a device for generating an indoor three-dimensional model according to an embodiment of the present invention.

[0044] Fig.11 It is a schematic block diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0045] The present disclosure is further described in detail below in conjunction with the accompanying drawings and examples. It is understood that the specific examples described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0046] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0047] In the field of home decoration, a layout estimation-based solution is usually used to estimate the layout of the indoor two-dimensional perspective view to obtain the indoor structure, and the indoor structure is used to generate the indoor three-dimensional model. However, when faced with a more complex indoor structure, this solution is difficult to accurately obtain the indoor structure.

[0048] The embodiment of the present disclosure provides a method for generating an indoor three-dimensional model, which can be implemented on a server. For example, the server is configured with a system for generating an indoor three-dimensional model, which is a software program for executing the method for generating an indoor three-dimensional model, and can generate an indoor three-dimensional model according to an indoor picture uploaded by a client.

[0049] Figure 1 FIG. 1 is a schematic diagram showing the overall process of a method M100 for generating an indoor three-dimensional model according to an embodiment of the present disclosure. Figure 1 The method shown includes steps S110 to S140, wherein the method may be executed by a server device.

[0050] Specifically, Figure 1 The methods shown include:

[0051] S110: Obtain a two-dimensional indoor depth map.

[0052] The two-dimensional indoor depth map is an image containing the depth information of the indoor space. The grayscale value of each pixel in the two-dimensional indoor depth map represents a depth value, and different grayscale values ​​can be mapped to different depth values, so that the grayscale value of the pixel of the two-dimensional indoor depth map can represent the depth information of the indoor space.

[0053] In an example, the two-dimensional indoor depth map may be uploaded by the client to the three-dimensional model generation system, so that the three-dimensional model generation system can obtain the two-dimensional indoor depth map.

[0054] In another example, a three-dimensional model generation system can be executed by a computer device, which is electrically connected to a depth sensor. The depth sensor is used to scan the depth of an indoor space to obtain depth information. The computer device converts the depth information obtained by the depth sensor into a two-dimensional indoor depth map and transmits it to the three-dimensional model generation system.

[0055] In another example, the 3D model generation system can convert the 2D indoor perspective image uploaded by the user into a 2D indoor depth map based on the depth estimation process. This solution will be described in detail later and will not be described here.

[0056] S120, performing semantic segmentation processing on the two-dimensional indoor depth map, obtaining and retaining a plurality of pixel points representing the spatial structure in the two-dimensional indoor depth map, and obtaining a two-dimensional pixel map based on the retained pixel points.

[0057] Semantic segmentation refers to labeling each pixel in the two-dimensional indoor depth map with the label of its category. In the field of home decoration, the category labels of semantic segmentation usually include furniture labels, such as tables, sofas, etc.; and spatial structure labels, such as ceilings, walls, floors, door openings, window openings, etc. In other words, retaining the pixels that represent the spatial structure in the two-dimensional indoor depth map is equivalent to retaining the pixels with spatial structure labels.

[0058] The basis for establishing an indoor three-dimensional model is to obtain the structural information of the empty room, including at least the position, size and other information of the ceiling, wall, floor and other structures, and exclude the furniture information in the room. Based on this, step S120 retains the pixel points representing the spatial structure in the two-dimensional indoor depth map to obtain a two-dimensional pixel map, so that the obtained two-dimensional pixel map can exclude the pixel points representing structures such as furniture, so that the two-dimensional pixel map eliminates interference and focuses on representing the spatial structure information, ensuring that the represented spatial structure information is accurate.

[0059] In one example, the generation system of the three-dimensional model performs semantic segmentation processing on the two-dimensional indoor depth map. The generation system of the three-dimensional model can transmit the two-dimensional indoor depth map to the semantic segmentation model for semantic segmentation processing to obtain the processing result output by the semantic segmentation model, and then filter out the pixels other than the spatial structure class according to the category label annotated in the processing result output by the semantic segmentation model, and only retain the pixels representing the spatial structure to obtain the two-dimensional pixel map. Among them, the semantic segmentation model can adopt the FNC model, DeepLabV3 model, LR-ASPP model, etc., which are not limited here.

[0060] S130 , projecting pixels in the two-dimensional pixel map into a one-dimensional point cloud, and obtaining a one-dimensional depth map based on the one-dimensional point cloud.

[0061] Among them, projecting the pixels in the two-dimensional pixel map into a one-dimensional point cloud means converting the pixel points in the two-dimensional pixel map into a one-dimensional depth vector along a specific path from a certain perspective. Each one-dimensional depth vector is a point in the one-dimensional point cloud, representing a coordinate position and a depth value. For example, from a top-down perspective, a one-dimensional depth vector in the two-dimensional pixel map is extracted along a path from left to right, and the obtained one-dimensional point cloud represents the depth characteristics of the wall structure. For another example, from a vertical perspective (perpendicular to the wall), a one-dimensional depth vector in the two-dimensional pixel map is extracted along a path from top to bottom, and the obtained one-dimensional point cloud represents the depth characteristics of the ceiling and floor structure.

[0062] A one-dimensional depth map refers to an image in which each pixel represents a coordinate position and a depth value. Each pixel in the one-dimensional depth map of the embodiment of the present disclosure originates from a point in the one-dimensional point cloud, so that the coordinate position represented by each pixel in the one-dimensional depth map is the coordinate position of the indoor space structure (wall, ceiling, or floor). Combined with the previous description, the one-dimensional point cloud that can be projected includes a one-dimensional point cloud that represents the depth characteristics of the wall structure, which can provide the coordinate position and depth value of the wall structure; and a one-dimensional point cloud that represents the depth characteristics of the ceiling and floor structures, which can provide the coordinate position and depth value of the ceiling and floor structures. Thus, the information provided by these two one-dimensional point clouds can be combined to obtain a one-dimensional depth map of the coordinate position and depth information of the structure between each pixel.

[0063] Specifically, in one example, a one-dimensional point cloud representing the depth features of the wall structure is recorded as the first point cloud, and a one-dimensional point cloud representing the depth features of the ceiling and floor structure is recorded as the second point cloud. The depth values ​​of the points in the first point cloud at a coordinate position are stored in the first channel, and the depth values ​​of the points in the second point cloud at the same coordinate position are stored in the second channel, so that the points at the same coordinate position have two depth values ​​of the dual channels, and a third point cloud is obtained by fusing the first point cloud and the second point cloud. At this time, each point in the third point cloud is a dual-channel feature point, and simultaneously represents the depth features of the wall structure at a coordinate position, as well as the depth features of the ceiling and floor structure. The dual-channel third point cloud thus obtained is the first depth map.

[0064] Compared with the two-dimensional depth map, the one-dimensional depth map can, in the first aspect, emphasize the characteristics of the indoor space structure more. When building a three-dimensional model based on the one-dimensional depth map, it can help the modeling program or software to focus on the indoor space structure to build a three-dimensional model, so as to obtain a more accurate three-dimensional model of the empty room. In the second aspect, the characteristics of non-structural elements, such as the characteristics of furniture, will not affect the structure of the three-dimensional model of the empty room (i.e., the indoor three-dimensional model, the same below). In the one-dimensional depth map of the present disclosure, the characteristics of non-structural elements can be ignored to reduce interference factors. In the third aspect, the one-dimensional feature can better characterize the linear structure of the object or space (such as the edge of the wall), which helps to maintain the geometric consistency in the three-dimensional model, so that the edge structure such as the right angle in the three-dimensional model of the empty room is accurately represented. In the fourth aspect, the one-dimensional feature can better express the topological relationship between objects (such as connection points, boundaries, etc.), which can make the established three-dimensional model of the empty room more accurate. In the fifth aspect, the one-dimensional depth map reduces the dimension of the original data, which can significantly reduce the amount of data to be processed, speed up the data processing speed and the data transmission speed. In the sixth aspect, lower dimensions mean lower complexity requirements for algorithms and models, and easier implementation and optimization.

[0065] S140: Establish a three-dimensional model based on the one-dimensional depth map to obtain an indoor three-dimensional model.

[0066] Establishing a three-dimensional model based on a one-dimensional depth map involves related technologies for reconstructing three-dimensional scenes using depth information. In one example, each pixel in the one-dimensional depth map can be converted into a three-dimensional coordinate point based on the coordinate information represented by each pixel in the one-dimensional depth map and the depth information of the dual channels, and multiple three-dimensional coordinate points constitute a three-dimensional point cloud. By triangulating or networking the three-dimensional cloud, a continuous geometric surface, that is, the surface of the three-dimensional model, can be constructed based on the three-dimensional point cloud. In this way, the three-dimensional model generated based on the three-dimensional point cloud obtained by converting the one-dimensional depth map is the indoor three-dimensional model.

[0067] See also Figure 2 , Figure 2 This is an application scenario of the method for generating an indoor three-dimensional model according to an embodiment of the present disclosure.

[0068] Figure 2 The blue dashed line in the figure is the correct outline of the floor area. Because there is treatment on the right side of the room, the right side of the blue dashed line extends along the corner.

[0069] When there are corner structures in the room, the layout estimation-based solution may ignore the existence of the corners and estimate the attached space that should belong to the corners as walls, resulting in inaccurate indoor structures obtained based on layout estimation, and further inaccurate indoor 3D models based on this indoor structure. Figure 2 The part selected by the red box is the area where the wall, ceiling, and floor are estimated based on the layout estimation solution. This solution cannot identify the auxiliary space at the corner, and it is easy to mistake the auxiliary space at the corner as part of the wall and include it in the area where the wall is located, so the accuracy is limited.

[0070] In some embodiments of the present disclosure, for the same indoor structure, a two-dimensional indoor depth map of the indoor environment is obtained. The two-dimensional indoor depth map has strong robustness and can retain the depth information at the corners, so that when the three-dimensional model is subsequently reconstructed based on the depth information at the corners, the structure of the corners and the attached space at the corners can be restored, which is more accurate than the solution based on layout estimation. When the two-dimensional indoor depth map is obtained, the depth of non-structural elements such as furniture and door openings will be displayed in the two-dimensional indoor depth map. Based on this, it is necessary to perform semantic segmentation processing on the two-dimensional indoor depth map, and only retain the pixel points that represent the spatial structure to obtain a two-dimensional pixel map, so that only the position features and depth features of the structural elements (walls, ceilings, floors, etc.) are retained in the two-dimensional pixel map, so that only the position features and depth features of the structural elements are calculated in subsequent calculations, so that when establishing the indoor three-dimensional model, more focus is placed on restoring the indoor spatial structure.

[0071] For the two-dimensional pixel map obtained through semantic segmentation, it is converted into a one-dimensional point cloud by performing a specific viewing angle and direction projection. Each point in the one-dimensional point cloud only needs one-dimensional depth information to represent the spatial structure of a coordinate position, which is more concise and closer to the geometric essence of the spatial structure, which is conducive to obtaining a more accurate indoor spatial structure when building an indoor three-dimensional model. Finally, a one-dimensional depth map is obtained based on the one-dimensional point cloud, and an indoor three-dimensional model is established based on the one-dimensional depth map, so that the indoor three-dimensional model can accurately represent the spatial structure of the indoor scene, such as Figure 2 The corner outline in the application scenario, as well as the space attached to the corner.

[0072] Regarding step S110, in some embodiments of the present disclosure, it may include: Figure 3 Steps S1101 to S1103 are shown.

[0073] S1101. Obtain a two-dimensional indoor perspective view.

[0074] The two-dimensional indoor perspective view refers to a two-dimensional indoor image of a perspective view. In one example, the two-dimensional indoor perspective view preferably includes three structural elements, namely, a wall, a ceiling, and a floor, so that the ultimately established indoor three-dimensional model includes a richer indoor space structure.

[0075] S1102: Perform depth estimation on pixel points of the two-dimensional indoor perspective image to obtain depth information of each pixel point of the two-dimensional indoor perspective image.

[0076] The two-dimensional indoor perspective image is usually a single RGB image. The depth of the pixels of the two-dimensional indoor perspective image can be estimated based on the monocular depth estimation solution to obtain the depth information of each pixel of the two-dimensional indoor perspective image. Among them, the monocular depth estimation solution can be implemented by depth estimation models such as the Monodepth2 model, the MiDaS model, and the Depth-HRC model. For example, the three-dimensional model generation system transmits the acquired two-dimensional indoor perspective image to the depth estimation model, and obtains the depth estimation result output by the depth estimation model. The depth estimation result includes the depth information of each pixel of the two-dimensional indoor perspective image.

[0077] S1103: Rendering depth information of each pixel point of the two-dimensional indoor perspective image into each pixel point of the two-dimensional indoor depth map to obtain the two-dimensional indoor depth map.

[0078] In one example, the two-dimensional indoor depth map is a grayscale image, each grayscale value can be mapped to a depth value, and the grayscale value of each pixel of the two-dimensional indoor depth map represents the depth value of the pixel. Based on this, when the depth information of each pixel of the two-dimensional indoor depth map is obtained, the depth value represented by the depth information can be mapped to the corresponding grayscale value, and the grayscale value is rendered as the color of the pixel to obtain the pixel of the two-dimensional indoor depth map, thereby obtaining the two-dimensional indoor depth map.

[0079] Regarding step S120, in some embodiments of the present disclosure, it may include the following steps: Figure 4 Steps S1201 to S1202 are shown.

[0080] S1201. Perform semantic segmentation processing on the two-dimensional indoor depth map to obtain a furniture segmentation mask representing the furniture.

[0081] The furniture segmentation mask is the area where the pixels representing the furniture are located in the two-dimensional indoor depth map. Each pixel within the furniture segmentation mask is marked as having a furniture class label.

[0082] In indoor scenes, the features of structural elements are often not prominent and are usually difficult to identify. However, the features of furniture elements are relatively prominent and easy to identify. Based on this, in order to obtain the pixel points representing the spatial structure, we can first find the easily recognizable pixel points representing furniture, and then filter out these pixel points representing furniture. The remaining pixel points are the pixel points representing the spatial structure.

[0083] In one example, semantic segmentation processing of a two-dimensional indoor depth map can be implemented using a pre-trained semantic segmentation model. A pre-trained semantic segmentation model refers to a model that is pre-trained using labeled data so that it can learn feature representations from unlabeled data to identify multiple types of objects, such as DeepLabV3 model, HRNet model, U-Net model, etc. The three-dimensional model generation system transmits the two-dimensional indoor depth map to the pre-trained semantic segmentation model, and obtains the semantic segmentation result generated by the pre-trained semantic segmentation model, and the semantic segmentation result includes a furniture segmentation mask that represents the furniture.

[0084] S1202: Filter out pixels in the area where the furniture segmentation mask is located in the two-dimensional indoor depth map, so that the pixels representing the spatial structure are retained, and obtain a two-dimensional pixel map based on the retained pixels.

[0085] Among them, the pixels in the area where the furniture segmentation mask is located all have classification labels of the furniture category. Based on this, the pixels in the area where the furniture segmentation mask is located can be screened out in the two-dimensional indoor depth map by screening the pixels with classification labels of the furniture category in the two-dimensional indoor depth map and deleting these pixels. After the pixels in the area where the furniture segmentation mask is located in the two-dimensional indoor depth map are screened out, the remaining pixels are the pixels that represent the spatial structure, and the pixel map composed of the remaining pixels that represent the spatial structure is the two-dimensional pixel map obtained in step S1202.

[0086] Regarding step S120, in some embodiments of the present disclosure, it may include the following steps: Figure 5 Steps S1203 to S1204 are shown.

[0087] S1203: Perform semantic segmentation processing on the two-dimensional indoor depth map to obtain a hole structure segmentation mask representing the door opening and / or window opening.

[0088] Similar to the furniture segmentation mask, the hole structure segmentation mask is the area in the 2D indoor depth map where the pixels representing the door or window openings are located. For example, if the indoor scene includes two window openings and one door opening, two window hole structure segmentation masks and one door hole structure segmentation mask can be obtained. Each pixel within the hole structure segmentation mask is marked as having a hole structure class label.

[0089] Hole structures such as door openings and window openings belong to indoor space structures, but compared with space structures such as walls, ceilings and floors, the characteristics of hole structures are relatively easy to identify. In addition, when generating indoor 3D models based on the information of indoor space structures, reconstructing space structures such as walls, ceilings and floors only involves the surface generation of 3D geometric objects, while reconstructing hole structures further involves the processing method of openings in 3D models based on the surface generation of 3D geometric objects. Based on this, it is necessary to mark the hole structures when obtaining the information of indoor space structures, so as to find the location of the hole structures and perform corresponding processing when generating indoor 3D models based on the information of space structures.

[0090] In one example, the furniture segmentation mask and the hole structure segmentation mask can be obtained at the same time during a semantic segmentation process of step S1201. In this case, the semantic segmentation process referred to by step S1204 and the semantic segmentation process referred to by step S1201 are the same semantic segmentation process. In another example, the furniture segmentation mask and the hole structure segmentation mask can be obtained by performing two semantic segmentation processes in step S1201 and step S1204, respectively. In this way, the furniture segmentation mask can be obtained by using a semantic segmentation model trained to perform semantic segmentation on furniture elements, and the hole structure segmentation mask can be obtained by using a semantic segmentation model trained to perform semantic segmentation on hole structure elements. By training the model in a targeted manner, the effect of semantic segmentation can be enhanced, making the obtained segmentation mask more accurate.

[0091] S1204 , marking the outline of the hole structure segmentation mask in the two-dimensional pixel image to obtain a hole structure region marking frame.

[0092] The hole structure segmentation mask includes each pixel point covering the door or window hole, which is used to characterize the position of the door or window hole. However, the position of the door or window hole can also be characterized by the outline of the door or window hole, and the required data is more lightweight. Based on this, on the basis of obtaining the hole structure segmentation mask, the outline of the hole structure segmentation mask can be further annotated to obtain the hole structure area annotation box, which represents the outline of the area where a hole structure is located. In this way, the position of a hole structure can be characterized by information that is as lightweight as possible.

[0093] Regarding step S140, in some embodiments of the present disclosure, it may include: Figure 6 Steps S1401 to S1404 are shown.

[0094] S1401 , superimposing a hole structure region annotation frame on the one-dimensional depth map to obtain a one-dimensional composite image.

[0095] The two-dimensional pixel map is projected into a one-dimensional depth map based on a projection method, and the obtained one-dimensional depth map has the same size as the two-dimensional pixel map, that is, the height and width of the one-dimensional depth map are respectively the same as the height and width of the two-dimensional pixel map. Based on this, when the hole structure area annotation box in the two-dimensional pixel map is superimposed on the one-dimensional depth map in a manner of keeping the coordinates of the annotation box unchanged, the position of the hole structure represented by the hole structure area annotation box remains unchanged, so that the hole structure area annotation box in the two-dimensional pixel map can be used to annotate the position of the hole structure in the one-dimensional depth map to obtain a one-dimensional synthetic image.

[0096] S1402: Generate a three-dimensional geometric surface based on the coordinates and depth information of the pixel points in the one-dimensional synthetic image.

[0097] In an example, for each pixel point (u, v) in the one-dimensional composite image, it can be converted into a three-dimensional coordinate (X, Y, Z) using the camera's intrinsic parameter matrix K by the following formula 1.

[0098] Formula 1:

[0099] Where d is the depth value of the pixel (u, v).

[0100] By converting each pixel point in the one-dimensional composite image into a three-dimensional coordinate point, a three-dimensional point cloud composed of multiple three-dimensional coordinate points can be obtained.

[0101] In one example, a random sample consistency algorithm can be used to fit the three-dimensional point cloud into a plane. Also, a Poisson surface reconstruction algorithm or a Marching Cubes algorithm can be used to convert the plane obtained by fitting the three-dimensional point cloud into multiple triangular meshes, and the geometric surface formed by connecting the various triangular meshes is the three-dimensional geometric surface obtained in step S1402, which represents a wall, a ceiling, or a floor according to its location.

[0102] S1403, based on the hole structure area annotation frame in the one-dimensional composite image, perform hole opening processing on the three-dimensional geometric surface to obtain an indoor three-dimensional model.

[0103] When reconstructing a 3D model, it is often necessary to restore the hole structure in the 3D model through hole processing, such as forming holes through corrosion operations in morphological operations. The hole boundary refers to the hole boundary during the hole processing for 3D model reconstruction. For an indoor 3D model, the hole structure is the hole structure of the indoor 3D model, and the outline of the hole structure is the hole boundary of the indoor 3D model. The hole processing is performed on the 3D geometric surface based on the hole structure area annotation box in the one-dimensional composite image.

[0104] In one example, a three-dimensional geometric surface is composed of multiple triangular meshes. Based on the hole structure area annotation box in the one-dimensional composite image, the three-dimensional geometric surface is opened by deleting the triangular mesh inside the hole structure area annotation box. After deleting some triangular meshes, the mesh needs to be rebuilt to ensure the integrity of the topological structure.

[0105] Regarding step S130, in some embodiments of the present disclosure, it may include the following steps: Figure 7 Steps S1301 to S1303 are shown.

[0106] S1301: Project pixels in the two-dimensional pixel map into a one-dimensional point cloud along a top-down perspective as a one-dimensional wall structure depth map.

[0107] Please combine Figure 8 , Figure 8 The process of converting the 2D indoor depth map into a 2D pixel map is omitted, and this process can be performed during the projection process. Based on the top-down projection, it is equivalent to projecting the pixel points representing the wall in the 2D pixel map to the x-axis. The pixel points projected to the x-axis are the points in the 1D top-down point cloud. The 1D top-down point cloud thus obtained represents the position information and depth information of the wall structure.

[0108] S1302: Project pixels in the two-dimensional pixel map along the vertical viewing angle into a one-dimensional vertical viewing angle point cloud as a one-dimensional top-to-ground structure depth map.

[0109] Please combine Figure 8 , based on the vertical perspective projection, it is equivalent to projecting the pixel points representing the ceiling and floor in the two-dimensional pixel map to the y-axis. The pixel points projected to the y-axis are the points in the one-dimensional vertical perspective point cloud. The one-dimensional vertical perspective point cloud thus obtained represents the position information and depth information of the ceiling and floor structures.

[0110] S1303: synthesize the one-dimensional wall structure depth map and the one-dimensional ceiling structure depth map into a dual-channel pixel map as a one-dimensional depth map.

[0111] Among them, a pixel point in the dual-channel pixel map simultaneously represents the depth value of two channel images, and the two channel images are respectively a one-dimensional wall structure depth map and a one-dimensional top-ground structure depth map. The one-dimensional wall structure depth map and the one-dimensional top-ground structure depth map obtained based on the top-view projection and the vertical view projection are of the same size, so the one-dimensional wall structure depth map and the one-dimensional top-ground structure depth map can be completely overlapped. On this basis, after the one-dimensional wall structure depth map and the one-dimensional top-ground structure depth map are completely overlapped, the pixel point at a coordinate position has two depth values ​​at the same time, and the pixel point at the same position with two depth values ​​at the same time is recorded as a dual-channel pixel point, and the image composed of multiple dual-channel pixels is a dual-channel pixel map, that is, a one-dimensional depth map. In this way, a pixel in the one-dimensional depth map contains a position information and depth information of two channels at the same time. The position information of the dual-channel pixel points and the depth information of the channel where the one-dimensional wall structure depth map is located can be used to restore the three-dimensional coordinate information of the wall. The position information of the dual-channel pixel points and the depth information of the channel where the one-dimensional ceiling and floor structure depth map is located can be used to restore the three-dimensional coordinate information of the ceiling and floor. Therefore, one-dimensional features can be used to store three-dimensional feature information, making the data lighter and more concise, and making the abstract features represented by the one-dimensional depth map closer to the geometric essence of the walls, ceilings and floors.

[0112] In some embodiments of the present disclosure, the method M100 for generating an indoor three-dimensional model may include: Fig. 9 Steps S150 to S160 are shown.

[0113] S150, adding furniture models to the indoor three-dimensional model to obtain a three-dimensional decoration effect model.

[0114] Among them, the indoor 3D model is equivalent to a model of an empty room. Adding furniture models to the indoor 3D model can simulate the placement effect of indoor home decoration. After adding the furniture model, the indoor 3D model and all the furniture models in it are collectively called the 3D decoration effect model.

[0115] In one example, the material of an indoor 3D model may be replaced to simulate the decoration effect of a wall, ceiling, or floor.

[0116] S160: Render the three-dimensional decoration effect model into a two-dimensional image to obtain a two-dimensional decoration effect diagram.

[0117] Rendering a three-dimensional decoration effect model into a two-dimensional image means projecting the three-dimensional decoration effect model into a two-dimensional RGB image at a viewing angle. The obtained two-dimensional RGB image is a two-dimensional decoration effect diagram, which is used to preview the decoration effect.

[0118] Based on any of the above embodiments, the present disclosure also provides a device for generating an indoor three-dimensional model. Fig.10It is a schematic block diagram of the structure of a device for generating an indoor three-dimensional model according to an embodiment of the present invention.

[0119] like Fig.10 As shown, the generating device of the indoor three-dimensional model includes:

[0120] The depth map acquisition module 110 is used to acquire a two-dimensional indoor depth map.

[0121] The semantic segmentation module 120 is used to perform semantic segmentation processing on the two-dimensional indoor depth map, obtain and retain multiple pixel points representing the spatial structure in the two-dimensional indoor depth map, and obtain a two-dimensional pixel map based on the retained pixel points.

[0122] The projection module 130 is used to project pixels in the two-dimensional pixel map into a one-dimensional point cloud, and obtain a one-dimensional depth map based on the one-dimensional point cloud.

[0123] The model generation module 140 is used to establish a three-dimensional model based on the one-dimensional depth map to obtain an indoor three-dimensional model.

[0124] The above-mentioned generating device can be in the form of computer software, and each module of the generating device can be in the form of computer software modules. The implementation process of the functions and effects of each module is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0125] The execution subject of the method for generating an indoor three-dimensional model in the specific implementation manner of the present disclosure may be an electronic device such as a server device.

[0126] Therefore, based on any one of the above embodiments, the present disclosure further provides an electronic device, which can execute the method for generating an indoor three-dimensional model of any one of the embodiments described above in the present disclosure.

[0127] Fig.11 1 is a schematic block diagram of the structure of an electronic device 1000 according to an embodiment of the present disclosure.

[0128] The hardware structure of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnected buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300 and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0129] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the figure only uses one connecting line, but does not mean that there is only one bus or one type of bus.

[0130] The present disclosure also provides a readable storage medium, in which a computer program is stored, and the computer program is used to implement the above method when executed by a processor. "Readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use in an instruction execution system, device or equipment or in combination with these instruction execution systems, devices or equipment. More specific examples of readable storage media include the following: an electrical connection portion (electronic device) with one or more wirings, a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM), etc.

[0131] The present disclosure also provides a computer program product. The method of the present disclosure can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed, the process or function of the present disclosure is executed in whole or in part.

[0132] A computer program or instruction may be stored in a readable storage medium or transmitted from one readable storage medium to another readable storage medium, for example, a computer program or instruction may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The readable storage medium may be any available medium that can be accessed or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid state drive. The computer readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0133] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable indoor three-dimensional model generation device to produce a machine, so that the instructions executed by the processor of the computer or other programmable indoor three-dimensional model generation device generate instructions for implementing the processes in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0135] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable indoor three-dimensional model generation device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0136] These computer program instructions can also be loaded onto a computer or other programmable indoor three-dimensional model generation device, so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0137] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments / methods or examples described in this specification and features of different embodiments / methods or examples, unless they are contradictory.

[0138] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0139] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A method for generating an indoor three-dimensional model, characterized in that: include: Get a 2D indoor depth map; Performing semantic segmentation processing on the two-dimensional indoor depth map to obtain and retain a plurality of pixel points representing the spatial structure in the two-dimensional indoor depth map, and obtaining a two-dimensional pixel map based on the retained pixel points; Projecting pixels in the two-dimensional pixel map into a one-dimensional point cloud, and obtaining a one-dimensional depth map based on the one-dimensional point cloud, wherein the one-dimensional depth map is an image in which each pixel point represents a coordinate position and a depth value, and the coordinate position represented by each pixel point in the one-dimensional depth map is the coordinate position of the indoor space structure; as well as A three-dimensional model is established based on the one-dimensional depth map to obtain an indoor three-dimensional model.

2. The method for generating an indoor three-dimensional model according to claim 1, characterized in that: Get a 2D indoor depth map, including: Get a 2D interior perspective image; Performing depth estimation on pixel points of the two-dimensional indoor perspective image to obtain depth information of each pixel point of the two-dimensional indoor perspective image; and The depth information of each pixel point of the two-dimensional indoor perspective image is rendered as each pixel point of the two-dimensional indoor depth map to obtain the two-dimensional indoor depth map.

3. The method for generating an indoor three-dimensional model according to claim 1, characterized in that: Performing semantic segmentation processing on the two-dimensional indoor depth map to obtain and retain a plurality of pixel points representing the spatial structure in the two-dimensional indoor depth map, and obtaining a two-dimensional pixel map based on the retained pixel points, including: Performing semantic segmentation processing on the two-dimensional indoor depth map to obtain a furniture segmentation mask representing the furniture; and In the two-dimensional indoor depth map, the pixels in the area where the furniture segmentation mask is located are filtered out so that the pixels representing the spatial structure are retained, and the two-dimensional pixel map is obtained based on the retained pixels.

4. The method for generating an indoor three-dimensional model according to claim 3, characterized in that: The method further comprises: performing semantic segmentation processing on the two-dimensional indoor depth map to obtain and retain a plurality of pixel points representing the spatial structure in the two-dimensional indoor depth map, and obtaining a two-dimensional pixel map based on the retained pixel points. Performing semantic segmentation processing on the two-dimensional indoor depth map to obtain a hole structure segmentation mask representing the door opening and / or window opening; and The outline of the hole structure segmentation mask is marked in the two-dimensional pixel image to obtain a hole structure area marking box.

5. The method for generating an indoor three-dimensional model according to claim 4, characterized in that: Establishing a three-dimensional model based on the one-dimensional depth map to obtain an indoor three-dimensional model includes: Superimposing the hole structure area annotation frame on the one-dimensional depth map to obtain a one-dimensional synthetic image; Generate a three-dimensional geometric surface based on the coordinates and depth information of the pixel points in the one-dimensional synthetic image; and The three-dimensional geometric surface is subjected to hole opening processing based on the hole structure area annotation frame in the one-dimensional synthetic image to obtain the indoor three-dimensional model.

6. The method for generating an indoor three-dimensional model according to claim 1, characterized in that: Projecting pixels in the two-dimensional pixel map into a one-dimensional point cloud, and obtaining a one-dimensional depth map based on the one-dimensional point cloud, including: Projecting pixels in the two-dimensional pixel image into a one-dimensional point cloud along a top-down angle as a one-dimensional wall structure depth map; Projecting the pixels in the two-dimensional pixel map into a one-dimensional vertical viewing angle point cloud along the vertical viewing angle as a one-dimensional top-to-bottom structure depth map; and The one-dimensional wall structure depth map and the one-dimensional ceiling structure depth map are synthesized into a dual-channel pixel map as the one-dimensional depth map.

7. The method for generating an indoor three-dimensional model according to claim 1, characterized in that: Also includes: Adding furniture models to the indoor three-dimensional model to obtain a three-dimensional decoration effect model; and The three-dimensional decoration effect model is rendered into a two-dimensional image to obtain a two-dimensional decoration effect diagram.

8. An electronic device, characterized in that: include: A memory storing execution instructions; as well as A processor, wherein the processor executes the execution instructions stored in the memory, so that the processor executes the method for generating an indoor three-dimensional model according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that: The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the method for generating an indoor three-dimensional model according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating an indoor three-dimensional model according to any one of claims 1 to 7 is implemented.

Images