Method, device, readable storage medium and electronic device for constructing house model

Abstract

The present application discloses a method, device, readable storage medium and electronic device for constructing a house model. The method for constructing a house model includes: obtaining first image data in RGB color mode obtained by photographing a house with a photographing device at a preset point, a depth map corresponding to the first image data and position information of the photographing device; determining a point cloud model according to the position information, the first image data and the depth map; performing semantic segmentation on the first image data to obtain a semantic point cloud with semantic priors; determining a semantic segmentation result of the point cloud model according to the semantic point cloud with semantic priors and the point cloud model; and determining the house model according to the semantic segmentation result of the point cloud model.

Description

Technical Field

[0001] The present application belongs to the field of data processing, and specifically relates to a method, device, readable storage medium and electronic device for constructing a house model. Background Art

[0002] In the past technology, depth acquisition equipment was often introduced to collect the geometric shape of the house for modeling. However, the depth acquisition equipment often introduces certain errors in the collected data due to the limitations of its own acquisition accuracy.

[0003] In addition, in the actual acquisition process, the photographer may also cause a significant decrease in the acquisition accuracy due to misoperation. In the case of a large acquisition error, the semantic information will be examined and used to compensate for the acquisition error. Specifically, the solution is as follows:

[0004] The walls, floors, ceilings, etc. are semantically recognized in two dimensions, and the surfaces of the walls, floors, and ceilings are reconstructed in accordance with the CAD (Computer Aided Design, which uses computers and their graphic devices to help designers perform design work) mapping method.

[0005] The above scheme actually adopts pure two-dimensional semantic segmentation. If the model is reconstructed using the above scheme, modeling is prone to failure in some cases. Summary of the invention

[0006] The purpose of the embodiments of the present application is to provide a method, device, readable storage medium and electronic device for constructing a house model, which can solve the problem of modeling failure in some cases if the model is reconstructed using the above solution.

[0007] In a first aspect, an embodiment of the present application provides a method for constructing a house model, comprising: obtaining first image data in RGB color mode obtained by photographing a house with a photographing device at a preset position, a depth map corresponding to the first image data, and posture information of the photographing device; determining a point cloud model according to the posture information, the first image data, and the depth map; performing semantic segmentation on the first image data to obtain a semantic point cloud with semantic priors; determining a semantic segmentation result of the point cloud model according to the semantic point cloud with semantic priors and the point cloud model; and determining the house model according to the semantic segmentation result of the point cloud model.

[0008] In a second aspect, an embodiment of the present application provides a device for constructing a house model, comprising: an acquisition unit, for acquiring first image data in RGB color mode obtained by photographing a house with a photographing device at a preset point, a depth map corresponding to the first image data, and posture information of the photographing device; a first determination unit, for determining a point cloud model according to the posture information, the first image data, and the depth map; a segmentation unit, for performing semantic segmentation on the first image data to obtain a semantic point cloud with semantic priors; a second determination unit, for determining a semantic segmentation result of the point cloud model based on the semantic point cloud with semantic priors and the point cloud model; and a construction unit, for determining the house model according to the semantic segmentation result of the point cloud model.

[0009] In a third aspect, an embodiment of the present application provides a device for constructing a house model, comprising: a processor and a memory, the memory storing programs or instructions that can be executed on the processor, and the programs or instructions, when executed by the processor, implement the steps of any method in the first aspect.

[0010] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method of the first aspect are implemented.

[0011] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a device for constructing a house model as in the second aspect or the third aspect; and / or a readable storage medium as in the fourth aspect.

[0012] In a sixth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method of the first aspect.

[0013] In a seventh aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method of the first aspect.

[0014] In the embodiment of the present application, semantic segmentation is performed on the first image data to obtain a verified semantic point cloud, and under the guidance of the semantic point cloud, the point cloud model is segmented. In this process, the two-dimensional semantic and three-dimensional point cloud instance segmentation are combined to enhance the overall modeling, improve the flatness and integrity of the modeling, and thus reduce the probability of modeling failure.

[0015] In addition, the existing pure two-dimensional semantic segmentation cannot perform accurate segmentation in some scenes due to the lack of prior information such as shape, such as hanging unknown objects or clothes on the wall. It is difficult to judge the category of such scenes, and they are likely to be mistakenly classified as walls. In the embodiments of the present application, a semantic point cloud with semantic priors is used for three-dimensional point cloud instance segmentation, which can effectively segment the above scenes, thereby increasing the chance of successful modeling.

[0016] Specifically, since the point cloud model is determined based on the posture information, first image data and depth map of the house captured by the shooting device at a preset point, the depth information of the house is retained in the result of semantic segmentation. Therefore, the obtained house model can accurately restore the real details of the house, achieve high-precision modeling, and restore the real situation of the house to the greatest extent, such as the surface of an object with a home decoration design sense and an uneven surface, so that users can accurately know the decoration design sense of the house through the house model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of a method for constructing a house model in an embodiment of the present application;

[0018] Figure 2 is a schematic diagram of a house before semantic segmentation in an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of a house after semantic segmentation in an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of a 6DoF posture in an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of a hole existing in a point cloud in an embodiment of the present application;

[0022] Figure 6 is a schematic diagram of a point cloud filled with holes in an embodiment of the present application;

[0023] Figure 7 is a schematic diagram of a house model before extracting a plane and performing mesh deformation on a reconstructed mesh corresponding to a relevant point cloud in an embodiment of the present application;

[0024] Figure 8 is a schematic diagram of a house model after extracting a plane and performing mesh deformation on a reconstructed mesh corresponding to a relevant point cloud in an embodiment of the present application;

[0025] Fig. 9 is a schematic block diagram of a device for constructing a house model in an embodiment of the present application;

[0026] Fig.10is a schematic block diagram of a device for constructing a house model in an embodiment of the present application;

[0027] Fig.11 It is a schematic diagram of the hardware structure of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0029] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0030] In conjunction with the accompanying drawings, the following detailed description is given of the construction method, device, readable storage medium and electronic device of the house model provided in the embodiments of the present application through specific embodiments and their application scenarios.

[0031] In one embodiment, if Figure 1 As shown, a method for constructing a house model is proposed, including:

[0032] Step 102, obtaining first image data in RGB color mode obtained by photographing the house by a photographing device at a preset point, a depth map corresponding to the first image data, and position information of the photographing device.

[0033] In one embodiment, the photographing device may be an RGB-D camera, such as a depth camera that uses Light Coding (structured light) or TOF for imaging.

[0034] In one embodiment, the preset point can be manually selected when the structure of the house is determined, wherein the point can be understood as the placement position of the camera.

[0035] In one of the technical solutions, the posture information can be calculated using SFM (Structure From Motion), SLAM (Simultaneous Localization and Mapping), or inertial navigation fusion. The calculation process of SFM (Structure From Motion), SLAM (Simultaneous Localization and Mapping), or inertial navigation fusion will not be repeated here.

[0036] In one of the embodiments, the depth map may be represented in the form of a depth picture.

[0037] Step 104: determine a point cloud model according to the pose information, the first image data, and the depth map.

[0038] In this embodiment, since the first image data and the depth map are obtained by collecting the house, the point cloud model can be understood as a model composed of point clouds corresponding to the house. By constructing the point cloud model, the house can be displayed in the form of a point cloud.

[0039] Among them, the method for generating the point cloud model is as follows:

[0040] P cloud =[R|t] -1 K -1 p rgbd , where P cloud is a 3D point in the point cloud model, p rgbd The RGB value of a target selected point in the first image data and the depth value of the target selected point in the depth map are obtained.

[0041] Wherein, [R|t] is the external parameter matrix of the camera using the first image data and the depth map, and K is the internal parameter.

[0042] Step 106: perform semantic segmentation on the first image data to obtain a semantic point cloud with semantic priors.

[0043] Among them, semantic segmentation can use various backbone deep learning networks as semantic segmentation networks, such as transformer, Resnet, etc. The specific segmentation method will not be repeated here.

[0044] In this embodiment, the semantic point cloud with semantic priors can be understood as the semantics corresponding to the segmented point cloud is verified and credible.

[0045] Among them, P cloud(semantic) =[R|t]-1 K -1 p rgbd(semantic) , where P cloud(semantic) is a 3D point in the semantic point cloud with semantic prior, p rgbd(semantic) The RGB value of the pixel corresponding to the two-dimensional semantic segmentation and the corresponding depth value in the depth map. Where [R|t] is the external parameter matrix of the camera using the first image data and the depth map, and K is the internal parameter.

[0046] Step 108 , determining a semantic segmentation result of the point cloud model according to the semantic point cloud with semantic priors and the point cloud model.

[0047] Step 110, determining a house model according to the semantic segmentation result of the point cloud model.

[0048] In the embodiment of the present application, semantic segmentation is performed on the first image data to obtain a verified semantic point cloud, and under the guidance of the semantic point cloud, the point cloud model is segmented. In this process, the two-dimensional semantic and three-dimensional point cloud instance segmentation are combined to enhance the overall modeling, improve the flatness and integrity of the modeling, and thus reduce the probability of modeling failure.

[0049] In addition, the existing pure two-dimensional semantic segmentation cannot perform accurate segmentation in some scenes due to the lack of prior information such as shape, such as hanging unknown objects or clothes on the wall. It is difficult to judge the category of such scenes, and they are likely to be mistakenly classified as walls. In the embodiments of the present application, a semantic point cloud with semantic priors is used for three-dimensional point cloud instance segmentation, which can effectively segment the above scenes, thereby increasing the chance of successful modeling.

[0050] Specifically, since the point cloud model is determined based on the posture information, first image data and depth map of the house captured by the shooting device at a preset point, the depth information of the house is retained in the result of semantic segmentation. Therefore, the obtained house model can accurately restore the real details of the house, achieve high-precision modeling, and restore the real situation of the house to the greatest extent, such as the surface of an object with a home decoration design sense and an uneven surface, so that users can accurately know the decoration design sense of the house through the house model.

[0051] In one of the embodiments, a semantic segmentation result of a point cloud model is determined based on a semantic point cloud with semantic priors and a point cloud model, including: determining a semantic type corresponding to the semantic point cloud with semantic priors; matching the semantic point cloud with semantic priors to the point cloud model based on the semantic type, and determining a first target point cloud corresponding to the semantic point cloud with semantic priors; associating the semantic type with the first target point cloud to obtain a semantic segmentation result of the point cloud model.

[0052] In this embodiment, the semantic type corresponding to the semantic point cloud with semantic prior is determined, so that when the semantic type is determined, the point cloud in the point cloud model that matches the semantic point cloud with semantic prior is picked up according to the semantic type as a category, that is, the first target point cloud is determined, thereby realizing the segmentation of the point cloud model.

[0053] In the case where there are multiple semantic types, the multiple semantic types are sorted, and the first target point cloud is determined in sequence according to the sorting results until the first target point clouds corresponding to all the semantic types are determined.

[0054] For example, when the semantic type includes walls and beds, the walls and beds are sorted to obtain a sorting result in which the walls are in front and the beds are in the back, and based on this result, the first target point cloud corresponding to the wall and the first target point cloud corresponding to the bed are determined in sequence.

[0055] In one of the embodiments, it also includes: matching the first target point cloud with a template in a preset point cloud library to obtain a first matching result; determining a second matching result of the first target point cloud with preset parameters within a preset threshold range of a preset semantic point cloud or an instance point cloud; and updating the semantic point cloud with semantic priors according to the first matching result and the second matching result.

[0056] In this embodiment, a preset point cloud library is set to verify the first target point cloud, thereby ensuring the accuracy of semantic segmentation. Specifically, when the first target point cloud belongs to the preset point cloud library, the segmentation is considered accurate; conversely, when the first target point cloud does not belong to the preset point cloud library, the segmentation is considered inaccurate.

[0057] Among them, the first target point cloud corresponding to the semantic point cloud with semantic prior can be understood as the point cloud obtained by segmenting the point cloud model according to the semantic type, which is expressed as follows:

[0058] P cloud(instance) =[R|t] -1 K -1 p rgbd(instance) , where P cloud(instance) is a 3D point in the point cloud obtained by segmenting the point cloud model according to the semantic type, p rgbd(instance) The RGB value of a pixel of a 3D point in the point cloud obtained by segmenting the point cloud model according to the semantic type and the corresponding depth value in the depth map.

[0059] Among them, the point cloud obtained by segmenting the point cloud model according to the semantic type can be a 3D point cloud, and the template in the preset point cloud library is matched to extract the features of the 3D point cloud and the point cloud in the template, such as sift features and orb features, and match them using similarity matching.

[0060] Specifically, the formula for similarity matching is as follows:

[0061]

[0062] in, is the category of the first matching result, P class A category for templates in the preset point cloud library.

[0063] Among them, the second matching result of the preset parameters of the first target point cloud within the preset threshold range of the preset semantic point cloud or the instance point cloud can be understood as fine tuning the point cloud within a certain range (that is, the preset threshold range mentioned above) according to the result of similarity matching, and its specific operations include:

[0064]

[0065] Among them, P c , loud(instance) It is a 3D point in the fine tune point cloud. Boundary is the best result within a preset threshold range (such as 0 cm to 50 cm). This result can be used to obtain the maximum similarity matching value.

[0066] Wherein, [R|t] is the external parameter matrix of the camera using the first image data and the depth map, and K is the internal parameter.

[0067] In one embodiment, the preset point cloud library may include but is not limited to point cloud data of tables and chairs, point cloud data of beds, point cloud data of lamps, and point cloud data of electrical appliances, wherein the electrical appliances may be refrigerators, washing machines, water heaters, air conditioners, etc.

[0068] In this embodiment, a second matching result of preset parameters of the first target point cloud within a preset threshold range of a preset semantic point cloud or an instance point cloud is calculated to measure the difference between the first target point cloud and the preset semantic point cloud or the instance point cloud, thereby optimizing the semantic point cloud with semantic priors according to the difference to ensure the accuracy of the segmentation of the point cloud model.

[0069] In one of the embodiments, the preset threshold range can be selected according to the actual usage scenario, and its specific value is not repeated here.

[0070] For example, the preset threshold range may be a value between 0 cm and 50 cm within the field of a preset semantic point cloud or an instance point cloud.

[0071] In one of the embodiments, the second matching result of the preset parameters of the first target point cloud within the preset threshold range of the preset semantic point cloud or the instance point cloud can be understood as whether the preset parameters of the first target point cloud are consistent or inconsistent with the preset parameters within the preset threshold range of the preset semantic point cloud or the instance point cloud.

[0072] In one of the embodiments, when the first target point cloud does not belong to a preset point cloud library and the first target point cloud does not match preset parameters within a preset threshold range of a preset semantic point cloud or an instance point cloud, a semantic point cloud with semantic priors is updated to ensure the accuracy of the segmentation of the point cloud model.

[0073] In one embodiment, the preset parameters include: a normal vector, a color value, and a depth value.

[0074] In one of the embodiments, the method further includes: determining a blank portion in the first target point cloud; and filling the blank portion according to the first matching result.

[0075] In this embodiment, the blank parts in the first target point cloud are filled to improve the completeness of the house model.

[0076] In one of the embodiments, the blank portion is filled with the first matching result, thereby ensuring the accuracy of the filling.

[0077] In one of the embodiments, filling the blank portion includes: determining a target position corresponding to the blank portion in the depth map; and performing depth filling on the target position.

[0078] In this embodiment, possible options for filling the blank parts are provided. By determining the target position corresponding to the blank part in the depth map, it is ensured that the filling position accurately corresponds to the blank part, thereby reducing the impact of filling the position that does not need to be filled or the inability to accurately fill the blank position on the integrity of the house model.

[0079] In one of the embodiments, performing depth filling on the target position may be understood as modifying the depth value.

[0080] In one of the embodiments, a traversal algorithm is used to find all holes where point clouds are missing (wherein a hole refers to a connected domain without point clouds), which is the blank part mentioned above.

[0081] In one of the embodiments, for each blank portion, a shape equation of the plane or surface where the hole is located is estimated, and a point cloud of the blank portion is generated based on the shape equation. The color of the point cloud generated for the blank portion is replaced by the color average of the point clouds surrounding the blank portion.

[0082] In this embodiment, when there is a lack of point cloud collection in the point cloud model (for example, desktop reflection, structured light, etc., the depth camera cannot correctly collect the point cloud), the use of precise semantic information can well supplement the corresponding point cloud and improve the modeling integrity.

[0083] In one of the embodiments, it also includes: determining a plane part in a semantic point cloud with semantic priors; determining a second target point cloud corresponding to the plane part in the first target point cloud; and performing mesh deformation on a reconstructed mesh corresponding to the second target point cloud.

[0084] In this embodiment, by determining the planar part in the semantic point cloud with semantic priors, after finding the second target point cloud corresponding to the planar part in the first target point cloud, the reconstructed grid corresponding to the second target point cloud is deformed, so that the part of the constructed house model that should be planar appears to be planar, thereby maximizing the flatness of the constructed house model.

[0085] In one of the embodiments, the plane part may be extracted using a preset plane extraction method, wherein the plane extraction method may be a 3D plane extraction method.

[0086] In one of the embodiments, the posture information includes: position information of the camera and three-dimensional orientation information of the camera.

[0087] In one of the embodiments, it also includes: determining the texture corresponding to the semantic segmentation result of the point cloud model; and determining the house model according to the texture and the semantic segmentation result of the point cloud model.

[0088] In this embodiment, when constructing a house model, the texture corresponding to the semantic segmentation result of the point cloud model is combined to improve the authenticity of the house model, which helps users feel the actual decoration design.

[0089] In one embodiment, the first image data includes panoramic image data.

[0090] In this embodiment, it can be understood that the panoramic image data is also a panoramic image.

[0091] In one embodiment, a method for constructing a house model includes:

[0092] Step 1: Input the original picture or original image of the collected house, and perform picture semantic segmentation or panorama semantic segmentation. In this step, various backbone deep learning networks can be used as semantic segmentation networks, such as transformer, Resnet, etc.

[0093] For example, in the original image Figure 2 In the case of , after semantic segmentation, we get Figure 3 The segmentation result.

[0094] Step 2: Calculate the 6DoF pose (including the acquisition position and the 3D orientation of the camera) of the panoramic image / original image collected at each point. Generally, SFM, SLAM, or inertial navigation fusion can be used for calculation.

[0095] like Figure 4 As shown, the vertex of the obtuse angle of the triangle represents the acquisition position, and the normal direction of the side corresponding to the obtuse angle of the triangle represents the three-dimensional direction of the acquisition.

[0096] Step 3: Generate a point cloud model based on the 6DoF pose of each point and the corresponding raw data (panoramic image / original image-RGB, depth image-depth). And generate a point cloud with 2D semantic priors based on the results of 2D semantic segmentation.

[0097] Step 4: Input the point cloud model and the point cloud with 2D semantic priors, and calculate the semantic segmentation result of the 3D point cloud. The specific steps are as follows:

[0098] 4-1: Extract the 3D point cloud of each instance according to the semantic type, and perform template matching on the 3D point cloud library (furniture, etc.). At the same time, calculate the normal vector N, color (RGB), and depth (D) within a certain threshold range (eg, 0cm-50cm) of the semantic / instance point cloud field, and re-fine tune the semantic point cloud based on the matching value results and template results;

[0099] 4-2: For the semantic point cloud, check whether there are holes. If there are holes, fill the hole part (that is, the blank part in this application) point cloud according to the template matching result. The specific method can directly modify the depth corresponding to the relevant hole on the depth map and perform depth filling. Repeat 4-1. If there are no holes, skip step 4-2.

[0100] like Figure 5 As shown, there are holes in the point cloud. By filling them, we get the following Figure 6 The semantic point cloud shown.

[0101] 4-3: For the plane part in the semantic point cloud, the 3D plane extraction method is used to extract the plane, and the reconstructed mesh corresponding to the relevant point cloud is deformed to make it conform to the plane features.

[0102] Before extracting the plane and deforming the reconstructed mesh corresponding to the relevant point cloud, the house model is as follows Figure 7 As shown in the figure, the plane is extracted by the 3D plane extraction method, and the reconstructed mesh corresponding to the relevant point cloud is deformed to obtain the following Figure 8 Model house shown.

[0103] Step 4: Texture estimation and obtain the final house model.

[0104] In this embodiment, the 2D semantic / instance segmentation is mixed with the 3D point cloud semantic segmentation method to improve the accuracy of semantic segmentation; at the same time, for hollow objects and point cloud errors caused by acquisition (the wall is not a single layer, but has a certain thickness), the integrity (no holes) and flatness (the ceiling, floor, table tops, etc. in the house are smoother) of the reconstructed model can be greatly improved.

[0105] The construction method of the house model provided in the embodiment of the present application can be executed by a construction device of the house model. In the embodiment of the present application, the construction method of the house model is executed by a construction device of the house model as an example to illustrate the construction device of the house model provided in the embodiment of the present application.

[0106] In one embodiment, if Fig. 9 As shown, a device 900 for constructing a house model is proposed, including: an acquisition unit 902, used to obtain first image data in RGB color mode obtained by photographing a house with a photographing device at a preset point, a depth map corresponding to the first image data, and posture information of the photographing device; a first determination unit 904, used to determine a point cloud model according to the posture information, the first image data and the depth map; a segmentation unit 906, used to perform semantic segmentation on the first image data to obtain a semantic point cloud with semantic priors; a second determination unit 908, used to determine the semantic segmentation result of the point cloud model according to the semantic point cloud with semantic priors and the point cloud model; a construction unit 910, used to determine the house model according to the semantic segmentation result of the point cloud model.

[0107] In one of the embodiments, the second determination unit 908 is used to: determine the semantic type corresponding to the semantic point cloud with semantic priors; match the semantic point cloud with semantic priors to the point cloud model according to the semantic type, and determine the first target point cloud corresponding to the semantic point cloud with semantic priors; associate the semantic type with the first target point cloud to obtain the semantic segmentation result of the point cloud model.

[0108] In one of the embodiments, the second determination unit 908 is also used to: match the first target point cloud with a template in a preset point cloud library to obtain a first matching result; determine a second matching result of the first target point cloud with preset parameters within a preset threshold range of a preset semantic point cloud or an instance point cloud; and update the semantic point cloud with semantic priors based on the first matching result and the second matching result.

[0109] In one embodiment, the preset parameters include: a normal vector, a color value, and a depth value.

[0110] In one embodiment, the second determining unit 908 is further used to: determine a blank portion in the first target point cloud; and fill the blank portion according to the first matching result.

[0111] In one of the embodiments, the second determining unit 908 is specifically configured to: determine a target position corresponding to the blank portion in the depth map; and perform depth filling on the target position.

[0112] In one of the embodiments, the second determination unit 908 is further used to: determine a plane part in a semantic point cloud with semantic priors; determine a second target point cloud corresponding to the plane part in the first target point cloud; and perform mesh deformation on a reconstructed mesh corresponding to the second target point cloud.

[0113] In one of the embodiments, the posture information includes: position information of the camera and three-dimensional orientation information of the camera.

[0114] In one of the embodiments, the construction unit is used to: determine the texture corresponding to the semantic segmentation result of the point cloud model; and determine the house model according to the texture and the semantic segmentation result of the point cloud model.

[0115] In one embodiment, the first image data includes panoramic image data.

[0116] The construction device of the house model in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than the terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a handheld computer, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0117] The house model construction device 900 in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0118] The building model construction device 900 provided in the embodiment of the present application can realize Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0119] Alternatively, if Fig.10 As shown, the embodiment of the present application also provides a device 1000 for constructing a house model, including a processor 1002 and a memory 1004. The memory 1004 stores a program or instruction that can be executed on the processor 1002. When the program or instruction is executed by the processor 1002, the various steps of the above-mentioned house model construction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0120] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0121] In one of the embodiments, an electronic device is provided, comprising: a device for constructing a house model as described above; and / or a readable storage medium as described above.

[0122] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0123] Fig.11 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0124] The electronic device 1100 includes but is not limited to: a radio frequency unit 1101, a network module 1102, an audio output unit 1103, an input unit 1104, a sensor 1105, a display unit 1106, a user input unit 1107, an interface unit 1108, a memory 1109 and a processor 1110 and other components.

[0125] Those skilled in the art will appreciate that the electronic device 1100 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.11The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0126] In one embodiment, the processor 1110 is used to: obtain first image data in RGB color mode obtained by photographing a house with a photographing device at a preset position, a depth map corresponding to the first image data, and posture information of the photographing device; determine a point cloud model according to the posture information, the first image data, and the depth map; perform semantic segmentation on the first image data to obtain a semantic point cloud with semantic priors; determine a semantic segmentation result of the point cloud model based on the semantic point cloud with semantic priors and the point cloud model; and determine a house model based on the semantic segmentation result of the point cloud model.

[0127] In one of the embodiments, a semantic segmentation result of a point cloud model is determined based on a semantic point cloud with semantic priors and a point cloud model, and the processor 1110 is used to: determine a semantic type corresponding to the semantic point cloud with semantic priors; match the semantic point cloud with semantic priors to the point cloud model based on the semantic type, and determine a first target point cloud corresponding to the semantic point cloud with semantic priors; and associate the semantic type with the first target point cloud to obtain a semantic segmentation result of the point cloud model.

[0128] In one embodiment, the processor 1110 is also used to: match the first target point cloud with a template in a preset point cloud library to obtain a first matching result; determine a second matching result of the first target point cloud with preset parameters within a preset threshold range of a preset semantic point cloud or an instance point cloud; and update the semantic point cloud with semantic priors based on the first matching result and the second matching result.

[0129] In one embodiment, the preset parameters include: a normal vector, a color value, and a depth value.

[0130] In one embodiment, the processor 1110 is further configured to: determine a blank portion in the first target point cloud; and fill the blank portion according to the first matching result.

[0131] In one of the embodiments, the blank portion is filled, and the processor 1110 is used to: determine a target position corresponding to the blank portion in the depth map; and perform depth filling on the target position.

[0132] In one of the embodiments, the processor 1110 is further used to: determine a planar portion in a semantic point cloud with semantic priors; determine a second target point cloud corresponding to the planar portion in the first target point cloud; and perform mesh deformation on a reconstructed mesh corresponding to the second target point cloud.

[0133] In one of the embodiments, the posture information includes: position information of the camera and three-dimensional orientation information of the camera.

[0134] In one of the embodiments, the processor 1110 is further used to: determine the texture corresponding to the semantic segmentation result of the point cloud model; and determine the house model according to the texture and the semantic segmentation result of the point cloud model.

[0135] In one embodiment, the first image data includes panoramic image data.

[0136] In the embodiment of the present application, semantic segmentation is performed on the first image data to obtain a verified semantic point cloud, and under the guidance of the semantic point cloud, the point cloud model is segmented. In this process, the two-dimensional semantic and three-dimensional point cloud instance segmentation are combined to enhance the overall modeling, improve the flatness and integrity of the modeling, and thus reduce the probability of modeling failure.

[0137] In addition, the existing pure two-dimensional semantic segmentation cannot perform accurate segmentation in some scenes due to the lack of prior information such as shape, such as hanging unknown objects or clothes on the wall. It is difficult to judge the category of such scenes, and they are likely to be mistakenly classified as walls. In the embodiments of the present application, a semantic point cloud with semantic priors is used for three-dimensional point cloud instance segmentation, which can effectively segment the above scenes, thereby increasing the chance of successful modeling.

[0138] Specifically, since the point cloud model is determined based on the posture information, first image data and depth map of the house captured by the shooting device at a preset point, the depth information of the house is retained in the result of semantic segmentation. Therefore, the obtained house model can accurately restore the real details of the house, achieve high-precision modeling, and restore the real situation of the house to the greatest extent, such as the surface of an object with a home decoration design sense and an uneven surface, so that users can accurately know the decoration design sense of the house through the house model.

[0139] It should be understood that in the embodiment of the present application, the input unit 1104 may include a graphics processor (Graphics Processing Unit, GPU) 11041 and a microphone 11042, and the graphics processor 11041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1106 may include a display panel 11061, and the display panel 11061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1107 includes a touch panel 11071 and at least one of other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include two parts: a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0140] The memory 1109 can be used to store software programs and various data. The memory 1109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1109 may include a volatile memory or a non-volatile memory, or the memory 1109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct RAM bus random access memory (DRRAM). The memory 1109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory. The processor 1110 may include one or more processing units; optionally, the processor 1110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations involving an operating system, a user interface and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1110.

[0141] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned symmetrical wallpaper display method and / or symmetrical wallpaper generation method embodiments, and can achieve the same technical effect. To avoid repetition, they are not repeated here.

[0142] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0143] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned symmetrical wallpaper display method and / or symmetrical wallpaper generation method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0144] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "including one" do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0145] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0146] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for constructing a house model, characterized in that: include: Acquire first image data in RGB color mode obtained by photographing a house by a photographing device at a preset point, a depth map corresponding to the first image data, and position information of the photographing device; Determine a point cloud model according to the pose information, the first image data and the depth map; Performing semantic segmentation on the first image data to obtain a semantic point cloud with semantic priors; Determine a semantic segmentation result of the point cloud model according to the semantic point cloud with semantic prior and the point cloud model; Determine the house model according to the semantic segmentation result of the point cloud model; Determining a semantic segmentation result of the point cloud model according to the semantic point cloud with semantic prior and the point cloud model includes: Determining the semantic type corresponding to the semantic point cloud with semantic prior; According to the semantic type, matching the semantic point cloud with semantic priors to the point cloud model to determine a first target point cloud corresponding to the semantic point cloud with semantic priors; Associating the semantic type with the first target point cloud to obtain a semantic segmentation result of the point cloud model; Matching the first target point cloud with a template in a preset point cloud library to obtain a first matching result; Determine a second matching result of the first target point cloud with preset parameters within a preset threshold range of a preset semantic point cloud or an instance point cloud; The semantic point cloud with semantic prior is updated according to the first matching result and the second matching result.

2. The method for constructing a house model according to claim 1, characterized in that: The preset parameters include: Normal vector, color value and depth value.

3. The method for constructing a house model according to claim 1, characterized in that: Also includes: Determine a blank portion in the first target point cloud; The blank portion is filled according to the first matching result.

4. The method for constructing a house model according to claim 3, characterized in that: The filling of the blank portion comprises: Determine a target position in the depth map corresponding to the blank portion; The target position is depth filled.

5. The method for constructing a house model according to claim 1, characterized in that: Also includes: Determining a planar portion in the semantic point cloud with semantic priors; Determine a second target point cloud corresponding to the plane portion in the first target point cloud; Perform mesh deformation on the reconstructed mesh corresponding to the second target point cloud.

6. The method for constructing a house model according to claim 1, characterized in that: The posture information includes: The position information of the photographing device and the three-dimensional orientation information of the photographing device.

7. The method for constructing a house model according to any one of claims 1 to 6, characterized in that: Also includes: Determining a texture corresponding to a semantic segmentation result of the point cloud model; The house model is determined according to the texture and the semantic segmentation result of the point cloud model.

8. The method for constructing a house model according to claim 7, characterized in that: The first image data includes panoramic image data.

9. A device for constructing a house model, characterized in that: include: An acquisition unit, configured to acquire first image data in RGB color mode obtained by photographing a house by a photographing device at a preset point, a depth map corresponding to the first image data, and position information of the photographing device; A first determining unit, configured to determine a point cloud model according to the pose information, the first image data and the depth map; A segmentation unit, used for performing semantic segmentation on the first image data to obtain a semantic point cloud with semantic priors; A second determining unit, configured to determine a semantic segmentation result of the point cloud model according to the semantic point cloud with semantic priors and the point cloud model; A construction unit, configured to determine the house model according to a semantic segmentation result of the point cloud model; The second determination unit is further used to determine the semantic type corresponding to the semantic point cloud with semantic prior; according to the semantic type, the semantic point cloud with semantic prior is matched with the point cloud model to determine the first target point cloud corresponding to the semantic point cloud with semantic prior; the semantic type is associated with the first target point cloud to obtain the semantic segmentation result of the point cloud model; The second determination unit is further used to match the first target point cloud with a template in a preset point cloud library to obtain a first matching result; Determine a second matching result of the first target point cloud with preset parameters within a preset threshold range of a preset semantic point cloud or an instance point cloud; and update the semantic point cloud with semantic prior according to the first matching result and the second matching result.

10. A device for constructing a house model, characterized in that: include: A processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. An electronic device, characterized in that: include: The device for constructing a house model as claimed in claim 9 or 10; and / or The readable storage medium as claimed in claim 11.

Images