Household type vector generation method, device and storage medium

Abstract

The present disclosure relates to a method, device and storage medium for generating a vectorized floor plan. By obtaining multiple key frames of panoramic videos captured in a target floor plan, determining the room layout and door / window positions respectively based on each key frame of the panoramic video, determining the confidence levels of room wall lines and room areas based on the room layouts corresponding to the multiple key frames of the panoramic video, generating a vectorized floor plan layout based on the confidence levels of the room wall lines and room areas, and projecting the door / window positions onto the floor plan layout, a floor plan vector of the target floor plan is obtained. Thus, the automatic generation of the floor plan vector is achieved, reducing the technical barriers and labor costs for drawing the floor plan vector.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, and in particular, to a method, device, and storage medium for generating a housing unit vector. Background Art

[0002] In the related art, a housing unit vector in Computer Aided Design (CAD) format can be drawn by a designer, and then the housing unit vector in CAD format can be converted into a housing unit vector in a standard vector format. Alternatively, a housing unit vector in a standard vector format can also be directly drawn using a Building Information Modeling (BIM) standardization tool. However, both of the above two housing unit vector generation methods have relatively high technical barriers and high labor costs. Summary of the Invention

[0003] To solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a method, device, and storage medium for generating a housing unit vector.

[0004] In a first aspect, an embodiment of the present disclosure provides a method for generating a housing unit vector, the method including:

[0005] Obtaining a plurality of panoramic video key frames captured in a target housing unit;

[0006] Respectively determining a room layout and window and door positions based on each panoramic video key frame;

[0007] Determining a confidence level of a house wall line and a confidence level of a room area based on the room layouts corresponding to the plurality of panoramic video key frames;

[0008] Generating a vectorized housing unit layout based on the confidence level of the house wall line and the confidence level of the room area;

[0009] Projecting the window and door positions onto the housing unit layout to obtain a housing unit vector of the target housing unit.

[0010] Optionally, the respectively determining a room layout and window and door positions based on each panoramic video key frame includes:

[0011] Respectively determining a room layout based on each panoramic video key frame;

[0012] Adjusting the room layout based on the Manhattan assumption so that the adjusted room layout conforms to the Manhattan assumption.

[0013] Optionally, the determining a confidence level of a house wall line and a confidence level of a room area based on the room layouts corresponding to the plurality of panoramic video key frames includes:

[0014] Screen out room layouts that do not conform to physical reality from the room layouts corresponding to the multiple panoramic video key frames;

[0015] Based on the remaining room layouts, determine the confidence of the house wall lines and the confidence of the room areas.

[0016] Optionally, the screening out of room layouts that do not conform to physical reality from the room layouts corresponding to the multiple panoramic video key frames includes:

[0017] Rotate the room layout corresponding to the panoramic video key frame based on the rotation relationship between the camera pose corresponding to the panoramic video key frame and the camera pose at the reference position to obtain the rotated room layout;

[0018] In response to the rotated room layout not conforming to the Manhattan assumption, screen out the room layout;

[0019] And / or

[0020] Input the panoramic video key frame into a preset estimation model, and based on the estimation model, determine the ratio of the distance from the camera to the ground to the distance from the camera to the ceiling;

[0021] In response to the ratio exceeding a preset first numerical range, screen out the room layout corresponding to the panoramic video key frame;

[0022] And / or

[0023] Based on the SLAM algorithm, predict the height of the camera when shooting the panoramic video key frame;

[0024] In response to the height exceeding a preset second numerical range, screen out the room layout corresponding to the panoramic video key frame;

[0025] And / or

[0026] For any panoramic video key frame, calculate the house area of the panoramic video key frame and the panoramic video key frames shot at the previous moment and / or the next moment of the panoramic video key frame;

[0027] In response to the difference between the room area of the panoramic video key frame and the room areas of the panoramic video key frames at the previous moment and / or the next moment being greater than a preset threshold, screen out the room layout corresponding to the panoramic video key frame.

[0028] Optionally, the determining of the confidence of the house wall lines and the confidence of the room areas based on the room layouts corresponding to the multiple panoramic video key frames includes:

[0029] Align the room layouts corresponding to the multiple panoramic video key frames;

[0030] For any house wall line in any room layout, determine the number of room layouts aligned with the house wall line, and determine the number of the room layouts as the confidence level of the house wall line.

[0031] Optionally, the determining the confidence level of the house wall line and the confidence level of the room area based on the room layouts corresponding to the multiple panoramic video key frames includes:

[0032] Based on the room layouts corresponding to the multiple panoramic video key frames, determine the degree of overlap between the room layouts;

[0033] Determine the room layouts with a degree of overlap higher than a preset degree of overlap with each other as the room layouts in the same room area;

[0034] Determine the number of room layouts corresponding to the same room area as the confidence level of the room area.

[0035] Optionally, the generating a vectorized house type layout based on the confidence level of the house wall line and the confidence level of the room area includes:

[0036] Based on the house wall lines with a confidence level of the house wall line higher than a first confidence level and the room areas with a confidence level of the room area higher than a second confidence level, generate a room outline;

[0037] Project the room outline into a three-dimensional space so that the projection of the room outline in the three-dimensional space conforms to the Manhattan assumption.

[0038] Optionally, after generating the vectorized house type layout based on the confidence level of the house wall line and the confidence level of the room area, the method further includes:

[0039] In the vectorized house type layout, perform a separation process on the wall between adjacent rooms.

[0040] In a second aspect, an embodiment of the present disclosure provides a house type vector generation device, and the device includes:

[0041] An acquisition module, configured to acquire a plurality of panoramic video key frames captured in a target house type;

[0042] A first determination module, configured to respectively determine a room layout and a door and window position based on each panoramic video key frame;

[0043] A second determination module, configured to determine the confidence level of the house wall line and the confidence level of the room area based on the room layouts corresponding to the multiple panoramic video key frames;

[0044] A generation module, configured to generate a vectorized house type layout based on the confidence level of the house wall line and the confidence level of the room area;

[0045] A projection module, configured to project the positions of the doors and windows onto the housing layout to obtain the housing vector of the target housing type.

[0046] Optionally, a first determination module, configured to:

[0047] Determine the room layout respectively based on each panoramic video key frame;

[0048] Adjust the room layout based on the Manhattan assumption so that the adjusted room layout conforms to the Manhattan assumption.

[0049] Optionally, a second determination module, configured to:

[0050] Filter out the room layouts that do not conform to physical reality from the room layouts corresponding to the multiple panoramic video key frames;

[0051] Based on the remaining room layouts, determine the confidence level of the house wall lines and the confidence level of the room areas.

[0052] Optionally, a second determination module, configured to:

[0053] Rotate the room layout corresponding to the panoramic video key frame based on the rotation relationship between the camera pose corresponding to the panoramic video key frame and the camera pose of the reference position to obtain the rotated room layout;

[0054] In response to the rotated room layout not conforming to the Manhattan assumption, filter out the room layout;

[0055] And / or

[0056] Input the panoramic video key frame into a preset estimation model, and based on the estimation model, determine the ratio of the distance from the camera to the ground to the distance from the camera to the ceiling;

[0057] In response to the ratio exceeding a preset first numerical range, filter out the room layout corresponding to the panoramic video key frame;

[0058] And / or

[0059] Predict the height of the camera when shooting the panoramic video key frame based on the SLAM algorithm;

[0060] In response to the height exceeding a preset second numerical range, filter out the room layout corresponding to the panoramic video key frame;

[0061] And / or

[0062] For any panoramic video key frame, calculate the housing area of the panoramic video key frame and the panoramic video key frames captured at the previous moment and / or the next moment of the panoramic video key frame.

[0063] In response to the difference between the room area of the panoramic video key frame and the room areas of the panoramic video key frames at the previous moment and / or the next moment being greater than a preset threshold, the room layout corresponding to the panoramic video key frame is screened out.

[0064] Optionally, the second determination module is configured to:

[0065] Align the room layouts corresponding to the multiple panoramic video key frames;

[0066] For any house wall line on any room layout, determine the number of room layouts aligned to the house wall line, and determine the number of the room layouts as the confidence level of the house wall line.

[0067] Optionally, the second determination module is configured to:

[0068] Based on the room layouts corresponding to the multiple panoramic video key frames, determine the degree of coincidence between the room layouts;

[0069] Determine the room layouts with a coincidence degree higher than a preset coincidence degree with each other as the room layouts in the same room area;

[0070] Determine the number of room layouts corresponding to the same room area as the confidence level of the room area.

[0071] Optionally, the generation module is configured to:

[0072] Generate a room contour based on the house wall lines with a confidence level higher than a first confidence level and the room areas with a confidence level higher than a second confidence level;

[0073] Project the room contour into a three-dimensional space so that the projection of the room contour in the three-dimensional space conforms to the Manhattan assumption.

[0074] Optionally, the type vector generation device may further include:

[0075] A separation processing module for performing separation processing on the wall between adjacent rooms in the vectorized house type layout.

[0076] In a third aspect, an embodiment of the present disclosure provides a computer device, where the computer device includes:

[0077] A memory;

[0078] A processor; and

[0079] A computer program;

[0080] Among them, the computer program is stored in the memory and is configured to be executed by the processor to implement the method described in the first aspect.

[0081] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the method described in the first aspect.

[0082] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including computer program instructions, and when the computer program instructions are executed by a processor, the method described in the first aspect can be implemented.

[0083] The method, device, and storage medium for generating a house type vector provided by the embodiments of the present disclosure obtain multiple panoramic video key frames captured in a target house type, respectively determine the room layout and door / window positions based on each panoramic video key frame, determine the confidence of the room wall lines and the confidence of the room area based on the room layouts corresponding to the multiple panoramic video key frames, generate a vectorized house type layout based on the confidence of the room wall lines and the confidence of the room area, and project the door / window positions onto the house type layout to obtain the house type vector of the target house type. Thus, the automatic generation of the house type vector is realized, and the technical barrier and labor cost of drawing the house type vector are reduced. Description of the Drawings

[0084] The drawings here are incorporated into the description and form a part of this description, showing embodiments consistent with the present disclosure, and are used together with the description to explain the principles of the present disclosure.

[0085] To more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0086] Figure 1 is a flowchart of a method for generating a house type vector provided by an embodiment of the present disclosure;

[0087] Figure 2 is a flowchart of a method for determining the confidence of house wall lines provided by an embodiment of the present disclosure;

[0088] Figure 3a is a schematic diagram of a room layout provided by an embodiment of the present disclosure;

[0089] Figure 3b is Figure 3a a schematic diagram of the alignment method of the room layout shown in

[0090] Figure 4It is a schematic diagram of the entropy value obtained by plotting the entropy value with different numbers of room layouts and the value of s * ;

[0091] Figure 5 It is s * The visualization diagram of the room layout when the value is 0.4 and 0.8;

[0092] Figure 6 It is a schematic diagram of a method for extracting points on the room layout provided by an embodiment of the present disclosure;

[0093] Figure 7 It is a flowchart of a method for determining the confidence of a room area provided by an embodiment of the present disclosure;

[0094] Figure 8 It is a schematic diagram of a method for determining the coincidence degree of room layouts;

[0095] Figure 9a It is a schematic diagram of the room contour generated based on the confidence of the house wall line and the confidence of the room area;

[0096] Figure 9b It is for Figure 9a The schematic diagram of the room contour obtained after adjusting the room contour shown;

[0097] Figure 9c It is for Figure 9b The floor plan layout obtained after separating the walls of adjacent rooms in;

[0098] Figure 10 It is a flowchart of a method for determining the confidence of the house wall line and the confidence of the room area provided by an embodiment of the present disclosure;

[0099] Figure 11 It is a schematic diagram of the structure of a floor plan vector generation device provided by an embodiment of the present disclosure;

[0100] Figure 12 It is a schematic diagram of the structure of an embodiment of a computer device provided by an embodiment of the present disclosure. Detailed implementation manners

[0101] In order to be able to more clearly understand the above objects, features and advantages of the present disclosure, the solution of the present disclosure will be further described below. It should be noted that, without conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other.

[0102] In the following description, many specific details are set forth in order to fully understand the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only a part of the embodiments of the present disclosure, rather than all of the embodiments.

[0103] Referring to the background art, the methods for drawing house type vectors based on BIM standard tools provided by the related art, as well as the method of drawing house type vectors in CAD format and then converting them into house type vectors in standard vector format, have the following deficiencies:

[0104] 1. Using BIM standard tools involves the development of a complete set of applications. Even after development, promoting it to designers will also involve issues such as changing their work processes, with relatively high technical barriers and costs.

[0105] 2. Converting house type vectors in CAD format into house type vectors in standard vector format also has very high technical barriers and relatively poor accuracy.

[0106] 3. Manual drawing has the problem of high cost and no room for optimization.

[0107] In view of the above technical problems, the embodiments of the present disclosure provide a method, device, and storage medium for generating house type vectors. Only by obtaining multiple panoramic video key frames continuously captured in each room of the target house type can the house type vector of the target house type be automatically generated, with relatively low technical barriers and labor costs.

[0108] The following describes the solution provided by the embodiments of the present disclosure in conjunction with exemplary embodiments.

[0109] Exemplarily, Figure 1 is a flowchart of a method for generating house type vectors provided by the embodiments of the present disclosure. This method can be executed by a computer device exemplarily. The computer device can be any device with image processing capabilities and computing capabilities, such as terminal devices like mobile phones and computers, computers and servers equipped with image processing systems, etc., but is not limited to the devices listed here. As Figure 1 shown, in some embodiments, the method for generating house type vectors provided by the embodiments of the present disclosure may include steps 101 - 105.

[0110] Step 101: Obtain multiple panoramic video key frames captured in the target house type.

[0111] Among them, the target house type refers to the house type for which the house type vector needs to be generated.

[0112] Exemplarily, in some embodiments, the multiple panoramic video key frames referred to in the embodiments of the present disclosure may exemplarily include multiple panoramic video key frames captured in at least one room of the target house type. For the sake of easy understanding, in this embodiment, it can be exemplarily understood that the multiple panoramic video key frames include one or more panoramic video frames captured in each room of the target house type.

[0113] In some embodiments, the multiple panoramic video key frames referred to in the embodiments of the present disclosure can be extracted from the panoramic video stream captured by a panoramic camera in a target house type. The extraction method can be set as needed, and the embodiments of the present disclosure do not make specific limitations.

[0114] Step 102: Based on each panoramic video key frame, determine the room layout and the positions of doors and windows.

[0115] Exemplarily, in some embodiments, the embodiments of the present disclosure can identify the room layout and the positions of doors and windows in the panoramic video key frame through one or more preset models. For example, in one embodiment, the LGT-Net model can be used, but not limited to the LGT-Net model, to estimate the room layout in the panoramic video key frame. For another example, in another embodiment, a preset semantic segmentation model can be used to perform semantic segmentation on the panoramic video key frame to obtain the positions of areas such as walls, floors, ceilings, doors, and windows included in the panoramic video key frame. Further, according to the position of the floor in the panoramic video key frame, the floor contour is obtained, floor contour points are extracted from the floor contour, and the projection points of the floor contour points in the three-dimensional space are obtained by projecting the floor contour points into the three-dimensional space. Thus, the geometric area enclosed by the projection points of the floor contour points in the three-dimensional space can be used as the room layout. However, due to possible errors in the extraction of the floor contour points, the estimation of the room layout may be inaccurate. To improve the accuracy of the room layout estimation, the embodiments of the present disclosure can also adjust the room layout based on the Manhattan assumption (i.e., the floors, walls, and ceilings of indoor scenes are usually aligned in three mutually perpendicular main directions) so that the adjusted room layout conforms to the Manhattan assumption. For example, in one embodiment, the floor contour points can be projected into the three-dimensional space through the Manhattan assumption, and then the projection points of the floor contour points are clustered in the x-axis and y-axis directions of the floor coordinate system to obtain point clusters in the x-axis and y-axis directions. Multiple lines parallel to the x-axis and y-axis are obtained by performing linear fitting within each point cluster. Further, the lines with a distance greater than a preset distance from each other in the x-axis and y-axis directions are used as target lines, and thus the area enclosed by the target lines is the room layout that conforms to the Manhattan assumption. Of course, this is only an illustrative example here and not the only limitation.

[0116] Exemplarily, in some other embodiments, the embodiments of the present disclosure can also perform three-dimensional reconstruction on the panoramic video key frame through a preset three-dimensional reconstruction model to obtain the point cloud data of the room. Then, based on a preset ground point cloud extraction algorithm, the ground point cloud data is extracted from the point cloud data of the room, and then the ground contour is obtained based on the ground point cloud data, and thus the ground contour is used as the room layout.

[0117] Step 103: Determine the confidence of the house wall lines and the confidence of the room areas based on the room layouts corresponding to multiple panoramic video key frames.

[0118] Exemplarily, Figure 2 is a flowchart of a method for determining the confidence of house wall lines provided by an embodiment of the present disclosure. As Figure 2 shown, in some embodiments, the confidence of the house wall lines can be determined by the following method of steps 201 - 202.

[0119] Step 201: Align the room layouts corresponding to the multiple panoramic video key frames.

[0120] In some embodiments, the method of point - to - point matching can be used to align the room layouts corresponding to multiple panoramic video key frames. For example, Figure 3a is a schematic diagram of the room layout provided by an embodiment of the present disclosure. Figure 3b is Figure 3a a schematic diagram of the alignment method of the room layout shown in Figure 3a As shown, T O and T i are two room layouts, t i is the rotation and translation parameter between the two room layouts, s * is the scaling scale. As Figure 3b shown, based on t i and s * T O or T i can be scaled, rotated, and translated to achieve the alignment between T O and T i Among them, when determining the value of s * , first, entropy values can be plotted with different numbers ( Figure 3a in the example, the number of room layouts is exemplarily 2) of room layouts and the values of s * , and then the value corresponding to the minimum entropy value is used as the final value of s * . Among them, the method of plotting the entropy values of T O and T i with different values can refer to the prior art and will not be elaborated here. For example, Figure 4 is a schematic diagram of the entropy values obtained by plotting entropy values with different numbers of room layouts and the values of s * , Figure 5 is a visualization diagram of the room layout when the value of s * is 0.4 and 0.8. As Figure 4 and Figure 5 shown, in Figure 4 , the vertical coordinate is the entropy value (Entropy), the horizontal coordinate is s (scale), and s is s* The value of, where N is the number of room layouts. As Figure 4 shown, when the value of s * is 0.8, the entropy value is less than when the value of s * is 0.4. As Figure 5 shown, through the 2D density map, for the value of s * being 0.4 (i.e., (a) in Figure 5 ) and 0.8 (i.e., (b) in Figure 5 ), after visualizing the room layouts, it can be seen that Figure 5 in (b), when the value of s * is 0.8, the room alignment effect is the best. Therefore, the final value of s * is 0.8. Among them, the method of visualizing the room layouts for the values of s * being 0.4 and 0.8 can refer to the related technology and will not be elaborated here.

[0121] In some other embodiments, the same number of points can be extracted from each room layout according to the same extraction method, and then the room layouts can be aligned based on the points extracted from each room layout. For example, Figure 6 is a schematic diagram of a method for extracting points on a room layout provided by an embodiment of the present disclosure. In Figure 6 , point o is the geometric center of the room layout, and the coordinate axes x and y are the coordinate axes of the plane coordinate system where the room layout is located. As Figure 6 shown, in some embodiments, starting from the geometric center o, a ray can be made every time a preset angle Q is rotated, and the intersection points between the ray and the room layout can be obtained. By analogy, multiple points can be obtained from the room layout. Similarly, starting from the geometric centers of the room layouts corresponding to each panoramic video key frame, the same number of points can be obtained from the room layouts corresponding to each panoramic video key frame using the same extraction method (such as the same rotation direction and the same angle between the first ray made from the geometric center and the x-axis and y-axis). Further, in 3D space, as long as two contours have the same points, the Plücker analysis method can be used to align the two contours. Therefore, in an exemplary embodiment, the Plücker analysis method can be used to align the points extracted from the above room layouts to obtain the scale relationship, rotation angle, and translation relationship between the room layouts.

[0122] By extracting the same number of points from each room layout in the same extraction method and then aligning the room layouts based on the points extracted from each room layout, the problem of greedy matching can be avoided, the error of room layout alignment can be reduced, and the accuracy of room layout alignment can be improved.

[0123] Step 202: For any house wall line in a room layout, determine the number of room layouts aligned to this house wall line, and determine the number of room layouts aligned to this house wall line as the confidence level of this house wall line.

[0124] For example, assume that for any edge in room layout a, if there is an edge in room layouts b, c, and d that can be aligned to this edge in room layout a, then the confidence level of this edge as a house wall line is 3. Of course, this is only an example for illustration and not the only limitation.

[0125] By aligning the room layouts, for any house wall line in any room layout, determining the number of room layouts aligned to this house wall line as the confidence level of this house wall line can accurately obtain the confidence level information of each house wall line. Furthermore, based on the confidence level information of each house wall line, the house wall lines with high confidence levels can be determined.

[0126] Exemplarily, Figure 7 is a flowchart of a method for determining the confidence level of a room area provided by an embodiment of the present disclosure. As Figure 7 shown, in some embodiments, the confidence level of the room area can be determined based on the following method of steps 701 - 703.

[0127] Step 701: Based on the room layouts corresponding to multiple panoramic video key frames, determine the overlap degree between the room layouts.

[0128] Exemplarily, in some embodiments, the room layouts can be aligned first. The alignment method can refer to the method in step 201 above and will not be elaborated here.

[0129] Further, after completing the alignment of the room layouts, a feasible implementation is to extract the overlapping areas between the room layouts and calculate the overlap degree between the rooms based on the overlapping areas. The method for calculating the overlap degree of room areas can refer to the related art and will not be elaborated here. Another feasible way is to use the projection of the area within a preset range around the shooting position of the panoramic video key frame in the room layout as the high-confidence area, and determine the overlap degree between the high-confidence area of one room layout and the overlapping part between this room layout and another or multiple room layouts as the overlap degree between this room layout and the other or multiple room layouts. For example, Figure 8 is a schematic diagram of a method for determining the overlap degree of room layouts, where T i-1 、T i 、T i+1 are the high-confidence areas of three room layouts. As Figure 8 shown, T i-1 and T i+1 only partially fall within the overlapping part of the three room layouts (i.e.,Figure 8 the area enclosed by the middle broken line), T i all fall within the overlapping part of the three room layouts, then T i the overlapping degree between the corresponding room layout and the other two room layouts is 100%, while T i-1 and T i+1 the overlapping degree between the corresponding room layout and the other two room layouts is: T i-1 and T i+1 the ratio of the area falling within the overlapping part of the room layout to the overlapping part of the room layout. Of course Figure 8 is only an exemplary method for determining the overlapping degree of room layouts, rather than the only method.

[0130] Step 702: Determine the room area based on the room layouts with an overlapping degree higher than the preset overlapping degree among each other.

[0131] Taking Figure 8 as an example, where T i-1 , T i , T i+1 the overlapping degree among the corresponding three room layouts is higher than the preset overlapping degree, then it can be determined that the overlapping part of T i-1 , T i , T i+1 the corresponding three room layouts is the room area. Among them, the preset overlapping degree can be set as needed and does not have to be limited to a specific value.

[0132] Step 703: Determine the number of room layouts containing the same room area as the confidence level of this room area.

[0133] For example, in the example shown in Figure 8 3 room layouts contain the same room area (i.e., the overlapping part of the three room layouts), then the confidence level of the overlapping part of these three room layouts as the room area is 3. Of course, this is only for illustrative purposes and not the only limitation.

[0134] By calculating the overlapping degree between room layouts, determining the room area based on the room layouts with an overlapping degree higher than the preset overlapping degree among each other, and determining the number of room layouts containing the same room area as the confidence level of this room area, the confidence level of the room area can be accurately estimated, so as to accurately obtain the room area with a high confidence level.

[0135] Step 104: Generate a vectorized house type layout based on the confidence level of the house wall lines and the confidence level of the room area.

[0136] Exemplarily, in some embodiments, a room contour may be generated based on house wall lines with a confidence level higher than a first confidence level and room areas with a confidence level higher than a second confidence level. Then, the room contour is projected into three-dimensional space, and the room contour projected into three-dimensional space is adjusted through the Manhattan assumption so that the room contour projected into three-dimensional space conforms to the Manhattan assumption.

[0137] Further, in some embodiments, the wall between adjacent rooms may also be separated to make the house type layout conform to physical reality.

[0138] For example, Figure 9a is a schematic diagram of a room contour generated based on the confidence levels of house wall lines and room areas. As Figure 9a shown, the scales of the room contours obtained based on the confidence levels of house wall lines and room areas are not unified, and the wall lines of adjacent rooms overlap. Based on this, the Figure 9a room contours can be unified in scale, and the room contours after scale unification are adjusted through the Manhattan assumption, or a weighted undirected graph is constructed for the room contours after scale unification, and shortest path optimization algorithms such as Dijkstra and Improved Shortest Augmenting Path are used to adjust the room contours after scale unification to obtain the Figure 9b room contour shown. Further, by separating the walls between adjacent rooms in Figure 9b , the house type layout shown in Figure 9c can be obtained. Among them, the specific method of constructing a weighted undirected graph of the room contour and using the shortest path optimization algorithm to adjust the room contour after scale unification can refer to related technologies and will not be elaborated here.

[0139] By unifying the scale of the room contour, adjusting the room contour through the Manhattan assumption, and separating the walls between adjacent rooms, the house type vector can be made to conform more to physical reality and the generation effect of the house type vector can be improved.

[0140] Step 105: Project the door and window positions onto the house type layout to obtain the house type vector of the target house type.

[0141] In an embodiment of the present disclosure, by obtaining a plurality of panoramic video key frames captured in a target house type, respectively determining the room layout and the positions of doors and windows based on each panoramic video key frame, determining the confidence of room wall lines and the confidence of room areas based on the room layouts corresponding to the plurality of panoramic video key frames, generating a vectorized house type layout based on the confidence of room wall lines and the confidence of room areas, and projecting the positions of doors and windows onto the house type layout, a house type vector of the target house type is obtained. Thus, the automatic generation of the house type vector is realized, and the technical barrier and labor cost of drawing the house type vector are reduced.

[0142] Exemplarily, Figure 10 is a flowchart of a method for determining the confidence of house wall lines and the confidence of room areas provided by an embodiment of the present disclosure. As Figure 10 shown, in some embodiments, the confidence of house wall lines and the confidence of room areas can be determined through the following steps:

[0143] Step 1001: Screen out the room layouts that do not conform to physical reality from the room layouts corresponding to the plurality of panoramic video key frames.

[0144] Among them, there are various methods to screen out the room layouts that do not conform to physical reality from the room layouts corresponding to the plurality of panoramic video key frames. The following takes several screening methods as examples for illustration. However, it can be understood that the screening methods listed below are only exemplary methods rather than the only methods, and in some embodiments, one or more screening methods can be used simultaneously:

[0145] In one embodiment, the room layout corresponding to the panoramic video key frame can be rotated based on the rotation relationship between the camera pose corresponding to the panoramic video key frame and the camera pose of the reference position to obtain the rotated room layout; in response to the rotated room layout not conforming to the Manhattan assumption, the room layout is screened out. Among them, the camera pose corresponding to the panoramic video key frame can be estimated based on the Simultaneous Localization and Mapping (SLAM) algorithm, and the reference position can be the shooting position of any panoramic video key frame. For the sake of understanding, in the embodiment of the present disclosure, it can be exemplarily understood as the shooting position corresponding to the first panoramic video key frame. The camera pose of the reference position can be understood as the camera pose corresponding to the first panoramic video key frame.

[0146] In yet another embodiment, the panoramic video key frame can be input into a preset estimation model, and based on the estimation model, the ratio of the distance from the camera to the ground to the distance from the camera to the ceiling can be determined; in response to the ratio exceeding a preset first numerical range, the room layout corresponding to the panoramic video key frame is screened out. Among them, the estimation model can be trained by using the training method provided by related technologies, and the present disclosure does not specifically limit the training method of the estimation model.

[0147] In yet another embodiment, the height of the camera when shooting the panoramic video key frame can be predicted based on the SLAM algorithm; in response to the height exceeding a preset second numerical range, the room layout corresponding to the panoramic video key frame is screened out, where the second numerical range can be set as needed. For example, if the height of the camera predicted based on the SLAM algorithm is 1.1 meters, and the second numerical range is from 1.6 meters to 1.8 meters, then the room layout corresponding to the panoramic video key frame at this position is screened out.

[0148] In yet another embodiment, for any panoramic video key frame, the house area of the panoramic video key frame and the panoramic video key frames shot at the previous moment and / or the next moment of the panoramic video key frame can be calculated; in response to the difference between the room area of the panoramic video key frame and the room areas of the panoramic video key frames at the previous moment and / or the next moment being greater than a preset threshold, the room layout corresponding to the panoramic video key frame is screened out. Among them, when calculating the room area, the room layouts corresponding to each panoramic video key frame can be unified to the same scale first, and then the room area can be calculated.

[0149] Step 1002: Based on the remaining room layouts, determine the confidence of the house wall lines and the confidence of the room areas.

[0150] Among them, the method for determining the confidence of the house wall lines and the confidence of the room areas based on the remaining room layouts can refer to the relevant parts of the above embodiments, and will not be elaborated here.

[0151] By screening out the room layouts that do not conform to physical reality from the room layouts corresponding to multiple panoramic video key frames, and determining the confidence of the house wall lines and the confidence of the room areas based on the remaining room layouts, the embodiments of the present disclosure can improve the accuracy of determining the confidence of the house wall lines and the confidence of the room areas.

[0152] Figure 11 It is a schematic structural diagram of a house type vector generation device provided by the embodiments of the present disclosure. The house type vector generation device can be exemplarily understood as the computer device or a partial functional module in the computer device in the above method embodiments, such as Figure 11 As shown, in some embodiments, the house type vector generation device 1100 provided by the embodiments of the present disclosure may include:

[0153] An acquisition module 1101, configured to acquire multiple panoramic video key frames captured in a target house type;

[0154] A first determination module 1102, configured to determine a room layout and door / window positions respectively based on each panoramic video key frame;

[0155] A second determination module 1103, configured to determine the confidence level of the house wall lines and the confidence level of the room areas based on the room layouts corresponding to the multiple panoramic video key frames;

[0156] A generation module 1104, configured to generate a vectorized house type layout based on the confidence level of the house wall lines and the confidence level of the room areas;

[0157] A projection module 1105, configured to project the door / window positions onto the house type layout to obtain the house type vector of the target house type.

[0158] Optionally, the first determination module 1102 is configured to:

[0159] Determine the room layout respectively based on each panoramic video key frame;

[0160] Adjust the room layout based on the Manhattan assumption so that the adjusted room layout conforms to the Manhattan assumption.

[0161] Optionally, the second determination module 1103 is configured to:

[0162] Filter out the room layouts that do not conform to physical reality from the room layouts corresponding to the multiple panoramic video key frames;

[0163] Determine the confidence level of the house wall lines and the confidence level of the room areas based on the remaining room layouts.

[0164] Optionally, the second determination module 1103 is configured to:

[0165] Rotate the room layout corresponding to the panoramic video key frame based on the rotation relationship between the camera pose corresponding to the panoramic video key frame and the camera pose of the reference position to obtain the rotated room layout;

[0166] In response to the rotated room layout not conforming to the Manhattan assumption, filter out the room layout;

[0167] And / or

[0168] Input the panoramic video key frame into a preset estimation model, and determine the ratio of the distance from the camera to the ground to the distance from the camera to the ceiling based on the estimation model;

[0169] In response to the ratio exceeding a preset first numerical range, the room layout corresponding to the panoramic video key frame is screened out;

[0170] and / or

[0171] Based on the SLAM algorithm, predict the height of the camera when shooting the panoramic video key frame;

[0172] In response to the height exceeding a preset second numerical range, the room layout corresponding to the panoramic video key frame is screened out;

[0173] and / or

[0174] For any panoramic video key frame, calculate the house area of the panoramic video key frame and the panoramic video key frames shot at the previous moment and / or the next moment of the panoramic video key frame;

[0175] In response to the difference between the room area of the panoramic video key frame and the room areas of the panoramic video key frames at the previous moment and / or the next moment being greater than a preset threshold, the room layout corresponding to the panoramic video key frame is screened out.

[0176] Optionally, the second determination module 1103 is configured to:

[0177] Align the room layouts corresponding to the multiple panoramic video key frames;

[0178] For any house wall line on any room layout, determine the number of room layouts aligned to the house wall line, and determine the number of room layouts as the confidence level of the house wall line.

[0179] Optionally, the second determination module 1103 is configured to:

[0180] Based on the room layouts corresponding to the multiple panoramic video key frames, determine the degree of overlap between the room layouts;

[0181] Determine the room layouts with an overlap degree higher than a preset overlap degree between each other as the room layouts in the same room area;

[0182] Determine the number of room layouts corresponding to the same room area as the confidence level of the room area.

[0183] Optionally, the generation module 1104 is configured to:

[0184] Generate a room contour based on the house wall lines with a confidence level higher than a first confidence level and the room areas with a confidence level higher than a second confidence level;

[0185] Project the room contour into a three-dimensional space so that the projection of the room contour in the three-dimensional space conforms to the Manhattan assumption.

[0186] Optionally, the type vector generation device may further include:

[0187] A separation processing module for separating the walls between adjacent rooms in the vectorized housing layout.

[0188] The device provided by the embodiments of the present disclosure can execute the methods of any of the above method embodiments, and the execution manners and beneficial effects are similar and will not be elaborated here.

[0189] It should also be noted that the division of modules in the above camera calibration device in the embodiments of the present disclosure is illustrative, only a logical function division, and there may be other division manners in actual implementation. In addition, in each embodiment of the present application, each functional module may be integrated in one processing module, or each module may exist physically alone, or two or more modules may be integrated in one module. The above integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0190] If the above integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may be stored in a processor-readable storage medium. Based on such an understanding, the technical solution of the present disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in the embodiments of the present disclosure.

[0191] Figure 12 It is a schematic structural diagram of a computer device embodiment provided by the embodiments of the present disclosure. As Figure 12 shown, the computer device includes a memory 121 and a processor 122.

[0192] The memory 121 is used to store programs. In addition to the above programs, the memory 121 may also be configured to store various other data to support operations on the computer device. Examples of these data include instructions for any application program or method for operating on the computer device, contact member data, phone book member data, messages, pictures, videos, etc.

[0193] The memory 121 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0194] The processor 122 is coupled to the memory 121 and executes the programs stored in the memory 121 for:

[0195] Obtaining a plurality of panoramic video key frames captured in the target housing type;

[0196] Based on each panoramic video key frame respectively, determining the room layout and the positions of doors and windows;

[0197] Based on the room layouts corresponding to the plurality of panoramic video key frames, determining the confidence of the house wall lines and the confidence of the room areas;

[0198] Based on the confidence of the house wall lines and the confidence of the room areas, generating a vectorized housing type layout;

[0199] Projecting the positions of the doors and windows onto the housing type layout to obtain the housing type vector of the target housing type.

[0200] Furthermore, as Figure 12 shown, the computer device may further include: other components such as a communication component 123, a power supply component 124, an audio component 125, a display 126, etc. Figure 12 Only some components are schematically shown herein, and it does not mean that the computer device only includes Figure 12 the components shown.

[0201] The communication component 123 is configured to facilitate communication between the computer device and other devices in a wired or wireless manner. The computer device can access a wireless network based on communication standards, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 123 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 123 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0202] The power supply component 124 provides power for various components of the computer device. The power supply component 124 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the computer device.

[0203] The audio component 125 is configured to output and / or input audio signals. For example, the audio component 125 includes a microphone (MIC), which is configured to receive external audio signals when the computer device is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 121 or transmitted via the communication component 123. In some embodiments, the audio component 125 further includes a speaker for outputting audio signals.

[0204] The display 126 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operations.

[0205] In addition, embodiments of the present disclosure further provide a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the method described in any of the above method embodiments.

[0206] In embodiments of the present disclosure, the above computer-readable storage medium may be any available medium or data storage device accessible by a processor, including but not limited to magnetic memories (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical memories (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor memories (such as ROMs, EPROMs, EEPROMs, non-volatile memories (NAND FLASH), solid-state drives (SSD)).

[0207] Those skilled in the art should understand that embodiments of the present disclosure may be provided as a method, a system, or a computer program product. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories and optical memories, etc.) containing computer-usable program code.

[0208] Embodiments of the present disclosure provide a computer program product, including computer program instructions, and when the computer program instructions are executed by a processor, they can implement the method described in any of the above method embodiments.

[0209] It should be noted that, in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0210] The above are only specific embodiments of the present disclosure, enabling those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a housing type vector, characterized in that: The method comprises: Acquire multiple panoramic video key frames shot in the target apartment; Determine the room layout and door and window locations based on each panoramic video key frame; Determining, based on the room layouts corresponding to the multiple panoramic video key frames, the confidence of the house wall line and the confidence of the room area on the room layout; Generate a vectorized apartment layout based on the confidence of the house wall line and the confidence of the room area; Projecting the door and window positions onto the apartment layout to obtain an apartment vector of the target apartment; The determining the confidence of the house wall line and the confidence of the room area based on the room layout corresponding to the multiple panoramic video key frames includes: Aligning room layouts corresponding to the multiple panoramic video key frames; For any house wall line on any room layout, determine the number of room layouts aligned to the house wall line, and determine the number of room layouts as the confidence of the house wall line; Determining the overlap between the room layouts based on the room layouts corresponding to the multiple panoramic video key frames; Determine room areas based on room layouts whose overlap is higher than a preset overlap; The number of room layouts containing the room region is determined as the confidence level of the room region.

2. The method according to claim 1, characterized in that The determining of the room layout and the door and window positions based on each panoramic video key frame respectively includes: Determine the room layout based on each panoramic video key frame respectively; The room layout is adjusted based on the Manhattan hypothesis, so that the adjusted room layout conforms to the Manhattan hypothesis.

3. The method according to claim 1, characterized in that The determining the confidence of the house wall line and the confidence of the room area based on the room layout corresponding to the multiple panoramic video key frames includes: Screening out room layouts that do not conform to physical reality from room layouts corresponding to the multiple panoramic video key frames; Based on the remaining room layout, the confidence of the house wall lines and the confidence of the room areas are determined.

4. The method according to claim 3, characterized in that The step of screening out room layouts that do not conform to physical reality from the room layouts corresponding to the multiple panoramic video key frames includes: Based on the rotation relationship between the camera pose corresponding to the panoramic video key frame and the camera pose at the reference position, the room layout corresponding to the panoramic video key frame is rotated to obtain a rotated room layout; In response to the rotated room layout not meeting the Manhattan assumption, screening out the room layout; and / or Inputting the panoramic video key frame into a preset estimation model, and determining the ratio of the distance from the camera to the ground and the distance from the camera to the ceiling based on the estimation model; In response to the ratio exceeding a preset first value range, filtering out the room layout corresponding to the panoramic video key frame; and / or Predicting the height of the camera when shooting the key frame of the panoramic video based on the SLAM algorithm; In response to the height exceeding a preset second numerical range, filtering out the room layout corresponding to the panoramic video key frame; and / or For any panoramic video key frame, calculating the house area of ​​the panoramic video key frame and the panoramic video key frames shot at a moment before and / or after the panoramic video key frame; In response to the difference between the room area of ​​the panoramic video key frame and the room area of ​​the panoramic video key frame at a previous moment and / or a next moment being greater than a preset threshold, the room layout corresponding to the panoramic video key frame is screened out.

5. The method according to claim 1, characterized in that The generating a vectorized apartment layout based on the confidence of the house wall line and the confidence of the room area includes: Generate a room outline based on the house wall line with a confidence level higher than the first confidence level and the room area with a confidence level higher than the second confidence level; The room outline is projected into a three-dimensional space, so that the projection of the room outline in the three-dimensional space conforms to the Manhattan hypothesis.

6. The method according to claim 1, characterized in that After generating the vectorized apartment layout based on the confidence of the house wall line and the confidence of the room area, the method further includes: In the vectorized apartment layout, walls between adjacent rooms are separated.

7. A computer device, characterized in that: The computer device comprises: Memory; Processor; and Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer program product, comprising computer program instructions, wherein when the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images