House type vector generation method and device and storage medium

Abstract

The invention relates to a house type vector generation method and device and a storage medium, and the method comprises the steps: obtaining a plurality of panoramic video key frames obtained through shooting in a target house type, determining a room layout and door and window positions based on each panoramic video key frame, and obtaining a plurality of house type vectors based on the room layouts corresponding to the plurality of panoramic video key frames; and determining a room wall line confidence coefficient and a room area confidence coefficient, generating a vectorized house type layout based on the room wall line confidence coefficient and the room area confidence coefficient, and projecting the door and window positions to the house type layout to obtain a house type vector of the target house type. Therefore, automatic generation of the house type vector is realized, and technical barriers and labor cost of house type vector drawing are reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a method, device and storage medium for generating a house type vector. Background Art

[0002] In the related technology, a designer can draw a house type vector in a Computer Aided Design (CAD) format, and then convert the house type vector in the CAD format into a house type vector in a standard vector format. Alternatively, the designer can directly use the Building Information Modeling (BIM) standardized tool to draw a house type vector in a standard vector format. However, both of the above two methods of generating house type vectors have high technical barriers and high labor costs. Summary of the invention

[0003] In order to solve the above technical problem or at least partially solve the above technical problem, the present disclosure provides a method, device and storage medium for generating a house type vector.

[0004] In a first aspect, an embodiment of the present disclosure provides a method for generating a housing type vector, the method comprising: Acquire multiple panoramic video key frames shot in the target apartment; Determine the room layout and door and window locations based on each panoramic video key frame; Determining a house wall line confidence and a room area confidence based on the room layouts corresponding to the multiple panoramic video key frames; Generate a vectorized apartment layout based on the house wall line confidence and the room area confidence; The door and window positions are projected onto the apartment layout to obtain an apartment vector of the target apartment.

[0005] Optionally, determining the room layout and door and window positions based on each panoramic video key frame respectively includes: Determine the room layout based on each panoramic video key frame respectively; The room layout is adjusted based on the Manhattan hypothesis, so that the adjusted room layout conforms to the Manhattan hypothesis.

[0006] Optionally, the determining the house wall line confidence and the room area confidence based on the room layouts corresponding to the multiple panoramic video key frames includes: Screening out room layouts that do not conform to physical reality from room layouts corresponding to the multiple panoramic video key frames; Based on the remaining room layout, the house wall line confidence and room area confidence are determined.

[0007] Optionally, screening out room layouts that do not conform to physical reality from the room layouts corresponding to the multiple panoramic video key frames includes: Based on the rotation relationship between the camera pose corresponding to the panoramic video key frame and the camera pose at the reference position, the room layout corresponding to the panoramic video key frame is rotated to obtain a rotated room layout; In response to the rotated room layout not meeting the Manhattan assumption, screening out the room layout; and / or Inputting the panoramic video key frame into a preset estimation model, and determining the ratio of the distance from the camera to the ground and the distance from the camera to the ceiling based on the estimation model; In response to the ratio exceeding a preset first value range, filtering out the room layout corresponding to the panoramic video key frame; and / or Predicting the height of the camera when shooting the key frame of the panoramic video based on the SLAM algorithm; In response to the height exceeding a preset second numerical range, filtering out the room layout corresponding to the panoramic video key frame; and / or For any panoramic video key frame, calculating the house area of ​​the panoramic video key frame and the panoramic video key frames shot at a moment before and / or after the panoramic video key frame; In response to the difference between the room area of ​​the panoramic video key frame and the room area of ​​the panoramic video key frame at a previous moment and / or a next moment being greater than a preset threshold, the room layout corresponding to the panoramic video key frame is screened out.

[0008] Optionally, the determining the house wall line confidence and the room area confidence based on the room layouts corresponding to the multiple panoramic video key frames includes: Aligning room layouts corresponding to the multiple panoramic video key frames; For any house wall line on any room layout, the number of room layouts aligned to the house wall line is determined, and the number of the room layouts is determined as the confidence of the house wall line.

[0009] Optionally, the determining the house wall line confidence and the room area confidence based on the room layouts corresponding to the multiple panoramic video key frames includes: Determining the overlap between the room layouts based on the room layouts corresponding to the multiple panoramic video key frames; Determine room layouts whose overlap degree is higher than a preset overlap degree as room layouts of the same room area; The number of room layouts corresponding to the same room region is determined as the confidence of the room region.

[0010] Optionally, generating a vectorized apartment layout based on the house wall line confidence and the room area confidence includes: Generate a room outline based on the house wall line whose confidence level is higher than the first confidence level and the room area whose confidence level is higher than the second confidence level; The room outline is projected into a three-dimensional space, so that the projection of the room outline in the three-dimensional space conforms to the Manhattan hypothesis.

[0011] Optionally, after generating a vectorized apartment layout based on the house wall line confidence and the room area confidence, the method further includes: In the vectorized apartment layout, walls between adjacent rooms are separated.

[0012] In a second aspect, an embodiment of the present disclosure provides a device for generating a house type vector, the device comprising: An acquisition module, used to acquire multiple panoramic video key frames shot in a target apartment; A first determination module is used to determine the room layout and the door and window positions based on each panoramic video key frame respectively; A second determination module is used to determine the house wall line confidence and the room area confidence based on the room layout corresponding to the multiple panoramic video key frames; A generating module, used for generating a vectorized apartment layout based on the house wall line confidence and the room area confidence; A projection module is used to project the door and window positions onto the apartment layout to obtain an apartment vector of the target apartment.

[0013] Optionally, the first determining module is used to: Determine the room layout based on each panoramic video key frame respectively; The room layout is adjusted based on the Manhattan hypothesis, so that the adjusted room layout conforms to the Manhattan hypothesis.

[0014] Optionally, the second determining module is used to: Screening out room layouts that do not conform to physical reality from room layouts corresponding to the multiple panoramic video key frames; Based on the remaining room layout, the house wall line confidence and room area confidence are determined.

[0015] Optionally, the second determining module is used to: Based on the rotation relationship between the camera pose corresponding to the panoramic video key frame and the camera pose at the reference position, the room layout corresponding to the panoramic video key frame is rotated to obtain a rotated room layout; In response to the rotated room layout not meeting the Manhattan assumption, screening out the room layout; and / or Inputting the panoramic video key frame into a preset estimation model, and determining the ratio of the distance from the camera to the ground and the distance from the camera to the ceiling based on the estimation model; In response to the ratio exceeding a preset first value range, filtering out the room layout corresponding to the panoramic video key frame; and / or Predicting the height of the camera when shooting the key frame of the panoramic video based on the SLAM algorithm; In response to the height exceeding a preset second numerical range, filtering out the room layout corresponding to the panoramic video key frame; and / or For any panoramic video key frame, calculating the house area of ​​the panoramic video key frame and the panoramic video key frames shot at a moment before and / or after the panoramic video key frame; In response to the difference between the room area of ​​the panoramic video key frame and the room area of ​​the panoramic video key frame at a previous moment and / or a next moment being greater than a preset threshold, the room layout corresponding to the panoramic video key frame is screened out.

[0016] Optionally, the second determining module is used to: Aligning room layouts corresponding to the multiple panoramic video key frames; For any house wall line on any room layout, the number of room layouts aligned to the house wall line is determined, and the number of the room layouts is determined as the confidence of the house wall line.

[0017] Optionally, the second determining module is used to: Determining the overlap between the room layouts based on the room layouts corresponding to the multiple panoramic video key frames; Determine room layouts whose overlap degree is higher than a preset overlap degree as room layouts of the same room area; The number of room layouts corresponding to the same room region is determined as the confidence of the room region.

[0018] Optionally, generate a module for: Generate a room outline based on the house wall line whose confidence level is higher than the first confidence level and the room area whose confidence level is higher than the second confidence level; The room outline is projected into a three-dimensional space, so that the projection of the room outline in the three-dimensional space conforms to the Manhattan hypothesis.

[0019] Optionally, the model vector generating device may further include: The separation processing module is used to separate the walls between adjacent rooms in the vectorized apartment layout.

[0020] In a third aspect, an embodiment of the present disclosure provides a computer device, the computer device comprising: Memory; Processor; and Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect.

[0021] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method described in the first aspect.

[0022] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including computer program instructions, which can implement the method described in the first aspect when executed by a processor.

[0023] The apartment type vector generation method, device and storage medium provided by the embodiments of the present disclosure obtain multiple panoramic video key frames shot in the target apartment type, determine the room layout and door and window positions based on each panoramic video key frame, determine the room wall line confidence and room area confidence based on the room layout corresponding to the multiple panoramic video key frames, generate a vectorized apartment type layout based on the room wall line confidence and room area confidence, and project the door and window positions onto the apartment type layout to obtain the apartment type vector of the target apartment type. This realizes the automatic generation of apartment type vectors and reduces the technical barriers and labor costs of apartment type vector drawing. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0026] Figure 1 is a flow chart of a method for generating a house type vector provided by an embodiment of the present disclosure; Figure 2 is a flow chart of a method for determining the confidence level of a house wall line provided by an embodiment of the present disclosure; Figure 3ais a schematic diagram of a room layout provided by an embodiment of the present disclosure; Figure 3b yes Figure 3a Schematic diagram of the alignment method for the room layout shown in; Figure 4 It is based on different number of room layouts and s * The schematic diagram of the entropy value obtained by plotting the entropy value by taking the value of ; Figure 5 Yes * Visualization of room layout when the value of is 0.4 and 0.8; Figure 6 is a schematic diagram of a method for extracting points on a room layout provided by an embodiment of the present disclosure; Figure 7 is a flow chart of a method for determining room area confidence provided by an embodiment of the present disclosure; Figure 8 It is a schematic diagram of a method for determining the overlap of room layouts; Figure 9a It is a schematic diagram of the room outline generated based on the house wall line confidence and room area confidence; Figure 9b Yes Figure 9a A schematic diagram of a room outline obtained after adjusting the room outline shown; Fig.9c Yes Figure 9b A schematic diagram of the apartment layout obtained after separating the walls of adjacent rooms; Fig.10 It is a flow chart of a method for determining a house wall line confidence and a room area confidence provided by an embodiment of the present disclosure; Fig.11 It is a structural schematic diagram of a housing type vector generating device provided by an embodiment of the present disclosure; Fig.12 A schematic diagram of the structure of a computer device embodiment provided in the present disclosure. DETAILED DESCRIPTION

[0027] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0028] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0029] Referring to the background technology, the method of drawing the apartment type vector based on the BIM standard tool provided in the related technology and the method of drawing the apartment type vector in the CAD format and then converting the apartment type vector in the CAD format into the apartment type vector in the standard vector format have the following shortcomings: 1. The use of BIM standard tools involves the development of a complete set of applications. Even after the development is completed, promoting it to designers will involve changing their workflow and other issues, and the technical barriers and costs are high.

[0030] 2. There are also very high technical barriers to converting apartment layout vectors in CAD format into apartment layout vectors in standard vector format, and the accuracy is poor.

[0031] 3. Manual drawing has the problem of high cost and no room for optimization.

[0032] In response to the above technical problems, the embodiments of the present disclosure provide a method, device and storage medium for generating an apartment type vector. It only requires obtaining multiple panoramic video key frames continuously shot in each room of the target apartment type to automatically generate the apartment type vector of the target apartment type, with low technical barriers and labor costs.

[0033] The solution provided by the embodiment of the present disclosure is described below in conjunction with exemplary embodiments.

[0034] For example, Figure 1 is a flow chart of a method for generating a house type vector provided by an embodiment of the present disclosure. The method can be exemplarily executed by a computer device, which can be any device with image processing and computing capabilities, such as a mobile phone, a computer, a server, etc., equipped with an image processing system, but is not limited to the devices listed here. Figure 1 As shown, in some implementations, the method for generating a housing type vector provided by an embodiment of the present disclosure may include steps 101 to 105.

[0035] Step 101: Acquire multiple panoramic video key frames shot in a target apartment.

[0036] The target housing type refers to the housing type for which a housing type vector needs to be generated.

[0037] For example, in some implementations, the multiple panoramic video key frames referred to in the embodiments of the present disclosure may exemplarily include multiple panoramic video key frames captured in at least one room of the target apartment. For ease of understanding, in this embodiment, it can be exemplarily understood that the multiple panoramic video key frames include one or more panoramic video frames captured in each room of the target apartment.

[0038] In some implementations, the multiple panoramic video key frames referred to in the embodiments of the present disclosure may be extracted from a panoramic video stream captured by a panoramic camera in the target apartment. The extraction method may be set as required, and the embodiments of the present disclosure do not specifically limit it.

[0039] Step 102: Determine the room layout and door and window positions based on each panoramic video key frame.

[0040] For example, in some embodiments, the disclosed embodiment can obtain the room layout and door and window positions in the panoramic video key frame through one or more preset model recognitions. For example, in one embodiment, the room layout in the panoramic video key frame can be estimated through the LGT-Net model, but not limited to the LGT-Net model. For another example, in another embodiment, the panoramic video key frame can be semantically segmented through a preset semantic segmentation model to obtain the positions of the walls, ground, ceiling, doors, windows and other areas contained in the panoramic video key frame. Further, according to the position of the ground in the panoramic video key frame, the ground contour is obtained, and the ground contour points are extracted from the ground contour, and the projection points of the ground contour points in the three-dimensional space are obtained by projecting the ground contour points into the three-dimensional space. Thus, the geometric area surrounded by the projection points of the ground contour points in the three-dimensional space can be used as the room layout. However, since there may be errors in the extraction of the ground contour points, the room layout estimation is inaccurate. In order to improve the accuracy of the room layout estimation, the disclosed embodiment can also adjust the room layout based on the Manhattan hypothesis (that is, the ground, wall, and ceiling of the indoor scene are usually aligned in three mutually perpendicular main directions) so that the adjusted room layout conforms to the Manhattan hypothesis. For example, in one embodiment, the ground contour points can be projected into three-dimensional space by the Manhattan hypothesis, and then the projection points of the ground contour points are clustered in the x-axis and y-axis directions of the ground coordinate system, respectively, to obtain point clusters in the x-axis direction and the y-axis direction, and multiple straight lines parallel to the x-axis and the y-axis are obtained by performing straight line fitting in each point cluster. Further, the straight lines in the x-axis and y-axis directions whose distances are greater than a preset distance are taken as target straight lines, so that the area enclosed by the target straight lines is the room layout that conforms to the Manhattan hypothesis. Of course, this is only an example, not the only limitation.

[0041] For example, in some other implementations, the disclosed embodiments can also perform 3D reconstruction on the panoramic video key frames through a preset 3D reconstruction model to obtain point cloud data of the room. Then, based on a preset ground point cloud extraction algorithm, ground point cloud data is extracted from the point cloud data of the room, and then the ground contour is obtained based on the ground point cloud data, so as to use the ground contour as the room layout.

[0042] Step 103: Determine the house wall line confidence and the room area confidence based on the room layouts corresponding to the multiple panoramic video key frames.

[0043] For example, Figure 2 is a flow chart of a method for determining the confidence level of a house wall line provided by an embodiment of the present disclosure, such as Figure 2 As shown, in some implementations, the house wall line confidence can be determined by the following method of steps 201-202.

[0044] Step 201: Align the room layouts corresponding to the multiple panoramic video key frames.

[0045] In some implementations, a point pair matching method may be used to align the room layouts corresponding to multiple panoramic video key frames. Figure 3a It is a schematic diagram of the room layout provided by the embodiment of the present disclosure. Figure 3b yes Figure 3a Schematic diagram of the alignment method for the room layout shown in . Figure 3a As shown, T O and T i It is a two-room layout. i is the rotation and translation parameter between the two room layouts, s * is the scaling factor, such as Figure 3b As shown, it can be based on t i and * To T O or T i Scaling, rotation, and translation to achieve T O and T i Among them, in determining s * When choosing the value of , you can first use different quantities ( Figure 3a In the example, the number of room layouts is exemplarily 2) and the number of room layouts is s * The entropy value is plotted, and the value with the minimum entropy value is taken as s * The final value of . Among them, T is plotted with different values O and T i The method of calculating the entropy value can refer to the prior art and will not be described here. For example, Figure 4 It is based on different number of room layouts and s * The schematic diagram of the entropy value obtained by plotting the entropy value is shown in Figure 2. Figure 5 Yes * Visualization of room layout when the value of is 0.4 and 0.8. Figure 4 and Figure 5 As shown, in Figure 4 In the figure, the ordinate is entropy, the abscissa is s (scale), and s is s* The value of , N is the number of room layouts. Figure 4 As shown, in s * The entropy value when the value is 0.8 is less than s * The entropy value of is 0.4, such as Figure 5 As shown, through the 2D density map s * The value of is 0.4 (i.e. Figure 5 (a)) and 0.8 (i.e. Figure 5 After visualizing the room layout in (b), it can be seen that Figure 5 In (b) when s * The room alignment effect is best when the value of is 0.8. * The final value of is 0.8. * The method for visualizing the room layout when the value of is 0.4 and 0.8 can be found in the relevant technology, which will not be repeated here.

[0046] In other embodiments, the same number of points may be extracted from each room layout in the same extraction manner, and then the room layouts may be aligned based on the points extracted from each room layout. Figure 6 is a schematic diagram of a method for extracting points on a room layout provided by an embodiment of the present disclosure, Figure 6 The midpoint o is the geometric center of the room layout, and the coordinate axes x and y are the coordinate axes of the plane coordinate system where the room layout is located. Figure 6 As shown, in some embodiments, starting from the geometric center o, a ray can be made each time a preset angle Q is rotated to obtain the intersection between the ray and the room layout, and so on, multiple points can be obtained from the room layout. Similarly, starting from the geometric center of the room layout corresponding to each panoramic video key frame, the same extraction method (such as the same rotation direction, the first ray made from the geometric center and the same angle between the x-axis and the y-axis) can be used to obtain the same number of points from the room layout corresponding to each panoramic video key frame. Furthermore, in 3D space, as long as two contours have the same points, the Prucker analysis method can be used to align the two contours. Therefore, in an exemplary embodiment, the points extracted from the above-mentioned room layout can be aligned by the Prucker analysis method to obtain the scale relationship, rotation angle and translation relationship between the room layouts.

[0047] By extracting the same number of points from each room layout in the same extraction method and then aligning the room layouts based on the points extracted from each room layout, the problem of greedy matching can be avoided, the error of room layout alignment can be reduced, and the accuracy of room layout alignment can be improved.

[0048] Step 202: for any house wall line on any room layout, determine the number of room layouts aligned to the house wall line, and determine the number of room layouts aligned to the house wall line as the confidence of the house wall line.

[0049] For example, assuming that for any edge on room layout a, assuming that an edge on room layout b, c, d can be aligned to the edge of room layout a, then the confidence that the edge is the wall line of the house is 3. Of course, this is only an example and not a sole limitation.

[0050] By aligning the room layouts, for any house wall line on any room layout, the number of room layouts aligned to the house wall line is determined as the confidence of the house wall line, so that the confidence information of each house wall line can be accurately obtained, and then the house wall line with high confidence can be determined according to the confidence information of each house wall line.

[0051] For example, Figure 7 is a flow chart of a method for determining room area confidence provided by an embodiment of the present disclosure, such as Figure 7 As shown, in some implementations, the room area confidence may be determined based on the following method of steps 701-703.

[0052] Step 701: Determine the overlap between room layouts based on room layouts corresponding to multiple panoramic video key frames.

[0053] For example, in some implementations, the layouts of the rooms may be aligned first. The alignment method may refer to the method in step 201 above, which will not be described in detail here.

[0054] Furthermore, after completing the alignment of the room layouts, a feasible implementation method is to extract the overlapping areas between the room layouts, and calculate the overlap between the rooms based on the overlapping areas. The room area overlap calculation method can refer to the relevant technology and will not be repeated here. Another feasible way is to use the projection of the area within a preset range around the panoramic video key frame shooting position in the room layout as a high confidence area, and determine the overlap between the high confidence area of ​​a room layout and the overlapping part of the room layout and another one or more room layouts as the overlap between the room layout and the other one or more room layouts. For example, Figure 8 is a schematic diagram of a method for determining the overlap of room layouts, where T i-1 , T i , T i+1 are the high confidence regions of the three room layouts, such as Figure 8 As shown, T i-1 and T i+1 Only some of them fall in the overlapping parts of the three room layouts (i.e. Figure 8The area enclosed by the mid-fold line), T i All of them fall in the overlapping part of the three room layouts, then T i The overlap between the corresponding room layout and the other two room layouts is 100%, while T i-1 and T i+1 The overlap between the corresponding room layout and the other two room layouts is: T i-1 and T i+1 The ratio of the area that falls within the overlapped portion of the room layout to the overlapped portion of the room layout. Figure 8 This is only an exemplary method of determining the room layout overlap, and is not the only method.

[0055] Step 702: Determine room areas based on room layouts whose overlap is higher than a preset overlap.

[0056] by Figure 8 For example, T i-1 , T i , T i+1 The overlap between the corresponding three room layouts is higher than the preset overlap, then it can be determined that T i-1 , T i , T i+1 The overlapping parts of the corresponding three room layouts are the room areas. The preset overlap degree can be set as needed, and need not be limited to a specific value.

[0057] Step 703: Determine the number of room layouts that include the same room area as the confidence level of the room area.

[0058] For example, in Figure 8 In the example shown, the three room layouts contain the same room area (ie, the overlapped part of the three room layouts), and the confidence that the overlapped part of the three room layouts is the room area is 3. Of course, this is only an example and not a sole limitation.

[0059] By calculating the overlap between room layouts, the room area is determined based on the room layouts whose overlap is higher than the preset overlap, and the number of room layouts that include the same room area is determined as the confidence of the room area. The confidence of the room area can be accurately estimated, thereby accurately obtaining the room area with high confidence.

[0060] Step 104: Generate a vectorized apartment layout based on the house wall line confidence and the room area confidence.

[0061] For example, in some embodiments, a room outline can be generated based on house wall lines having a confidence level higher than a first confidence level and a room area having a confidence level higher than a second confidence level, and then the room outline can be projected into a three-dimensional space. The room outline projected into the three-dimensional space can be adjusted using the Manhattan hypothesis so that the room outline projected into the three-dimensional space conforms to the Manhattan hypothesis.

[0062] Furthermore, in some implementations, the walls between adjacent rooms may be separated so that the apartment layout conforms to physical reality.

[0063] for example, Figure 9a This is a schematic diagram of the room outline generated based on the house wall line confidence and room area confidence. Figure 9a As shown in Figure 1, the scales of the room outlines obtained based on the house wall line confidence and the room area confidence are not uniform, and the wall lines of adjacent rooms overlap. Based on this, we can Figure 9a The room outlines in the image are scaled uniformly, and the scaled room outlines are adjusted by the Manhattan hypothesis, or a weighted undirected graph is constructed for the scaled room outlines, and a shortest path optimization algorithm such as Dijkstra and Improved Shortest Augmenting Path is used to adjust the scaled room outlines to obtain Figure 9b The room outline shown, further, by Figure 9b By separating the walls of adjacent rooms, we can get Fig.9c The specific method of constructing a weighted undirected graph of room outlines and using the shortest path optimization algorithm to adjust the room outlines after the scale is unified can be found in the relevant technology, which will not be repeated here.

[0064] By unifying the scale of room outlines, adjusting room outlines through the Manhattan hypothesis, and separating the walls of adjacent rooms, the apartment vector can be made more consistent with physical reality and the generation effect of the apartment vector can be improved.

[0065] Step 105: Project the door and window positions onto the apartment layout to obtain an apartment vector of the target apartment.

[0066] The disclosed embodiment obtains multiple panoramic video key frames shot in the target apartment, determines the room layout and door and window positions based on each panoramic video key frame, determines the room wall line confidence and room area confidence based on the room layout corresponding to the multiple panoramic video key frames, generates a vectorized apartment layout based on the room wall line confidence and room area confidence, and projects the door and window positions onto the apartment layout to obtain the apartment vector of the target apartment. This realizes the automatic generation of apartment vectors and reduces the technical barriers and labor costs of apartment vector drawing.

[0067] For example, Fig.10 is a flow chart of a method for determining a house wall line confidence and a room area confidence provided by an embodiment of the present disclosure, such as Fig.10 As shown, in some embodiments, the house wall line confidence and the room area confidence can be determined by the following steps: Step 1001: Screen out room layouts that do not conform to physical reality from room layouts corresponding to multiple panoramic video key frames.

[0068] There are many methods for filtering out room layouts that do not conform to physical reality from room layouts corresponding to multiple panoramic video key frames. Several filtering methods are used as examples for explanation below. However, it can be understood that the filtering methods listed below are only exemplary methods rather than the only methods, and in some implementations, one or more filtering methods can be used simultaneously: In one embodiment, the room layout corresponding to the panoramic video key frame can be rotated based on the rotation relationship between the camera pose corresponding to the panoramic video key frame and the camera pose at the reference position to obtain the rotated room layout; in response to the rotated room layout not conforming to the Manhattan hypothesis, the room layout is screened out. Among them, the camera pose corresponding to the panoramic video key frame can be estimated based on the Simultaneous Localization and Mapping (SLAM) algorithm, and the reference position can be the shooting position of any panoramic video key frame. For ease of understanding, the embodiment of the present disclosure can be exemplarily understood as the shooting position corresponding to the first panoramic video key frame. The camera pose at the reference position can be understood as the camera pose corresponding to the first panoramic video key frame.

[0069] In another embodiment, the panoramic video key frame can be input into a preset estimation model, and the ratio of the distance from the camera to the ground to the distance from the camera to the ceiling can be determined based on the estimation model; in response to the ratio exceeding a preset first numerical range, the room layout corresponding to the panoramic video key frame is screened out. The estimation model can be trained using a training method provided by a related technology, and the present disclosure does not specifically limit the training method of the estimation model.

[0070] In another embodiment, the height of the camera when shooting the panoramic video key frame can be predicted based on the SLAM algorithm; in response to the height exceeding a preset second numerical range, the room layout corresponding to the panoramic video key frame is screened out, wherein the second numerical range can be set as needed. For example, if the camera height predicted based on the SLAM algorithm is 1.1 meters, and the second numerical range is 1.6 meters to 1.8 meters, then the room layout corresponding to the panoramic video key frame at this position is screened out.

[0071] In another embodiment, for any panoramic video key frame, the house area of ​​the panoramic video key frame and the panoramic video key frames taken at the previous moment and / or the next moment of the panoramic video key frame can be calculated; in response to the difference between the room area of ​​the panoramic video key frame and the room area of ​​the panoramic video key frame at the previous moment and / or the next moment being greater than a preset threshold, the room layout corresponding to the panoramic video key frame is screened out. When calculating the room area, the room layouts corresponding to the panoramic video key frames can be unified to the same scale first, and then the room area is calculated.

[0072] Step 1002: Based on the remaining room layouts, determine the house wall line confidence and the room area confidence.

[0073] Among them, the method of determining the house wall line confidence and the room area confidence based on the remaining room layout can refer to the relevant parts of the above embodiment, which will not be repeated here.

[0074] The disclosed embodiment screens out room layouts that do not conform to physical reality from the room layouts corresponding to multiple panoramic video key frames, and determines the house wall line confidence and the room area confidence based on the remaining room layouts, thereby improving the accuracy of determining the house wall line confidence and the room area confidence.

[0075] Fig.11 is a schematic diagram of the structure of a housing type vector generating device provided in an embodiment of the present disclosure. The housing type vector generating device can be exemplarily understood as the computer device in the above method embodiment or some functional modules in the computer device, such as Fig.11 As shown, in some implementations, the housing type vector generating device 1100 provided in the embodiment of the present disclosure may include: An acquisition module 1101 is used to acquire a plurality of panoramic video key frames shot in a target apartment; A first determination module 1102 is used to determine the room layout and the door and window positions based on each panoramic video key frame respectively; A second determination module 1103 is used to determine the house wall line confidence and the room area confidence based on the room layout corresponding to the multiple panoramic video key frames; A generating module 1104, configured to generate a vectorized apartment layout based on the house wall line confidence and the room area confidence; The projection module 1105 is used to project the door and window positions onto the apartment layout to obtain an apartment vector of the target apartment.

[0076] Optionally, the first determining module 1102 is configured to: Determine the room layout based on each panoramic video key frame respectively; The room layout is adjusted based on the Manhattan hypothesis, so that the adjusted room layout conforms to the Manhattan hypothesis.

[0077] Optionally, the second determining module 1103 is configured to: Screening out room layouts that do not conform to physical reality from room layouts corresponding to the multiple panoramic video key frames; Based on the remaining room layout, the house wall line confidence and room area confidence are determined.

[0078] Optionally, the second determining module 1103 is configured to: Based on the rotation relationship between the camera pose corresponding to the panoramic video key frame and the camera pose at the reference position, the room layout corresponding to the panoramic video key frame is rotated to obtain a rotated room layout; In response to the rotated room layout not meeting the Manhattan assumption, screening out the room layout; and / or Inputting the panoramic video key frame into a preset estimation model, and determining the ratio of the distance from the camera to the ground and the distance from the camera to the ceiling based on the estimation model; In response to the ratio exceeding a preset first value range, filtering out the room layout corresponding to the panoramic video key frame; and / or Predicting the height of the camera when shooting the key frame of the panoramic video based on the SLAM algorithm; In response to the height exceeding a preset second numerical range, filtering out the room layout corresponding to the panoramic video key frame; and / or For any panoramic video key frame, calculating the house area of ​​the panoramic video key frame and the panoramic video key frames shot at a moment before and / or after the panoramic video key frame; In response to the difference between the room area of ​​the panoramic video key frame and the room area of ​​the panoramic video key frame at a previous moment and / or a next moment being greater than a preset threshold, the room layout corresponding to the panoramic video key frame is screened out.

[0079] Optionally, the second determining module 1103 is configured to: Aligning room layouts corresponding to the multiple panoramic video key frames; For any house wall line on any room layout, the number of room layouts aligned to the house wall line is determined, and the number of the room layouts is determined as the confidence of the house wall line.

[0080] Optionally, the second determining module 1103 is configured to: Determining the overlap between the room layouts based on the room layouts corresponding to the multiple panoramic video key frames; Determine room layouts whose overlap degree is higher than a preset overlap degree as room layouts of the same room area; The number of room layouts corresponding to the same room region is determined as the confidence of the room region.

[0081] Optionally, the generating module 1104 is used to: Generate a room outline based on the house wall lines whose confidence level is higher than the first confidence level and the room areas whose confidence level is higher than the second confidence level; The room outline is projected into a three-dimensional space, so that the projection of the room outline in the three-dimensional space conforms to the Manhattan hypothesis.

[0082] Optionally, the model vector generating device may further include: The separation processing module is used to separate the walls between adjacent rooms in the vectorized apartment layout.

[0083] The device provided in the embodiments of the present disclosure can execute the method of any of the above method embodiments, and its execution method and beneficial effects are similar and will not be repeated here.

[0084] It should also be noted that the division of modules in the above-mentioned camera calibration device in the embodiments of the present disclosure is schematic and is only a logical functional division. There may be other division methods in actual implementation. In addition, the functional modules in the various embodiments of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0085] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present disclosure.

[0086] Fig.12 The schematic diagram of the structure of the computer device embodiment provided by the present disclosure embodiment. Fig.12 As shown, the computer device includes a memory 121 and a processor 122 .

[0087] The memory 121 is used to store programs. In addition to the above programs, the memory 121 can also be configured to store various other data to support operations on the computer device. Examples of such data include instructions for any application or method operating on the computer device, contact member data, phone book member data, messages, pictures, videos, etc.

[0088] The memory 121 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0089] The processor 122 is coupled to the memory 121 and executes the program stored in the memory 121 to: A plurality of panoramic video key frames shot in the target apartment will be obtained; Determine the room layout and door and window locations based on each panoramic video key frame; Determining a house wall line confidence and a room area confidence based on the room layouts corresponding to the multiple panoramic video key frames; Generate a vectorized apartment layout based on the house wall line confidence and the room area confidence; The door and window positions are projected onto the apartment layout to obtain an apartment vector of the target apartment.

[0090] Further, if Fig.12 As shown, the computer device may also include: a communication component 123, a power component 124, an audio component 125, a display 126 and other components. Fig.12 Only some components are shown schematically, and it does not mean that the computer equipment only includes Fig.12 Components shown.

[0091] The communication component 123 is configured to facilitate wired or wireless communication between the computer device and other devices. The computer device can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 123 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 123 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared member data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0092] The power supply component 124 provides power to various components of the computer device. The power supply component 124 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the computer device.

[0093] The audio component 125 is configured to output and / or input audio signals. For example, the audio component 125 includes a microphone (MIC), and when the computer device is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 121 or sent via the communication component 123. In some embodiments, the audio component 125 also includes a speaker for outputting audio signals.

[0094] The display 126 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0095] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the method described in any of the above method embodiments.

[0096] In the embodiments of the present disclosure, the above-mentioned computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO)), optical storage (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drives (SSDs)), etc.

[0097] It should be understood by those skilled in the art that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program codes.

[0098] An embodiment of the present disclosure provides a computer program product, including computer program instructions, which can implement the method described in any of the above method embodiments when executed by a processor.

[0099] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0100] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a housing type vector, characterized in that: The method comprises: Acquire multiple panoramic video key frames shot in the target apartment; Determine the room layout and door and window locations based on each panoramic video key frame; Determining a house wall line confidence and a room area confidence based on the room layouts corresponding to the multiple panoramic video key frames; Generate a vectorized apartment layout based on the house wall line confidence and the room area confidence; The door and window positions are projected onto the apartment layout to obtain an apartment vector of the target apartment.

2. The method according to claim 1, characterized in that The determining of the room layout and the door and window positions based on each panoramic video key frame respectively includes: Determine the room layout based on each panoramic video key frame respectively; The room layout is adjusted based on the Manhattan assumption, so that the adjusted room layout conforms to the Manhattan assumption.

3. The method according to claim 1, characterized in that The determining of the house wall line confidence and the room area confidence based on the room layouts corresponding to the multiple panoramic video key frames includes: Screening out room layouts that do not conform to physical reality from room layouts corresponding to the multiple panoramic video key frames; Based on the remaining room layout, the house wall line confidence and room area confidence are determined.

4. The method according to claim 3, characterized in that The step of screening out room layouts that do not conform to physical reality from the room layouts corresponding to the multiple panoramic video key frames includes: Based on the rotation relationship between the camera pose corresponding to the panoramic video key frame and the camera pose at the reference position, the room layout corresponding to the panoramic video key frame is rotated to obtain a rotated room layout; In response to the rotated room layout not meeting the Manhattan assumption, screening out the room layout; and / or Inputting the panoramic video key frame into a preset estimation model, and determining the ratio of the distance from the camera to the ground and the distance from the camera to the ceiling based on the estimation model; In response to the ratio exceeding a preset first value range, filtering out the room layout corresponding to the panoramic video key frame; and / or Predicting the height of the camera when shooting the key frame of the panoramic video based on the SLAM algorithm; In response to the height exceeding a preset second numerical range, filtering out the room layout corresponding to the panoramic video key frame; and / or For any panoramic video key frame, calculate the house area of ​​the panoramic video key frame and the panoramic video key frames shot at a moment before and / or after the panoramic video key frame; In response to the difference between the room area of ​​the panoramic video key frame and the room area of ​​the panoramic video key frame at a previous moment and / or a next moment being greater than a preset threshold, the room layout corresponding to the panoramic video key frame is screened out.

5. The method according to claim 1, characterized in that The determining of the house wall line confidence and the room area confidence based on the room layouts corresponding to the multiple panoramic video key frames includes: Aligning room layouts corresponding to the multiple panoramic video key frames; For any house wall line on any room layout, the number of room layouts aligned to the house wall line is determined, and the number of the room layouts is determined as the confidence of the house wall line.

6. The method according to claim 1, characterized in that The determining of the house wall line confidence and the room area confidence based on the room layouts corresponding to the multiple panoramic video key frames includes: Determining the overlap between the room layouts based on the room layouts corresponding to the multiple panoramic video key frames; Determine room areas based on room layouts whose overlap is higher than a preset overlap; The number of room layouts containing the room region is determined as the confidence level of the room region.

7. The method according to any one of claims 1, 5 or 6, characterized in that: The generating a vectorized apartment layout based on the house wall line confidence and the room area confidence includes: Generate a room outline based on the house wall line whose confidence level is higher than the first confidence level and the room area whose confidence level is higher than the second confidence level; The room outline is projected into a three-dimensional space, so that the projection of the room outline in the three-dimensional space conforms to the Manhattan hypothesis.

8. The method according to claim 1, characterized in that After generating the vectorized apartment layout based on the house wall line confidence and the room area confidence, the method further includes: In the vectorized apartment layout, walls between adjacent rooms are separated.

9. A computer device, characterized in that: The computer device comprises: Memory; Processor; and Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

11. A computer program product, comprising computer program instructions, which implement the method according to any one of claims 1 to 8 when executed by a processor.

Images