House plan generation method and device and computer readable storage medium

Abstract

The embodiment of the invention provides a floor plan generation method and device and a computer readable storage medium. According to the embodiment of the invention, the method comprises the steps: scanning a three-dimensional space comprising at least one room through a panorama camera, obtaining a panorama video of the three-dimensional space, and obtaining a scanning track of the panorama camera for the panorama video; acquiring a layout density map of the three-dimensional space according to the panoramic video and the scanning track; acquiring at least one of an object density map and a room type density map of the three-dimensional space according to the panoramic video and the scanning track; and according to the layout density map of the three-dimensional space and in combination with at least one of the object density map and the room type density map of the three-dimensional space, generating a house type map of the three-dimensional space.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method, device and computer-readable storage medium for generating a floor plan. Background Art

[0002] For a three-dimensional space containing rooms, its corresponding floor plan is a simple and clear display of the room structure of the three-dimensional space. At present, how to simply and conveniently generate a floor plan corresponding to the three-dimensional space is a technical problem that the image processing field hopes to solve.

[0003] In traditional methods, it is generally necessary to calculate based on the top view or horizontal cross-section of the three-dimensional space to generate a floor plan. However, such methods usually require obtaining three-dimensional data of each room in the three-dimensional space, which is costly to obtain and complex to calculate.

[0004] Therefore, a simple and accurate method and device is needed to automatically generate a floor plan of a three-dimensional space. Summary of the invention

[0005] To solve the above technical problems, according to one aspect of the present invention, a method for generating a floor plan is provided, comprising: using a panoramic camera to scan a three-dimensional space including at least one room, obtaining a panoramic video of the three-dimensional space, and obtaining a scanning trajectory of the panoramic camera for the panoramic video; obtaining a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory; obtaining at least one of an object density map and a room type density map of the three-dimensional space based on the panoramic video and the scanning trajectory; generating a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and in combination with at least one of the object density map and the room type density map of the three-dimensional space.

[0006] According to another aspect of the present invention, a floor plan generating device is provided, comprising: a scanning unit, configured to use a panoramic camera to scan a three-dimensional space including at least one room, to obtain a panoramic video of the three-dimensional space, and to obtain a scanning trajectory of the panoramic camera on the panoramic video; a first acquisition unit, configured to obtain a layout density map of the three-dimensional space according to the panoramic video and the scanning trajectory; a second acquisition unit, configured to obtain at least one of an object density map and a room type density map of the three-dimensional space according to the panoramic video and the scanning trajectory; a generation unit, configured to generate a floor plan of the three-dimensional space according to the layout density map of the three-dimensional space and in combination with at least one of the object density map and the room type density map of the three-dimensional space.

[0007] According to another aspect of the present invention, there is provided a floor plan generating device, comprising: a processor; and a memory, wherein computer program instructions are stored in the memory, wherein when the computer program instructions are executed by the processor, the processor performs the following steps: using a panoramic camera to scan a three-dimensional space including at least one room, to obtain a panoramic video of the three-dimensional space, and to obtain a scanning trajectory of the panoramic camera for the panoramic video; obtaining a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory; obtaining at least one of an object density map and a room type density map of the three-dimensional space based on the panoramic video and the scanning trajectory; generating a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and in combination with at least one of the object density map and the room type density map of the three-dimensional space.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the following steps: use a panoramic camera to scan a three-dimensional space including at least one room, obtain a panoramic video of the three-dimensional space, and obtain a scanning trajectory of the panoramic camera for the panoramic video; obtain a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory; obtain at least one of an object density map and a room type density map of the three-dimensional space based on the panoramic video and the scanning trajectory; generate a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and in combination with at least one of the object density map and the room type density map of the three-dimensional space.

[0009] According to the above-mentioned floor plan generation method, device and computer-readable storage medium of the embodiment of the present invention, it is possible to obtain a layout density map of the three-dimensional space by using a panoramic video obtained by scanning the three-dimensional space with a panoramic camera, and further obtain at least one of an object density map and a room type density map, thereby generating a floor plan of the three-dimensional space. The floor plan generation method, device and computer-readable storage medium of the embodiment of the present invention do not need to obtain and calculate the three-dimensional data of each room in the three-dimensional space, which reduces the calculation cost and improves the user experience.

[0010] In addition, in the floor plan generation method, device and computer-readable storage medium of the embodiments of the present invention, door detection information on the room walls in three-dimensional space can be obtained in the object density map to obtain information such as the door opening direction, so as to accurately generate a floor plan in three-dimensional space, thereby optimizing the generation method of the floor plan. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the present invention will become more apparent by describing in detail the embodiments of the present invention in conjunction with the accompanying drawings.

[0012] Figure 1 A flow chart showing a method for generating a floor plan according to an embodiment of the present invention is shown;

[0013] Figure 2 A schematic diagram showing a scanning trajectory of a panoramic camera according to an example of an embodiment of the present invention is shown;

[0014] Figure 3 A schematic diagram showing a room layout according to an example of an embodiment of the present invention;

[0015] Figure 4 A layout projection diagram of a room in a three-dimensional space according to an example of an embodiment of the present invention is shown;

[0016] Figure 5 A layout projection diagram of the same room in a three-dimensional space obtained according to an example of an embodiment of the present invention is shown;

[0017] Figure 6 Show that Figure 5 An example of a revised layout projection diagram obtained after the scale of the layout projection diagrams of two identical rooms shown is adjusted;

[0018] Figure 7 An example of a layout density map of a three-dimensional space according to an example of an embodiment of the present invention is shown;

[0019] Figure 8 A schematic diagram of two-dimensional detection of doors and windows according to an example of an embodiment of the present invention is shown;

[0020] Fig. 9 A schematic diagram of three-dimensional detection of placed objects according to an example of an embodiment of the present invention is shown;

[0021] Fig.10 An object density map according to an example of an embodiment of the present invention is shown;

[0022] Fig.11 A room type density map according to an example of an embodiment of the present invention is shown;

[0023] Fig.12 An example of a floor plan of a three-dimensional space according to an example of an embodiment of the present invention is shown;

[0024] Fig.13 A block diagram showing a device for generating a floor plan according to an embodiment of the present invention is shown;

[0025] Fig.14 A block diagram of a floor plan generating device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0026] The following will describe a floor plan generation method, device, and computer-readable storage medium according to an embodiment of the present invention with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same elements throughout. It should be understood that the embodiments described herein are merely illustrative and should not be construed as limiting the scope of the present invention.

[0027] Generally speaking, in order to obtain a floor plan of a three-dimensional space containing at least one room, it is necessary to first obtain the three-dimensional data of each room in the three-dimensional space, and then generate the corresponding plane floor plan by calculating the top view or horizontal cross-section view of the three-dimensional space. This often requires a large amount of three-dimensional data and calculation process, and the steps are relatively cumbersome.

[0028] In this regard, refer to Figure 1 A method for generating a floor plan according to an embodiment of the present invention is described. Figure 1 A flow chart of the floor plan generating method 100 is shown.

[0029] like Figure 1 As shown, in step S101, a three-dimensional space including at least one room is scanned using a panoramic camera to obtain a panoramic video of the three-dimensional space, and a scanning trajectory of the panoramic camera on the panoramic video is obtained.

[0030] In this embodiment, the three-dimensional space may be the entirety or a portion of a three-dimensional building including at least one room, and accordingly, the generated floor plan may be a plan floor plan of the three-dimensional building.

[0031] In this embodiment, a panoramic camera (such as a fisheye camera, a binocular camera, etc.) can be used to obtain a panoramic video of a three-dimensional space. In addition, when obtaining the panoramic video of the three-dimensional space, the position information and attitude information of the panoramic camera along with the timestamp can also be obtained according to the panoramic video to generate the scanning trajectory of the panoramic camera. Optionally, the panoramic video can be input into a visual simultaneous localization and mapping (SLAM), such as OpenVSLAM, to generate the trajectory of the camera scanning during the panoramic video shooting process. Optionally, the format of the generated scanning trajectory of the panoramic camera may include: timestamp, camera position information, camera attitude information. For example, assuming that the panoramic camera always maintains a vertical state and keeps the camera's orientation unchanged during the shooting process, the trajectory can only include the timestamp Ti and the horizontal position of the camera (Xi, Yi), expressed as (Ti, Xi, Yi).

[0032] In step S102, a layout density map of the three-dimensional space is acquired according to the panoramic video and the scanning trajectory.

[0033] In this embodiment, obtaining the layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory may include: projecting the panoramic video to obtain a layout projection map of the panoramic video; associating the layout projection map with the scanning trajectory and determining the scale of the layout projection map; and obtaining the layout density map of the three-dimensional space based on the layout projection map and the scale.

[0034] Optionally, the video frames of the panoramic video may first be input into a model for predicting the room layout, such as HorizonNet, and projected to obtain a layout projection diagram of each video frame in a three-dimensional space. In an embodiment of the present invention, the layout of the room may show the boundary between the ceiling and the wall, and the boundary between the ground and the wall in the room. In the process of obtaining the layout projection diagram using the video frames of the panoramic video, only some key frames in the panoramic video may be selected for calculation to improve the computational efficiency.

[0035] After obtaining the layout projection map, the timestamp corresponding to each video frame and the position information of the camera can be used to place the corresponding camera position in the layout projection map of the video frame to associate the layout projection map with the corresponding scanning trajectory. Subsequently, based on the entropy minimization method, the relative scale of the layout projection map of each video frame and the scanning trajectory can be corrected so that the layout projection maps of the same room calculated and shown by each video frame overlap as much as possible, so as to obtain the layout density map of the entire three-dimensional space by superimposing the layout projection map of each video frame and the corrected determined scale.

[0036] In step S103, at least one of an object density map and a room type density map of the three-dimensional space is acquired according to the panoramic video and the scanning trajectory.

[0037] In this embodiment, the video frames of the panoramic video can first be input into the model for object detection, and projected to obtain the object projection map of each video frame in the three-dimensional space. In the embodiment of the present invention, the object detection of the three-dimensional space can include two-dimensional detection of doors and windows on the wall of the room in the video frame, and can also include three-dimensional detection of objects placed on the floor of the room (such as furniture, ornaments, etc.). Specifically, it can include: according to the panoramic video, at least one of the doors and windows on the wall of the room in each video frame in the three-dimensional space is detected to obtain door detection information and / or window detection information, wherein the door detection information includes at least one of the door type, the door opening direction, and the door outline; and / or according to the panoramic video, the objects placed on the floor of the room in the three-dimensional space in each video frame are detected to obtain the detection information of the placed objects. In the process of obtaining the object projection map using the video frames of the panoramic video, only some key frames in the panoramic video can be selected for calculation to improve the operation efficiency.

[0038] After object detection is performed on each video frame, the detected object can be projected in the layout projection map to obtain the corresponding object projection map. Specifically, the line segment corresponding to the object detected on the wall in the room on the ground-wall boundary of the room layout can be first intercepted and projected into the layout projection map, and different types of objects can be further displayed as different colors. Among them, when projecting according to the door detection information in the layout projection map, the indicating line segment of the door opening direction can also be projected into the layout projection map according to the information of the door opening direction contained in the door detection information, thereby providing more detailed information about the object projection, which is conducive to the accurate judgment and generation process of the floor plan. In addition, for the objects obtained by three-dimensional detection, the bottom surface of the three-dimensional frame of the placed object detected on the ground in the room can also be intercepted and projected into the layout projection map, and different types of objects can also be further displayed as different colors.

[0039] After acquiring the object projection map, the object projection map of each video frame may be adjusted using the scale determined after associating each acquired video frame with the scanning trajectory, and the object density map of the three-dimensional space may be superimposed and acquired.

[0040] Optionally, when obtaining the room type density map, the video frames of the panoramic video may be first input into a room type classification model (such as a VGG-based classifier) ​​and projected to obtain a room type projection map of each room for each video frame in the three-dimensional space. In the process of obtaining the room type projection map using the video frames of the panoramic video, only some key frames in the panoramic video may be selected for calculation to improve the computational efficiency.

[0041] After acquiring the room type projection map, the room type projection map of each video frame may be adjusted using the scale determined after associating each acquired video frame with the scanning trajectory, and the room type density map of the three-dimensional space may be superimposed and acquired.

[0042] In step S104, a floor plan of the three-dimensional space may be generated according to the layout density map of the three-dimensional space and in combination with at least one of the object density map and the room type density map of the three-dimensional space.

[0043] In this embodiment, the layout density map of the three-dimensional space, and at least one of the object density map and the room type density map of the three-dimensional space can be spliced ​​as different channels and sent to the floor plan generation model to facilitate the floor plan generation model to generate the floor plan of the three-dimensional space. The floor plan generation model can be a generation method based on GAN, or a detection method such as RoomFormer, etc., which are not limited here. The floor plan generation model of the embodiment of the present invention can display the polygonal shape, room type and the position of objects (such as beds, sofas, doors and windows, etc.) in each room in the three-dimensional space according to the layout density map of the three-dimensional space and at least one of the object density map and the room type density map of the three-dimensional space, wherein when the input object density map includes the information of the door opening direction in the door detection information, the generated floor plan can display the information of the door with the opening direction accordingly. In addition, in the obtained floor plan of the three-dimensional space, the same type of rooms can also be displayed as the same color according to the relevant information of the room type density map.

[0044] According to the floor plan generation method of the embodiment of the present invention, it is possible to obtain a layout density map of the three-dimensional space by using a panoramic video obtained by scanning the three-dimensional space with a panoramic camera, and further obtain at least one of an object density map and a room type density map, thereby generating a floor plan of the three-dimensional space. The floor plan generation method of the embodiment of the present invention does not need to obtain and calculate the three-dimensional data of each room in the three-dimensional space, which reduces the calculation cost and improves the user experience.

[0045] In addition, in the floor plan generation method of the embodiment of the present invention, door detection information on the room wall in three-dimensional space can be obtained in the object density map to obtain information such as the door opening direction, so as to accurately generate a floor plan in three-dimensional space, thereby optimizing the generation method of the floor plan.

[0046] An example of a method for generating a floor plan according to an embodiment of the present invention is shown below.

[0047] In this example, first, a panoramic camera is used to scan a three-dimensional space including multiple rooms, a panoramic video of the three-dimensional space is obtained, and a scanning trajectory of the panoramic camera for the panoramic video is obtained. Specifically, in this example, a panoramic camera can be used to obtain a panoramic video of a three-dimensional space. In addition, when obtaining a panoramic video of the three-dimensional space, the panoramic video can also be input into OpenVSLAM to generate a trajectory of the camera scanning during the panoramic video shooting process. Among them, when the panoramic camera always maintains a vertical state and keeps the direction of the camera unchanged, the scanning trajectory of the generated panoramic camera can only include the timestamp Ti and the horizontal position (Xi, Yi) of the camera, which is expressed as (Ti, Xi, Yi). Figure 2FIG. 4 shows a schematic diagram of a scanning trajectory of a panoramic camera according to an example of an embodiment of the present invention. Figure 2 As shown, when the panoramic camera maintains a vertical state and keeps a constant direction for scanning, the horizontal position coordinates corresponding to different time stamps can be used to show the scanning trajectory of the panoramic camera in the process of scanning the panoramic video.

[0048] Subsequently, the layout density map of the three-dimensional space can be obtained according to the panoramic video and the scanning trajectory. Specifically, the panoramic video can be projected first to obtain a layout projection map of the panoramic video; the layout projection map is associated with the scanning trajectory, and the scale of the layout projection map is determined; and the layout density map of the three-dimensional space is obtained according to the layout projection map and the scale.

[0049] In this example, the video frames of the panoramic video can first be input into a model for room layout prediction, such as HorizonNet, to obtain the layout of the room in three-dimensional space. The layout of the room can show the boundaries between the ceiling and the wall, and the boundaries between the ground and the wall. Among them, only some key frames in the panoramic video can be selected to obtain and calculate the room layout to improve the operation efficiency. Figure 3 A schematic diagram of a room layout according to an example of an embodiment of the present invention is shown. Specifically, Figure 3 FIG. 4 shows the ceiling-wall boundary and the ground-wall boundary extracted from the video frames of the input panoramic video, which are used to reflect the layout of the room in the three-dimensional space.

[0050] After the layout of the room in the three-dimensional space is obtained, the layout may be projected to obtain a layout projection image corresponding to each video frame. Figure 4 FIG. 2 shows a layout projection diagram of a room in a three-dimensional space according to an example of an embodiment of the present invention. Figure 4 As shown, the layout of the room can be projected into the top-down plan view to form a corresponding room layout projection view.

[0051] Subsequently, the layout projection image may be associated with the position of the scanning track of the corresponding video frame by using the timestamp corresponding to each video frame and the position information of the camera. Figure 5 FIG. 1 shows an example of an embodiment of the present invention, in which a layout projection diagram of the same room in a three-dimensional space is obtained based on different video frames in a panoramic video and combined with a scanning trajectory. Figure 5 As shown in Figure 1, it can be seen that if the camera centers of different video frames at different times are loaded into the corresponding scanning trajectories respectively, the layout projections of the same room may not be effectively overlapped due to the scale problem of the layout projections corresponding to each video frame and the combined scanning trajectory (e.g. Figure 5As shown, the difference between the two is t). In this regard, during the scanning and calculation process, the relative scale of the layout projection diagram of each video frame and the scanning trajectory can be corrected based on the entropy minimization method, so that the layout projection diagrams of the same room calculated by each video frame overlap as much as possible. Figure 6 Shows the Figure 5 An example of a modified layout projection diagram obtained after the scale of the layout projection diagrams of two identical rooms is adjusted. Figure 6 As shown, the difference t between the two layout projection images can be adjusted by introducing a scale relative to the scanning trajectory of the adjusted layout projection image, i.e., a coefficient s*, so that the layout projection images of the same room calculated in each video frame overlap as much as possible.

[0052] After determining the corrected scale parameter s* through the above steps, the corresponding scale of the layout projection map of each video frame used relative to the scanning trajectory can be adjusted by a similar method, and the layout projection map corresponding to each video frame can be superimposed with the adjusted scale to obtain the layout density map of the entire three-dimensional space. Figure 7 An example of a layout density map of a three-dimensional space according to an example of an embodiment of the present invention is shown. Figure 7 As shown, the layout projection diagrams of each key frame selected in the panoramic video can be superimposed according to the adjusted scale to obtain a layout density diagram of the superimposed three-dimensional space. In the layout density diagram, the approximate room structure and apartment type of the three-dimensional space can be shown.

[0053] On the basis of obtaining the layout density map of the three-dimensional space, at least one of an object density map and a room type density map of the three-dimensional space can also be obtained according to the panoramic video and the scanning trajectory. In the process of obtaining the object density map and / or the room type density map, only some key frames in the panoramic video can be selected for calculation to improve the calculation efficiency.

[0054] In addition, in the above acquisition process, the scale of the object density map and the room type density map when they are superimposed can be adjusted in combination with the corresponding scale of the layout projection map of each video frame used previously relative to the scanning trajectory.

[0055] In the process of obtaining the three-dimensional space object density map, the video frames of the panoramic video can first be input into the model for object detection, and projected to obtain the object projection map of each video frame in the three-dimensional space. Optionally, two-dimensional detection can be performed on the doors and windows on the walls of the room in the video frame to obtain door and window detection information, and after detection, the detected objects are projected in the layout projection map to obtain the corresponding object projection map.

[0056] Figure 8 FIG. 2 shows a schematic diagram of two-dimensional door and window detection according to an example of an embodiment of the present invention. Figure 8 The left figure shows the door detection information, window detection information, etc. obtained by detecting the doors and windows on the walls of the room in the three-dimensional space according to the panoramic video. The door detection information may include at least one of the door type (such as inward-opening door, outward-opening door, sliding door, etc.), door opening direction, and door outline (such as the door's enclosing frame). The window detection information may include information such as the window outline. Figure 8 The right figure shows the corresponding object projection map obtained by projecting the detected object on the layout projection map. Specifically, the line segment corresponding to the door or window detected on the wall in the room on the floor-wall boundary of the room layout can be first intercepted and projected into the layout projection map. In addition, different types of objects can be further displayed in different colors. Among them, when projecting according to the door detection information in the layout projection map, such as Figure 8 As shown in the right figure, based on the information of the door opening direction contained in the door detection information, the indicating line segment of the door opening direction can also be projected into the layout projection diagram, thereby providing more detailed information about the object projection, which is beneficial to the accurate judgment and generation process of the floor plan.

[0057] In the object detection process, optionally, the objects placed on the floor of the room in the three-dimensional space in each video frame may be detected according to the panoramic video to obtain the object detection information. Subsequently, for the objects obtained by the three-dimensional detection, the bottom surface of the three-dimensional frame of the objects detected on the floor in the room may be intercepted and projected into the layout projection map. Fig. 9 FIG. 2 is a schematic diagram of three-dimensional detection of placed objects according to an example of an embodiment of the present invention. Fig. 9 The left figure shows the object detection information obtained by detecting the furniture and other objects placed on the floor of the room in the three-dimensional space based on the panoramic video. Fig. 9 The right figure shows the corresponding object projection image obtained by projecting the detected placed objects on the layout projection image. Specifically, the geometric shape corresponding to the placed objects detected on the ground in the room can be first intercepted and projected into the layout projection image. In addition, different types of placed objects can be further displayed in different colors, such as beds, cabinets, sofas, etc. can be displayed in different colors.

[0058] After obtaining the object projection map by combining the two-dimensional and three-dimensional detection and projection results, the object projection map of each video frame can be corrected by using a similar method to the previous layout projection map generation process, using the scale adjusted after the determined video frames are associated with the scanning trajectory, and the object density map of the three-dimensional space can be superimposed and obtained. Fig.10 FIG. 2 shows an object density map according to an example of an embodiment of the present invention. Fig.10 As shown, the object density map may include the distribution of two-dimensional objects (doors, windows, etc.) and / or three-dimensional objects (placed objects such as furniture, etc.) included in each room in the three-dimensional space.

[0059] In addition, when obtaining the room type density map, the video frames of the panoramic video can first be input into a room type classification model (such as a VGG-based classifier), and different room types can be projected in different colors to obtain a room type projection map of each room in each video frame in the three-dimensional space. After obtaining the room type projection map, the room type projection map of each video frame can be adjusted using the scale determined after the obtained video frames are associated with the scanning trajectory, and the room type density map of the three-dimensional space can be superimposed and obtained. Fig.11 FIG. 2 shows a room type density diagram according to an example of an embodiment of the present invention. Fig.11 As shown, different room types in the three-dimensional space can be known according to different colors.

[0060] Finally, the floor plan of the three-dimensional space may be generated according to the layout density map of the three-dimensional space and in combination with at least one of the object density map and the room type density map of the three-dimensional space.

[0061] In this example, the layout density map of the three-dimensional space, and at least one of the object density map and the room type density map of the three-dimensional space can be spliced ​​as different channels and sent to the floor plan generation model to generate the floor plan of the three-dimensional space using a GAN-based generation method, or a floor plan generation model such as a RoomFormer detection method. Fig.12 An example of a floor plan of a three-dimensional space according to an example of an embodiment of the present invention is shown. Fig.12 The floor plan is generated by splicing the layout density map, object density map and room type density map of the three-dimensional space through the floor plan generation model, which shows the location, shape and other information of each different room in the three-dimensional space, and shows the connection relationship between different rooms in detail through the schematic diagram of the door opening direction. In addition, in the obtained three-dimensional space floor plan, rooms of the same type can also be displayed in the same color according to the relevant information of the room type density map.

[0062] Below, refer to Fig.13 To describe the floor plan generating device according to an embodiment of the present invention. Fig.13 FIG. 1 is a block diagram of a floor plan generating device 1300 according to an embodiment of the present invention. Fig.13As shown in FIG. 1 , the floor plan generating device 1300 includes a scanning unit 1310, a first acquiring unit 1320, a second acquiring unit 1330, and a generating unit 1340. In addition to these units, the floor plan generating device 1300 may also include other components. However, since these components are irrelevant to the content of the embodiment of the present invention, their illustration and description are omitted here. In addition, since the specific details of the following operations performed by the floor plan generating device 1300 according to the embodiment of the present invention are the same as those described above with reference to FIG. Figure 1-Figure 12 The details described are the same, so repeated description of the same details is omitted here to avoid repetition.

[0063] Fig.13 The scanning unit 1310 of the floor plan generating device 1300 uses a panoramic camera to scan a three-dimensional space including at least one room, obtains a panoramic video of the three-dimensional space, and obtains a scanning trajectory of the panoramic camera on the panoramic video.

[0064] In this embodiment, the three-dimensional space may be the entirety or a portion of a three-dimensional building including at least one room, and accordingly, the generated floor plan may be a plan floor plan of the three-dimensional building.

[0065] In this embodiment, the scanning unit 1310 can use a panoramic camera (such as a fisheye camera, a binocular camera, etc.) to obtain a panoramic video of a three-dimensional space. In addition, when obtaining the panoramic video of the three-dimensional space, the position information and attitude information of the panoramic camera along with the timestamp can also be obtained according to the panoramic video to generate the scanning trajectory of the panoramic camera. Optionally, the panoramic video can be input into a visual simultaneous localization and mapping (SLAM), such as OpenVSLAM, to generate a trajectory of the camera scanning during the panoramic video shooting process. Optionally, the format of the scanning trajectory of the generated panoramic camera may include: timestamp, camera position information, camera attitude information. For example, assuming that the panoramic camera maintains a vertical state and keeps the camera's orientation unchanged during the shooting process, the trajectory may only include the timestamp Ti and the horizontal position (Xi, Yi) of the camera, expressed as (Ti, Xi, Yi).

[0066] The first acquisition unit 1320 acquires a layout density map of the three-dimensional space according to the panoramic video and the scanning trajectory.

[0067] In this embodiment, the first acquisition unit 1320 projects the panoramic video to obtain a layout projection map of the panoramic video; associates the layout projection map with the scanning trajectory and determines the scale of the layout projection map; and obtains the layout density map of the three-dimensional space based on the layout projection map and the scale.

[0068] Optionally, the first acquisition unit 1320 may first input the video frames of the panoramic video into a model for room layout prediction, such as HorizonNet, and perform projection to obtain a layout projection diagram of each video frame in three-dimensional space. In an embodiment of the present invention, the layout of the room may show the boundary between the ceiling and the wall, and the boundary between the ground and the wall in the room. In the process of obtaining the layout projection diagram using the video frames of the panoramic video, only some key frames in the panoramic video may be selected for calculation to improve the operation efficiency.

[0069] After acquiring the layout projection map, the first acquisition unit 1320 can use the timestamp corresponding to each video frame and the position information of the camera to place the corresponding camera position in the layout projection map of the video frame to associate the layout projection map with the corresponding scanning trajectory. Subsequently, based on the entropy minimization method, the relative scale of the layout projection map of each video frame and the scanning trajectory can be corrected so that the layout projection maps of the same room calculated and shown by each video frame overlap as much as possible, so as to obtain the layout density map of the entire three-dimensional space by superimposing the layout projection map of each video frame and the corrected determined scale.

[0070] The second acquisition unit 1330 acquires at least one of an object density map and a room type density map of the three-dimensional space according to the panoramic video and the scanning trajectory.

[0071] In this embodiment, the second acquisition unit 1330 may first input the video frames of the panoramic video into the model for object detection, and perform projection to obtain the object projection map of each video frame in the three-dimensional space. In the embodiment of the present invention, object detection in the three-dimensional space may include two-dimensional detection of doors and windows on the walls of the room in the video frame, and may also include three-dimensional detection of objects placed on the floor of the room (such as furniture, ornaments, etc.). Specifically, the second acquisition unit 1330 may detect at least one of the doors and windows on the walls of the room in the three-dimensional space in each video frame according to the panoramic video, and obtain door detection information and / or window detection information, wherein the door detection information includes at least one of the door type, the door opening direction, and the door outline; and / or detect the objects placed on the floor of the room in the three-dimensional space in each video frame according to the panoramic video, and obtain the object detection information. In the process of obtaining the object projection map using the video frames of the panoramic video, only some key frames in the panoramic video may be selected for calculation to improve the operation efficiency.

[0072] After performing object detection on each video frame, the second acquisition unit 1330 can project the detected object in the layout projection map to obtain the corresponding object projection map. Specifically, the line segment corresponding to the object detected on the wall in the room on the ground-wall boundary of the room layout can be first intercepted and projected into the layout projection map, and different types of objects can be further displayed as different colors. Among them, when projecting according to the door detection information in the layout projection map, the indicating line segment of the door opening direction can also be projected into the layout projection map according to the information of the door opening direction contained in the door detection information, thereby providing more detailed information about the object projection, which is conducive to the accurate judgment and generation process of the floor plan. In addition, for the objects obtained by three-dimensional detection, the bottom surface of the three-dimensional frame of the placed object detected on the ground in the room can also be intercepted and projected into the layout projection map, and different types of objects can also be further displayed as different colors.

[0073] After acquiring the object projection map, the second acquisition unit 1330 may adjust the object projection map of each video frame using the scale determined after associating each acquired video frame with the scanning trajectory, and superimpose and acquire the object density map of the three-dimensional space.

[0074] Optionally, when acquiring the room type density map, the second acquisition unit 1330 may first input the video frames of the panoramic video into a room type classification model (such as a VGG-based classifier), and perform projection to obtain a room type projection map of each room for each video frame in the three-dimensional space. In the process of acquiring the room type projection map using the video frames of the panoramic video, only some key frames in the panoramic video may be selected for calculation to improve the operation efficiency.

[0075] After acquiring the room type projection map, the second acquisition unit 1330 may adjust the room type projection map of each video frame using the scale determined by associating each of the previously acquired video frames with the scanning trajectory, and superimpose and acquire the room type density map of the three-dimensional space.

[0076] The generating unit 1340 may generate a floor plan of the three-dimensional space according to the layout density map of the three-dimensional space and in combination with at least one of the object density map and the room type density map of the three-dimensional space.

[0077] In this embodiment, the generation unit 1340 can splice the layout density map of the three-dimensional space, and at least one of the object density map and the room type density map of the three-dimensional space as different channels, and send them to the floor plan generation model, so that the floor plan generation model can generate the floor plan of the three-dimensional space. The floor plan generation model can be a generation method based on GAN, or a detection method such as RoomFormer, etc., which are not limited here. The floor plan generation model of the embodiment of the present invention can display the polygonal shape, room type and the position of objects (such as beds, sofas, doors and windows, etc.) in each room in the three-dimensional space according to the layout density map of the three-dimensional space and at least one of the object density map and the room type density map of the three-dimensional space, wherein when the input object density map includes the information of the door opening direction in the door detection information, the generated floor plan can display the information of the door with the opening direction accordingly. In addition, in the obtained floor plan of the three-dimensional space, the same type of rooms can also be displayed as the same color according to the relevant information of the room type density map.

[0078] According to the floor plan generation device of the embodiment of the present invention, it is possible to obtain a layout density map of the three-dimensional space by using a panoramic video obtained by scanning the three-dimensional space with a panoramic camera, and further obtain at least one of an object density map and a room type density map, thereby generating a floor plan of the three-dimensional space. The floor plan generation method of the embodiment of the present invention does not need to obtain and calculate the three-dimensional data of each room in the three-dimensional space, thereby reducing the calculation cost and improving the user experience.

[0079] In addition, in the floor plan generation device of the embodiment of the present invention, door detection information on the room wall in the three-dimensional space can be obtained in the object density map to obtain information such as the door opening direction, so as to accurately generate a floor plan in the three-dimensional space, thereby optimizing the generation method of the floor plan.

[0080] Below, refer to Fig.14 To describe the floor plan generating device according to an embodiment of the present invention. Fig.14 FIG. 1 is a block diagram of a floor plan generating device 1400 according to an embodiment of the present invention. Fig.14 As shown, the device 1400 may be a computer or a server.

[0081] like Fig.14 As shown, the floor plan generating device 1400 includes one or more processors 1410 and a memory 1420. Of course, in addition to this, the floor plan generating device 1400 may also include an input device, an output device (not shown), etc. These components may be interconnected via a bus system and / or other forms of connection mechanisms. It should be noted that Fig.14The components and structures of the floor plan generating device 1400 shown are merely exemplary and non-restrictive. The floor plan generating device 1400 may also have other components and structures as required.

[0082] The processor 1410 can be a central processing unit (CPU) or other form of processing unit with data processing capability and / or instruction execution capability, and can use the computer program instructions stored in the memory 1420 to perform the desired functions, which may include: using a panoramic camera to scan a three-dimensional space including at least one room, obtaining a panoramic video of the three-dimensional space, and obtaining a scanning trajectory of the panoramic camera for the panoramic video; obtaining a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory; obtaining at least one of an object density map and a room type density map of the three-dimensional space based on the panoramic video and the scanning trajectory; generating a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and in combination with at least one of the object density map and the room type density map of the three-dimensional space.

[0083] The memory 1420 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1410 may run the program instructions to implement the functions of the floor plan generation device of the embodiment of the present invention described above and / or other desired functions, and / or may execute the floor plan generation method according to the embodiment of the present invention. Various applications and various data may also be stored in the computer-readable storage medium.

[0084] Below, a computer-readable storage medium according to an embodiment of the present invention is described, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the following steps: use a panoramic camera to scan a three-dimensional space including at least one room, obtain a panoramic video of the three-dimensional space, and obtain a scanning trajectory of the panoramic camera for the panoramic video; obtain a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory; obtain at least one of an object density map and a room type density map of the three-dimensional space based on the panoramic video and the scanning trajectory; generate a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and in combination with at least one of the object density map and the room type density map of the three-dimensional space.

[0085] Of course, the above-mentioned specific embodiments are merely examples rather than limitations, and those skilled in the art can, based on the concept of the present invention, merge and combine some steps and devices from the various embodiments described separately above to achieve the effects of the present invention. Such merged and combined embodiments are also included in the present invention, and such merges and combinations are not described one by one herein.

[0086] Note that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details of the above invention are only for the purpose of illustration and facilitating understanding, not limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0087] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0088] The step flow charts and the above method descriptions in the present invention are only illustrative examples and are not intended to require or imply that the steps of each embodiment must be performed in the order given. As will be appreciated by those skilled in the art, the order of the steps in the above embodiments can be performed in any order. Words such as "thereafter", "then", "next", etc. are not intended to limit the order of the steps; these words are only used to guide the reader through the description of these methods. In addition, any reference to an element in the singular, such as using "one", "one", or "the" is not to be construed as limiting the element to the singular.

[0089] In addition, the steps and devices in the various embodiments of this document are not limited to being implemented in a certain embodiment. In fact, based on the concept of the present invention, relevant partial steps and partial devices in the various embodiments of this document can be combined to conceive new embodiments, and these new embodiments are also included in the scope of the present invention.

[0090] Each operation of the method described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors.

[0091] The various illustrated logic blocks, modules and circuits described may be implemented or performed using a general purpose processor, digital signal processor (DSP), ASIC, field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but as an alternative, the processor may be any commercially available processor, controller, microcontroller or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0092] The steps of the method or algorithm described in conjunction with the present invention can be directly embedded in hardware, in a software module executed by a processor, or in a combination of the two. A software module can exist in any form of tangible storage medium. Some examples of storage media that can be used include random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, etc. A storage medium can be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative manner, the storage medium can be integral with the processor. A software module can be a single instruction or many instructions, and can be distributed on several different code segments, between different programs, and across multiple storage media.

[0093] The method invented herein includes one or more actions for implementing the described method. The method and / or action can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and / or use of specific actions can be modified without departing from the scope of the claims.

[0094] The functions described can be implemented by hardware, software, firmware or any combination thereof. If implemented in software, the functions can be stored as one or more instructions on a tangible computer-readable medium. The storage medium can be any available tangible medium that can be accessed by a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device or any other tangible medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. As used herein, a disc includes a compact disc (CD), a laser disc, an optical disc, a digital versatile disc (DVD), a floppy disk and a blue disc.

[0095] Thus, a computer program product may perform the operations presented herein. For example, such a computer program product may be a computer-readable tangible medium having instructions tangibly stored (and / or encoded) thereon, which instructions may be executed by one or more processors to perform the operations described herein. The computer program product may include packaging materials.

[0096] Software or instructions may also be transmitted via a transmission medium. For example, the software may be transmitted from a website, server or other remote source using a transmission medium such as coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology such as infrared, radio or microwave.

[0097] In addition, the module and / or other appropriate means for performing the methods and techniques described herein can be downloaded and / or otherwise obtained by the user terminal and / or base station when appropriate. For example, such a device can be coupled to a server to facilitate the transmission of the means for performing the methods described herein. Alternatively, the various methods described herein can be provided via a storage component (e.g., RAM, ROM, a physical storage medium such as a CD or a floppy disk, etc.) so that the user terminal and / or base station can obtain the various methods when being coupled to the device or providing a storage component to the device. In addition, any other appropriate technology for providing the methods and techniques described herein to a device can be utilized.

[0098] Other examples and implementations are within the scope and spirit of the present invention and the appended claims. For example, due to the nature of software, the functions described above can be implemented using software executed by a processor, hardware, firmware, hard wiring, or any combination of these. Features that implement the functions can also be physically located in various locations, including being distributed so that parts of the functions are implemented at different physical locations. Moreover, as used herein, including as used in the claims, "or" used in the enumeration of items beginning with "at least one" indicates a separate enumeration, so that, for example, the enumeration of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). In addition, the wording "exemplary" does not mean that the example described is preferred or better than other examples.

[0099] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present invention is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0100] The above description of the invented aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features of the present invention.

[0101] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present invention to the form invented herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for generating a floor plan, comprising: scanning a three-dimensional space including at least one room using a panoramic camera to obtain a panoramic video of the three-dimensional space, and obtaining a scanning trajectory of the panoramic camera for the panoramic video; obtaining a layout density map of the three-dimensional space according to the panoramic video and the scanning trajectory; obtaining at least one of an object density map and a room type density map of the three-dimensional space according to the panoramic video and the scanning trajectory; generating a floor plan of the three-dimensional space according to the layout density map of the three-dimensional space and in combination with at least one of the object density map and the room type density map of the three-dimensional space.

2. The method according to claim 1, wherein, scanning a three-dimensional space including at least one room using a panoramic camera to obtain a panoramic video of the three-dimensional space, and obtaining a scanning trajectory of the panoramic camera for the panoramic video includes: obtaining position information and attitude information of the panoramic camera over time stamps according to the panoramic video, and generating a scanning trajectory of the panoramic camera.

3. The method according to claim 1, wherein, obtaining a layout density map of the three-dimensional space according to the panoramic video and the scanning trajectory includes: projecting the panoramic video to obtain a layout projection map of the panoramic video; associating the layout projection map with the scanning trajectory, and determining a scale of the layout projection map; obtaining the layout density map of the three-dimensional space according to the layout projection map and the scale.

4. The method according to claim 3, wherein, obtaining an object density map of the three-dimensional space according to the panoramic video and the scanning trajectory includes: performing object detection on the three-dimensional space according to the panoramic video; projecting the objects detected in the panoramic video to obtain an object projection map of the three-dimensional space; adjusting the object projection map according to the scanning trajectory and the scale to obtain an object density map of the three-dimensional space.

5. The method according to claim 4, wherein, performing object detection on the three-dimensional space according to the panoramic video includes: detecting at least one of doors and windows on the room walls in the three-dimensional space according to the panoramic video to obtain door detection information and / or window detection information, where the door detection information includes at least one of door type, door opening direction, and door contour; and / or detecting objects placed on the room floor in the three-dimensional space according to the panoramic video to obtain object placement detection information.

6. The method according to claim 3, wherein, obtaining a room type density map of the three-dimensional space according to the panoramic video and the scanning trajectory includes: obtaining the room type of each room in the three-dimensional space according to the panoramic video; projecting according to the room type of each room in the three-dimensional space to obtain a room type projection map of the three-dimensional space; adjusting the room type projection map according to the scanning trajectory and the scale to obtain a room type density map of the three-dimensional space.

7. The method according to claim 1, wherein, Generating the floor plan of the three-dimensional space according to the layout density map of the three-dimensional space and in combination with at least one of the object density map and the room type density map of the three-dimensional space includes: Splicing the layout density map of the three-dimensional space and at least one of the object density map and the room type density map of the three-dimensional space as different channels, and using a floor plan generation model to generate the floor plan of the three-dimensional space.

8. A floor plan generation device Comprising: A scanning unit configured to scan a three-dimensional space including at least one room using a panoramic camera, obtain a panoramic video of the three-dimensional space, and obtain the scanning trajectory of the panoramic camera for the panoramic video; A first obtaining unit configured to obtain the layout density map of the three-dimensional space according to the panoramic video and the scanning trajectory; A second obtaining unit configured to obtain at least one of the object density map and the room type density map of the three-dimensional space according to the panoramic video and the scanning trajectory; A generating unit configured to generate the floor plan of the three-dimensional space according to the layout density map of the three-dimensional space and in combination with at least one of the object density map and the room type density map of the three-dimensional space.

9. A floor plan generation device Comprising: A processor; And a memory in which computer program instructions are stored, wherein, when the computer program instructions are run by the processor, the processor is caused to perform the following steps: Scanning a three-dimensional space including at least one room using a panoramic camera, obtaining a panoramic video of the three-dimensional space, and obtaining the scanning trajectory of the panoramic camera for the panoramic video; Obtaining the layout density map of the three-dimensional space according to the panoramic video and the scanning trajectory; Obtaining at least one of the object density map and the room type density map of the three-dimensional space according to the panoramic video and the scanning trajectory; Generating the floor plan of the three-dimensional space according to the layout density map of the three-dimensional space and in combination with at least one of the object density map and the room type density map of the three-dimensional space.

10. A computer-readable storage medium having computer program instructions stored thereon, wherein, when the computer program instructions are executed by a processor, the following steps are implemented: Scanning a three-dimensional space including at least one room using a panoramic camera, obtaining a panoramic video of the three-dimensional space, and obtaining the scanning trajectory of the panoramic camera for the panoramic video; Obtaining the layout density map of the three-dimensional space according to the panoramic video and the scanning trajectory; Obtaining at least one of the object density map and the room type density map of the three-dimensional space according to the panoramic video and the scanning trajectory; Generating the floor plan of the three-dimensional space according to the layout density map of the three-dimensional space and in combination with at least one of the object density map and the room type density map of the three-dimensional space.

Images