Floor plan generation method, device and computer-readable storage medium

Abstract

To provide a floor plan generating method, an apparatus, and a computer-readable storage medium.SOLUTION: A floor plan generating method according to an embodiment of the present invention includes the steps of: using a panoramic camera to scan a three-dimensional space including at least one room, acquiring a panoramic video of the three-dimensional space, and acquiring a scanning locus of the panoramic camera relating to the panoramic video; acquiring a layout density diagram of the three-dimensional space based on the panoramic video and the scanning locus; acquiring at least one of an object density diagram and a room-type density diagram of the three-dimensional space based on the panoramic video and the scanning locus; and generating a floor plan of the three-dimensional space based on the layout density diagram of the three-dimensional space and based on at least the object density diagram or the room type density diagram of the three-dimensional space.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present application relates to the field of image processing, and in particular to a floor plan generating method, apparatus, and computer-readable storage medium. [Background technology]

[0002] For a three-dimensional space including a room, the corresponding floor plan is a simple and clear display diagram of the room structure of the three-dimensional space. At present, how to easily and conveniently generate a floor plan corresponding to a three-dimensional space is a technical problem to be solved in the field of image processing.

[0003] In conventional methods, in order to generate a floor plan, calculations must be performed from a top view or horizontal cross section of a three-dimensional space. However, such methods usually require obtaining three-dimensional data for each room in the three-dimensional space, which is costly to obtain and requires complex calculations.

[0004] Therefore, there is a need for a simple and accurate method and apparatus that can automatically generate floor plans of three-dimensional spaces. Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present application is to provide a floor plan generating method, apparatus, and computer-readable storage medium. [Means for solving the problem]

[0006] In order to solve the technical problem, according to one aspect of the present invention, a floor plan generating method is provided, the floor plan generating method including the steps of: using a panoramic camera to scan a three-dimensional space including at least one room, obtaining a panoramic video of the three-dimensional space, and obtaining a scanning trajectory of the panoramic camera for the panoramic video; obtaining a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory; obtaining at least one of an object density map and a room type density map of the three-dimensional space based on the panoramic video and the scanning trajectory; and generating a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and further based on at least one of the object density map and the room type density map of the three-dimensional space.

[0007] According to another aspect of the present invention, there is provided a floor plan generating device, the floor plan generating device including: a scanning unit that uses a panoramic camera to scan a three-dimensional space including at least one room, acquires a panoramic video of the three-dimensional space, and acquires a scanning trajectory of the panoramic camera for the panoramic video; a first acquisition unit that acquires a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory; a second acquisition unit that acquires at least one of an object density map and a room type density map of the three-dimensional space based on the panoramic video and the scanning trajectory; and a generation unit that generates a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and further based on at least one of the object density map and the room type density map of the three-dimensional space.

[0008] According to another aspect of the present invention, there is provided a floor plan generating device comprising a processor and a memory in which computer program instructions are stored, wherein the processor executes the computer program instructions to cause the processor to perform the following steps: scanning a three-dimensional space including at least one room using a panoramic camera, acquiring a panoramic video of the three-dimensional space, and acquiring a scanning trajectory of the panoramic camera for the panoramic video; acquiring a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory; acquiring at least one of an object density map and a room type density map of the three-dimensional space based on the panoramic video and the scanning trajectory; and generating a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and further based on the object density map and at least one of the room type density map of the three-dimensional space.

[0009] According to another aspect of the present invention, there is provided a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, perform the following steps: scanning a three-dimensional space including at least one room using a panoramic camera, acquiring a panoramic video of the three-dimensional space, and acquiring a scanning trajectory of the panoramic camera for the panoramic video; acquiring a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory; acquiring at least one of an object density map and a room type density map of the three-dimensional space based on the panoramic video and the scanning trajectory; and generating a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and further based on the object density map and the room type density map of the three-dimensional space. Effect of the Invention

[0010] According to the floor plan generating method, device, and computer-readable storage medium of the embodiment of the present invention, a floor plan of a three-dimensional space can be generated by obtaining a layout density map of the three-dimensional space from a panoramic video obtained by scanning the three-dimensional space with a panoramic camera, and further obtaining at least one of an object density map and a room type density map. The floor plan generating method, device, and computer-readable storage medium of the embodiment of the present invention do not need to obtain and calculate three-dimensional data of each room in the three-dimensional space, which can reduce calculation costs and improve user experience.

[0011] In addition, in the floor plan generation method, device, and computer-readable storage medium according to the embodiments of the present invention, door detection information on the wall of a room in three-dimensional space is obtained from the object density map, and information such as the door opening direction is obtained, thereby enabling the floor plan of the three-dimensional space to be accurately generated and the floor plan generation method to be optimized. [Brief description of the drawings]

[0012] The above and other objects, features and advantages of the present invention will become more apparent from the detailed description of the embodiments of the present invention taken in conjunction with the drawings. [Figure 1] FIG. 1 shows a flowchart of a floor plan generating method according to an embodiment of the present invention. [Diagram 2] FIG. 2 shows a schematic diagram of a scanning trajectory of a panoramic camera according to an embodiment of the present invention. [Diagram 3] FIG. 3 shows a schematic diagram of a room layout according to an example embodiment of the present invention. [Figure 4] FIG. 4 shows a layout projection of a room in a three-dimensional space according to an example embodiment of the present invention. [Diagram 5] FIG. 5 shows a layout projection of the same room in a three-dimensional space according to an example embodiment of the present invention. [Figure 6] FIG. 6 shows an example of a modified layout projection obtained after scaling the two layout projections of the same room shown in FIG. [Figure 7] FIG. 7 illustrates an example of a layout density map of a three-dimensional space according to an embodiment of the present invention. [Figure 8] FIG. 8 shows a schematic diagram of two-dimensional detection of doors and windows according to an example of an embodiment of the present invention. [Figure 9] FIG. 9 is a schematic diagram of three-dimensional detection of a placed object according to an embodiment of the present invention. [Figure 10] FIG. 10 illustrates an object density diagram according to an example embodiment of the present invention. [Figure 11] FIG. 11 illustrates a room type density diagram according to an example embodiment of the present invention. [Figure 12] FIG. 12 illustrates an example of a floor plan of a three-dimensional space according to an embodiment of the present invention. [Figure 13] FIG. 13 shows a block diagram of a floor plan generating device according to an embodiment of the present invention. [Figure 14] FIG. 14 shows a block diagram of a floor plan generating device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, a floor plan generating method, an apparatus, and a computer-readable storage medium according to an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same reference numerals denote the same elements throughout. The embodiments described herein are merely examples and are not intended to limit the scope of the present invention.

[0014] Generally, to obtain a floor plan of a three-dimensional space including at least one room, it is necessary to first obtain three-dimensional data of each room in the three-dimensional space, and then generate a corresponding planar floor plan by calculating a top view or a horizontal cross-section view of the three-dimensional space, which always requires a large amount of three-dimensional data and calculation processes, and the steps are cumbersome.

[0015] In this regard, a floor plan generating method according to an embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 shows a flowchart of a floor plan generating method 100.

[0016] As shown in FIG. 1, in step S101, a panoramic camera is used to scan a three-dimensional space including at least one room, a panoramic video of the three-dimensional space is obtained, and a scanning trajectory of the panoramic camera for the panoramic video is obtained.

[0017] In this embodiment, the three-dimensional space may be a whole or a part of a three-dimensional building including at least one room, and accordingly, the generated floor plan may be a plan floor plan of the three-dimensional building.

[0018] In this embodiment, a panoramic video of a three-dimensional space can be acquired using a panoramic camera (e.g., a fisheye camera, a binocular camera, etc.). When acquiring the panoramic video of the three-dimensional space, position information and orientation information of the panoramic camera along with a time stamp can be further acquired based on the panoramic video, and a scanning trajectory of the panoramic camera can be generated. Optionally, the panoramic video can be input into a visual real-time localization and mapping (SLAM) such as OpenVSLAM to generate a trajectory of a camera scan in the panoramic video shooting process. Optionally, the format of the generated scanning trajectory of the panoramic camera can include a time stamp, a camera position information, and a camera orientation information. For example, assuming that the panoramic camera maintains a vertical state during shooting and the camera orientation remains unchanged, the trajectory may only include a time stamp Ti and a horizontal position (Xi, Yi) of the camera, represented as (Ti, Xi, Yi).

[0019] In step S102, a layout density map of the 3D space is obtained based on the panoramic video and the scanning trajectory.

[0020] In this embodiment, the step of obtaining a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory may include: projecting the panoramic video and obtaining a layout projection view of the panoramic video; associating the layout projection view with the scanning trajectory and determining a scale of the layout projection view; and obtaining the layout density map of the three-dimensional space based on the layout projection view and the scale.

[0021] Optionally, the video frames of the panoramic video can be first input to a model for room layout prediction, such as HorizonNet, and then projected to obtain a layout projection of each video frame in a three-dimensional space. In an embodiment of the present invention, the room layout can indicate the ceiling-wall boundaries and floor-wall boundaries of the room. Here, in the process of obtaining the layout projection using the video frames of the panoramic video, only some key frames in the panoramic video can be selected for calculation, thereby improving the computation efficiency.

[0022] After obtaining the layout projection, the time stamp and the position information of the camera corresponding to each video frame can be used to locate the corresponding camera position in the layout projection of the video frame, so as to associate the layout projection with the corresponding scanning trajectory. Then, based on the entropy minimization method, the relative scale between the layout projection of each video frame and the scanning trajectory can be corrected, and the layout projection of the same room shown in the calculation of each video frame is superimposed as much as possible based on the layout projection of each video frame and the scale determined after correction, to obtain the layout density map of the entire three-dimensional space.

[0023] In step S103, at least one of an object density map and a room type density map of the 3D space is obtained based on the panoramic video and the scanning trajectory.

[0024] In this embodiment, the video frames of the panoramic video are first input into a model for object detection, and then projected to obtain an object projection of each video frame in a three-dimensional space. In the embodiment of the present invention, performing object detection in the three-dimensional space may include performing two-dimensional detection of doors, windows, etc. on the wall of a room in the video frames, and may also include performing three-dimensional detection of objects (furniture, ornaments, etc.) placed on the floor of the room. Specifically, the method may include detecting at least one of doors and windows on the wall of the room in the three-dimensional space in each video frame based on the panoramic video, and obtaining door detection information and / or window detection information, where the door detection information includes at least one of door type, door opening direction, and door outline, and / or detecting objects on the floor of the room in the three-dimensional space in each video frame based on the panoramic video, and obtaining object detection information. Here, in the process of obtaining the object projection using the video frames of the panoramic video, only some key frames in the panoramic video can be selected for calculation, thereby improving the calculation efficiency.

[0025] After object detection is performed on each video frame, the detected object can be projected onto a layout projection to obtain a corresponding object projection. Specifically, a line segment corresponding to the floor-wall boundary of the room layout of the object detected on the wall surface in the room can be extracted first, and then projected onto the layout projection, and different types of objects can be displayed in different colors. Here, when projecting on the layout projection based on the door detection information, a line segment indicating the door opening direction can be projected onto the layout projection based on the door opening direction information contained in the door detection information, thereby providing more detailed information on the object projection, which is favorable for the accurate judgment and generation process of the floor plan. In addition, for the object obtained by 3D detection, the bottom surface of the 3D frame of the object detected on the ground in the room can be extracted, and then projected onto the layout projection, and similarly different types of objects can be displayed in different colors.

[0026] After obtaining the object projection view, the scale determined after associating each of the obtained video frames with the scanning trajectory can be used to adjust the object projection view of each video frame, and a superimposed object density view of the three-dimensional space can be obtained.

[0027] Optionally, when obtaining a room type density map, the video frames of the panoramic video may be first input into a room type classification model (e.g., a classifier based on VGG), and projection may be performed to obtain a room type projection map for each room of each video frame in a three-dimensional space. Here, in the process of obtaining the room type projection map using the video frames of the panoramic video, only some key frames in the panoramic video may be selected for calculation, thereby improving the computation efficiency.

[0028] After obtaining the room-type projection map, the scale determined after associating each of the obtained video frames with a scanning trajectory can be used to adjust the room-type projection map of each video frame, and a room-type density map of the three-dimensional space can be obtained by superimposing the room-type projection map.

[0029] In step S104, based on the layout density map of the three-dimensional space, moreover A floor plan of the three-dimensional space can be generated based on at least one of an object density map and a room type density map of the three-dimensional space.

[0030] In this embodiment, the layout density map of the three-dimensional space, and at least one of the object density map and room type density map of the three-dimensional space are connected as different channels and sent to a floor plan generation model, and the floor plan of the three-dimensional space is generated using the floor plan generation model. The floor plan generation model may be a generation method based on GAN, such as a detection method of RoomFormer, and the like, and is not limited here. The floor plan generation model of the embodiment of the present invention can display the polygonal shape, room type, and position of objects in the room (e.g., bed, sofa, door, window, etc.) of each room in the three-dimensional space based on at least one of the layout density map of the three-dimensional space, the object density map of the three-dimensional space, and the room type density map, where, if the input object density map includes information on the opening direction of the door in the door detection information, the generated floor plan can display information on the door with the opening direction accordingly. In addition, in the acquired floor plan of the three-dimensional space, rooms of the same type can also be displayed in the same color based on the related information of the room type density map.

[0031] According to the floor plan generating method of the embodiment of the present invention, a layout density map of the three-dimensional space is obtained using a panoramic video obtained by scanning the three-dimensional space with a panoramic camera, and at least one of an object density map and a room type density map is obtained, so that a floor plan of the three-dimensional space can be generated. The floor plan generating method of the embodiment of the present invention does not need to obtain and calculate three-dimensional data of each room in the three-dimensional space, so that the calculation cost can be reduced and the user experience can be improved.

[0032] Furthermore, in the floor plan generation method according to an embodiment of the present invention, door detection information on the wall of a room in three-dimensional space is obtained from the object density map, and information such as the door opening direction is obtained, thereby enabling the floor plan of the three-dimensional space to be accurately generated and the floor plan generation method to be optimized.

[0033] An example of a floor plan generating method according to an embodiment of the present invention will be described below.

[0034] In this example, a panoramic camera is first used to scan a three-dimensional space including a plurality of rooms, a panoramic video of the three-dimensional space is obtained, and a scanning trajectory of the panoramic camera for the panoramic video is obtained. Specifically, in this example, a panoramic camera can be used to obtain a panoramic video of the three-dimensional space. In addition, when the panoramic video of the three-dimensional space is obtained, the panoramic video can be further input into OpenVSLAM to generate a camera scanning trajectory in the shooting process of the panoramic video. Here, when the panoramic camera maintains a vertical state and the camera orientation does not change, the generated scanning trajectory of the panoramic camera only includes a time stamp Ti and a horizontal position (Xi, Yi) of the camera, and can be expressed as (Ti, Xi, Yi). FIG. 2 shows a schematic diagram of the scanning trajectory of the panoramic camera according to an embodiment of the present invention. As shown in FIG. 2, when the panoramic camera scans while maintaining a vertical state and orientation, the horizontal position coordinates corresponding to different time stamps can be used to show the scanning trajectory in the panoramic video scanning process of the panoramic camera.

[0035] Then, a layout density map of the three-dimensional space can be obtained based on the panoramic video and the scanning trajectory. Specifically, the panoramic video is first projected, a layout projection map of the panoramic video is obtained, the layout projection map is associated with the scanning trajectory, and a scale of the layout projection map is determined, and the layout density map of the three-dimensional space can be obtained based on the layout projection map and the scale.

[0036] In this example, the video frames of the panoramic video are first input into a model used for room layout prediction, such as HorizonNet, to obtain the layout of the room in three-dimensional space. The room layout may indicate the ceiling-wall and floor-wall boundaries of the room. Here, only some key frames in the panoramic video may be selected to obtain and calculate the floor plan, thereby improving the computation efficiency. FIG. 3 shows a schematic diagram of a room layout according to an example of an embodiment of the present invention. Specifically, in FIG. 3, the ceiling-wall and floor-wall boundaries extracted from the video frames of the input panoramic video are shown to represent the layout of the room in three-dimensional space.

[0037] After obtaining the layout of the room in the three-dimensional space, the layout can be projected to obtain a layout projection corresponding to each video frame. Figure 4 shows a layout projection of a room in the three-dimensional space according to an example of an embodiment of the present invention. As shown in Figure 4, the layout of the room can be projected onto a top view to form a corresponding room layout projection.

[0038] Then, the layout projection can be associated with the position of the scanning trajectory of the corresponding video frame by using the time stamp and the position information of the camera corresponding to each video frame. FIG. 5 shows an example of an embodiment of the present invention, which is a layout projection of the same room in a three-dimensional space obtained based on different video frames in a panoramic video and based on a scanning trajectory. As shown in FIG. 5, when the camera centers of different video frames at different times are loaded into the corresponding scanning trajectories, it can be seen that the layout projections of the same room cannot be efficiently superimposed due to the problem of the scale of the scanning trajectory based on the layout projections corresponding to each video frame (the difference between the two is t, as shown in FIG. 5). In contrast, in the process of scanning and calculation, the relative scale between the layout projections of each video frame and the scanning trajectory can be corrected based on the entropy minimization method, so that the layout projections of the same room shown in the calculation of each video frame can be superimposed as much as possible. FIG. 6 shows an example of a modified layout projection obtained after the scale adjustment of the two layout projections of the same room shown in FIG. 5. As shown in FIG. 6, the difference t between the two layout projections is the relative scale between the adjusted layout projections and the scanning trajectory, i.e., by introducing the coefficient s*, the relationship between the two layout projections can be adjusted so that the layout projections of the same room shown by each video frame calculation can be superimposed as much as possible.

[0039] After the modified scale parameter s* is determined by the above step, the corresponding scale of the layout projection of each video frame used for scanning trajectory is adjusted by a similar method, and the layout projection of each video frame is superimposed with the scale determined after adjustment, thereby obtaining the layout density map of the entire three-dimensional space. Figure 7 shows an example of a layout density map of a three-dimensional space according to an embodiment of the present invention. As shown in Figure 7, the layout projection of each key frame selected from the panoramic video can be superimposed according to the adjusted scale to obtain a layout density map of the superimposed three-dimensional space. The layout density map can show the rough room structure and floor plan outline of the three-dimensional space.

[0040] In addition to obtaining a layout density map of a three-dimensional space, at least one of an object density map and a room type density map of a three-dimensional space can be obtained based on the panoramic video and the scanning trajectory. In the process of obtaining the object density map and / or the room type density map, only some key frames in the panoramic video can be selected for calculation, thereby improving computation efficiency.

[0041] In addition, during the above acquisition process, the scale at which the object density map and the room type density map are superimposed can be adjusted according to the corresponding scale of the scanning trajectory of the layout projection map of each video frame previously used.

[0042] In the process of obtaining the object density map in the three-dimensional space, the video frames of the panoramic video can be input into the model for object detection first, and projection can be performed to obtain the object projection map of each video frame in the three-dimensional space. Optionally, two-dimensional detection can be performed for the doors and windows on the walls of the room in the video frames, and the detection information of the doors and windows can be obtained, and the detected objects can be projected onto the layout projection map after detection to obtain the corresponding object projection map.

[0043] FIG. 8 shows a schematic diagram of two-dimensional detection of doors and windows according to an embodiment of the present invention. The left diagram of FIG. 8 shows door detection information, window detection information, etc. obtained by detecting doors and windows on the wall of a room in a three-dimensional space from the panoramic video. Here, the door detection information can include at least one of door type (e.g., inward-opening door, outward-opening door, sliding door, etc.), door opening direction, and door contour (e.g., door frame). The window detection information can include information such as window contour. The right diagram of FIG. 8 shows a corresponding object projection obtained by projecting the detected object onto a layout projection. Specifically, first, the door or window detected on the wall in the room cuts out the corresponding line segment on the floor-wall boundary of the room layout, and projects it onto the layout projection, and further, different types of objects can be displayed with different colors. Here, when projecting based on the door detection information in the layout projection drawing, as shown in the right diagram of FIG. 8, a line indicating the door opening direction can be projected onto the layout projection drawing based on the door opening direction information contained in the door detection information, thereby providing more detailed information on the object projection and facilitating the accurate judgment and generation process of the floor plan.

[0044] In the object detection process, as an option, objects on the floor of the room in the three-dimensional space in each video frame can be detected based on the panoramic video to obtain object detection information. Then, for the objects obtained by the three-dimensional detection, the bottom surface of the three-dimensional frame of the object detected on the floor in the room can be cut out and projected onto a layout projection. FIG. 9 shows a schematic diagram of three-dimensional object detection according to an embodiment of the present invention. The left diagram of FIG. 9 shows object detection information obtained by detecting objects such as furniture on the floor of the room in the three-dimensional space from the panoramic video. The right diagram of FIG. 9 shows a corresponding object projection obtained by projecting the detected objects onto the layout projection. Specifically, first, a geometric shape corresponding to the object detected on the floor in the room is cut out and projected onto the layout projection, and further, different types of objects can be displayed in different colors, for example, a bed, a cabinet, a sofa, etc. can be displayed in different colors.

[0045] After the two-dimensional and three-dimensional detection and projection results are combined to obtain the object projection diagram, the scale adjusted after each determined video frame is associated with the scanning trajectory can be used to correct the object projection diagram of each video frame in a similar manner to the previous process of generating the layout projection diagram, and the object density diagram of the three-dimensional space can be obtained by superimposing the object projection diagram of each video frame. Figure 10 shows an object density diagram according to an example of an embodiment of the present invention. As shown in Figure 10, the object density diagram can include the distribution of two-dimensional objects (such as doors and windows) and / or three-dimensional objects (such as furniture) in each room in the three-dimensional space.

[0046] In addition, when obtaining a room type density map, the video frames of the panoramic video can be input into a room type classification model (e.g., a classifier based on VGG) first, and different room types can be projected with different colors, and a room type projection map for each room of each video frame in a three-dimensional space is obtained. After obtaining the room type projection map, the room type projection map of each video frame can be adjusted by using the scale determined after each obtained video frame is associated with a scanning trajectory, and a room type density map in a three-dimensional space can be obtained by superimposing it. Figure 11 shows a room type density map according to an example of an embodiment of the present invention. As shown in Figure 11, different room types in a three-dimensional space can be known from different colors.

[0047] Finally, a floor plan of a three-dimensional space may be generated according to a layout density map of the three-dimensional space in combination with at least one of an object density map and a room type density map of said three-dimensional space.

[0048] In this example, the layout density map of the three-dimensional space, and at least one of the object density map and the room type density map of the three-dimensional space are stitched together as different channels and fed into a floor plan generation model, and a floor plan generation model such as a generation method based on GAN or a detection method of RoomFormer is used to generate a floor plan of the three-dimensional space. FIG. 12 shows an example of a floor plan of a three-dimensional space according to an embodiment of the present invention. The floor plan of FIG. 12 is generated by a floor plan generation model after stitching together the layout density map, the object density map, and the room type density map of the three-dimensional space, which shows information such as the position, shape, etc. of each different room in the three-dimensional space, and shows the connection relationship between different rooms in detail by showing the outline of the direction of opening the door. In addition, in the acquired floor plan of the three-dimensional space, rooms of the same type can also be displayed in the same color based on the related information of the room type density map.

[0049] Hereinafter, a floor plan generating device according to an embodiment of the present invention will be described with reference to FIG. 13. FIG. 13 shows a block diagram of a floor plan generating device 1300 according to an embodiment of the present invention. As shown in FIG. 13, the floor plan generating device 1300 includes a scanning unit 1310, a first acquisition unit 1320, a second acquisition unit 1330, and a generating unit 1340. The floor plan generating device 1300 may include other components in addition to these units, but these components are not related to the contents of the embodiment of the present invention, so they will not be illustrated or described here. In addition, the details of the following operations performed by the floor plan generating device 1300 according to the embodiment of the present invention are the same as those described above with reference to FIGS. 1 to 12, so repeated description of the same details will be omitted.

[0050] The scanning unit 1310 of the floor plan generating device 1300 in FIG. 13 uses a panoramic camera to scan a three-dimensional space including at least one room, obtain a panoramic video of the three-dimensional space, and obtain a scanning trajectory of the panoramic camera for the panoramic video.

[0051] In this embodiment, the three-dimensional space may be a whole or a part of a three-dimensional building including at least one room, and accordingly, the generated floor plan may be a plan floor plan of the three-dimensional building.

[0052] In this embodiment, the scanning unit 1310 can use a panoramic camera (e.g., a fisheye camera, a binocular camera, etc.) to acquire a panoramic video in a three-dimensional space. In addition, when acquiring the panoramic video in the three-dimensional space, the scanning unit 1310 can further acquire position information and orientation information associated with a timestamp of the panoramic camera based on the panoramic video, and generate a scanning trajectory of the panoramic camera. Optionally, the panoramic video is input to a visual real-time localization and mapping (SLAM) such as OpenVSLAM to generate a trajectory of the camera scan in the panoramic video shooting process. Optionally, the format of the generated scanning trajectory of the panoramic camera can include a timestamp, a camera position information, and a camera orientation information. For example, assuming that the panoramic camera maintains a vertical state during shooting and the camera orientation remains unchanged, the trajectory may only include a timestamp Ti and a horizontal position (Xi, Yi) of the camera, represented as (Ti, Xi, Yi).

[0053] A first acquisition unit 1320 acquires a layout density map of a three-dimensional space according to the panoramic video and the scanning trajectory.

[0054] In this embodiment, a first acquisition unit 1320 projects the panoramic video, obtains a layout projection view of the panoramic video, associates the layout projection view with the scanning trajectory, and determines a scale of the layout projection view, and obtains the layout density view of the three-dimensional space based on the layout projection view and the scale.

[0055] Optionally, the first acquisition unit 1320 may first input the video frames of the panoramic video into a model for room layout prediction, such as HorizonNet, and then perform projection to obtain a layout projection of each video frame in three-dimensional space. In this embodiment, the room layout may indicate the ceiling-wall boundary and floor-wall boundary of the room. Here, in the process of using the video frames of the panoramic video to obtain the layout projection, only some key frames in the panoramic video may be selected for calculation, thereby improving computation efficiency.

[0056] After obtaining the layout projection, the first acquisition unit 1320 uses the time stamp and camera position information corresponding to each video frame to set the corresponding camera position in the layout projection of the video frame, and associates the layout projection with the corresponding scanning trajectory. Then, according to an entropy minimization method, the relative scale between the layout projection of each video frame and the scanning trajectory is modified, and the layout projection of the same room calculated and displayed by each video frame is superimposed as much as possible according to the scale determined after the modification, and the layout projection of each video frame is superimposed to obtain the layout density map of the entire three-dimensional space.

[0057] A second acquisition unit 1330 acquires at least one of an object density map and a room type density map of a three-dimensional space based on the panoramic video and the scanning trajectory.

[0058] In this embodiment, the second acquisition unit 1330 can first input the video frames of the panoramic video into a model for object detection and project them to obtain an object projection of each video frame in a three-dimensional space. In this embodiment of the present invention, performing object detection on the three-dimensional space may include performing two-dimensional detection on doors, windows, etc. on the wall of a room in a video frame, and may also include performing three-dimensional detection on objects (furniture, ornaments, etc.) placed on the floor of the room. Specifically, the second acquisition unit 1330 can include detecting at least one of doors and windows on the wall of the room in the three-dimensional space in each video frame based on the panoramic video, and obtaining door detection information and / or window detection information, where the door detection information includes at least one of door type, door opening direction, and door contour, and / or detecting objects on the floor of the room in the three-dimensional space in each video frame based on the panoramic video, and obtaining object detection information. Here, in the process of obtaining the object projection using the video frames of the panoramic video, only some key frames in the panoramic video can be selected and calculated, thereby improving calculation efficiency.

[0059] After performing object detection for each video frame, the second acquisition unit 1330 can project the detected object onto a layout projection to obtain a corresponding object projection. Specifically, first, a line segment corresponding to the floor-wall boundary of the room layout of the object detected on the wall surface in the room can be cut out, and then projected onto the layout projection, and different types of objects can be displayed in different colors. Here, when projecting based on the door detection information in the layout projection, a line segment indicating the door opening direction can be projected onto the layout projection based on the door opening direction information included in the door detection information, thereby providing more detailed information on the object projection, which is favorable for the accurate judgment and generation process of the floor plan. In addition, for the object acquired by the three-dimensional detection, the bottom surface of the three-dimensional frame of the object detected on the ground in the room can be cut out, and then projected onto the layout projection, and similarly, different types of objects can be displayed in different colors.

[0060] After acquiring the object projection view, the second acquisition unit 1330 can adjust the object projection view of each video frame using the scale determined after associating each acquired video frame with the scanning trajectory, and acquire an object density view of the three-dimensional space by superimposing it.

[0061] Optionally, when obtaining a room type density map, the second acquisition unit 1330 may first input the video frames of the panoramic video into a room type classification model (e.g., a classifier based on VGG), and perform projection to obtain a room type projection map for each room of each video frame in a three-dimensional space. Here, in the process of obtaining the room type projection map using the video frames of the panoramic video, only some key frames in the panoramic video can be selected and calculated, thereby improving computation efficiency.

[0062] After acquiring the room type projection map, the second acquisition unit 1330 can adjust the room type projection map of each video frame using the scale determined after associating each acquired video frame with a scanning trajectory, and acquire a room type density map of the three-dimensional space by superimposing it.

[0063] The generating unit 1340 may generate a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space, and further based on at least one of an object density map and a room type density map of the three-dimensional space.

[0064] In this embodiment, the generation unit 1340 stitches together the layout density map of the three-dimensional space, and at least one of the object density map and room type density map of the three-dimensional space as different channels, sends them to a floor plan generation model, and generates a floor plan of the three-dimensional space using the floor plan generation model. The floor plan generation model may be a generation method based on GAN, such as the detection method of RoomFormer, and the like, and is not limited here. The floor plan generation model of the embodiment of the present invention can display the polygonal shape, room type, and position of objects in the room (e.g., bed, sofa, door, window, etc.) of each room in the three-dimensional space based on at least one of the layout density map of the three-dimensional space, the object density map of the three-dimensional space, and the room type density map, where, if the input object density map includes information on the door opening direction in the door detection information, the generated floor plan can display door information with the opening direction accordingly. In addition, in the acquired three-dimensional space floor plan, rooms of the same type can also be displayed with the same color based on the related information of the room type density map.

[0065] According to the floor plan generating device of the embodiment of the present invention, a layout density map of the three-dimensional space is obtained using a panoramic video obtained by scanning the three-dimensional space with a panoramic camera, and at least one of an object density map and a room type density map is further obtained, so that a floor plan of the three-dimensional space can be generated. The floor plan generating device of the embodiment of the present invention does not need to obtain and calculate three-dimensional data of each room in the three-dimensional space, so that the calculation cost can be reduced and the user experience can be improved.

[0066] Furthermore, in the floor plan generation device according to an embodiment of the present invention, door detection information on the walls of a room in three-dimensional space can be obtained from the object density map, and information such as the door opening direction can be known, enabling the floor plan of the three-dimensional space to be accurately generated and the floor plan generation method to be optimized.

[0067] Next, a floor plan generating device according to an embodiment of the present invention will be described with reference to Fig. 14. Fig. 14 shows a block diagram of a floor plan generating device 1400 according to an embodiment of the present invention. As shown in Fig. 14, the device 1400 may be a computer or a server.

[0068] As shown in Fig. 14, the floor plan generating device 1400 includes one or more processors 1410 and memory 1420, and of course the floor plan generating device 1400 may further include input devices, output devices (not shown), etc., which may be interconnected by a bus system and / or other form of connection mechanism. It should be noted that the components and structure of the floor plan generating device 1400 shown in Fig. 14 are merely exemplary and not limiting, and the floor plan generating device 1400 may have other components and structures as needed.

[0069] The processor 141 may be a central processing unit (CPU) or other form of processing device having data processing and / or instruction execution capabilities, and may utilize computer program instructions stored in memory 1420 to perform desired functions, including scanning a three-dimensional space including at least one room using a panoramic camera to obtain a panoramic video of the three-dimensional space and obtain a scanning trajectory of the panoramic camera for the panoramic video, obtaining a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory, obtaining at least one of an object density map and a room type density map of the three-dimensional space based on the panoramic video and the scanning trajectory, and generating a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and further based on at least one of the object density map and the room type density map of the three-dimensional space.

[0070] The memory 1420 includes one or more computer program products, which may include various computer-readable storage media, such as volatile and / or non-volatile memory. One or more computer program instructions are stored in the computer-readable storage media. The processor 1410 executes the computer program instructions to realize the functions of the floor plan generating device according to the embodiment of the present invention and / or other desired functions, and / or to execute the floor plan generating method according to the embodiment of the present invention. In addition, various application programs and data may be stored in the computer-readable storage media.

[0071] A computer-readable storage medium according to an embodiment of the present invention is described below, which stores computer program instructions on the storage medium and is executed by a processor to perform the following steps: scanning a three-dimensional space including at least one room using a panoramic camera to obtain a panoramic video of the three-dimensional space and obtain a scanning trajectory of the panoramic camera for the panoramic video, obtaining a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory, obtaining at least one of an object density map and a room type density map of the three-dimensional space based on the panoramic video and the scanning trajectory, and generating a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and further based on at least one of the object density map and the room type density map of the three-dimensional space.

[0072] Of course, the above-mentioned embodiments are merely examples and are not limiting. Those skilled in the art can achieve the effects of the present invention by combining or integrating some processes or devices from each of the above-mentioned embodiments described separately based on the concept of the present invention. Such combined or combined embodiments are also included in the present invention, and detailed description thereof will be omitted here.

[0073] The advantages, benefits, and effects mentioned in the present invention are merely examples and are not limiting. Please note that these advantages, benefits, and effects cannot be considered to be necessarily provided by each embodiment of the present invention. In addition, the specific details of the invention are provided to show exemplary and easily understandable effects and are not limiting. In addition, the details described above do not limit the present invention from being realized using these details.

[0074] Block diagrams of devices, apparatus, facilities, and systems related to the present invention are merely illustrative examples and are not intended to require or suggest that they be connected, arranged, or configured in the manner shown in the block diagrams. As will be recognized by those skilled in the art, these devices, apparatus, facilities, and systems may be connected, arranged, or configured in any manner. Terms such as "include", "includes", and "having" are general terms meaning "including but not limited to" and are used interchangeably. In this specification, the terms "or" and "and" mean and are used interchangeably with the term "and / or" unless the context clearly indicates otherwise. Also, in this specification, the term "such as" means and is interchangeable with the phrase "such as but not limited to."

[0075] The flow of the present invention and the above method description are merely illustrative examples and are not intended to require or suggest that each embodiment be performed in the order shown. As will be appreciated by one of ordinary skill in the art, the steps of the above embodiments may be performed in any order. Terms such as "then," "and," and "next" are not intended to limit the order of the steps, but are used to guide the description of the method. It should be noted that any reference to a singular word using, for example, the articles "a," "one," or "the" does not limit the word to the singular.

[0076] In addition, the steps and devices relating to each embodiment in the present text are not limited to being performed in a specific embodiment, and a new embodiment can be constructed by combining some of the steps and some of the devices relating to each embodiment in the present text based on the spirit of the present invention, and is included in the scope of the present invention.

[0077] Each operation of the methods described above may be performed by any suitable means having the corresponding functionality, including but not limited to a circuit, an application specific integrated circuit (ASIC), or a processor, including various hardware and / or software components and / or modules.

[0078] Each of the illustrative logic blocks, modules, and circuits may be implemented or performed using a general purpose processor, a digital signal processor (DSP), an ASIC, a field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but alternatively, the processor may be any commercially available processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing equipment, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or multiple microprocessors in cooperation with a DSP core, or any other such configuration.

[0079] The steps of a method or algorithm according to the present invention may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may be stored in any form of tangible storage medium. Examples of available storage media include random access memory (RAM), read only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, etc. The storage medium may be coupled to the processor such that the processor can read information from and write information to the storage medium. Alternatively, the storage medium and the processor may be integrated. A software module may be either a single instruction or multiple instructions, and may be distributed over several different code segments, among different programs and across multiple storage media.

[0080] Additionally, the methods of the present invention may include one or more operations for implementing the method. The methods and / or operations may be interchanged without departing from the scope of the claims. In other words, unless otherwise specified, the order and / or use of the operations may be modified without departing from the scope of the claims.

[0081] The above-mentioned functions are implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions are stored as one or more instructions on a tangible computer-readable medium. The storage medium is any practical medium accessible by a computer. By way of example and not limitation, such computer-readable media include RAM, ROM, EEPROM, CD-ROM or other optical disks, magnetic disks or other magnetic memory devices, or any other physical medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disks as used herein include, for example, CDs (Compact Disks), Laser Disks, Optical Disks, DVDs (Digital Versatile Disks), Floppy Disks, and Blu-ray Disks.

[0082] Thus, a computer program product can perform the operations described herein. For example, such a computer program product is a computer-readable tangible medium having instructions tangibly stored (and / or encoded) thereon. The instructions are executed by one or more processors to implement the operations described herein. The computer program product may be included in a package.

[0083] The software or instructions may be transmitted over a transmission medium. For example, the software may be transmitted from a website, server, or other remote source using a transmission medium such as coaxial cable, fiber optic cable, twisted wire, Digital Subscriber Line (DSL), or wireless techniques such as infrared, radio, microwave, or the like.

[0084] Modules and / or other suitable means for performing the methods and techniques described above may be downloaded and / or otherwise obtained by the user terminal and / or base station as appropriate. For example, such equipment may be connected to a server to provide for the transmission of means for performing the methods described above. Alternatively, the various methods described above may be provided via storage means (e.g. physical storage media such as RAM, ROM, CD or floppy) and the user terminal and / or base station may obtain the various methods when coupled to or providing the storage means to the equipment. Furthermore, any other suitable technology may be used to provide the methods and techniques described above to the equipment.

[0085] Other examples and implementations are within the scope and spirit of the present invention and the claims. For example, due to the nature of software, the functions described above may be implemented by software executed by a processor, hardware, firmware, hardwire, or any combination thereof. Features implementing the functions may be located in various physical locations. This includes functions distributed such that portions of the functions are implemented in different physical locations. As used herein and in the claims, "or" used in a list preceding "at least one" refers to a disjunctive list, e.g., the list "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, "exemplary" does not imply that the described example is preferred or preferable over other examples.

[0086] Various changes, substitutions and modifications to the techniques described above may be made without departing from the limitations of the claims. Furthermore, the scope of the claims is not limited to the specific processes, machines, manufacture, combinations of events, means, methods and acts described above. Any processes, machines, manufacture, combinations of events, means, methods and acts now known or later developed that perform substantially the same functions or achieve substantially the same results as those described herein may be utilized. Thus, the scope of the claims includes any processes, machines, manufacture, combinations of events, means, methods or acts that fall within their scope.

[0087] The above description is provided to enable one skilled in the art to use or realize the present invention. Various modifications to these descriptions will be apparent to those skilled in the art, and the general principles defined herein may be applied elsewhere without departing from the scope of the present invention. Therefore, the present invention is not limited to the above description, but is to be accorded the widest scope consistent with the concept and novel features of the invention.

[0088] The foregoing description has been presented for purposes of illustration and description, but is not intended to be limiting, and variations, additions, modifications and subcombinations of the embodiments described above will occur to those skilled in the art.

Claims

1. 1. A computer implemented method for generating a floor plan, comprising: Using a panoramic camera to scan a three-dimensional space including at least one room, to obtain a panoramic video of the three-dimensional space, and to obtain a scanning trajectory of the panoramic camera for the panoramic video; obtaining a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory; obtaining at least one of an object density map and a room type density map of the three-dimensional space based on the panoramic video and the scanning trajectory; generating a floor plan of the three-dimensional space based on a layout density map of the three-dimensional space and further based on at least one of an object density map and a room type density map of the three-dimensional space; A floor plan generating method comprising:

2. The steps of scanning a three-dimensional space including at least one room using a panoramic camera to obtain a panoramic video of the three-dimensional space, and obtaining a scanning trajectory of the panoramic camera for the panoramic video include: The floor plan generating method according to claim 1 , further comprising: acquiring position information and attitude information associated with a time stamp of the panoramic camera based on the panoramic video, and generating a scanning trajectory of the panoramic camera.

3. The step of obtaining a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory includes: projecting the panoramic video and obtaining a layout projection of the panoramic video; Associating the layout projection with the scanning trajectory and determining a scale of the layout projection; and obtaining the layout density map of the three-dimensional space based on the layout projection map and the scale.

4. The step of obtaining an object density map of the three-dimensional space based on the panoramic video and the scanning trajectory includes: performing object detection in the three-dimensional space based on the panoramic video; projecting the detected object in the panoramic video to obtain a projection of the object in the three-dimensional space; and adjusting the object projection view based on the scanning trajectory and the scale to obtain an object density view of the three-dimensional space.

5. Performing object detection on the three-dimensional space based on the panoramic video includes: Detecting at least one of a door and a window on a wall surface of the room in the three-dimensional space based on the panoramic video, and obtaining door detection information and / or window detection information, the door detection information including at least one of a door type, a door opening direction, and a door contour; and / or The floor plan generating method according to claim 4 , further comprising: detecting objects on a floor surface of the room in the three-dimensional space based on the panoramic video, and acquiring object detection information.

6. The step of obtaining a room type density map of the three-dimensional space based on the panoramic video and the scanning trajectory includes: obtaining a room type for each room in the three-dimensional space based on the panoramic video; performing projection based on a room type of each room in the three-dimensional space to obtain a room type projection drawing of the three-dimensional space; and adjusting the room-type projection map based on the scanning trajectory and the scale to obtain a room-type density map of the three-dimensional space.

7. generating a floor plan of the three-dimensional space based on a layout density map of the three-dimensional space and based on at least one of an object density map and a room type density map of the three-dimensional space, 2. The floor plan generating method of claim 1, further comprising stitching together at least one of the layout density map of the three-dimensional space, and the object density map and room type density map of the three-dimensional space as different channels, and generating a floor plan of the three-dimensional space using a floor plan generation model.

8. A floor plan generating device, comprising: a scanning unit for scanning a three-dimensional space including at least one room using a panoramic camera to obtain a panoramic video of the three-dimensional space and obtain a scanning trajectory of the panoramic camera with respect to the panoramic video; a first acquisition unit for acquiring a layout density map of the three-dimensional space according to the panoramic video and the scanning trajectory; a second acquisition unit for acquiring at least one of an object density map and a room type density map of the three-dimensional space based on the panoramic video and the scanning trajectory; a generating unit for generating a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space, and further based on at least one of an object density map and a room type density map of the three-dimensional space; A floor plan generating device comprising:

9. A processor; A floor plan generator comprising: Execution of the computer program instructions by the processor causes the processor to: Using a panoramic camera to scan a three-dimensional space including at least one room, to obtain a panoramic video of the three-dimensional space, and to obtain a scanning trajectory of the panoramic camera for the panoramic video; obtaining a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory; obtaining at least one of an object density map and a room type density map of the three-dimensional space based on the panoramic video and the scanning trajectory; generating a floor plan of the three-dimensional space based on a layout density map of the three-dimensional space and further based on at least one of an object density map and a room type density map of the three-dimensional space; A floor plan generating device characterized by executing the above.

10. A computer-readable storage medium, A computer program instruction is stored, and the computer program instruction is executed by a processor to Using a panoramic camera to scan a three-dimensional space including at least one room, to obtain a panoramic video of the three-dimensional space, and to obtain a scanning trajectory of the panoramic camera for the panoramic video; obtaining a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory; obtaining at least one of an object density map and a room type density map of the three-dimensional space based on the panoramic video and the scanning trajectory; generating a floor plan of the three-dimensional space based on a layout density map of the three-dimensional space and further based on at least one of an object density map and a room type density map of the three-dimensional space; A computer-readable storage medium for executing the method.

11. A program for causing a computer to execute the floor plan generating method according to any one of claims 1 to 7.

Images