Floor plan generation method, device, and computer-readable storage medium

Abstract

To provide a floor plan generating method, an apparatus, and a computer-readable storage medium.SOLUTION: A floor plan generating method according to an embodiment of the present invention includes the steps of: using a panoramic camera to scan a three-dimensional space including at least one room, acquiring a panoramic video of the three-dimensional space, and acquiring a scanning locus of the panoramic camera relating to the panoramic video; acquiring a layout density diagram of the three-dimensional space based on the panoramic video and the scanning locus; acquiring at least one of an object density diagram and a room-type density diagram of the three-dimensional space based on the panoramic video and the scanning locus; and generating a floor plan of the three-dimensional space based on the layout density diagram of the three-dimensional space and based on at least the object density diagram or the room type density diagram of the three-dimensional space.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present application relates to the field of image processing, and in particular to a floor plan generation method, apparatus, and computer-readable storage medium. [Background technology]

[0002] For a three-dimensional space including rooms, the corresponding floor plan is a simple and clear display diagram of the room structure of the three-dimensional space. At present, how to easily and conveniently generate a floor plan corresponding to a three-dimensional space is a technical problem to be solved in the field of image processing.

[0003] In conventional methods, to generate a floor plan, calculations are generally required from a top view or horizontal cross section of a three-dimensional space. However, such methods typically require the acquisition of three-dimensional data for each room in the three-dimensional space, which is costly to acquire and requires complex calculations.

[0004] Therefore, there is a need for a simple and accurate method and apparatus that can automatically generate floor plans of three-dimensional spaces. Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present application is to provide a floor plan generation method, an apparatus, and a computer-readable storage medium. [Means for solving the problem]

[0006] In order to solve the above technical problems, according to one aspect of the present invention, there is provided a floor plan generation method, comprising the steps of: using a panoramic camera to scan a three-dimensional space including at least one room, obtaining a panoramic video of the three-dimensional space, and obtaining a scanning trajectory of the panoramic camera for the panoramic video; obtaining a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory; obtaining at least one of an object density map and a room type density map of the three-dimensional space based on the panoramic video and the scanning trajectory; and generating a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and further based on at least one of the object density map and the room type density map of the three-dimensional space.

[0007] According to another aspect of the present invention, there is provided a floor plan generation device including: a scanning unit that uses a panoramic camera to scan a three-dimensional space including at least one room, acquires a panoramic video of the three-dimensional space, and acquires a scanning trajectory of the panoramic camera for the panoramic video; a first acquisition unit that acquires a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory; a second acquisition unit that acquires at least one of an object density map and a room type density map of the three-dimensional space based on the panoramic video and the scanning trajectory; and a generation unit that generates a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and further based on at least one of the object density map and the room type density map of the three-dimensional space.

[0008] According to another aspect of the present invention, there is provided a floor plan generation device comprising a processor and a memory in which computer program instructions are stored, wherein the execution of the computer program instructions by the processor causes the processor to perform the following steps: scanning a three-dimensional space including at least one room using a panoramic camera to obtain a panoramic video of the three-dimensional space and obtain a scanning trajectory of the panoramic camera for the panoramic video; obtaining a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory; obtaining at least one of an object density map and a room type density map of the three-dimensional space based on the panoramic video and the scanning trajectory; and generating a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and further based on at least one of the object density map and the room type density map of the three-dimensional space.

[0009] According to another aspect of the present invention, there is provided a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, perform the following steps: scanning a three-dimensional space including at least one room using a panoramic camera to obtain a panoramic video of the three-dimensional space and obtain a scanning trajectory of the panoramic camera for the panoramic video; obtaining a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory; obtaining at least one of an object density map and a room type density map of the three-dimensional space based on the panoramic video and the scanning trajectory; and generating a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and further based on at least one of the object density map and the room type density map of the three-dimensional space. [Effects of the Invention]

[0010] According to the floor plan generating method, device, and computer-readable storage medium of the present invention, a floor plan of a three-dimensional space can be generated by acquiring a layout density map of the three-dimensional space from a panoramic video obtained by scanning the three-dimensional space using a panoramic camera, and further acquiring at least one of an object density map and a room type density map. The floor plan generating method, device, and computer-readable storage medium of the present invention do not require acquiring and calculating three-dimensional data for each room in the three-dimensional space, thereby reducing calculation costs and improving user experience.

[0011] In addition, in the floor plan generation method, device, and computer-readable storage medium according to the embodiments of the present invention, door detection information on the wall of a room in three-dimensional space is obtained from the object density map, and information such as the door opening direction is obtained, thereby enabling the accurate generation of a floor plan in three-dimensional space and optimizing the floor plan generation method. [Brief explanation of the drawings]

[0012] The above and other objects, features and advantages of the present invention will become more apparent from the detailed description of the embodiments of the present invention taken in conjunction with the drawings. [Figure 1] FIG. 1 shows a flowchart of a floor plan generating method according to an embodiment of the present invention. [Figure 2] FIG. 2 shows a schematic diagram of a scanning trajectory of a panoramic camera according to an embodiment of the present invention. [Figure 3] FIG. 3 shows a schematic diagram of a room layout according to an example embodiment of the present invention. [Figure 4] FIG. 4 shows a layout projection of a room in a three-dimensional space according to an example embodiment of the present invention. [Figure 5] FIG. 5 shows a layout projection of the same room in three-dimensional space according to an example embodiment of the present invention. [Figure 6] FIG. 6 shows an example of a modified layout projection obtained after scaling the two layout projections of the same room shown in FIG. [Figure 7] FIG. 7 illustrates an example of a layout density map of a three-dimensional space according to an embodiment of the present invention. [Figure 8] FIG. 8 shows a schematic diagram of two-dimensional detection of doors and windows according to an example embodiment of the present invention. [Figure 9] FIG. 9 is a schematic diagram of three-dimensional detection of a placed object according to an embodiment of the present invention. [Figure 10] FIG. 10 shows an object density diagram according to an example embodiment of the present invention. [Figure 11] FIG. 11 illustrates a room type density diagram according to an example embodiment of the present invention. [Figure 12] FIG. 12 shows an example of a floor plan of a three-dimensional space according to an embodiment of the present invention. [Figure 13] FIG. 13 shows a block diagram of a floor plan generating device according to an embodiment of the present invention. [Figure 14] FIG. 14 shows a block diagram of a floor plan generating device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] A floor plan generating method, an apparatus, and a computer-readable storage medium according to embodiments of the present invention will be described below with reference to the drawings. In the drawings, like reference numerals denote like elements throughout. The embodiments described herein are merely examples and are not intended to limit the scope of the present invention.

[0014] Generally, to obtain a floor plan of a three-dimensional space including at least one room, it is necessary to first obtain three-dimensional data of each room in the three-dimensional space, and then generate a corresponding planar floor plan by calculating a top view or horizontal cross section of the three-dimensional space, which always requires a large amount of three-dimensional data and calculation processes, and the steps are cumbersome.

[0015] In this regard, a floor plan generating method according to an embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 shows a flowchart of a floor plan generating method 100.

[0016] As shown in FIG. 1, in step S101, a panoramic camera is used to scan a three-dimensional space including at least one room, a panoramic video of the three-dimensional space is obtained, and a scanning trajectory of the panoramic camera for the panoramic video is obtained.

[0017] In this embodiment, the three-dimensional space may be a whole or part of a three-dimensional building including at least one room, and accordingly, the generated floor plan may be a plan floor plan of the three-dimensional building.

[0018] In this embodiment, a panoramic video of a three-dimensional space can be captured using a panoramic camera (e.g., a fisheye camera, a binocular camera, etc.). Furthermore, when capturing the panoramic video of a three-dimensional space, position information and orientation information of the panoramic camera along with a timestamp can be further acquired based on the panoramic video, and a scanning trajectory of the panoramic camera can be generated. Optionally, the panoramic video can be input into a visual real-time localization and mapping (SLAM) software such as OpenVSLAM to generate a camera scanning trajectory during the panoramic video capture process. Optionally, the format of the generated panoramic camera scanning trajectory can include a timestamp, camera position information, and camera orientation information. For example, assuming that the panoramic camera maintains a vertical state during capture and the camera orientation remains unchanged, the trajectory can only include a timestamp Ti and the horizontal position (Xi, Yi) of the camera, represented as (Ti, Xi, Yi).

[0019] In step S102, a layout density map of the three-dimensional space is obtained based on the panoramic video and the scanning trajectory.

[0020] In this embodiment, the step of obtaining a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory can include: projecting the panoramic video and obtaining a layout projection map of the panoramic video; associating the layout projection map with the scanning trajectory and determining a scale of the layout projection map; and obtaining the layout density map of the three-dimensional space based on the layout projection map and the scale.

[0021] Optionally, video frames of the panoramic video can be first input to a model used for room layout prediction, such as HorizonNet, and then projected to obtain a layout projection of each video frame in 3D space. In an embodiment of the present invention, the room layout can indicate the ceiling-wall boundaries and floor-wall boundaries of the room. Here, in the process of obtaining the layout projection using video frames of the panoramic video, only some key frames in the panoramic video can be selected for calculation, thereby improving computational efficiency.

[0022] After obtaining the layout projection, the timestamp and camera position information corresponding to each video frame can be used to locate the corresponding camera position in the layout projection of the video frame, thereby associating the layout projection with the corresponding scanning trajectory. Then, based on an entropy minimization method, the relative scale between the layout projection of each video frame and the scanning trajectory can be corrected, and the layout projection of the same room shown in the calculation for each video frame is superimposed as much as possible based on the scale determined after correction with the layout projection of each video frame to obtain the layout density map of the entire three-dimensional space.

[0023] In step S103, at least one of an object density map and a room type density map of the 3D space is obtained based on the panoramic video and the scanning trajectory.

[0024] In this embodiment, video frames of a panoramic video are first input into an object detection model and then projected to obtain an object projection of each video frame in a three-dimensional space. In this embodiment of the present invention, performing object detection in the three-dimensional space may include performing two-dimensional detection of doors, windows, etc. on the walls of a room in the video frames, or performing three-dimensional detection of objects (furniture, ornaments, etc.) placed on the floor of the room. Specifically, the method may include detecting at least one of doors and windows on the walls of the room in the three-dimensional space in each video frame based on the panoramic video to obtain door detection information and / or window detection information, where the door detection information includes at least one of door type, door opening direction, and door outline, and / or detecting objects on the floor of the room in the three-dimensional space in each video frame based on the panoramic video to obtain object detection information. Here, in the process of obtaining the object projection using the video frames of the panoramic video, only some key frames in the panoramic video can be selected for calculation, thereby improving computational efficiency.

[0025] After object detection is performed on each video frame, the detected objects can be projected onto a layout projection to obtain a corresponding object projection. Specifically, first, line segments corresponding to the floor-wall boundaries of the room layout of the detected objects on the wall surfaces within the room are extracted and projected onto the layout projection, and different types of objects can be displayed in different colors. Here, when projecting onto the layout projection based on door detection information, line segments indicating the door opening direction can be projected onto the layout projection based on the door opening direction information contained in the door detection information, thereby providing more detailed information about the object projection and facilitating the accurate determination and generation of floor plans. Furthermore, for objects obtained through 3D detection, the bottom surfaces of the 3D frames of objects detected on the ground within the room can be extracted and projected onto the layout projection, and different types of objects can be displayed in different colors.

[0026] After obtaining the object projection view, the scale determined after associating each of the obtained video frames with the scanning trajectory can be used to adjust the object projection view of each video frame, and a superimposed object density view of the three-dimensional space can be obtained.

[0027] Optionally, when obtaining a room type density map, the video frames of the panoramic video can be first input into a room type classification model (e.g., a classifier based on VGG) and projection can be performed to obtain a room type projection map for each room in each video frame in three-dimensional space. Here, in the process of obtaining the room type projection map using the video frames of the panoramic video, only some key frames in the panoramic video can be selected for calculation, thereby improving computational efficiency.

[0028] After obtaining the room type projection map, the scale determined after associating each of the obtained video frames with the scanning trajectory can be used to adjust the room type projection map of each video frame, and a room type density map of the three-dimensional space can be obtained by superimposing it.

[0029] In step S104, based on the layout density map of the three-dimensional space, moreover A floor plan of the three-dimensional space can be generated based on at least one of an object density map and a room type density map of the three-dimensional space.

[0030] In this embodiment, the three-dimensional space layout density map and at least one of the three-dimensional space object density map and room type density map are connected as separate channels and sent to a floor plan generation model, which then generates a floor plan of the three-dimensional space using the floor plan generation model. The floor plan generation model may be a GAN-based generation method, such as the RoomFormer detection method, but is not limited thereto. The floor plan generation model of this embodiment of the present invention can display the polygonal shape, room type, and location of objects (e.g., beds, sofas, doors, windows, etc.) in each room in the three-dimensional space based on the three-dimensional space layout density map and at least one of the three-dimensional space object density map and room type density map. If the input object density map includes door opening direction information in the door detection information, the generated floor plan can display door information with opening direction accordingly. In addition, in the acquired three-dimensional space floor plan, rooms of the same type can be displayed in the same color based on the related information in the room type density map.

[0031] According to a floor plan generating method of an embodiment of the present invention, a layout density map of the three-dimensional space is obtained using a panoramic video acquired by scanning the three-dimensional space with a panoramic camera, and at least one of an object density map and a room type density map is further obtained to generate a floor plan of the three-dimensional space. The floor plan generating method of the embodiment of the present invention does not require acquiring and calculating three-dimensional data for each room in the three-dimensional space, thereby reducing calculation costs and improving user experience.

[0032] Furthermore, in the floor plan generation method according to an embodiment of the present invention, door detection information on the wall of a room in three-dimensional space is obtained from the object density map, and information such as the door opening direction is obtained, thereby enabling the accurate generation of a floor plan in three-dimensional space and optimizing the floor plan generation method.

[0033] An example of a floor plan generating method according to an embodiment of the present invention will be described below.

[0034] In this example, a panoramic camera is first used to scan a three-dimensional space including multiple rooms, to obtain a panoramic video of the three-dimensional space, and a scanning trajectory of the panoramic camera for the panoramic video is then obtained. Specifically, in this example, a panoramic camera is used to obtain a panoramic video of the three-dimensional space. Furthermore, when the panoramic video of the three-dimensional space is obtained, the panoramic video can be input into OpenVSLAM to generate a camera scanning trajectory during the panoramic video shooting process. Here, if the panoramic camera maintains a vertical state and the camera orientation does not change, the generated panoramic camera scanning trajectory only includes timestamp Ti and the camera's horizontal position (Xi, Yi), and can be expressed as (Ti, Xi, Yi). Figure 2 shows a schematic diagram of a panoramic camera scanning trajectory according to an embodiment of the present invention. As shown in Figure 2, when the panoramic camera maintains a vertical state and scans while maintaining its orientation, the scanning trajectory during the panoramic video scanning process of the panoramic camera can be represented using horizontal position coordinates corresponding to different timestamps.

[0035] Then, a layout density map of the three-dimensional space can be obtained based on the panoramic video and the scanning trajectory. Specifically, the panoramic video is first projected, a layout projection map of the panoramic video is obtained, the layout projection map is associated with the scanning trajectory, and a scale of the layout projection map is determined. The layout density map of the three-dimensional space can be obtained based on the layout projection map and the scale.

[0036] In this example, video frames of a panoramic video are first input into a model used for room layout prediction, such as HorizonNet, to obtain the room layout in three-dimensional space. The room layout may indicate the ceiling-wall and floor-wall boundaries of the room. Here, only some key frames in the panoramic video can be selected to acquire and calculate the floor plan, thereby improving computational efficiency. FIG. 3 shows a schematic diagram of a room layout according to an example embodiment of the present invention. Specifically, FIG. 3 shows the ceiling-wall and floor-wall boundaries extracted from the video frames of the input panoramic video to represent the room layout in three-dimensional space.

[0037] After obtaining the room layout in the three-dimensional space, the layout can be projected to obtain a layout projection corresponding to each video frame. Figure 4 shows a layout projection of a room in the three-dimensional space according to an example embodiment of the present invention. As shown in Figure 4, the room layout can be projected onto a top view to form a corresponding room layout projection.

[0038] Then, the layout projections can be associated with the position of the scanning trajectory in the corresponding video frame using the timestamp and camera position information corresponding to each video frame. Figure 5 shows an example of an embodiment of the present invention, illustrating layout projections of the same room in three-dimensional space obtained based on different video frames in a panoramic video and based on the scanning trajectory. As shown in Figure 5, when the camera centers of different video frames at different times are loaded into the corresponding scanning trajectories, it can be seen that the layout projections of the same room cannot be efficiently superimposed due to the scale issue of the scanning trajectories based on the layout projections corresponding to each video frame (the difference between the two is t, as shown in Figure 5). In contrast, during the scanning and calculation process, the relative scale between the layout projections of each video frame and the scanning trajectory can be corrected based on the entropy minimization method, so that the layout projections of the same room shown in the calculation of each video frame can be superimposed as much as possible. Figure 6 shows an example of a corrected layout projection obtained after scaling the two layout projections of the same room shown in Figure 5. As shown in Figure 6, the difference t between the two layout projections can be adjusted by introducing the relative scale between the adjusted layout projections and the scanning trajectory, i.e., the coefficient s*, to adjust the relationship between the two layout projections, so that the layout projections of the same room shown by each video frame calculation can be superimposed as much as possible.

[0039] After determining the modified scale parameter s* through the above step, the corresponding scale of the layout projection view of each video frame used for the scanning trajectory is adjusted in a similar manner, and the layout projection views corresponding to each video frame are superimposed at the determined scale after adjustment, thereby obtaining the layout density view of the entire three-dimensional space. Figure 7 shows an example of a layout density view of a three-dimensional space according to an embodiment of the present invention. As shown in Figure 7, the layout projection views of each key frame selected from the panoramic video are superimposed according to the adjusted scale to obtain the superimposed layout density view of the three-dimensional space. The layout density view can show the rough room structure and floor plan outline of the three-dimensional space.

[0040] In addition to obtaining a layout density map of the 3D space, at least one of an object density map and a room type density map of the 3D space can be obtained based on the panoramic video and the scanning trajectory. In the process of obtaining the object density map and / or the room type density map, only some key frames in the panoramic video can be selected for calculation, thereby improving computation efficiency.

[0041] In addition, during the above acquisition process, the scale at which the object density map and the room type density map are superimposed can be adjusted to match the corresponding scale of the scanning trajectory of the layout projection map of each video frame previously used.

[0042] In the process of obtaining the object density map in the three-dimensional space, the video frames of the panoramic video can be first input into the object detection model, and projection can be performed to obtain the object projection map of each video frame in the three-dimensional space. Optionally, two-dimensional detection can be performed on the doors and windows on the walls of the rooms in the video frames to obtain the detection information of the doors and windows. After detection, the detected objects can be projected onto the layout projection map to obtain the corresponding object projection map.

[0043] FIG. 8 is a schematic diagram of two-dimensional door and window detection according to an embodiment of the present invention. The left diagram of FIG. 8 shows door detection information, window detection information, etc., obtained by detecting doors and windows on the walls of a room in three-dimensional space from the panoramic video. Here, the door detection information can include at least one of the door type (e.g., inward-opening door, outward-opening door, sliding door, etc.), door opening direction, and door outline (e.g., door frame). The window detection information can include information such as window outline. The right diagram of FIG. 8 shows a corresponding object projection obtained by projecting the detected object onto a layout projection. Specifically, first, a door or window detected on a wall in the room is cut out, and the corresponding line segment on the floor-wall boundary of the room layout is projected onto the layout projection. Furthermore, different types of objects can be displayed in different colors. Here, when projecting based on the door detection information in the layout projection drawing, as shown in the right diagram of Figure 8, based on the door opening direction information contained in the door detection information, a line segment indicating the door opening direction can be projected onto the layout projection drawing, thereby providing more detailed information about the object projection and facilitating the accurate determination and generation process of the floor plan.

[0044] During the object detection process, optionally, objects on the floor of the room in the three-dimensional space can be detected in each video frame based on the panoramic video to obtain object detection information. Then, for the objects detected through 3D detection, the bottom of the 3D frame of the object detected on the floor in the room can be cropped and projected onto a layout projection. Figure 9 shows a schematic diagram of 3D object detection according to an embodiment of the present invention. The left diagram of Figure 9 shows object detection information obtained by detecting objects such as furniture on the floor of the room in the three-dimensional space from the panoramic video. The right diagram of Figure 9 shows a corresponding object projection obtained by projecting the detected objects onto the layout projection. Specifically, geometric shapes corresponding to the objects detected on the floor in the room can be cropped and projected onto the layout projection. Different types of objects can be displayed in different colors, such as beds, cabinets, and sofas.

[0045] After combining the two-dimensional and three-dimensional detection and projection results to obtain the object projection map, the scale adjusted after each determined video frame is associated with the scanning trajectory can be used to correct the object projection map of each video frame, in a similar manner to the previous process of generating the layout projection map, and the object density map of the three-dimensional space can be obtained by overlaying it. Figure 10 shows an object density map according to an example embodiment of the present invention. As shown in Figure 10, the object density map can include the distribution of two-dimensional objects (doors, windows, etc.) and / or three-dimensional objects (furniture, etc.) contained in each room in the three-dimensional space.

[0046] In addition, when obtaining a room type density map, the video frames of the panoramic video can first be input into a room type classification model (e.g., a VGG-based classifier), and different room types can be projected with different colors to obtain a room type projection map for each room in each video frame in three-dimensional space. After obtaining the room type projection map, the scale determined after each captured video frame is associated with the scanning trajectory can be used to adjust the room type projection map for each video frame and obtain a room type density map in three-dimensional space by superimposing it. Figure 11 shows a room type density map according to an example embodiment of the present invention. As shown in Figure 11, different colors can indicate different room types in three-dimensional space.

[0047] Finally, a floor plan of a three-dimensional space may be generated according to a layout density map of the three-dimensional space in combination with at least one of an object density map and a room type density map of the three-dimensional space.

[0048] In this example, the three-dimensional space layout density map and at least one of the three-dimensional space object density map and room type density map are stitched together as separate channels and fed into a floor plan generation model. A floor plan of the three-dimensional space is generated using a floor plan generation model, such as a GAN-based generation method or the RoomFormer detection method. Figure 12 shows an example of a three-dimensional space floor plan according to an embodiment of the present invention. The floor plan of Figure 12 is generated by a floor plan generation model after stitching together the three-dimensional space layout density map, object density map, and room type density map. It shows information such as the location and shape of each different room in the three-dimensional space, and outlines the door opening directions to show the connection relationships between different rooms in detail. In addition, in the acquired three-dimensional space floor plan, rooms of the same type can be displayed in the same color based on the related information in the room type density map.

[0049] A floor plan generating device according to an embodiment of the present invention will be described below with reference to FIG. 13. FIG. 13 shows a block diagram of a floor plan generating device 1300 according to an embodiment of the present invention. As shown in FIG. 13, the floor plan generating device 1300 includes a scanning unit 1310, a first acquisition unit 1320, a second acquisition unit 1330, and a generation unit 1340. In addition to these units, the floor plan generating device 1300 may include other components. However, since these components are not relevant to the content of the embodiment of the present invention, they will not be illustrated or described here. Furthermore, the details of the following operations performed by the floor plan generating device 1300 according to the embodiment of the present invention are the same as those described above with reference to FIGS. 1 to 12, and therefore, repeated description of the same details will be omitted.

[0050] The scanning unit 1310 of the floor plan generation device 1300 in FIG. 13 uses a panoramic camera to scan a three-dimensional space including at least one room, obtain a panoramic video of the three-dimensional space, and obtain a scanning trajectory of the panoramic camera for the panoramic video.

[0051] In this embodiment, the three-dimensional space may be a whole or part of a three-dimensional building including at least one room, and accordingly, the generated floor plan may be a plan floor plan of the three-dimensional building.

[0052] In this embodiment, the scanning unit 1310 can capture a panoramic video of a three-dimensional space using a panoramic camera (e.g., a fisheye camera, a binocular camera, etc.). When capturing the panoramic video of a three-dimensional space, the scanning unit 1310 can further acquire time-stamped position information and orientation information of the panoramic camera based on the panoramic video, thereby generating a scanning trajectory of the panoramic camera. Optionally, the panoramic video can be input to a visual real-time localization and mapping (SLAM) software such as OpenVSLAM to generate a camera scanning trajectory during the panoramic video capture process. Optionally, the format of the generated panoramic camera scanning trajectory can include a timestamp, camera position information, and camera orientation information. For example, assuming that the panoramic camera maintains a vertical state during capture and the camera orientation remains unchanged, the trajectory can include only a timestamp Ti and the horizontal position (Xi, Yi) of the camera, represented as (Ti, Xi, Yi).

[0053] A first acquisition unit 1320 acquires a layout density map of a three-dimensional space according to the panoramic video and the scanning trajectory.

[0054] In this embodiment, the first acquisition unit 1320 projects the panoramic video, acquires a layout projection of the panoramic video, associates the layout projection with the scanning trajectory, and determines the scale of the layout projection, and acquires the layout density map of the three-dimensional space based on the layout projection and the scale.

[0055] Optionally, the first acquisition unit 1320 may first input the video frames of the panoramic video into a model used for room layout prediction, such as HorizonNet, and then perform projection to obtain a layout projection of each video frame in three-dimensional space. In this embodiment, the room layout may indicate the ceiling-wall boundaries and floor-wall boundaries of the room. Here, in the process of obtaining the layout projection using the video frames of the panoramic video, only some key frames in the panoramic video may be selected for calculation, thereby improving computational efficiency.

[0056] After obtaining the layout projection, the first acquisition unit 1320 uses the timestamp and camera position information corresponding to each video frame to locate the corresponding camera position in the layout projection of the video frame, and associates the layout projection with the corresponding scanning trajectory. Then, based on an entropy minimization method, adjust the relative scale between the layout projection of each video frame and the scanning trajectory, so that the layout projections of the same room calculated and displayed by each video frame are overlapped as much as possible, and are overlapped with the layout projection of each video frame based on the adjusted scale to obtain the layout density map of the entire three-dimensional space.

[0057] A second acquisition unit 1330 acquires at least one of an object density map and a room type density map of a three-dimensional space based on the panoramic video and the scanning trajectory.

[0058] In this embodiment, the second acquisition unit 1330 first inputs video frames of the panoramic video into a model for object detection and projects them to obtain an object projection view of each video frame in a three-dimensional space. In this embodiment of the present invention, performing object detection in the three-dimensional space may include performing two-dimensional detection of doors, windows, etc. on the walls of a room in the video frames, and may also include performing three-dimensional detection of objects (furniture, ornaments, etc.) placed on the floor of the room. Specifically, the second acquisition unit 1330 may detect at least one of doors and windows on the walls of the room in the three-dimensional space in each video frame based on the panoramic video to obtain door detection information and / or window detection information, where the door detection information includes at least one of door type, door opening direction, and door outline, and / or detect objects on the floor of the room in the three-dimensional space in each video frame based on the panoramic video to obtain object detection information. Here, in the process of obtaining the object projection view using the video frames of the panoramic video, only some key frames in the panoramic video may be selected for calculation, thereby improving computational efficiency.

[0059] After performing object detection on each video frame, the second acquisition unit 1330 can project the detected objects onto a layout projection to obtain a corresponding object projection. Specifically, first, line segments corresponding to the floor-wall boundaries of the room layout of the detected objects on the wall surfaces within the room are extracted and projected onto the layout projection, and different types of objects can be displayed in different colors. Here, when projecting onto the layout projection based on door detection information, line segments indicating the door opening direction can be projected onto the layout projection based on the door opening direction information contained in the door detection information, thereby providing more detailed information about the object projection and favoring the accurate determination and generation process of the floor plan. In addition, for objects acquired through 3D detection, the bottom surfaces of the 3D frames of objects detected on the ground within the room can be extracted and projected onto the layout projection, and different types of objects can be displayed in different colors.

[0060] After acquiring the object projection view, the second acquisition unit 1330 can adjust the object projection view of each video frame using the scale determined after associating each acquired video frame with the scanning trajectory, and acquire a superimposed object density view of the three-dimensional space.

[0061] Optionally, when obtaining a room type density map, the second obtaining unit 1330 may first input the video frames of the panoramic video into a room type classification model (e.g., a classifier based on VGG) and perform projection to obtain a room type projection map for each room in each video frame in three-dimensional space. Here, in the process of obtaining the room type projection map using the video frames of the panoramic video, only some key frames in the panoramic video can be selected for calculation, thereby improving computation efficiency.

[0062] After acquiring the room type projection map, the second acquisition unit 1330 can adjust the room type projection map of each video frame using the scale determined after associating each acquired video frame with a scanning trajectory, and acquire a room type density map of the three-dimensional space by superimposing it.

[0063] The generating unit 1340 may generate a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space, and further based on at least one of an object density map and a room type density map of the three-dimensional space.

[0064] In this embodiment, the generation unit 1340 stitches together the layout density map of the 3D space and at least one of the object density map and room type density map of the 3D space as separate channels and sends them to a floor plan generation model, which then generates a floor plan of the 3D space using the floor plan generation model. The floor plan generation model may be a GAN-based generation method, such as the RoomFormer detection method, but is not limited thereto. The floor plan generation model of this embodiment of the present invention can display the polygonal shape, room type, and location of objects (e.g., beds, sofas, doors, windows, etc.) in each room in the 3D space based on the layout density map of the 3D space and at least one of the object density map and room type density map of the 3D space. If the input object density map includes door opening direction information in the door detection information, the generated floor plan can accordingly display door information with opening direction. In addition, in the acquired 3D floor plan, rooms of the same type can be displayed with the same color based on the related information in the room type density map.

[0065] According to a floor plan generating device of an embodiment of the present invention, a floor plan of a three-dimensional space can be generated by acquiring a layout density map of the three-dimensional space using a panoramic video acquired by scanning the three-dimensional space with a panoramic camera, and further acquiring at least one of an object density map and a room type density map. The floor plan generating device of the embodiment of the present invention does not need to acquire and calculate three-dimensional data for each room in the three-dimensional space, thereby reducing calculation costs and improving the user experience.

[0066] Furthermore, in the floor plan generation device according to an embodiment of the present invention, door detection information on the wall of a room in three-dimensional space can be obtained from the object density map, and information such as the door opening direction can be obtained, enabling accurate generation of floor plans in three-dimensional space and optimization of the floor plan generation method.

[0067] Next, a floor plan generating device according to an embodiment of the present invention will be described with reference to Fig. 14. Fig. 14 shows a block diagram of a floor plan generating device 1400 according to an embodiment of the present invention. As shown in Fig. 14, the device 1400 may be a computer or a server.

[0068] 14, floor plan generating device 1400 includes one or more processors 1410 and memory 1420, and of course floor plan generating device 1400 may further include input devices, output devices (not shown), etc., which may be interconnected by a bus system and / or other form of connection mechanism. It should be noted that the components and structure of floor plan generating device 1400 shown in FIG. 14 are merely exemplary and not limiting, and floor plan generating device 1400 may have other components and structure as needed.

[0069] The processor 141 may be a central processing unit (CPU) or other form of processing device having data processing and / or instruction execution capabilities, and may utilize computer program instructions stored in memory 1420 to perform desired functions, including using a panoramic camera to scan a three-dimensional space including at least one room, to obtain a panoramic video of the three-dimensional space, and to obtain a scanning trajectory of the panoramic camera relative to the panoramic video, to obtain a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory, to obtain at least one of an object density map and a room-type density map of the three-dimensional space based on the panoramic video and the scanning trajectory, and to generate a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and further based on at least one of the object density map and the room-type density map of the three-dimensional space.

[0070] The memory 1420 includes one or more computer program products, which may include various computer-readable storage media, such as volatile and / or non-volatile memory. One or more computer program instructions are stored on the computer-readable storage media. The processor 1410 executes the computer program instructions to achieve the functionality of the floor plan generation device according to an embodiment of the present invention and / or other desired functions, and / or to perform the floor plan generation method according to an embodiment of the present invention. The computer-readable storage media may also store various application programs and data.

[0071] A computer-readable storage medium according to an embodiment of the present invention is described below, which stores computer program instructions that are executed by a processor to perform the following steps: scanning a three-dimensional space including at least one room using a panoramic camera to obtain a panoramic video of the three-dimensional space and obtain a scanning trajectory of the panoramic camera relative to the panoramic video; obtaining a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory; obtaining at least one of an object density map and a room-type density map of the three-dimensional space based on the panoramic video and the scanning trajectory; and generating a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and at least one of the object density map and the room-type density map of the three-dimensional space.

[0072] Of course, the above-described embodiments are merely examples and are not limiting. Those skilled in the art can achieve the effects of the present invention by combining or integrating some processes or devices from the above-described embodiments based on the concept of the present invention. Such combined or combined embodiments are also included in the present invention, and detailed descriptions thereof will be omitted here.

[0073] The advantages, benefits, and effects mentioned in the present invention are merely examples and are not limiting. It should be noted that these advantages, benefits, and effects cannot be considered to be necessarily possessed by each embodiment of the present invention. Furthermore, the specific details of the invention are intended to show exemplary and easily understandable effects and are not limiting. Furthermore, the details described above do not limit the present invention from being realized using these details.

[0074] Block diagrams of devices, apparatus, facilities, and systems related to the present invention are merely illustrative examples and are not intended to require or suggest that they be connected, arranged, or configured in the manner shown in the block diagrams. As will be recognized by those skilled in the art, these devices, apparatus, facilities, and systems may be connected, arranged, or configured in any manner. Terms such as "comprise," "include," "includes," and "have" are general terms meaning "including but not limited to" and are used interchangeably. As used herein, the terms "or" and "and" mean and are used interchangeably with the term "and / or," unless the context clearly indicates otherwise. Also, as used herein, the term "such as" means and is interchangeable with the phrase "such as but not limited to."

[0075] The flow charts and method descriptions in the present invention are merely illustrative examples and are not intended to require or suggest that each embodiment be performed in the order shown. As will be recognized by those skilled in the art, the steps of the embodiments described above may be performed in any order. Terms such as "then," "and," and "next" are not intended to limit the order of the steps, but are used to guide the description of the method. Note that any reference to a singular word, for example, using the articles "one," "an," or "the," does not limit the word to the singular.

[0076] It should be noted that the steps and devices relating to each embodiment in the present text are not limited to being performed in a specific embodiment, and a new embodiment can be constructed by combining some of the steps and some of the devices relating to each embodiment in the present text based on the spirit of the present invention, and this is included in the scope of the present invention.

[0077] Each operation of the methods described above may be performed by any suitable means having the corresponding functionality, including, but not limited to, a circuit, an application specific integrated circuit (ASIC), or a processor, including various hardware and / or software components and / or modules.

[0078] Each of the illustrative logic blocks, modules, and circuits described above may be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any commercially available processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing equipment, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.

[0079] The steps of a method or algorithm according to the present invention may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may be stored on any form of tangible storage medium. Examples of usable storage media include random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, etc. A storage medium may be coupled to a processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium and the processor may be integrated. A software module may represent either a single instruction or multiple instructions, and may be distributed over several different code segments, among different programs, and across multiple storage media.

[0080] Additionally, a method according to the present invention comprises one or more acts for achieving the method. The methods and / or acts may be interchanged without departing from the scope of the claims. In other words, unless otherwise specified, the order and / or use of acts may be modified without departing from the scope of the claims.

[0081] The functions described above may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions are stored as one or more instructions on a tangible computer-readable medium. A storage medium is any practical medium accessible by a computer. By way of example and not limitation, such computer-readable media include RAM, ROM, EEPROM, CD-ROM or other optical disks, magnetic disks or other magnetic memory devices, or any other physical medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disks, as used herein, include, for example, compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and Blu-ray disks.

[0082] Thus, a computer program product can perform the operations described herein. For example, such a computer program product is a tangible computer-readable medium having instructions tangibly stored (and / or encoded) thereon. The instructions are executed by one or more processors to implement the described operations. The computer program product may be included in a package.

[0083] The software or instructions may be transmitted over a transmission medium. For example, the software may be transmitted from a website, server, or other remote source using a transmission medium such as coaxial cable, fiber optic cable, twisted wire, Digital Subscriber Line (DSL), or wireless techniques such as infrared, radio, microwave, or other wireless techniques.

[0084] Modules and / or other suitable means for performing the above-described methods and techniques may be downloaded and / or otherwise obtained by the user terminal and / or base station, as appropriate. For example, such equipment may be connected to a server and provided with transmission of the means for performing the above-described methods. Alternatively, the above-described methods may be provided via storage means (e.g., physical storage media such as RAM, ROM, CD, or floppy disk), and the user terminal and / or base station may obtain the methods when coupled to or providing the storage means to the equipment. Furthermore, any other suitable technology may be used to provide the above-described methods and techniques to the equipment.

[0085] Other examples and implementations are within the scope and spirit of the present invention and the claims. For example, due to the nature of software, the functions described above may be implemented by software executed by a processor, hardware, firmware, hardwire, or any combination thereof. Features implementing the functions may be located in various physical locations. This includes functions distributed such that portions of the functions are implemented in different physical locations. As used herein and in the claims, "or" used in a list preceding "at least one" refers to a disjunctive list; for example, the list "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, "exemplary" does not imply that the described example is preferred or preferable over other examples.

[0086] Various changes, substitutions, and alterations to the techniques described above may be made without departing from the limitations of the claims. Furthermore, the scope of the claims is not limited to the particular processes, machines, manufacture, combinations of events, means, methods, and acts described above. Any now-known or later-developed processes, machines, manufacture, combinations of events, means, methods, and acts that perform substantially the same function or achieve substantially the same result as those described herein may be utilized. Accordingly, the scope of the claims includes any processes, machines, manufacture, combinations of events, means, methods, or acts that fall within their scope.

[0087] The foregoing description is provided to enable one skilled in the art to use or realize the present invention. Various modifications to these descriptions will be apparent to those skilled in the art, and the general principles defined herein may be applied elsewhere without departing from the scope of the present invention. Therefore, the present invention is not limited to the foregoing description, but is to be accorded the widest scope consistent with the inventive concept and novel features.

[0088] The foregoing description is intended to be illustrative and explanatory, but not limiting, and variations, additions, modifications and subcombinations of the embodiments described above will occur to those skilled in the art.

Claims

1. 1. A computer-implemented method for generating a floor plan, comprising: using a panoramic camera to scan a three-dimensional space including at least one room, to obtain a panoramic video of the three-dimensional space, and to obtain a scanning trajectory of the panoramic camera for the panoramic video; obtaining a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory, the layout density map being obtained by superimposing layout projection maps corresponding to each of a plurality of video frames of the panoramic video; obtaining an object density map of the three-dimensional space based on the panoramic video and the scanning trajectory, the object density map being obtained by superimposing object projection maps corresponding to each of a plurality of video frames of the panoramic video; generating a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and further based on the object density map of the three-dimensional space; Including, generating a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and further based on the object density map of the three-dimensional space, A floor plan generation method comprising: connecting the layout density map of the three-dimensional space and the object density map of the three-dimensional space as different channels, inputting the channels into a floor plan generation model, and generating a floor plan of the three-dimensional space.

2. The step of scanning a three-dimensional space including at least one room using a panoramic camera to obtain a panoramic video of the three-dimensional space and obtaining a scanning trajectory of the panoramic camera for the panoramic video includes: The floor plan generating method according to claim 1, further comprising: acquiring position information and posture information associated with a timestamp of the panoramic camera based on the panoramic video, and generating a scanning trajectory of the panoramic camera.

3. The step of obtaining a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory includes: projecting the panoramic video and obtaining a layout projection of the panoramic video; Associating the layout projection with the scanning trajectory and determining a scale for the layout projection; and obtaining the layout density map of the three-dimensional space based on the layout projection map and the scale.

4. The step of obtaining an object density map of the three-dimensional space based on the panoramic video and the scanning trajectory includes: performing object detection in the three-dimensional space based on the panoramic video; projecting the detected object in the panoramic video to obtain a projection of the object in the three-dimensional space; and adjusting the object projection view based on the scanning trajectory and the scale to obtain an object density view of the three-dimensional space.

5. Performing object detection in the three-dimensional space based on the panoramic video includes: Detecting at least one of a door and a window on a wall surface of the room in the three-dimensional space based on the panoramic video, and obtaining door detection information and / or window detection information, wherein the door detection information includes at least one of a door type, a door opening direction, and a door contour; and / or The floor plan generating method according to claim 4 , further comprising: detecting objects on a floor surface of the room in the three-dimensional space based on the panoramic video, and acquiring object detection information.

6. A floor plan generation device, comprising: a scanning unit that uses a panoramic camera to scan a three-dimensional space including at least one room, obtains a panoramic video of the three-dimensional space, and obtains a scanning trajectory of the panoramic camera relative to the panoramic video; a first acquisition unit that acquires a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory, the layout density map being acquired by superimposing layout projection maps corresponding to each of a plurality of video frames of the panoramic video; a second acquisition unit that acquires an object density map of the three-dimensional space based on the panoramic video and the scanning trajectory, the object density map being acquired by superimposing object projection maps corresponding to each of a plurality of video frames of the panoramic video; a generating unit for generating a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and further based on the object density map of the three-dimensional space; Including, The generation unit connects the layout density map of the three-dimensional space and the object density map of the three-dimensional space as different channels, inputs them into a floor plan generation model, and generates a floor plan of the three-dimensional space.

7. a processor; a memory in which computer program instructions are stored, Execution of the computer program instructions by the processor causes the processor to: using a panoramic camera to scan a three-dimensional space including at least one room, to obtain a panoramic video of the three-dimensional space, and to obtain a scanning trajectory of the panoramic camera for the panoramic video; obtaining a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory, the layout density map being obtained by superimposing layout projection maps corresponding to each of a plurality of video frames of the panoramic video; obtaining an object density map of the three-dimensional space based on the panoramic video and the scanning trajectory, the object density map being obtained by superimposing object projection maps corresponding to each of a plurality of video frames of the panoramic video; generating a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and further based on the object density map of the three-dimensional space; Execute generating a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and further based on the object density map of the three-dimensional space, A floor plan generation device characterized by connecting the layout density map of the three-dimensional space and the object density map of the three-dimensional space as different channels and inputting them into a floor plan generation model to generate a floor plan of the three-dimensional space.

8. A computer-readable storage medium, Computer program instructions are stored, and the computer program instructions are executed by a processor to using a panoramic camera to scan a three-dimensional space including at least one room, to obtain a panoramic video of the three-dimensional space, and to obtain a scanning trajectory of the panoramic camera for the panoramic video; obtaining a layout density map of the three-dimensional space based on the panoramic video and the scanning trajectory, the layout density map being obtained by superimposing layout projection maps corresponding to each of a plurality of video frames of the panoramic video; obtaining an object density map of the three-dimensional space based on the panoramic video and the scanning trajectory, the object density map being obtained by superimposing object projection maps corresponding to each of a plurality of video frames of the panoramic video; generating a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and further based on the object density map of the three-dimensional space; Run generating a floor plan of the three-dimensional space based on the layout density map of the three-dimensional space and further based on the object density map of the three-dimensional space, A computer-readable storage medium that connects the layout density map of the three-dimensional space and the object density map of the three-dimensional space as different channels and inputs them into a floor plan generation model to generate a floor plan of the three-dimensional space.

9. A program for causing a computer to execute the floor plan generating method according to any one of claims 1 to 5.

Images