Space structure diagram and house type diagram generation method and device, equipment and storage medium

Abstract

The invention provides a space structure diagram and house type diagram generation method and device, equipment and a storage medium. The space structure diagram and house type diagram generation method comprises the steps of obtaining point cloud data and a panoramic image obtained by an information acquisition terminal at at least one shooting point in a target space; acquiring a space contour of the target space according to the point cloud data of the at least one shooting point location; a target medium identified in the target panoramic image is acquired, the target medium is an image of an entity medium in a target space in the target panoramic image, and the target panoramic image is a panoramic image used for identifying the target medium in the panoramic images of the at least one shooting point location; and determining a mapping medium corresponding to the target medium in the space contour to obtain a space structure diagram of the target space. According to the scheme, the panoramic image assists in determining the positions of the target media, namely a door body, a window body and the like, in the space contour of the target space, and the accuracy of the generated space structure diagram is improved.

Description

Method, device, equipment and storage medium for generating space structure diagram and floor plan Technical Field

[0001] The present invention relates to the field of model reconstruction technology, and in particular to a method, device, equipment and storage medium for generating a spatial structure diagram and a floor plan. Background Art

[0002] The two-dimensional model of the target space is usually used to help users understand the spatial structure information of the space. For example, in real estate service scenarios, the two-dimensional model of the target space is used to display the apartment structure, etc.

[0003] To obtain a 2D model of the target space, a 3D reconstruction of the target space is usually performed first, for example, using optical imaging or structured light scanning. The reconstructed 3D model is then cut into transverse sections to obtain the corresponding 2D model. However, the 2D models obtained using these methods often lack detail and are inaccurate.

[0004] Summary of the Invention

[0005] The present invention provides a method, device, equipment and storage medium for generating a space structure diagram and a floor plan, so as to improve the accuracy of the generated space structure diagram and floor plan.

[0006] In a first aspect, an embodiment of the present invention provides a method for generating a spatial structure diagram, which is applied to a control terminal. The method includes:

[0007] Acquire point cloud data and panoramic images obtained by the information acquisition terminal at least one shooting point in the target space;

[0008] Acquiring a spatial contour of the target space according to the point cloud data of the at least one shooting point;

[0009] Acquire a target medium identified in a target panoramic image, where the target medium is an image of a physical medium in the target space in the target panoramic image, and the target panoramic image is a panoramic image for identifying the target medium in the panoramic image of the at least one shooting point;

[0010] A mapping medium corresponding to the target medium is determined in the space outline to obtain a spatial structure diagram of the target space.

[0011] In a second aspect, an embodiment of the present invention provides a spatial structure diagram generating device, applied to a control terminal, the device comprising:

[0012] An acquisition module is used to acquire point cloud data and a panoramic image obtained by the information acquisition terminal at least one shooting point in the target space;

[0013] A processing module is used to obtain the spatial contour of the target space based on the point cloud data of the at least one shooting point; obtain the target medium identified in the target panoramic image, the target medium being the image of the physical medium in the target space in the target panoramic image, and the target panoramic image is a panoramic image for identifying the target medium in the panoramic image of the at least one shooting point; determine the mapping medium corresponding to the target medium in the spatial contour to obtain a spatial structure diagram of the target space. In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein executable code is stored on the memory, and when the executable code is executed by the processor, the processor executes the spatial structure diagram generation method as described in the first aspect.

[0014] In a fourth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the spatial structure diagram generation method described in the first aspect.

[0015] In a fifth aspect, an embodiment of the present invention provides a floor plan generation method, which is applied to a target control terminal, and the method includes:

[0016] Acquiring point cloud data and panoramic images corresponding to a plurality of subspaces in the target physical space obtained by the information collection terminal, respectively, to determine a plurality of spatial contours corresponding to the plurality of subspaces; wherein the plurality of subspaces correspond one-to-one to the plurality of spatial contours, and the point cloud data and panoramic image of any subspace are collected from at least one shooting point in the any subspace;

[0017] displaying a plurality of space contours corresponding to the plurality of subspaces for editing;

[0018] For a target subspace among the multiple subspaces, obtaining a target medium identified in a target panoramic image, where the target medium is an image of a physical medium in the target subspace in the target panoramic image, and the target panoramic image is a panoramic image captured by at least one shooting point in the target subspace and used to identify the target medium;

[0019] Determining a mapping medium corresponding to the target medium in a spatial contour of the target subspace to generate a spatial structure diagram of the target subspace;

[0020] In response to the completion of the operation of acquiring the multiple space structure diagrams corresponding to the multiple subspaces, a floor plan of the target physical space obtained by splicing the multiple space structure diagrams is acquired.

[0021] In a sixth aspect, an embodiment of the present invention provides a floor plan generating device, which is applied to a target control terminal, and the device includes:

[0022] an acquisition module, configured to acquire point cloud data and panoramic images corresponding to a plurality of subspaces in the target physical space obtained by the information acquisition terminal, so as to determine a plurality of spatial contours corresponding to the plurality of subspaces; wherein the plurality of subspaces correspond one-to-one to the plurality of spatial contours, and the point cloud data and panoramic image of any subspace are acquired from at least one shooting point in the any subspace;

[0023] A display module, configured to display a plurality of space contours corresponding to the plurality of subspaces for editing;

[0024] A processing module is used to obtain, for a target subspace among multiple subspaces, a target medium identified in a target panoramic image, where the target medium is an image of a physical medium in the target subspace in the target panoramic image, and the target panoramic image is a panoramic image captured by at least one shooting point of the target subspace, for identifying the target medium; determine a mapping medium corresponding to the target medium in the spatial contour of the target subspace to generate a spatial structure diagram of the target subspace; and in response to completion of an operation to obtain multiple spatial structure diagrams corresponding to the multiple subspaces, obtain a floor plan of the target physical space obtained by splicing the multiple spatial structure diagrams.

[0025] In a seventh aspect, an embodiment of the present invention provides a floor plan generation method, which is applied to a control terminal, and the method includes:

[0026] Acquire point cloud data and panoramic images corresponding to a plurality of subspaces in the target physical space, respectively, sent by the information collection terminal, wherein the point cloud data and panoramic image of any subspace are collected from at least one shooting point in the subspace;

[0027] In the process of sequentially performing spatial structure graph acquisition processing on the plurality of subspaces, for a target subspace currently to be edited, acquiring a target space contour of the target subspace based on point cloud data and / or panoramic images collected at at least one shooting point in the target subspace;

[0028] Acquire a target medium identified in a target panoramic image, where the target medium is an image of a physical medium in the target subspace in the target panoramic image, and the target panoramic image is a panoramic image used to identify the target medium in a panoramic image captured at at least one shooting point in the target subspace;

[0029] determining a mapping medium for representing the target medium on a target space contour of the target subspace to obtain a spatial structure diagram of the target subspace;

[0030] In response to the completion of the acquisition operation of the spatial structure diagram of the target subspace, if there is no subspace among the multiple subspaces for which the spatial structure diagram has not been acquired, a floor plan of the target physical space obtained by splicing the spatial structure diagrams of the multiple spaces is acquired.

[0031] In an eighth aspect, an embodiment of the present invention provides a floor plan generating device, which is applied to a control terminal, and the device includes:

[0032] an acquisition module, configured to acquire point cloud data and panoramic images corresponding to a plurality of subspaces in the target physical space, respectively, sent by the information acquisition terminal, wherein the point cloud data and panoramic image of any subspace are acquired from at least one shooting point in the subspace;

[0033] A processing module is used to obtain a target space outline of the target subspace according to the point cloud data and / or panoramic image collected at at least one shooting point of the target subspace in the process of sequentially performing spatial structure diagram acquisition processing on the multiple subspaces; obtain a target medium identified in the target panoramic image, wherein the target medium is an image of the physical medium in the target subspace in the target panoramic image, and the target panoramic image is a panoramic image used to identify the target medium in the panoramic image collected at at least one shooting point of the target subspace; determine a mapping medium for representing the target medium on the target space outline of the target subspace to obtain a spatial structure diagram of the target subspace; in response to the completion operation of the acquisition of the spatial structure diagram of the target subspace, if there is no subspace among the multiple subspaces for which the spatial structure diagram has not been obtained, obtain a floor plan of the target physical space obtained by splicing the spatial structure diagrams of the multiple spaces.

[0034] In a ninth aspect, an embodiment of the present invention provides a floor plan generation method, the method being used to generate a floor plan of a target physical space, the target physical space including at least N spaces, and being applied to a control terminal, the method comprising:

[0035] Step 1: Acquire point cloud data and panoramic images collected by an information collection terminal in each of the N spaces, wherein the point cloud data and panoramic images are collected from at least one shooting point in each of the spaces;

[0036] Step 2: Obtaining an M-th space outline of an M-th space among the N spaces and displaying it for editing, wherein the M-th space outline is obtained based on point cloud data and / or panoramic images collected from at least one shooting point of the M-th space;

[0037] Step 3: Acquire a target medium identified in the target panoramic image of the Mth space, so as to obtain a mapping medium of the target medium in the outline of the Mth space based on the target medium, so as to edit the outline of the Mth space based on the mapping medium to obtain a floor plan of the Mth space; the target panoramic image is a panoramic image for identifying the target medium in a panoramic image captured by at least one shooting point in the Mth space, and the target medium is an image of a physical medium in the Mth space in the target panoramic image;

[0038] Step 4: Determine whether the Mth space is the last space among the N spaces for which a floor plan diagram is generated;

[0039] If not, execute step 5, assign M to M+1 and return to step 2;

[0040] If so, execute step 6 to obtain a floor plan of the target physical space consisting of N floor plan structure diagrams for display, and the process ends; wherein M and N are natural numbers, and 1≤M≤N.

[0041] In a tenth aspect, an embodiment of the present invention provides a floor plan generating device, the device being used to generate a floor plan of a target physical space, the target physical space including at least N spaces, and being applied to a control terminal, the device comprising:

[0042] an acquisition module, configured to acquire point cloud data and panoramic images collected by the information collection terminal in each of the N spaces, wherein the point cloud data and panoramic images are collected from at least one shooting point in each of the spaces;

[0043] A first processing module is configured to obtain an M-th spatial outline of an M-th space among the N spaces for displaying for editing, wherein the M-th spatial outline is obtained based on point cloud data and / or a panoramic image acquired from at least one shooting point of the M-th space; obtain a target medium identified in a target panoramic image of the M-th space, so as to obtain a mapping medium of the target medium in the M-th spatial outline based on the target medium, so as to edit the M-th spatial outline based on the mapping medium, so as to obtain a floor plan diagram of the M-th space; the target panoramic image is a panoramic image for identifying the target medium in a panoramic image acquired from at least one shooting point of the M-th space, wherein the target medium is an image of a physical medium in the M-th space in the target panoramic image;

[0044] The second processing module is used to determine whether the Mth space is the last space among the N spaces for generating a floor plan diagram; if not, M is assigned a value of M+1 and the process returns to execute the first processing module; if so, the floor plan of the target physical space composed of the N floor plan diagrams is obtained for display, and the process ends; wherein M and N are natural numbers, and 1≤M≤N.

[0045] In an eleventh aspect, an embodiment of the present invention provides a floor plan generation method, the method being used to generate a floor plan of a target physical space, wherein the target physical space includes multiple subspaces and is applied to a control terminal, the method comprising:

[0046] Acquiring point cloud data and panoramic images corresponding to the multiple subspaces respectively obtained by the information collection terminal, wherein the point cloud data and panoramic image of any subspace are collected from at least one shooting point in the any subspace;

[0047] During the process of sequentially stitching the floor plan diagrams of the plurality of subspaces, for a first subspace to be stitched, obtaining a target spatial outline of the first subspace based on point cloud data and / or panoramic images collected at at least one shooting point in the first subspace;

[0048] Acquire a target medium identified in a target panoramic image, where the target medium is an image of a physical medium in the first subspace in the target panoramic image, and the target panoramic image is a panoramic image captured at at least one shooting point in the first subspace and used to identify the target medium;

[0049] Determining a mapping medium for representing the target medium on a target space outline of the first subspace to obtain a floor plan of the first subspace;

[0050] splicing the floor plan diagram of the first subspace with the floor plan diagram of the second subspace, where the second subspace is the subspace to which the floor plan diagram of the first subspace has been spliced;

[0051] If there is no subspace among the multiple subspaces that has not been subjected to the floor plan structure splicing process, the splicing process result is determined as the floor plan of the target physical space.

[0052] In a twelfth aspect, an embodiment of the present invention provides a floor plan generating device, the device being used to generate a floor plan of a target physical space, wherein the target physical space includes multiple subspaces and is applied to a control terminal, the device comprising:

[0053] an acquisition module, configured to acquire point cloud data and panoramic images corresponding to the plurality of subspaces respectively obtained by the information acquisition terminal, wherein the point cloud data and panoramic image of any subspace are acquired from at least one shooting point in the any subspace;

[0054] A stitching module is used to, in the process of stitching the floor plan diagrams of the multiple subspaces in sequence, obtain, for a first subspace to be stitched, a target space outline of the first subspace based on point cloud data and / or panoramic images collected at at least one shooting point of the first subspace; obtain a target medium identified in the target panoramic image, where the target medium is an image of a physical medium in the first subspace in the target panoramic image, and the target panoramic image is a panoramic image used to identify the target medium in the panoramic image collected at at least one shooting point of the first subspace; determine a mapping medium for representing the target medium on the target space outline of the first subspace to obtain a floor plan diagram of the first subspace; stitch the floor plan diagram of the first subspace with that of the second subspace, where the second subspace is a subspace to which the floor plan diagram has been stitched before;

[0055] The processing module is configured to determine the splicing result as the floor plan of the target physical space if there is no subspace in the plurality of subspaces that has not been spliced ​​with the floor plan structure diagram.

[0056] In a thirteenth aspect, an embodiment of the present invention provides a method for generating a floor plan, wherein the method is used to generate a floor plan of a target physical space, wherein the target physical space includes at least N spaces, and is applied to a control terminal, the method comprising:

[0057] Step 1: Acquire point cloud data and panoramic images collected by an information collection terminal in each of the N spaces, wherein the point cloud data and panoramic images are collected from at least one shooting point in each of the spaces;

[0058] Step 2: Obtain an Mth spatial outline for display for editing; wherein the Mth spatial outline is a spatial outline of the Mth space among the N spaces, and the Mth spatial outline is obtained based on point cloud data and / or panoramic images collected from at least one shooting point of the Mth space;

[0059] Step 3: Acquire a first target medium identified in the first target panoramic image, so as to obtain a first mapping medium of the first target medium in the outline of the Mth space based on the first target medium, so as to edit the outline of the Mth space based on the first mapping medium to obtain a floor plan of the Mth space; the first target panoramic image is a panoramic image for identifying the first target medium in the panoramic image captured by the at least one shooting point of the Mth space, and the first target medium is an image of a physical medium in the Mth space in the first target panoramic image;

[0060] Step 4: Obtain an M+1th spatial outline for display for editing; wherein the M+1th spatial outline is a spatial outline of the M+1th space among the N spaces, the M+1th space is an adjacent space to the Mth space, and the M+1th spatial outline is obtained based on point cloud data and / or panoramic images collected from at least one shooting point in the M+1th space;

[0061] Step 5: Acquire a second target medium identified in the second target panoramic image, so as to obtain a second mapping medium of the second target medium in the outline of the M+1th space based on the second target medium, so as to edit the outline of the M+1th space based on the second mapping medium to obtain a floor plan of the M+1th space; the second target panoramic image is a panoramic image for identifying the second target medium in the panoramic image captured by the at least one shooting point of the M+1th space, and the second target medium is an image of a physical medium in the M+1th space in the first target panoramic image;

[0062] Step 6: splice the floor plan diagram of the M+1th space with the floor plan diagram of the Mth space, and determine whether the M+1th space is the last space among the N spaces for which a floor plan diagram is generated;

[0063] If not, execute step 7, merge the Mth space and the M+1th space as the Mth space, and return to step 4;

[0064] If so, execute step 8, use the splicing result as the floor plan of the target physical space for display, and the process ends.

[0065] In a fourteenth aspect, an embodiment of the present invention provides a floor plan generating device, the device being configured to generate a floor plan of a target physical space, wherein the target physical space includes at least N spaces, and being applied to a control terminal, the device comprising:

[0066] A first acquisition module is configured to acquire point cloud data and panoramic images collected by the information collection terminal in each of the N spaces, wherein the point cloud data and panoramic images are collected from at least one shooting point in each of the spaces; acquire an Mth space outline for display for editing; wherein the Mth space outline is the space outline of the Mth space in the N spaces, and the Mth space outline is acquired based on the point cloud data and / or panoramic image collected from at least one shooting point in the Mth space; acquire a first target medium identified in a first target panoramic image, so as to acquire a first mapping medium of the first target medium in the Mth space outline based on the first target medium, so as to edit the Mth space outline based on the first mapping medium to obtain a floor plan of the Mth space; the first target panoramic image is a panoramic image for identifying the first target medium in the panoramic image collected from at least one shooting point in the Mth space, and the first target medium is an image of a physical medium in the Mth space in the first target panoramic image;

[0067] A second acquisition module is configured to acquire an M+1th spatial outline for display for editing; wherein the M+1th spatial outline is the spatial outline of the M+1th space among the N spaces, the M+1th space is an adjacent space to the Mth space, and the M+1th spatial outline is acquired based on point cloud data and / or panoramic images collected from at least one shooting point of the M+1th space; a second target medium identified in a second target panoramic image is acquired, so that a second mapping medium of the second target medium in the M+1th spatial outline is acquired based on the second target medium, so as to edit the M+1th spatial outline based on the second mapping medium to acquire a floor plan of the M+1th space; the second target panoramic image is a panoramic image for identifying the second target medium in the panoramic image collected from at least one shooting point of the M+1th space, and the second target medium is an image of a physical medium in the M+1th space in the first target panoramic image;

[0068] A processing module is configured to splice the floor plan diagram of the M+1th space with the floor plan diagram of the Mth space, and determine whether the M+1th space is the last space among the N spaces for which a floor plan diagram is generated; if not, merge the Mth space and the M+1th space as the Mth space, and return to execute the second acquisition module; if so, use the splicing result as the floor plan diagram of the target physical space for display, and the process ends.

[0069] In the fifteenth aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein, the memory stores executable code, and when the executable code is executed by the processor, the processor executes the floor plan generation method described in the fifth aspect, the seventh aspect, the ninth aspect, the eleventh aspect and / or the thirteenth aspect.

[0070] In the sixteenth aspect, an embodiment of the present invention provides a non-transitory machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the floor plan generation method described in the fifth aspect, the seventh aspect, the ninth aspect, the eleventh aspect and / or the thirteenth aspect.

[0071] In this solution, the mapping medium on the spatial outline in the floor plan is determined based on the panoramic image. Compared with point cloud data, the panoramic image can better reflect the actual position of doors and windows (i.e., target media) in the actual space. Therefore, based on the assistance of the panoramic image, the floor plan of each space is marked with more accurate door and window information, which can better reflect the scene information in the actual space. Therefore, the floor plan determined based on multiple floor plan diagrams can also accurately reflect the actual spatial structure of the target physical space. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0073] FIG1 is a schematic diagram of a spatial structure diagram generating system provided by a first embodiment of the present invention;

[0074] FIG2 is a flowchart of a method for generating a spatial structure diagram provided by the first embodiment of the present invention;

[0075] FIG3 is a schematic diagram of a point cloud image provided by the first embodiment of the present invention;

[0076] FIG4 is a schematic diagram of a spatial structure diagram provided by the first embodiment of the present invention;

[0077] FIG5 is a flowchart of another method for generating a spatial structure diagram provided by the first embodiment of the present invention;

[0078] FIG6 is a schematic structural diagram of a spatial structure diagram generating apparatus provided by the first embodiment of the present invention;

[0079] FIG7 is a schematic diagram of a system for generating a floor plan according to a second embodiment of the present invention;

[0080] FIG8 is a flow chart of a method for generating a floor plan according to a second embodiment of the present invention;

[0081] FIG9 is a schematic diagram of a floor plan of a target physical space provided by a second embodiment of the present invention;

[0082] FIG10 is an interactive flow chart of another floor plan generation method provided by the second embodiment of the present invention;

[0083] FIG11 is a schematic structural diagram of a floor plan generating device provided by a second embodiment of the present invention;

[0084] FIG12 is a flowchart of a method for generating a floor plan according to a third embodiment of the present invention;

[0085] FIG13 is a schematic diagram of a space profile provided by a third embodiment of the present invention;

[0086] FIG14 is a schematic structural diagram of a floor plan generating device provided by a third embodiment of the present invention;

[0087] FIG15 is a flowchart of a method for generating a floor plan according to a fourth embodiment of the present invention;

[0088] FIG16 is a schematic diagram of a scenario for generating a floor plan according to a fourth embodiment of the present invention;

[0089] FIG17 is a schematic diagram of an M-th spatial profile provided by a fourth embodiment of the present invention;

[0090] FIG18 is a schematic diagram of a floor plan generation process provided by a fourth embodiment of the present invention;

[0091] FIG19 is a schematic structural diagram of a floor plan generating device provided by a fourth embodiment of the present invention;

[0092] FIG20 is a flowchart of a method for generating a floor plan according to a fifth embodiment of the present invention;

[0093] FIG21 is a schematic diagram of a target space profile provided by a fifth embodiment of the present invention;

[0094] FIG22 is a schematic diagram of an actual spatial structure of a target physical space provided by a fifth embodiment of the present invention;

[0095] FIG23 is a schematic diagram of a housing structure diagram provided in accordance with a fifth embodiment of the present invention;

[0096] FIG24 is a schematic diagram of another apartment structure diagram provided in the fifth embodiment of the present invention;

[0097] FIG25 is a schematic structural diagram of a floor plan generating device provided by a fifth embodiment of the present invention;

[0098] FIG26 is a flowchart of a method for generating a floor plan according to a sixth embodiment of the present invention;

[0099] FIG27 is a schematic diagram of a scenario for generating a floor plan according to a sixth embodiment of the present invention;

[0100] FIG28 is a schematic diagram of a spatial outline of a space Z provided by a sixth embodiment of the present invention;

[0101] FIG29 is a schematic diagram of a unit structure diagram provided in accordance with a sixth embodiment of the present invention;

[0102] FIG30 is a schematic diagram of a spliced ​​apartment structure diagram provided by the sixth embodiment of the present invention;

[0103] FIG31 is a schematic diagram of another spliced ​​apartment structure diagram provided by the sixth embodiment of the present invention;

[0104] FIG32 is a schematic structural diagram of a floor plan generating device provided by a sixth embodiment of the present invention;

[0105] FIG33 is a schematic structural diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0106] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0107] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0108] The following describes the method for generating a floor plan structure diagram and the method for generating a floor plan provided by the present invention through multiple embodiments.

[0109] First embodiment

[0110] Generally, a space contains at least one subspace. For example, a residential building may include a living room, a kitchen, and two bedrooms. Each of these subspaces, or units, can be considered a subspace. To help users quickly understand a space's structure, a pre-generated two-dimensional floor plan, also known as a floor plan, is often used.

[0111] It is understood that for a space containing at least one subspace, the corresponding floor plan diagram is actually obtained by combining the spatial structure diagrams corresponding to the at least one subspace. The accuracy of the spatial structure diagram of any subspace will affect the accuracy of the final floor plan diagram.

[0112] To this end, an embodiment of the present invention provides a method for generating a spatial structure diagram, which is used to generate an accurate spatial structure diagram of a target space. It should be noted that the target space in the embodiment of the present invention refers to a unit space, that is, any of the above-mentioned subspaces.

[0113] FIG1 is a schematic diagram of a spatial structure diagram generation system provided by a first embodiment of the present invention. As shown in FIG1 , the spatial structure diagram generation system includes an information collection terminal and a control terminal. Optionally, the information collection terminal can be directly integrated with the control terminal, functioning as a single entity. Alternatively, the information collection terminal can be decoupled from the control terminal and separately configured, communicating with the control terminal via, for example, Bluetooth or a Wireless Fidelity (WiFi) hotspot.

[0114] The information collection terminal includes: a laser sensor, a camera, a motor, and a processor (such as a CPU). Among them, the laser sensor and camera serve as perception devices to collect scene information of the target space, namely, point cloud data and image data of the target space.

[0115] In practical applications, the target space corresponds to at least one shooting point. For each target shooting point, the information collection terminal, in response to an information collection instruction, drives the laser sensor 360 degrees via a drive motor to collect point cloud data corresponding to the target shooting point. The drive motor also drives the camera 360 degrees to capture images corresponding to the target point at multiple preset angles. The processor can stitch the images captured at multiple preset angles into a panoramic image using a panoramic image stitching algorithm, such as a feature matching algorithm. The target shooting point is any one of the at least one shooting point.

[0116] Optionally, the information collection instruction is sent by the control terminal, or the information collection instruction is triggered in response to a triggering operation of the user on the information collection terminal.

[0117] In this embodiment, there is no restriction on the order in which point cloud data and image data are collected at a given shooting point; both can be collected simultaneously or sequentially. In an alternative embodiment, if point cloud data is collected first and image data is collected later, to improve image quality, the camera can be activated simultaneously with point cloud data collection to collect scene lighting information at the current shooting point, perform photometry, and determine corresponding exposure parameters. The camera then captures image data based on the determined exposure parameters.

[0118] Optionally, when generating a panoramic image, High Dynamic Range Imaging (HDRI) may be combined to generate a high-quality panoramic image.

[0119] Optionally, multiple preset angles can be customized by the user according to the viewing angle of the camera, and images captured based on the multiple preset angles contain scene information within a 360-degree range of the current point.

[0120] Optionally, the information acquisition terminal also includes an inertial measurement unit (IMU). The IMU is used to correct the pose information corresponding to the collected point cloud data and image data to reduce errors caused by environmental or human factors (such as: the information acquisition terminal is not placed horizontally, etc.).

[0121] The control terminal can be a terminal device with data processing capabilities, such as a smartphone, tablet computer, or laptop computer. As shown in Figure 1, the control terminal is used to obtain point cloud data and panoramic images obtained by the information acquisition terminal at at least one shooting point in the target space to generate a spatial structure diagram of the target space.

[0122] Optionally, the spatial structure diagram generation system shown in Figure 1 also includes a cloud server, which can be a physical server or a virtual server in the cloud. The control terminal communicates with the cloud server by accessing a wireless network based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks.

[0123] In an optional embodiment, the cloud server can receive point cloud data and panoramic images of at least one captured point in the target space forwarded by the control terminal to generate a spatial structure diagram of the target space. This spatial structure diagram is then fed back to the control terminal for display. In practice, the cloud server generates the spatial structure diagram of the target space using the same process as the control terminal. However, due to the cloud server's greater computing power, it generates the spatial structure diagram of the target space more efficiently, further enhancing the user experience.

[0124] In another optional embodiment, the cloud server may also be directly connected to the information collection terminal to directly obtain the point cloud data and panoramic image obtained by the information collection terminal at at least one shooting point in the target space, and generate a spatial structure diagram of the target space.

[0125] The following description takes the control terminal generating a spatial structure diagram of a target space as an example.

[0126] FIG2 is a flowchart of a method for generating a spatial structure diagram according to a first embodiment of the present invention, which is applied to a control terminal in the spatial structure diagram generating system shown in FIG1. ​​As shown in FIG2, the method for generating a spatial structure diagram includes the following steps:

[0127] 201. Acquire point cloud data and a panoramic image obtained by an information collection terminal at least one shooting point in a target space.

[0128] 202. Acquire a spatial contour of the target space based on point cloud data of at least one shooting point.

[0129] 203. Acquire a target medium identified in the target panoramic image. The target medium is an image of a physical medium in the target space in the target panoramic image. The target panoramic image is a panoramic image of at least one shooting point used to identify the target medium.

[0130] 204. Determine a mapping medium corresponding to the target medium in the spatial outline to obtain a spatial structure diagram of the target space.

[0131] For ease of understanding, this embodiment is described by taking the target space as a building space as an example, but the present invention is not limited thereto. In fact, the target space can also be a certain spatial structure, container or vehicle, etc.

[0132] In this embodiment, after the control terminal and the information collection terminal establish a communication connection, the information collection terminal responds to the information collection instruction, obtains the point cloud data and panoramic image corresponding to each shooting point at at least one shooting point in the target space in turn, and sends the collected point cloud data and panoramic image of at least one shooting point to the control terminal.

[0133] The at least one shooting point may be selected by the user in the target space according to modeling needs, or may be a reference shooting point generated for the target space by the control terminal based on description information of the target space input by the user.

[0134] During specific implementations, the process for acquiring point cloud data and panoramic images corresponding to any of the at least one shooting point is consistent. Taking shooting point X as an example, assuming the information collection terminal is currently located at shooting point X, the information collection terminal responds to the information collection instruction and acquires the point cloud data and panoramic image corresponding to shooting point X. The specific process for acquiring the point cloud data and panoramic image corresponding to shooting point X by the information collection terminal can be found in the previous embodiment and will not be further described here.

[0135] When the information collection terminal sends the acquired point cloud data and panoramic images to the control terminal, optionally, the information collection terminal can feed back the acquired point cloud data and panoramic images to the control terminal each time it acquires the point cloud data and panoramic images of a shooting point; the information terminal can also send the point cloud data and panoramic images of all shooting points in the target space to the control terminal in a unified manner after acquiring the point cloud data and panoramic images of all shooting points.

[0136] It should be noted that if the information collection terminal is integrated into the control terminal, the control terminal and the information collection terminal synchronously acquire the point cloud data and panoramic image of the target space.

[0137] Afterwards, the control terminal obtains the spatial contour of the target space based on the obtained point cloud data of at least one shooting point in the target space.

[0138] It is understood that since the point cloud data for at least one shooting point is collected in the target space, the point cloud data for at least one shooting point can be fused based on the relative positional relationship between the at least one shooting point in the target space to determine the target point cloud data for the target space. Compared to the point cloud data for a single shooting point, the target point cloud data obtained after fusion contains more data, is denser, and better reflects the spatial structure of the target space. Furthermore, based on the target point cloud data, an accurate spatial outline of the target space can be obtained.

[0139] In a specific implementation, when determining the spatial contour of a target space based on target point cloud data, the target point cloud data is first mapped onto a two-dimensional plane to obtain a point cloud image of the target space. Then, the spatial contour of the target space is determined based on the point cloud image. As an optional implementation, the control terminal can identify the spatial contour corresponding to the point cloud image using, for example, an edge detection algorithm.

[0140] It can be understood that point cloud data is actually a series of three-dimensional coordinate points. Any three-dimensional coordinate point can be represented by three-dimensional Cartesian coordinates (x, y, z), where x, y, and z are the coordinate values ​​of the x-axis, y-axis, and z-axis, which have a common zero point and are orthogonal to each other.

[0141] When mapping the target point cloud data of the target space to a two-dimensional plane to obtain a point cloud image of the target space, in an optional embodiment, the three-dimensional coordinate points (x, y, z) corresponding to the target point cloud data can be converted to two-dimensional coordinate points (x, y), for example, by setting the z-axis coordinate value of the three-dimensional coordinate point to 0, and then, based on the converted two-dimensional coordinate point, a plane point cloud image of the target space is obtained. In another optional embodiment, a three-dimensional spatial structure diagram of the target space can be first generated based on the three-dimensional coordinate points (x, y, z) corresponding to the target point cloud data, and then a top view of the three-dimensional spatial structure diagram is obtained to serve as the two-dimensional point cloud image of the target space.

[0142] In order to obtain an accurate spatial contour of the target space, as an optional implementation method, after obtaining a point cloud image of the target space, the method further includes: receiving a user's correction operation on the point cloud image, and determining the spatial contour of the target space based on the target point cloud image obtained after the correction operation.

[0143] The aforementioned correction operation includes cropping. In practical applications, the target space often contains objects that affect point cloud data acquisition, such as glass and mirrors. These objects can cause interference data in the acquired point cloud data. This interference data is reflected in the point cloud image, specifically manifesting as the presence of part of the image outside the regular wall lines in the point cloud image (i.e., the interference data corresponds to the interference image), or the wall lines in the point cloud image are blurred. The wall lines in the point cloud image correspond to the walls in the target space.

[0144] FIG3 is a schematic diagram of a point cloud image provided by the first embodiment of the present invention. In actual applications, the user can optionally modify the point cloud image using the edit button on the point cloud image editing interface. For example, as shown in FIG3 , by adding wall lines, the interfering image outside the wall lines is trimmed to obtain the target point cloud image. The wall lines in the target point cloud image are clear, and a relatively accurate spatial contour can be identified based on the target point cloud image. It is understood that the spatial contour of the target space is composed of multiple contour lines, and the contour lines correspond to the wall lines in the point cloud image.

[0145] However, in actual applications, the spatial contour may include a target contour line that does not correspond to a wall line in the point cloud image. Therefore, in another optional embodiment, in response to a user's editing operation on the target contour line in the spatial contour of the target space, the shape and / or position of the target contour line can be adjusted to make the target contour line coincide with the wall line in the point cloud image. For example, the length and position of the target contour line l can be adjusted to make the target contour line l coincide with the wall line L in the point cloud image, where the target contour line l and the wall line L correspond to the same wall in the target space.

[0146] Optionally, the control terminal is preset with other contour line correction options for the space contour, such as: adding contour line option, deleting contour line option, etc.

[0147] After acquiring the spatial outline of the target space, it is necessary to annotate the physical media within the target space, such as doors and windows, within the spatial outline. In this embodiment, the target media is first identified based on the panoramic image, and then the corresponding mapping media are determined. For ease of distinction, the images corresponding to the doors and windows in the target space in the panoramic image are referred to as target media, and the corresponding markings of the doors and windows in the spatial outline are referred to as mapping media. Optionally, the target media in the panoramic image can be identified based on a pre-set image recognition algorithm.

[0148] In actual applications, the target medium may be included in the panoramic images corresponding to more than one shooting point. In an optional embodiment, to accelerate the control terminal's recognition of the target medium, before the control terminal recognizes the target medium in the panoramic image, it may also determine a target panoramic image from the panoramic images of at least one shooting point for use in identifying the target medium. The target panoramic image is a panoramic image that meets preset recognition requirements, such as a panoramic image with the widest field of view, the best lighting, or a panoramic image that includes user-marked information (e.g., the best panoramic image).

[0149] The shooting points corresponding to the target panoramic image can be the same as or different from the shooting points corresponding to the point cloud data used to generate the spatial outline. Assume that the target space contains two shooting points, shooting point A and shooting point B. Panoramic image A1 and point cloud data A2 are acquired at shooting point A, and panoramic image B1 and point cloud data B2 are acquired at shooting point B. If the spatial outline is generated based on point cloud data A2, then panoramic image A1 can be determined to be the target panoramic image, or panoramic image B1 can be determined to be the target panoramic image. Similarly, if the spatial outline is generated based on point cloud data B2, then panoramic image A1 can be determined to be the target panoramic image, or panoramic image B1 can be determined to be the target panoramic image.

[0150] It is understandable that the door body and the window body have corresponding size information. In order to facilitate the user to understand the spatial structure of the target space, the mapping medium added to the spatial outline of the target space should at least be able to reflect the position information, size information and type information of the door body and / or window body contained in the target space.

[0151] As an optional implementation manner, determining the mapping medium corresponding to the target medium in the spatial profile includes:

[0152] According to the mapping relationship between the target panoramic image and the spatial contour, the panoramic pixel coordinates corresponding to the target medium in the target panoramic image and the mapped spatial contour coordinates are obtained to determine the mapping medium corresponding to the target medium in the spatial contour.

[0153] The mapping medium is adapted to the target identifier and target display size of the target medium. The target identifier is used to distinguish target media of different types. For example, target media belonging to a door or a window correspond to different target identifiers.

[0154] The mapping relationship between the target panoramic image and the spatial outline is a mapping between the target panoramic image and the spatial outline established based on the coordinate mapping between the point cloud data and the target panoramic image.

[0155] It is understood that the relative position between the laser sensor and camera is pre-calibrated before point cloud data and panoramic image acquisition. Based on this pre-calibrated relative position and the relative positional relationship between the actual shooting points, the coordinate mapping between the 3D point cloud coordinates corresponding to the collected point cloud data and the panoramic pixel coordinates of the panoramic image can be determined.

[0156] In the embodiments of the present invention, the specific method of mapping the coordinates of the panoramic image and the point cloud data is not limited. Optionally, the panoramic pixel coordinates can be directly mapped to the three-dimensional point cloud coordinates, and the three-dimensional point cloud coordinates can be mapped to the panoramic pixel coordinates based on the relative posture relationship between the devices that obtain the panoramic image and the point cloud data. Alternatively, the panoramic pixel coordinates can be first mapped to the intermediate coordinates, and then mapped to the three-dimensional point cloud coordinates, with the help of the relative posture relationship and the intermediate coordinate system. Alternatively, the three-dimensional point cloud coordinates can be first mapped to the intermediate coordinates, and then mapped to the intermediate coordinates. Here, the specific type of the intermediate coordinate system is not limited, nor is the specific method used in the coordinate mapping process. The mapping method used will also be different depending on the different intermediate coordinate systems and the different relative posture relationships.

[0157] Figure 4 is a schematic diagram of a spatial structure diagram provided by the first embodiment of the present invention. Assume that the actual distribution of doors and windows in target space Y is as shown in the left figure of Figure 4, including door a, window b, and window c. Based on the spatial structure diagram generation method provided by this embodiment of the present invention, the generated spatial structure diagram of target space Y is shown in the right figure of Figure 4. The mapping media corresponding to the target media door a, window b, and window c in the target space are marked on the spatial outline j of the target space. The corresponding position, size, and type of the mapping media match the actual target media in the target space.

[0158] In this solution, by identifying the target medium in the target panoramic image (the corresponding images of the door and window in the target panoramic image), the corresponding mapping medium of the door and window in the spatial contour of the target space is determined, so that the spatial structure diagram of the target space finally obtained contains an accurate spatial contour, and the mapping medium used to represent the door and window is marked at the correct position on the spatial contour, so that the spatial structure diagram can accurately reflect the actual spatial structure of the target space.

[0159] FIG5 is a flowchart of another method for generating a spatial structure diagram according to the first embodiment of the present invention. As shown in FIG5 , the method for generating a spatial structure diagram includes the following steps:

[0160] 501. Acquire point cloud data and a panoramic image obtained by an information collection terminal at least one shooting point in a target space.

[0161] 502. Receive a user's confirmation operation on the validity of the received point cloud data of at least one shooting point and the panoramic image, so as to obtain a spatial contour of the target space based on the point cloud data of at least one shooting point.

[0162] 503. Acquire a target medium identified in the target panoramic image. The target medium is an image of a physical medium in the target space in the target panoramic image. The target panoramic image is a panoramic image of at least one shooting point and is used to identify the target medium.

[0163] 504. Determine a mapping medium corresponding to the target medium in the spatial outline to obtain a spatial structure diagram of the target space.

[0164] The specific implementation process of steps 501, 503 and 504 can be referred to the above embodiment and will not be repeated in this embodiment.

[0165] In this embodiment, confirming the validity of the point cloud data and the panoramic image includes: confirming whether the panoramic image received from at least one shooting point meets the shooting requirements, and confirming whether the point cloud data of multiple shooting points currently received can fully represent the target space.

[0166] During the specific implementation process, for a panoramic image of a certain shooting point, if the door or window in the panoramic image is blocked, or there is a stitching error in the panoramic image (for example, image misalignment), the panoramic image of the shooting point is considered to not meet the shooting requirements, that is, the panoramic image is invalid.

[0167] If a panoramic image is invalid, the information collection terminal is instructed to reacquire point cloud data and a panoramic image at the same shooting point. If a panoramic image is valid, a validity confirmation operation is performed on the panoramic image. Optionally, the validity confirmation operation is a confirmation operation triggered by the user on the control terminal interface for the panoramic image.

[0168] The purpose of validating point cloud data is to confirm whether the received point cloud data of at least one shooting point can fully represent the target space, that is, to confirm whether to add a new shooting point to collect point cloud data.

[0169] During the specific implementation process, if the user confirms that the point cloud data currently received from multiple shooting points can fully represent the target space, that is, the multiple point cloud data currently received contain all the scene information of the target space, then the validity confirmation operation is performed on the point cloud data. If the user confirms that the point cloud data currently received from multiple shooting points cannot fully represent the target space, then a new shooting point is added and point cloud data is obtained at the new shooting point to make up for the missing point cloud data corresponding to the target space in addition to the point cloud data of at least one shooting point currently received.

[0170] In an optional embodiment, to facilitate user confirmation of whether the currently received point cloud data for multiple shooting locations fully represents the target space, after acquiring the point cloud data for the current shooting location, the point cloud data for at least one currently received shooting location is fused and mapped onto a two-dimensional plane to produce a point cloud image for the at least one shooting location. Subsequently, the displayed point cloud image is determined to be consistent with the actual spatial structure of the target space to determine whether the currently received point cloud data for the at least one shooting location fully represents the target space. If so, the target space can be fully represented; if not, the target space cannot be fully represented.

[0171] In the above embodiment, the validity confirmation operation of the point cloud data and panoramic image of at least one shooting point is actually confirming the correctness and integrity of the original data used to generate the spatial structure diagram of the target space. Based on the confirmation operation of the correctness and integrity of the original data, an accurate spatial structure diagram can be generated.

[0172] The following describes in detail one or more embodiments of the spatial structure diagram generating apparatus of the present invention. Those skilled in the art will appreciate that these apparatuses can be configured using commercially available hardware components and the steps taught in this solution.

[0173] FIG6 is a schematic structural diagram of a spatial structure diagram generating device provided by the first embodiment of the present invention. The device is applied to a control terminal. As shown in FIG6 , the device includes: an acquisition module 11 and a processing module 12 .

[0174] The acquisition module 11 is used to acquire point cloud data and a panoramic image obtained by the information acquisition terminal at least one shooting point in the target space.

[0175] The processing module 12 is used to obtain the spatial outline of the target space based on the point cloud data of the at least one shooting point; obtain the target medium identified in the target panoramic image, where the target medium is the image of the physical medium in the target space in the target panoramic image, and the target panoramic image is a panoramic image for identifying the target medium in the panoramic image of the at least one shooting point; determine the mapping medium corresponding to the target medium in the spatial outline to obtain a spatial structure diagram of the target space.

[0176] Optionally, the processing module 12 is specifically used to fuse the point cloud data of the at least one shooting point to determine the target point cloud data of the target space; map the target point cloud data to a two-dimensional plane to obtain a point cloud image of the target space; and determine the spatial contour of the target space based on the point cloud image.

[0177] Optionally, the acquisition module 11 is further configured to receive a user's correction operation on the point cloud image.

[0178] Correspondingly, the processing module 12 is specifically configured to determine the spatial contour of the target space according to the target point cloud image obtained after the correction operation.

[0179] Optionally, the processing module 12 is specifically configured to adjust the shape and / or position of the target contour line in response to an editing operation on the target contour line on the spatial contour of the target space, so that the target contour line coincides with the wall line in the point cloud image.

[0180] Optionally, the processing module 12 is specifically used to obtain the panoramic pixel coordinates corresponding to the target medium in the target panoramic image and the mapped spatial contour coordinates based on the mapping relationship between the target panoramic image and the spatial contour, so as to determine the mapping medium corresponding to the target medium in the spatial contour; wherein the mapping medium is adapted to the target identification and target display size of the target medium, the target identification is used to distinguish target media belonging to different types, and the mapping relationship is a mapping between the target panoramic image and the spatial contour established based on the coordinate mapping between the point cloud data and the target panoramic image.

[0181] The device shown in FIG6 can execute the steps in the aforementioned embodiments. For detailed execution process and technical effects, please refer to the description in the aforementioned embodiments and will not be repeated here.

[0182] Second embodiment

[0183] FIG7 is a schematic diagram of a floor plan generation system according to a second embodiment of the present invention. As shown in FIG7 , the floor plan generation system includes an information collection terminal and a target control terminal. Optionally, the information collection terminal can be directly integrated with the target control terminal, functioning as a single entity. Alternatively, the information collection terminal can be decoupled from the target control terminal and separately configured, communicating with the target control terminal via, for example, Bluetooth or a Wireless Fidelity (WiFi) hotspot.

[0184] It is understandable that a physical space usually contains multiple subspaces. Taking the target physical space as a building space as an example, a building space may contain a living room, a kitchen and two bedrooms, among which the living room, kitchen and bedroom can all be considered as a subspace.

[0185] When generating a floor plan of the target physical space, the information collection terminal is used to collect point cloud data and panoramic images corresponding to multiple subspaces in the target physical space. The point cloud data and panoramic image of any subspace are collected at at least one shooting point in any subspace. For example, the point cloud data and panoramic image of subspace X include: point cloud data a and panoramic image a collected at shooting point a in subspace X, and point cloud data b and panoramic image b collected at shooting point b in subspace X. Shooting point a and shooting point b can be selected by the user in subspace X according to modeling needs; or they can be reference shooting points generated for subspace X by the target control terminal based on the description information of subspace X input by the user (such as the size of the space).

[0186] In reality, the information collection process for the information collection terminal is the same for any shooting point in any subspace. To facilitate understanding, the data collection process for the information collection terminal is described using the process of collecting point cloud data and panoramic images at the target shooting point Y in subspace X as an example.

[0187] As shown in Figure 7, the information acquisition terminal includes a laser sensor, a camera, a motor, and a processor (such as a CPU). The laser sensor and camera serve as sensing devices for acquiring scene information of subspace X, namely, point cloud data and panoramic images of subspace X.

[0188] For the target shooting point Y, the information collection terminal responds to the information collection instruction, drives the laser sensor to rotate 360 ​​degrees through the driving motor to collect the point cloud data corresponding to the target shooting point Y; and drives the camera to rotate 360 ​​degrees through the driving motor to collect the panoramic image corresponding to the target shooting point Y.

[0189] Optionally, the camera in the information collection terminal is a panoramic camera or a non-panoramic camera. If the camera in the information collection terminal is a non-panoramic camera, during the 360-degree rotation process, the camera is controlled to capture images corresponding to the target capture point Y at multiple preset angles. The processor can stitch the images captured at the multiple preset angles into a panoramic image using a panoramic image stitching algorithm, such as a feature matching algorithm.

[0190] Optionally, multiple preset angles can be customized by the user according to the camera's viewing angle. The images taken based on multiple preset angles contain scene information within a 360-degree range of the current point. For example, if the camera's viewing angle is 180 degrees, a certain reference direction can be set as 0 degrees, and a degree and (a+180) degrees based on the reference direction are determined as preset angles.

[0191] Optionally, the information collection instruction is sent by the target control terminal, or the information collection instruction is triggered in response to a triggering operation of the user on the information collection terminal.

[0192] At the target shooting point Y, the point cloud data and the panoramic image may be collected simultaneously, or may be collected sequentially in sequence, which is not limited in this embodiment.

[0193] In an optional embodiment, if point cloud data is collected first and then the panoramic image is captured, to improve the quality of the captured panoramic image, the camera can be activated simultaneously with the point cloud data acquisition process to collect scene lighting information at the current capture point, perform light measurement, and determine corresponding exposure parameters. The camera then captures the panoramic image based on the determined exposure parameters.

[0194] Optionally, when generating a panoramic image, High Dynamic Range Imaging (HDRI) may be combined to generate a high-quality panoramic image.

[0195] Optionally, the information acquisition terminal also includes an inertial measurement unit (IMU). The IMU is used to correct the pose information corresponding to the collected point cloud data and image data to reduce errors caused by environmental or human factors (such as: the information acquisition terminal is not placed horizontally, etc.).

[0196] Optionally, the floor plan generation system shown in Figure 7 also includes a cloud server, which can be a physical server or a virtual server in the cloud. The target control terminal communicates with the cloud server by accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks.

[0197] In an optional embodiment, the cloud server may receive point cloud data and panoramic images corresponding to multiple subspaces forwarded by the target control terminal to generate a floor plan of the target physical space, and then feed the floor plan back to the control terminal for display. In practice, the cloud server generates the floor plan of the target physical space through the same process as the target control terminal generates the spatial structure diagram. However, due to the cloud server's greater computing power, it generates the floor plan of the target physical space more efficiently, further enhancing the user experience.

[0198] In another optional embodiment, the cloud server may also be directly connected to the information collection terminal to directly obtain the point cloud data and panoramic images corresponding to the multiple subspaces collected by the information collection terminal to generate a floor plan of the target physical space.

[0199] In this embodiment, the target control terminal is used as an example to illustrate the process of generating a floor plan of the target physical space based on the point cloud data and panoramic images corresponding to multiple subspaces in the target physical space obtained by the information collection terminal. The floor plan can be understood as a two-dimensional planar structure diagram of the target physical space.

[0200] The target control terminal can be a terminal device with data processing capabilities, such as a smart phone, tablet computer, laptop computer, etc. The following describes the process of generating a floor plan of a target physical space from the perspective of the target control terminal in conjunction with specific embodiments.

[0201] FIG8 is a flow chart of a method for generating a floor plan according to a second embodiment of the present invention, which is applied to a target control terminal. As shown in FIG8 , the method for generating a floor plan includes the following steps:

[0202] 801. Obtain point cloud data and panoramic images corresponding to a plurality of subspaces in a target physical space obtained by an information collection terminal to determine a plurality of spatial contours corresponding to the plurality of subspaces.

[0203] The multiple subspaces correspond one-to-one to the multiple space contours, and the point cloud data and panoramic image of any subspace are collected from at least one shooting point in any subspace.

[0204] 802. Display multiple space contours corresponding to the multiple subspaces for editing.

[0205] 803. For a target subspace among multiple subspaces, obtain a target medium identified in a target panoramic image. The target medium is an image of a physical medium in the target subspace in the target panoramic image. The target panoramic image is a panoramic image captured by at least one shooting point of the target subspace and is used to identify the target medium.

[0206] 804. Determine a mapping medium corresponding to the target medium in the spatial contour of the target subspace to generate a spatial structure diagram of the target subspace.

[0207] 805 . In response to the completion of the operation of obtaining the multiple space structure diagrams corresponding to the multiple subspaces, obtain a floor plan of the target physical space obtained by splicing the multiple space structure diagrams.

[0208] In this embodiment, the target physical space may be a building space, a complex connected structure or container, a vehicle, etc. For ease of understanding, this embodiment uses a building space (e.g., an office area or an indoor area of ​​a residential building) as an example for description, but is not limited thereto.

[0209] In step 801, if the information acquisition terminal is integrated with the target control terminal, the target control terminal can directly and synchronously acquire the point cloud data and panoramic images of the multiple subspaces obtained by the information acquisition terminal. If the information acquisition terminal is connected to the target control terminal via a communication link, the target control terminal receives the point cloud data and panoramic images of the multiple subspaces transmitted by the information acquisition terminal based on the pre-established communication link. The process by which the information acquisition terminal acquires the point cloud data and panoramic images corresponding to the multiple subspaces in the target physical space can be referred to in the previous embodiment. This embodiment mainly describes the processing process after the target control terminal acquires the point cloud data and panoramic images corresponding to the multiple subspaces.

[0210] First, based on the acquired point cloud data and panoramic images corresponding to the multiple subspaces, the spatial contours corresponding to the multiple subspaces are determined.

[0211] Optionally, for any one of the multiple subspaces (referred to as the target subspace), the spatial contour of the target subspace is determined based on the point cloud data and panoramic image of the target subspace, including: determining the spatial contour of the target subspace based on the point cloud data of at least one shooting point in the target subspace, and / or, the panoramic image of at least one shooting point.

[0212] During the specific implementation process, a first spatial contour can be obtained based on the point cloud data of at least one shooting point, and the first spatial contour can be directly used as the spatial contour of the target subspace; or, a second spatial contour can be obtained based on the panoramic image of at least one shooting point, and the second spatial contour can be directly used as the spatial contour of the target subspace; or, a spatial contour with better contour line quality is selected from the above-mentioned first spatial contour and the above-mentioned second spatial contour as the spatial contour of the target subspace, or the contour lines of the above-mentioned first spatial contour and the above-mentioned second spatial contour are fused to obtain a spatial contour with better contour line quality, and the fused spatial contour can be directly used as the spatial contour of the target subspace.

[0213] Optionally, determining a spatial contour of the target subspace according to point cloud data of at least one shooting point in the target subspace includes:

[0214] Based on the relative positional relationship between at least one shooting point in the target subspace, the point cloud data of at least one shooting point are fused; the point cloud data after the fusion processing is determined to be the target point cloud data of the target subspace; the target point cloud data is mapped to a two-dimensional plane to obtain an initial point cloud image of the target subspace; a correction operation of the user on the initial point cloud image is received, and the point cloud image obtained after the correction operation is determined to be the point cloud image of the target subspace, and the spatial contour of the target subspace is determined by, for example, an edge detection algorithm.

[0215] It can be understood that point cloud data is actually a series of three-dimensional coordinate points. Any three-dimensional coordinate point can be represented by three-dimensional Cartesian coordinates (x, y, z), where x, y, and z are the coordinate values ​​of the x-axis, y-axis, and z-axis, which have a common zero point and are orthogonal to each other.

[0216] In an optional embodiment, the target point cloud data is mapped to a two-dimensional plane to obtain an initial point cloud image of the target subspace, including: converting the three-dimensional coordinate points (x, y, z) corresponding to the target point cloud data into two-dimensional coordinate points (x, y), for example, setting the z-axis coordinate value of the three-dimensional coordinate points to 0, and then obtaining an initial point cloud image of the target subspace based on the converted two-dimensional coordinate points, the initial point cloud image being a two-dimensional image. Alternatively, a three-dimensional spatial structure diagram of the target subspace is first generated based on the three-dimensional coordinate points (x, y, z) corresponding to the target point cloud data, and then obtaining a top view of the three-dimensional spatial structure diagram as the initial point cloud image of the target subspace.

[0217] The aforementioned correction operation on the initial point cloud image includes cropping. In practical applications, the target subspace often contains objects that affect point cloud data acquisition, such as glass and mirrors. These objects can cause some interference data in the acquired point cloud data. This interference data is reflected in the point cloud image, specifically manifesting as partial images outside the regular wall lines in the point cloud image (i.e., interference data corresponds to interference images), or the wall lines in the point cloud image are blurred. The wall lines in the point cloud image correspond to the walls in the target space.

[0218] Referring to Figure 3 of the first embodiment, in actual applications, users can optionally modify the point cloud image using the edit button on the point cloud image editing interface. For example, as shown in Figure 3, by adding a wall line, interfering imagery outside the wall line is trimmed to obtain the target point cloud image. The wall line is clearly visible in the target point cloud image, and a relatively accurate spatial outline can be identified based on the target point cloud image.

[0219] After determining the spatial contours corresponding to multiple subspaces, the multiple spatial contours can be displayed simultaneously to edit the multiple spatial contours at the same time, and the spatial structure diagrams corresponding to the multiple subspaces can be generated in parallel; or, the multiple spatial contours can be displayed in sequence to edit the multiple spatial contours one by one, and the spatial structure diagrams corresponding to the multiple subspaces can be generated one by one.

[0220] It is understandable that, whether the spatial structure graphs corresponding to multiple subspaces are generated in parallel or serially, the process for generating the spatial structure graph of any one of the multiple subspaces (referred to as the target subspace) is the same. Therefore, the following description uses the process of generating the spatial structure graph of the target subspace as an example.

[0221] The spatial outline of the target subspace is composed of multiple contour lines. In actual applications, the automatically identified spatial outline may contain target contour lines that do not correspond to the actual wall positions. Optionally, the target control terminal has preset correction options for the contour lines of the spatial outline, such as options for adjusting the shape and / or position of contour lines, adding contour lines, and deleting contour lines.

[0222] To ensure that the spatial outline accurately reflects the spatial structure of the target subspace, the spatial outline of the target subspace is displayed on a point cloud image of the target subspace. The point cloud image is determined based on point cloud data of at least one captured point in the target subspace, and the point cloud image includes wall lines corresponding to walls in the target subspace.

[0223] For target contour lines in the spatial contour that do not correspond to the actual wall position, the target contour line's shape and / or position can be optionally adjusted in response to a user's editing operation on the target subspace's spatial contour to align the target contour line with the wall line in the point cloud image. For example, the length and position of target contour line l can be adjusted to align with wall line L in the point cloud image, where target contour line l and wall line L correspond to the same wall in the target space. Alternatively, target contour lines that do not have a corresponding wall line can be deleted; alternatively, target contour lines corresponding to a specific wall line can be added.

[0224] After editing the spatial outline of the target subspace, the physical media within the target subspace, namely the door and window, need to be annotated within the resulting spatial outline. Specifically, the target media are first identified based on the panoramic image, and then the corresponding mapping media are determined. For ease of distinction, the images corresponding to the door and window in the target space in the panoramic image are referred to as target media, and the corresponding markings of the door and window in the spatial outline are referred to as mapping media. Alternatively, the target media in the panoramic image can be identified based on a pre-set image recognition algorithm.

[0225] In actual applications, the target medium may be included in the panoramic images corresponding to more than one shooting point. To improve the efficiency of the target control terminal in identifying the target medium, the target control terminal may optionally determine a target panoramic image from the panoramic images corresponding to at least one shooting point for use in identifying the target medium before identifying the target medium in the panoramic image.

[0226] The target panoramic image is a panoramic image that meets preset recognition requirements, such as a panoramic image with the widest field of view and the best lighting, or a panoramic image containing user marking information (such as the best panoramic image).

[0227] For the target subspace, the shooting points corresponding to the target panoramic image can be the same as or different from the shooting points corresponding to the point cloud data used to generate the spatial contour. Assume that the target space contains two shooting points, shooting point A and shooting point B. Panoramic image A1 and point cloud data A2 are acquired at shooting point A, and panoramic image B1 and point cloud data B2 are acquired at shooting point B. If the spatial contour is generated based on point cloud data A2, then panoramic image A1 can be determined to be the target panoramic image, or panoramic image B1 can be determined to be the target panoramic image. Similarly, if the spatial contour is generated based on point cloud data B2, then panoramic image A1 can be determined to be the target panoramic image, or panoramic image B1 can be determined to be the target panoramic image.

[0228] It can be understood that the door body and the window have corresponding size information. In order to facilitate the user to understand the spatial structure of the target subspace, the mapping medium added to the spatial outline of the target subspace should at least be able to reflect the position information, size information and type information of the door body and / or window contained in the target subspace.

[0229] As an optional implementation, determining the mapping medium corresponding to the target medium in the spatial contour of the target subspace includes:

[0230] According to the mapping relationship between the target panoramic image and the spatial contour of the target subspace, the panoramic pixel coordinates corresponding to the target medium in the target panoramic image and the mapped spatial contour coordinates are obtained to determine the mapping medium corresponding to the target medium in the spatial contour of the target subspace.

[0231] The mapping medium is adapted to the target identifier and target display size of the target medium. The target identifier is used to distinguish target media of different types. For example, target media belonging to a door or a window correspond to different target identifiers.

[0232] The above-mentioned mapping relationship between the target panoramic image and the space outline is a mapping between the target panoramic image and the space outline established based on the coordinate mapping between the point cloud data of the target subspace and the target panoramic image.

[0233] It is understood that the relative pose between the laser sensor and camera is pre-calibrated before point cloud data and panoramic image acquisition. Based on this pre-calibrated relative pose and the relative positional relationship between the actual captured points in the target subspace, the coordinate mapping between the 3D point cloud coordinates corresponding to the acquired point cloud data and the panoramic pixel coordinates of the panoramic image can be determined.

[0234] In the embodiments of the present invention, the specific method of mapping the coordinates of the panoramic image and the point cloud data is not limited. Optionally, the panoramic pixel coordinates can be directly mapped to the three-dimensional point cloud coordinates, and the three-dimensional point cloud coordinates can be mapped to the panoramic pixel coordinates based on the relative posture relationship between the devices that obtain the panoramic image and the point cloud data. Alternatively, the panoramic pixel coordinates can be first mapped to the intermediate coordinates, and then mapped to the three-dimensional point cloud coordinates, with the help of the relative posture relationship and the intermediate coordinate system. Alternatively, the three-dimensional point cloud coordinates can be first mapped to the intermediate coordinates, and then mapped to the intermediate coordinates. Here, the specific type of the intermediate coordinate system is not limited, nor is the specific method used in the coordinate mapping process. The mapping method used will also be different depending on the different intermediate coordinate systems and the different relative posture relationships.

[0235] Based on the above method, spatial structure diagrams corresponding to multiple subspaces can be obtained.

[0236] In response to the completion of the operation of obtaining the multiple space structure diagrams corresponding to the multiple subspaces, a floor plan of the target physical space obtained by splicing the multiple space structure diagrams is obtained.

[0237] It is understandable that multiple subspaces in the target physical space are connected to each other through doors or windows. For example, subspace 1 and subspace 2 are connected through the same target door. Since the target door is open when the panoramic image is captured, panoramic image 1 of subspace 1 may contain a partial image m of subspace 2. Spatially, the area corresponding to image m is outside subspace 1 but within the field of view of the camera when panoramic image 1 was captured. Therefore, in practical applications, based on panoramic image 1 of subspace 1 and panoramic image 2 of subspace 2, the spatial connection relationship between subspace 1 and subspace 2 can be determined through methods such as feature matching. Subsequently, based on this spatial connection relationship, the spatial structure diagram of subspace 1 can be spliced ​​with the spatial structure diagram of subspace 2. Similarly, the spatial structure diagrams of the other subspaces are spliced, and the resulting spliced ​​image is the floor plan of the target physical space.

[0238] Optionally, the spatial connection relationship between the subspaces may be determined based on the point cloud data corresponding to the subspaces.

[0239] For ease of understanding, for example, Figure 9 is a schematic diagram of a floor plan of a target physical space provided in the second embodiment of the present invention, wherein a in Figure 9 is the actual spatial structure of the target physical space, b in Figure 9 is a spatial structure diagram of multiple subspaces in the target physical space, and c in Figure 9 is the floor plan of the target physical space.

[0240] As shown in a in Figure 9, it is assumed that the target physical space contains three subspaces, namely bedroom 1, bedroom 2 and living room 3, where bedroom 1 and living room 3 are connected through door 1, bedroom 2 and living room 3 are connected through door 2, and bedroom 1 and bedroom 2 are connected through window 3.

[0241] Based on the method provided by an embodiment of the present invention, the generated spatial structure diagrams x for bedroom 1, y for bedroom 2, and z for living room 3 are shown in Figure 9(b). Each spatial structure diagram includes a space outline marked with mapping media representing doors and windows. The floor plan of the target physical space is obtained by concatenating spatial structure diagrams x, y, and z, as shown in Figure 9(c).

[0242] In this solution, for a target physical space with multiple subspaces, when generating a floor plan of the target physical space, a spatial structure diagram of each subspace is first generated, and then the multiple spatial structure diagrams corresponding to the multiple subspaces are spliced ​​together to obtain the floor plan of the target physical space. Among them, the spatial structure diagram of the target subspace includes the spatial outline of the subspace and the mapping media corresponding to the windows and doors (i.e., the target media) in the subspace. Based on the editing of the spatial outline, the accurate spatial outline corresponding to the target subspace can be obtained; based on the panoramic image, the position of the mapping medium corresponding to the target medium in the target subspace in the spatial outline can be determined, and the mapping medium corresponding to the target medium can be identified at the correct position on the spatial outline, so that the determined spatial structure diagram can accurately reflect the actual spatial structure information of the target subspace, and then the floor plan determined based on the multiple spatial structure diagrams can also accurately reflect the actual spatial structure of the target physical space.

[0243] To improve the efficiency of generating floor plans for the target physical space, multiple control terminals can be used to simultaneously edit the spatial outlines of multiple subspaces and generate corresponding spatial structure diagrams. This collaboration of multiple control terminals reduces the time required to generate floor plans for the target physical space.

[0244] FIG10 is an interactive flow chart of another floor plan generation method provided by the second embodiment of the present invention. The floor plan generation method includes the following steps:

[0245] 1001. The target control terminal obtains point cloud data and panoramic images corresponding to multiple subspaces in the target physical space obtained by the information collection terminal to determine multiple space contours corresponding to the multiple subspaces.

[0246] The multiple subspaces correspond one-to-one to the multiple space contours, and the point cloud data and panoramic image of any subspace are collected from at least one shooting point in any subspace.

[0247] 1002. The target control terminal displays multiple spatial contours corresponding to the multiple subspaces for synchronous editing with other control terminals (1002-1), and sends the multiple spatial contours corresponding to the multiple subspaces to other control terminals (1002-2), so that the multiple spatial contours corresponding to the multiple subspaces are displayed on the other control terminals for synchronous editing with the target control terminal (1002-3).

[0248] 1003. The target control terminal and other control terminals obtain the target medium identified in the target panoramic image for the target subspace among the multiple subspaces, where the target medium is the image of the physical medium in the target subspace in the target panoramic image; and determine the mapping medium corresponding to the target medium in the spatial contour of the target subspace to generate a spatial structure diagram of the target subspace.

[0249] The target panoramic image is a panoramic image captured by at least one shooting point in the target subspace and is used to identify the target medium.

[0250] 1004. The target control terminal obtains the spatial structure diagram of the subspace obtained by other control terminals.

[0251] 1005. In response to the completion of the operation of obtaining the multiple spatial structure diagrams corresponding to the multiple subspaces, the target control terminal obtains a floor plan of the target physical space obtained by splicing the multiple spatial structure diagrams.

[0252] In this embodiment, the target control terminal and other control terminals can respectively edit the spatial contours of different subspaces and generate spatial structure diagrams of the corresponding subspaces. For example: the target control terminal is used to edit the spatial contours of subspace 1 and generate a spatial structure diagram of subspace 1; the other control terminals are used to edit the spatial contours of subspace 2 and generate a spatial structure diagram of subspace 2. Among them, the process of editing the spatial contours and generating the spatial structure diagram of each device is the same, and reference may be made to the aforementioned embodiment, which will not be repeated in this embodiment. It should be noted that the other control terminals synchronize the point cloud data and panoramic images corresponding to multiple subspaces in the target physical space.

[0253] To facilitate distinguishing the space profiles currently being edited by different devices, the target control terminal and other control terminals, when displaying multiple space profiles, may optionally further include: displaying terminal device identifiers corresponding to each of the multiple space profiles. The terminal device identifiers are used to indicate the terminal device currently editing each space profile, and the terminal devices include the target control terminal and other control terminals.

[0254] The target control terminal is communicatively connected to other control terminals. Optionally, in the process of generating a floor plan of the target physical space, the target control terminal can synchronously update multiple space outlines on the display interface according to the editing data returned by other control terminals.

[0255] Optionally, after generating the spatial structure diagram, the other control terminals can proactively feed it back to the target control terminal; alternatively, in response to a spatial structure diagram acquisition instruction from the target control terminal, they can send the generated spatial structure diagram to the target control terminal. If the target control terminal acquires the spatial structure diagrams corresponding to all subspaces, the acquired spatial structure diagrams are concatenated to obtain the floor plan of the target physical space. In practical applications, the number of other control terminals can be more than one, for example, to match the number of subspaces in the target physical space. This embodiment does not limit the number of other control terminals.

[0256] The following describes in detail the floor plan generation device according to one or more embodiments of the present invention. Those skilled in the art will appreciate that these devices can be constructed using commercially available hardware components and configured according to the steps taught in this solution.

[0257] FIG11 is a schematic structural diagram of a floor plan generating device provided in a second embodiment of the present invention. The device is applied to a target control terminal. As shown in FIG11 , the device includes: an acquisition module 21 , a display module 22 and a processing module 23 .

[0258] The acquisition module 21 is used to acquire the point cloud data and panoramic images corresponding to multiple subspaces in the target physical space obtained by the information acquisition terminal, so as to determine the multiple spatial contours corresponding to the multiple subspaces; wherein the multiple subspaces correspond one-to-one to the multiple spatial contours, and the point cloud data and panoramic images of any subspace are collected from at least one shooting point in any subspace.

[0259] The display module 22 is configured to display a plurality of space contours corresponding to the plurality of subspaces for editing.

[0260] The processing module 23 is used to obtain, for a target subspace among multiple subspaces, a target medium identified in a target panoramic image, where the target medium is an image of a physical medium in the target subspace in the target panoramic image, and the target panoramic image is a panoramic image captured by at least one shooting point of the target subspace, for identifying the target medium; determine a mapping medium corresponding to the target medium in the spatial contour of the target subspace to generate a spatial structure diagram of the target subspace; and in response to the completion operation of obtaining multiple spatial structure diagrams corresponding to the multiple subspaces, obtain a floor plan of the target physical space obtained by splicing the multiple spatial structure diagrams.

[0261] Optionally, the device further includes a sending module for sending the multiple space contours corresponding to the multiple subspaces to other control terminals, so as to display the multiple space contours corresponding to the multiple subspaces on the other control terminals for synchronous editing with the target control terminal.

[0262] Correspondingly, the acquisition module 21 is further configured to acquire the spatial structure diagram of the target subspace acquired by the other control terminals.

[0263] Optionally, the display module 22 is further configured to display terminal device identifiers corresponding to the multiple space outlines, wherein the terminal device identifiers are used to indicate the terminal device currently editing each space outline, and the terminal devices include the target control terminal and the other control terminals.

[0264] Optionally, the processing module 23 is specifically used to obtain a first spatial contour based on the point cloud data of at least one shooting point of the target subspace; obtain a second spatial contour based on the panoramic image of at least one shooting point of the target subspace; and determine the spatial contour of the target subspace based on the first spatial contour and the second spatial contour.

[0265] Optionally, the display module 22 is specifically used to adjust the shape and / or position of the target contour line in response to an editing operation on the target contour line on the spatial contour of the target subspace, so that the target contour line coincides with the wall line in the point cloud image, and the point cloud image is determined based on the point cloud data of at least one shooting point of the target subspace, and the spatial contour of the target subspace is composed of multiple contour lines.

[0266] Optionally, the processing module 23 is further specifically used to determine the spatial connection relationship between the multiple subspaces based on the point cloud data and / or panoramic images corresponding to the multiple subspaces; and splice the spatial structure diagrams according to the spatial connection relationship to obtain the floor plan of the target physical space.

[0267] Optionally, the processing module 23 further obtains the panoramic pixel coordinates corresponding to the target medium in the target panoramic image and the mapped spatial contour coordinates based on the mapping relationship between the target panoramic image and the spatial contour of the target subspace, so as to determine the mapping medium corresponding to the target medium in the spatial contour of the target subspace; wherein the mapping medium is adapted to the target identifier and the target display size of the target medium, the target identifier is used to distinguish target media belonging to different types, and the mapping relationship is a mapping between the target panoramic image and the spatial contour established based on the coordinate mapping between the point cloud data of the target subspace and the target panoramic image.

[0268] The device shown in FIG11 can execute the steps in the aforementioned embodiments. For detailed execution process and technical effects, please refer to the description in the aforementioned embodiments and will not be repeated here.

[0269] Third embodiment

[0270] In this embodiment, the floor plan of the target physical space can be understood as a two-dimensional planar structural diagram of the target physical space. Based on the floor plan of the target physical space, the user can obtain information about the distribution of each subspace in the target physical space and the connections between subspaces. Taking the target physical space as a residential space as an example, based on the floor plan of a residential space, the user can understand the location of subspaces such as the living room and bedroom, as well as the orientation of windows or doors in the subspaces, and thus understand the lighting conditions of the building space.

[0271] Therefore, in practical applications, generating an accurate floor plan for a target physical space helps users better understand the target physical space. For example, in the case of second-hand housing sales, an accurate floor plan can better display the structure of the house for sale, helping buyers better understand the house for sale, thereby increasing the transaction rate.

[0272] Referring to FIG. 7 of the second embodiment, as shown in FIG. 7 , the floor plan generation system includes: an information collection terminal and a control terminal (i.e., the target control terminal in FIG. 7 ). Optionally, the information collection terminal can be directly integrated with the control terminal, functioning as a single entity. Alternatively, the information collection terminal can be decoupled from the control terminal and separately provided, communicating with the control terminal via, for example, Bluetooth or a Wireless Fidelity (WiFi) hotspot.

[0273] The information collection terminal is used to collect point cloud data and panoramic images corresponding to multiple subspaces within the target physical space. Any subspace may contain more than one shooting point. Therefore, in this embodiment, the point cloud data and panoramic image of any subspace are collected from at least one shooting point within that subspace. For example, the point cloud data and panoramic image of subspace X include: point cloud data Xa and panoramic image Xa collected at shooting point a in subspace X, and point cloud data Xb and panoramic image Xb collected at shooting point b in subspace X.

[0274] Optionally, the shooting points can be selected by the user in the subspace according to modeling needs; or they can be reference shooting points automatically generated for the subspace by the control terminal based on the description information of the subspace input by the user (such as the size of the space, etc.).

[0275] In actual applications, the information collection terminal sequentially collects point cloud data and panoramic images at multiple shooting points in the target physical space. The corresponding information collection process is the same for any shooting point in any subspace. For ease of understanding, the data collection process of the information collection terminal is described using the process of collecting point cloud data and panoramic images at the target shooting point Y in subspace X as an example.

[0276] As shown in Figure 7, the information acquisition terminal includes a laser sensor, a camera, a motor, and a processor (such as a CPU). The laser sensor and camera serve as sensing devices for acquiring scene information of subspace X, namely, point cloud data and panoramic images of subspace X.

[0277] For the target shooting point Y, the information collection terminal responds to the information collection instruction, drives the motor to drive the laser sensor to rotate 360 ​​degrees to collect the point cloud data corresponding to the target shooting point Y; drives the motor to drive the camera to rotate 360 ​​degrees to collect the panoramic image corresponding to the target shooting point Y.

[0278] Optionally, the information collection instruction is sent by the control terminal, or the information collection instruction is triggered in response to a triggering operation of the user on the information collection terminal.

[0279] Optionally, at the same shooting point, the motor can drive the laser sensor and the camera to rotate at the same time to collect point cloud data and panoramic images at the same time, or it can drive the laser sensor and the camera to rotate in sequence to collect point cloud data and panoramic images respectively. This embodiment does not impose any restrictions on this.

[0280] In an optional embodiment, if point cloud data is collected first and then the panoramic image is captured, to improve the quality of the captured panoramic image, the camera can be activated simultaneously with the point cloud data acquisition process to collect scene lighting information at the current capture point, perform light measurement, and determine corresponding exposure parameters. The camera then captures the panoramic image based on the determined exposure parameters.

[0281] Optionally, the camera in the information collection terminal is a panoramic camera or a non-panoramic camera. If the camera in the information collection terminal is a non-panoramic camera, during the 360-degree rotation process, the camera is controlled to capture images corresponding to the target capture point Y at multiple preset angles. The processor can stitch the images captured at the multiple preset angles into a panoramic image using a panoramic image stitching algorithm, such as a feature matching algorithm.

[0282] Optionally, multiple preset angles can be customized by the user according to the camera's viewing angle. The images taken based on multiple preset angles contain scene information within a 360-degree range of the current point. For example, if the camera's viewing angle is 180 degrees, a certain reference direction can be set as 0 degrees, and a degree and (a+180) degrees based on the reference direction are determined as preset angles.

[0283] Optionally, when generating a panoramic image, High Dynamic Range Imaging (HDRI) may be combined to generate a high-quality panoramic image.

[0284] Optionally, the information acquisition terminal also includes an inertial measurement unit (IMU). The IMU is used to correct the pose information corresponding to the collected point cloud data and image data to reduce errors caused by environmental or human factors (such as: the information acquisition terminal is not placed horizontally, etc.).

[0285] The control terminal is used to generate a floor plan of the target physical space based on the point cloud data and panoramic images corresponding to multiple subspaces in the target physical space sent by the information collection terminal. The control terminal can be a terminal device with data processing capabilities, such as a smartphone, tablet computer, or laptop computer.

[0286] Optionally, the floor plan generation system shown in Figure 7 also includes a cloud server, which can be a physical server or a virtual server in the cloud. The control terminal communicates with the cloud server by accessing a wireless network based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks.

[0287] In an optional embodiment, the cloud server can receive point cloud data and panoramic images corresponding to multiple subspaces forwarded by the control terminal to generate a floor plan of the target physical space and feed the floor plan back to the control terminal for display. In practice, the cloud server generates the floor plan of the target physical space using the same process as the control terminal generates the spatial structure diagram. However, due to the cloud server's greater computing power, it generates the floor plan more efficiently, further enhancing the user experience.

[0288] In another optional embodiment, the cloud server may also be directly connected to the information collection terminal to directly obtain the point cloud data and panoramic images corresponding to the multiple subspaces collected by the information collection terminal to generate a floor plan of the target physical space.

[0289] The following describes the process of generating a floor plan of a target physical space from the perspective of a control terminal, using a control terminal as an example in conjunction with specific embodiments.

[0290] FIG12 is a flow chart of a floor plan generation method provided by a third embodiment of the present invention, which is applied to a control terminal. As shown in FIG12 , the floor plan generation method includes the following steps:

[0291] 1201. Obtain point cloud data and panoramic images corresponding to multiple subspaces in the target physical space obtained by an information collection terminal, wherein the point cloud data and panoramic image of any subspace are collected at at least one shooting point in any subspace.

[0292] 1202. In the process of sequentially performing spatial structure diagram acquisition processing on multiple subspaces, for the target subspace currently to be edited, a target space contour of the target subspace is acquired based on point cloud data and / or panoramic images collected at at least one shooting point of the target subspace.

[0293] 1203. Obtain a target medium identified in a target panoramic image. The target medium is an image of a physical medium in a target subspace in the target panoramic image. The target panoramic image is a panoramic image captured by at least one shooting point in the target subspace and is used to identify the target medium.

[0294] 1204. Determine a mapping medium for representing the target medium on the target space contour of the target subspace to obtain a spatial structure diagram of the target subspace.

[0295] 1205. In response to the completion of the acquisition of the spatial structure diagram of the target subspace, if there is no subspace in the multiple subspaces for which the spatial structure diagram has not been acquired, then acquire a floor plan of the target physical space obtained by splicing the spatial structure diagrams of the multiple spaces.

[0296] Among them, the process of the information collection terminal obtaining the point cloud data and panoramic images corresponding to multiple subspaces in the target physical space can be referred to the aforementioned embodiment and will not be repeated in this embodiment.

[0297] In step 1201, if the information acquisition terminal is integrated into the target control terminal, the control terminal can directly and synchronously obtain the point cloud data and panoramic images of multiple subspaces obtained by the information acquisition terminal; if the information acquisition terminal is connected to the control terminal through a communication link, the control terminal receives the point cloud data and panoramic images of multiple subspaces sent by the information acquisition terminal based on the pre-established communication link.

[0298] In this embodiment, when generating a floor plan for a target physical space, the spatial structure diagrams corresponding to multiple subspaces are first obtained in sequence. These spatial structure diagrams are then concatenated to determine the floor plan for the target physical space. Generating the spatial structure diagrams for each subspace individually requires fewer computing resources and can accommodate the computing power of most control devices, thereby increasing the application scenarios of the floor plan generation method of this embodiment.

[0299] In the process of sequentially performing spatial structure graph acquisition processing on multiple subspaces, the spatial structure graph acquisition process corresponding to any subspace is the same. In this embodiment, the target subspace to be edited is used as an example for description.

[0300] It should be noted that, according to the acquisition of the spatial structure diagrams corresponding to multiple subspaces, the multiple subspaces in the target physical space can be divided into: subspaces for which the spatial structure diagram has been obtained, subspaces for which the spatial structure diagram has not been obtained, and subspaces for which the spatial structure diagram is being obtained, that is, the target subspace currently to be edited.

[0301] First, a target space contour of the target subspace is acquired based on point cloud data and / or panoramic images collected at at least one shooting point in the target subspace.

[0302] Optionally, a first spatial contour can be obtained based on the point cloud data of at least one shooting point, and the first spatial contour can be directly used as the target spatial contour of the target subspace; or, a second spatial contour can be obtained based on the panoramic image of at least one shooting point, and the second spatial contour can be directly used as the target spatial contour of the target subspace; or, a spatial contour with better contour line quality is selected from the above-mentioned first spatial contour and the above-mentioned second spatial contour as the target spatial contour of the target subspace; or, the contour lines of the above-mentioned first spatial contour and the above-mentioned second spatial contour are fused to obtain a spatial contour with better contour line quality, and the fused spatial contour is directly used as the spatial contour of the target subspace.

[0303] Optionally, before determining the first spatial contour and / or the second spatial contour as the target spatial contour, manual or automatic editing processing can be performed on the first spatial contour and / or the second spatial contour to use the edited spatial contour as the spatial contour of the target space.

[0304] The process of obtaining a spatial contour based on point cloud data of at least one shooting point and manually editing the spatial contour is taken as an example to illustrate the process of determining the target spatial contour.

[0305] During the specific implementation process, the point cloud data collected at at least one shooting point in the target subspace is first mapped to a two-dimensional plane to determine a two-dimensional point cloud image of the target subspace. Since the relative positional relationship of at least one shooting point in the target subspace is known, the point cloud data collected at at least one shooting point can be fused based on this relative positional relationship to obtain dense point cloud data, which is then mapped to obtain a two-dimensional plane point cloud image. Thereafter, a spatial contour identified based on the two-dimensional point cloud image is displayed in the two-dimensional point cloud image, where the spatial contour is composed of multiple contour lines. Finally, in response to a user's correction operation on the spatial contour, the shape and / or position of the target contour line in the spatial contour is adjusted so that the target contour line coincides with the wall line in the two-dimensional point cloud image, and the spatial contour formed by the contour lines coinciding with the wall line is determined to be the target spatial contour of the target subspace. The wall line in the point cloud image corresponds to the wall in the target subspace.

[0306] It can be understood that the point cloud data is actually a series of three-dimensional coordinate points, and any three-dimensional coordinate point can be represented by a three-dimensional Cartesian coordinate (x, y, z), where x, y, and z are the coordinate values ​​of the x-axis, y-axis, and z-axis, which have a common zero point and are orthogonal to each other.

[0307] In an optional embodiment, mapping point cloud data collected at at least one shooting point in the target subspace onto a two-dimensional plane to determine a two-dimensional point cloud image of the target subspace includes: converting three-dimensional coordinate points (x, y, z) corresponding to the point cloud data collected at at least one shooting point in the target subspace into two-dimensional coordinate points (x, y), for example, setting the z-axis coordinate value of the three-dimensional coordinate points to 0, and then obtaining a two-dimensional point cloud image of the target subspace based on the converted two-dimensional coordinate points. Alternatively, a three-dimensional spatial structure diagram of the target subspace is first generated based on the three-dimensional coordinate points (x, y, z) corresponding to the point cloud data collected at at least one shooting point in the target subspace, and then obtaining a top view of the three-dimensional spatial structure diagram as the two-dimensional point cloud image of the target subspace.

[0308] Figure 13 is a schematic diagram of a spatial contour provided by the third embodiment of the present invention. As shown in the left figure of Figure 13, contour line l in the spatial contour Z does not coincide with wall line L in the point cloud image. However, contour line l and wall line L actually correspond to the same wall in the target subspace. Based on this, contour line l can be used as the target contour line. The length and position of target contour line l can be adjusted using preset editing options to make it coincide with wall line L in the point cloud image. As shown in the right figure of Figure 13, the spatial contour Z' formed by the contour line that coincides with the wall line is used as the target spatial contour of the target subspace.

[0309] Optionally, for target contour lines in the space contour that do not match the actual wall positions, target contour lines that do not have corresponding wall lines may be deleted; or target contour lines corresponding to certain wall lines may be added.

[0310] Optionally, after obtaining the 2D point cloud image of the target subspace, the 2D point cloud image can be cropped to accurately identify the spatial contours. For example, regions with unclear boundaries can be cropped out. The point cloud data corresponding to these regions may be interference data collected by the laser sensor due to interference from glass, mirrors, and other objects in the target subspace. Alternatively, unclear wall lines in the 2D point cloud image can be highlighted to facilitate accurate spatial contour recognition.

[0311] After editing the spatial outline of the target subspace, the physical media within the target subspace, namely the door and window, need to be annotated in the resulting spatial outline. Specifically, the target medium is first identified based on the panoramic image, and then the corresponding mapping medium is determined to obtain a spatial structure diagram of the target subspace. For ease of distinction, the images corresponding to the door and window in the target space in the panoramic image are referred to as target media, and the corresponding identifiers of the door and window in the spatial outline are referred to as mapping media. Alternatively, the target medium can be identified from the panoramic image using, for example, an image recognition algorithm, and the mapping medium determined based on the target medium.

[0312] In practical applications, the target medium may be included in the panoramic images corresponding to more than one shooting point. To improve the control terminal's efficiency in identifying the target medium, the control terminal may optionally determine a target panoramic image from the panoramic images corresponding to at least one shooting point before identifying the target medium in the panoramic image. The target panoramic image is a panoramic image that meets preset recognition requirements, such as a panoramic image with the widest field of view, the best lighting, or a panoramic image that includes user-marked information (e.g., the best panoramic image).

[0313] For the target subspace, the shooting points corresponding to the target panoramic image can be the same as or different from the shooting points corresponding to the point cloud data used to generate the spatial contour. Assume that the target subspace contains two shooting points, shooting point A and shooting point B. Panoramic image A1 and point cloud data A2 were acquired at shooting point A, and panoramic image B1 and point cloud data B2 were acquired at shooting point B. If the spatial contour is generated based on point cloud data A2, then either panoramic image A1 or B1 can be determined to be the target panoramic image. Similarly, if the spatial contour is generated based on point cloud data B2, then either panoramic image A1 or B1 can be determined to be the target panoramic image.

[0314] It can be understood that the door body and the window have corresponding size information. In order to facilitate the user to understand the spatial structure of the target subspace, the mapping medium added to the spatial outline of the target subspace should at least be able to reflect the position information, size information and type information of the door body and / or window contained in the target subspace.

[0315] As an optional implementation, determining the mapping medium corresponding to the target medium in the spatial contour of the target subspace includes:

[0316] According to the mapping relationship between the target panoramic image and the spatial contour of the target subspace, the panoramic pixel coordinates corresponding to the target medium in the target panoramic image and the mapped spatial contour coordinates are obtained to determine the mapping medium corresponding to the target medium in the spatial contour of the target subspace.

[0317] The mapping medium is adapted to the target identifier and target display size of the target medium. The target identifier is used to distinguish target media of different types. For example, target media belonging to a door or a window correspond to different target identifiers.

[0318] The above-mentioned mapping relationship between the target panoramic image and the space outline is a mapping between the target panoramic image and the space outline established based on the coordinate mapping between the point cloud data of the target subspace and the target panoramic image.

[0319] The relative pose between the laser sensor and camera is pre-calibrated before point cloud data and panoramic image acquisition. Based on this pre-calibrated relative pose and the relative positional relationship between the actual captured points in the target subspace, the coordinate mapping between the 3D point cloud coordinates corresponding to the collected point cloud data and the panoramic pixel coordinates of the panoramic image is determined.

[0320] In the embodiments of the present invention, the specific method of mapping the coordinates of the panoramic image and the point cloud data is not limited. Optionally, the panoramic pixel coordinates can be directly mapped to the three-dimensional point cloud coordinates, and the three-dimensional point cloud coordinates can be mapped to the panoramic pixel coordinates based on the relative posture relationship between the devices that obtain the panoramic image and the point cloud data. Alternatively, the panoramic pixel coordinates can be first mapped to the intermediate coordinates, and then mapped to the three-dimensional point cloud coordinates, with the help of the relative posture relationship and the intermediate coordinate system. Alternatively, the three-dimensional point cloud coordinates can be first mapped to the intermediate coordinates, and then mapped to the intermediate coordinates. Here, the specific type of the intermediate coordinate system is not limited, nor is the specific method used in the coordinate mapping process. The mapping method used will also be different depending on the different intermediate coordinate systems and the different relative posture relationships.

[0321] In response to the completion of the acquisition operation of the spatial structure diagram of the target subspace, it is determined whether there are still subspaces whose spatial structure diagrams have not been acquired. If there is at least one subspace among the multiple subspaces whose spatial structure diagrams have not been acquired, a subspace to be edited is determined from the at least one subspace to acquire the spatial structure diagram of the subspace to be edited.

[0322] Optionally, determining a subspace to be edited from at least one subspace includes: randomly determining a subspace from at least one subspace as the subspace to be edited.

[0323] Optionally, determining a subspace to be edited from at least one subspace further includes: determining a subspace to be edited from at least one subspace based on the acquisition time point of the point cloud data and panoramic image corresponding to the at least one subspace, wherein the difference between the acquisition time point corresponding to the subspace to be edited and the current moment is greater than the difference between the acquisition time point corresponding to other subspaces in the at least one subspace and the current moment. For example, the subspaces for which the spatial structure diagram is not obtained include: subspace a, subspace b, and subspace c, and the acquisition times of the corresponding point cloud data are t1, t2, and t3, respectively, wherein t1 is earlier than t2, and t2 is earlier than t3. Based on the difference between t1, t2, and t3 and the current moment t, subspace a is determined to be the subspace to be edited. After the spatial structure diagram of subspace a is obtained, subspace b is determined to be the subspace to be edited, and so on, until the spatial structure diagrams of all subspaces are obtained.

[0324] Optionally, determining a subspace to be edited from at least one subspace further includes: determining a connection relationship between the multiple subspaces based on the point cloud data corresponding to the multiple subspaces; and determining a subspace to be edited from the at least one subspace based on the connection relationship, wherein the subspace to be edited is connected to the target subspace. Alternatively, determining a connection relationship between the multiple subspaces based on the panoramic images corresponding to the multiple subspaces; and determining a subspace to be edited from the at least one subspace based on the connection relationship, wherein the subspace to be edited is connected to the target subspace.

[0325] It is understandable that multiple subspaces in the target physical space are connected to each other through doors or windows. For example, subspace 1 and subspace 2 are connected through the same target door. Since the target door is open when collecting point cloud data and panoramic images, point cloud data 1 of subspace 1 and point cloud data 2 of subspace 2 may contain feature points m corresponding to the same object, and panoramic image 3 of subspace 1 and panoramic image 4 of subspace 2 may contain image n corresponding to the same object. Spatially speaking, the area corresponding to image n is outside subspace 1, but within the field of view of the camera when capturing panoramic image 3.

[0326] Therefore, in practical applications, the spatial connection relationship between subspace 1 and subspace 2 can be determined based on the point cloud data 1 of subspace 1 and the point cloud data 2 of subspace 2, and / or the panoramic image 3 of subspace 1 and the panoramic image 4 of subspace 2, by methods such as feature matching.

[0327] Optionally, the connection relationship may also be used to splice spatial structure graphs of multiple subspaces.

[0328] In practical applications, if no subspaces have a missing spatial structure diagram, it indicates that the spatial structure diagrams corresponding to all subspaces have been obtained. Subsequently, the spatial structure diagrams of the multiple subspaces can be spliced ​​together based on their connection relationships to obtain the floor plan of the target physical space.

[0329] To facilitate understanding of the above floor plan generation method, let's take Figure 9 of the second embodiment as an example. As shown in Figure 9a, assume that the target physical space contains three subspaces: bedroom 1, bedroom 2, and living room 3. Bedroom 1 and living room 3 are connected by door 1, bedroom 2 and living room 3 are connected by door 2, and bedroom 1 and bedroom 2 are connected by window 3.

[0330] Based on the method provided by an embodiment of the present invention, assume that a spatial structure diagram x of bedroom 1 is first generated. Next, bedroom 2 is identified as the subspace to be edited from among bedrooms 2 and living room 3, and a spatial structure diagram y for bedroom 2 is determined. Next, living room 3 is identified as the subspace to be edited, and a spatial structure diagram z for living room 3 is determined, as shown in FIG9(b). Each spatial structure diagram includes a spatial outline, which is marked with mapping media representing doors and windows. Since no subspaces have yet to obtain a spatial structure diagram after spatial structure diagram z is determined, the target physical space floor plan is obtained by concatenating spatial structure diagrams x, y, and z based on the connection relationship between bedrooms 1, 2, and living room 3, as shown in FIG9(c).

[0331] In this embodiment, for a target physical space having multiple subspaces, when generating a floor plan of the target physical space, the spatial structure diagrams corresponding to the multiple subspaces are first obtained in sequence, and then the spatial structure diagrams corresponding to the multiple subspaces are spliced ​​together to determine the floor plan of the target physical space. When generating the spatial structure diagram corresponding to each subspace one by one, fewer computing resources are required, and the computing processing capabilities of most control devices can be adapted, thereby increasing the application scenarios of the floor plan generation method of this embodiment. Moreover, the position of the mapping medium corresponding to the target medium in the target subspace in the spatial outline is determined based on the panoramic image, and the mapping medium corresponding to the target medium can be identified at the correct position on the spatial outline, so that the determined spatial structure diagram can accurately reflect the actual spatial structure information of the target subspace, and thus the floor plan determined based on the multiple spatial structure diagrams can also accurately reflect the actual spatial structure of the target physical space.

[0332] The following describes in detail the floor plan generation device according to one or more embodiments of the present invention. Those skilled in the art will appreciate that these devices can be constructed using commercially available hardware components and configured according to the steps taught in this solution.

[0333] FIG14 is a schematic structural diagram of a floor plan generating device provided in a third embodiment of the present invention. The device is applied to a control terminal. As shown in FIG14 , the device includes: an acquisition module 31 and a processing module 32 .

[0334] The acquisition module 31 is used to obtain point cloud data and panoramic images corresponding to multiple subspaces in the target physical space obtained by the information acquisition terminal, wherein the point cloud data and panoramic image of any subspace are collected from at least one shooting point in any subspace.

[0335] The processing module 32 is used to obtain the target space outline of the target subspace according to the point cloud data and / or panoramic image collected at at least one shooting point of the target subspace in the process of sequentially performing spatial structure diagram acquisition processing on the multiple subspaces, for the target subspace to be edited currently; obtain the target medium identified in the target panoramic image, the target medium being the image of the physical medium in the target subspace in the target panoramic image, and the target panoramic image being a panoramic image used to identify the target medium in the panoramic image collected at at least one shooting point of the target subspace; determine the mapping medium used to represent the target medium on the target space outline of the target subspace to obtain the spatial structure diagram of the target subspace; in response to the completion operation of the acquisition of the spatial structure diagram of the target subspace, if there is no subspace among the multiple subspaces for which the spatial structure diagram has not been obtained, obtain the floor plan of the target physical space obtained by splicing the spatial structure diagrams of the multiple spaces.

[0336] Optionally, the processing module 32 is also used to respond to the completion operation of obtaining the spatial structure diagram of the target subspace. If there is at least one subspace among the multiple subspaces whose spatial structure diagram has not been obtained, a subspace to be edited is determined from the at least one subspace to obtain the spatial structure diagram of the subspace to be edited.

[0337] Optionally, the processing module 32 is specifically used to determine a subspace to be edited from the at least one subspace based on the acquisition time point of the point cloud data and the panoramic image corresponding to the at least one subspace, wherein the difference between the acquisition time point corresponding to the subspace to be edited and the current moment is greater than the difference between the acquisition time point corresponding to other subspaces in the at least one subspace and the current moment.

[0338] Optionally, the processing module 32 is further specifically used to determine the connection relationship between multiple subspaces based on the point cloud data corresponding to the multiple subspaces; based on the connection relationship, determine a subspace to be edited from the at least one subspace, wherein the subspace to be edited is connected to the target subspace.

[0339] Optionally, the processing module 32 is further specifically used to map the point cloud data collected at at least one shooting point in the target subspace to a two-dimensional plane to determine the two-dimensional point cloud image of the target subspace; display a spatial contour identified based on the two-dimensional point cloud image, wherein the spatial contour is composed of multiple contour lines; in response to the user's correction operation on the spatial contour, adjust the shape and / or position of the target contour line in the spatial contour so that the target contour line coincides with the wall line in the two-dimensional point cloud image; and determine that the spatial contour composed of the contour lines coinciding with the wall line is the target spatial contour of the target subspace.

[0340] Optionally, the processing module 32 is further specifically used to obtain a mapping medium that matches the target identification and target display size of the target medium, wherein the target identification is used to distinguish target media belonging to different types; determine the three-dimensional point cloud coordinates corresponding to the mapping medium according to the panoramic pixel coordinates corresponding to the target medium in the target panoramic image, and the relative posture between the device for obtaining the target subspace point cloud data and the panoramic image; and add the mapping medium to the target space contour of the target subspace according to the three-dimensional point cloud coordinates corresponding to the mapping medium.

[0341] The device shown in FIG14 can execute the steps in the aforementioned embodiments. For detailed execution process and technical effects, please refer to the description in the aforementioned embodiments and will not be repeated here.

[0342] Fourth embodiment

[0343] FIG15 is a flow chart of a method for generating a floor plan according to a fourth embodiment of the present invention, which is applied to a control terminal. As shown in FIG15 , the method for generating a floor plan includes the following steps:

[0344] Step S151: Acquire point cloud data and panoramic images collected by the information collection terminal in each of N spaces in the target physical space, wherein the point cloud data and panoramic image are collected at at least one shooting point in each space.

[0345] Step S152: Obtain an Mth space outline of an Mth space among the N spaces for displaying for editing, where the Mth space outline is obtained based on point cloud data and / or panoramic images collected from at least one shooting point of the Mth space.

[0346] Step S153: Acquire the target medium identified in the target panoramic image of the Mth space, so as to obtain the mapping medium of the target medium in the Mth space outline according to the target medium, so as to edit the Mth space outline according to the mapping medium to obtain the floor plan of the Mth space.

[0347] The target panoramic image is a panoramic image captured by at least one shooting point in the Mth space and is used to identify the target medium. The target medium is an image of the physical medium in the Mth space in the target panoramic image.

[0348] Step S154: Determine whether the Mth space is the last space among the N spaces for which a floor plan diagram is generated; if not, execute step S155; if so, execute step S156.

[0349] Step S155: Assign M to M+1 and return to step S152.

[0350] Step S156: Obtain a floor plan of the target physical space consisting of N floor plan structure diagrams for display, and the process ends.

[0351] The target physical space includes at least N spaces, M and N are natural numbers, and 1≤M≤N.

[0352] The control terminal can be a terminal device with data processing capabilities, such as a smart phone, tablet computer, or laptop computer.

[0353] In this embodiment, for a target physical space comprising at least N spaces, when generating a floor plan for the target space, a floor plan for each space is first generated one by one in order from 1 to N. The floor plans for the N spaces are then concatenated to generate the floor plan for the target physical space. In this embodiment, generating the floor plan for each space individually requires fewer computing resources, accommodating the computing processing capabilities of most control devices. Furthermore, it facilitates the user to independently edit the floor plan for each space, thereby generating accurate floor plans for each of the N spaces and thus obtaining an accurate floor plan for the target physical space. Furthermore, in this embodiment, the mapping medium on the spatial outline in each space floor plan is determined based on a panoramic image. Since panoramic images better reflect the actual location of doors and windows (i.e., target media) in the actual space compared to point cloud data, the floor plan for each space, aided by the panoramic image, contains more accurate door and window information, better reflecting the scene information in the actual space.

[0354] The following describes the method for generating a floor plan shown in FIG15 in conjunction with the schematic diagram of a scene for generating a floor plan shown in FIG16 .

[0355] Figure 16 is a schematic diagram of a scenario for generating a floor plan according to the fourth embodiment of the present invention. As shown in Figure 16 , the information collection terminal and the control terminal are decoupled from each other. The information collection terminal collects point cloud data and panoramic images from each of N spaces and transmits them to the control terminal, which then generates a floor plan for the target physical space based on the received point cloud data and panoramic images. In practical applications, the information collection terminal can communicate with the control terminal via, for example, Bluetooth or a Wireless Fidelity (WiFi) hotspot.

[0356] Alternatively, the information collection terminal can be directly integrated with the control terminal. The control terminal, acting as a single entity, can directly and synchronously acquire the point cloud data and panoramic images collected by the information collection terminal in each of the N spaces, eliminating the need to transmit the point cloud data and panoramic images over an established communication connection. This can improve the efficiency of the control terminal in generating floor plans for the target physical space.

[0357] Optionally, a floor plan of the target physical space can be generated through a cloud server. The cloud server can be a physical server or a virtual server in the cloud, and the control terminal communicates with the cloud server by accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G, or other mobile communication network.

[0358] The cloud server can receive the point cloud data and panoramic images corresponding to the N spaces forwarded by the control terminal to generate a floor plan of the target physical space, and feed the floor plan back to the control terminal for display. Alternatively, the cloud server can also directly communicate with the information collection terminal to directly obtain the point cloud data and panoramic images corresponding to the multiple subspaces collected by the information collection terminal to generate a floor plan of the target physical space. In fact, the process of the cloud server generating the floor plan of the target physical space is the same as the process of the control terminal generating the spatial structure diagram, but because the cloud server has stronger computing power, it is more efficient in generating the floor plan of the target physical space, which can further enhance the user experience.

[0359] In this embodiment, an example is given in which the information collection terminal and the control terminal are decoupled from each other, and the control terminal generates a floor plan of the target physical space.

[0360] Since each space contains at least one shooting point, the point cloud data and panoramic image of each space are collected by the information collection terminal at at least one shooting point in that space. For example, the point cloud data and panoramic image of the Xth space include: point cloud data Xa and panoramic image Xa collected at shooting point a in the Xth space, and point cloud data Xb and panoramic image Xb collected at shooting point b in the Xth space.

[0361] When the information collection terminal collects point cloud data and panoramic images at each shooting point, the corresponding information collection process is the same. In this embodiment, a certain shooting point Y is taken as an example for explanation.

[0362] First, a brief description of the basic structure of the information acquisition terminal is given. The information acquisition terminal includes: a laser sensor, a camera, a motor, and a processor (such as a CPU). Among them, the laser sensor and the camera are used as sensing devices to collect scene information of each space at each shooting point, that is, point cloud data and panoramic images; the motor is used to drive the laser sensor and the camera to rotate so as to collect point cloud data and panoramic images from various angles. Optionally, the information acquisition terminal also includes an inertial measurement unit (IMU). The IMU is used to correct the posture information corresponding to the collected point cloud data and image data to reduce errors caused by environmental or human factors (such as: the information acquisition terminal is not placed horizontally, etc.).

[0363] Next, the data collection process of the information collection terminal at the shooting point Y is described.

[0364] For shooting point Y, the information collection terminal responds to the information collection instruction by driving the motor to rotate the laser sensor 360 degrees to collect point cloud data corresponding to the target shooting point Y; the driving motor also drives the camera 360 degrees to collect a panoramic image corresponding to the target shooting point Y. The information collection instruction is sent by the user through the control terminal, or is triggered in response to a trigger operation by the user on the information collection terminal.

[0365] Optionally, at the shooting point Y, the motor can drive the laser sensor and the camera to rotate at the same time to collect point cloud data and panoramic images at the same time, or it can drive the laser sensor and the camera to rotate in sequence to collect point cloud data and panoramic images respectively. This embodiment does not limit this.

[0366] In an optional embodiment, if point cloud data is collected first and then the panoramic image is captured, to improve the quality of the captured panoramic image, the camera can be activated simultaneously with the point cloud data acquisition process to collect scene lighting information at the current capture point, perform light measurement, and determine corresponding exposure parameters. The camera then captures the panoramic image based on the determined exposure parameters.

[0367] Optionally, the camera in the information collection terminal is a panoramic camera or a non-panoramic camera. If the camera in the information collection terminal is a non-panoramic camera, then during the above-mentioned 360-degree rotation process, the camera is controlled to capture images corresponding to the target shooting point Y at multiple preset angles, and the above-mentioned processor can stitch the images captured at multiple preset angles into a panoramic image through a panoramic image stitching algorithm such as a feature matching algorithm. For example: if the camera's viewing angle is 120 degrees, a certain reference direction can be set as 0 degrees, and a degree, (a+120) degrees, and (a+240) degrees based on the reference direction are determined as preset angles, and the camera is controlled to capture images 1, 2, and 3 at these three preset angles; then, images 1 to 3 are stitched together to obtain a panoramic image.

[0368] It is understood that the smaller the intervals between the preset angles, the more images are captured, and the more accurate the resulting panoramic image after stitching. In specific implementations, the number of preset angles can be customized by the user based on the camera's viewing angle. Images captured at multiple preset angles contain scene information within a 360-degree range at the current point.

[0369] Optionally, when generating a panoramic image, High Dynamic Range Imaging (HDRI) may be combined to generate a high-quality panoramic image.

[0370] After the information collection terminal has collected point cloud data and panoramic images at a shooting point, it can directly send the point cloud data and panoramic images of the shooting point to the control terminal; or, it can first store them, and then after completing the collection of point cloud data and panoramic images at all shooting points in the current space, send the point cloud data and panoramic images of all shooting points in the space to the control terminal. This embodiment does not impose any restrictions on this.

[0371] After receiving the point cloud data and panoramic images of each of the N spaces in the target physical space, the control terminal generates the floor plan structure diagrams of the N spaces one by one, as shown in FIG16 . For example, the floor plan structure diagram of the first space is generated first, and then the floor plan structure diagram of the second space is generated, and so on, until the floor plan structure diagram of the Nth space is generated.

[0372] In practical applications, the order in which floor plan diagrams are obtained for each of the N spaces can be determined based on the acquisition time of the point cloud data and / or panoramic images corresponding to each of the N spaces. This is done by sorting the N spaces. For example, assume that N is 3, meaning the target physical space contains three spaces: space a, space b, and space c. The point cloud data for space a was collected at time t1, the point cloud data for space b was collected at time t2, and the point cloud data for space c was collected at time t3, with t1 occurring earlier than t2, which in turn occurs earlier than t3. These three spaces can be sorted based on the order of t1, t2, and t3, for example: space a is the first space, space b is the first space, and space c is the third space. Floor plan diagrams for each space are then generated one by one, in order from 1 to 3. Alternatively, the order in which floor plan diagrams are generated for the N spaces can be randomly determined.

[0373] Next, the process of generating a floor plan diagram will be described using an example where the space for which the floor plan diagram is currently being generated is the Mth space among N spaces. It will be appreciated that when floor plans are generated for each space one by one in a certain order, the process for generating the floor plan diagram for each space is the same.

[0374] In this embodiment, the floor plan of the target physical space consists of floor plan diagrams for N spaces. Both the floor plan of the target physical space and the floor plan diagrams for each space can be understood as two-dimensional floor plans of that space. The difference between the two is that the floor plan of the target physical space corresponds to more spaces and is a "larger" two-dimensional floor plan, while the floor plan diagram for each space is only a "smaller" two-dimensional floor plan for the current space. In practical applications, such as AR house viewing scenarios, a two-dimensional floor plan is more commonly understood as a bird's-eye view of the physical space.

[0375] To represent physical space, each floor plan diagram consists of a space outline and a mapping medium. The space outline represents the walls in the physical space, while the mapping medium represents the windows and doors in the physical space. Therefore, the process of obtaining a floor plan diagram for each space can be further divided into determining the space outline and the mapping medium, each of which is explained below.

[0376] In addition, it should be noted that each space corresponds to a space profile. In this embodiment, for ease of description, the space profile of the Mth space is recorded as the Mth space profile. For example, the second space profile is the space profile of the second space, and the third space profile is the space profile of the third space.

[0377] For the Mth space, when determining the Mth space outline, the Mth space outline of the Mth space can be determined based on the point cloud data and / or panoramic image collected from at least one shooting point of the Mth space, and the Mth space outline can be displayed for editing.

[0378] Specifically, a first spatial contour can be obtained based on the point cloud data of at least one shooting point in the Mth space, and the first spatial contour can be directly used as the Mth spatial contour; or, a second spatial contour can be obtained based on a panoramic image of at least one shooting point in the Mth space, and the second spatial contour can be directly used as the Mth spatial contour; or, a spatial contour with better contour line quality is selected from the above-mentioned first spatial contour and the above-mentioned second spatial contour as the Mth spatial contour; or, the contour lines of the above-mentioned first spatial contour and the above-mentioned second spatial contour are fused to obtain a spatial contour with better contour line quality, and the fused spatial contour is directly used as the Mth spatial contour.

[0379] Optionally, manual or automatic editing may be performed on the first spatial profile and / or the second spatial profile, so that the edited spatial profile is used as the Mth spatial profile.

[0380] In actual applications, the Mth space outline contains multiple contour lines. Among them, there are some contour lines that do not match the actual positions of the walls in the Mth space. In order to ensure that the Mth space outline can accurately reflect the Mth space, the Mth space outline can be edited.

[0381] To facilitate editing of the Mth space outline, the Mth space outline can optionally be displayed in the two-dimensional point cloud image of the Mth space. Subsequently, in response to a user's editing operation on the Mth space outline, the contour lines of the Mth space outline are adjusted so that the contour lines of the Mth space outline coincide with the wall lines in the two-dimensional point cloud image. The wall lines in the two-dimensional point cloud image correspond to the walls in the Mth space.

[0382] The two-dimensional point cloud image of the Mth space is obtained by plane mapping the point cloud data of at least one shooting point in the Mth space. Since the relative position relationship of at least one shooting point in the Mth space is known, the point cloud data collected at the at least one shooting point can be fused based on the relative position relationship to obtain dense point cloud data, and then mapped to obtain a two-dimensional point cloud image. Specifically, the point cloud data of at least one shooting point in the Mth space among the N spaces are first fused to determine the target point cloud data of the Mth space; then, the target point cloud data is mapped to a two-dimensional plane to obtain the initial two-dimensional point cloud image of the Mth space; thereafter, based on the user's correction operation on the initial two-dimensional point cloud image (for example, cropping out the area with unclear boundaries in the two-dimensional point cloud image, highlighting the unclear wall lines in the two-dimensional point cloud image, etc.), the target two-dimensional point cloud image obtained after the correction operation is determined to be the two-dimensional point cloud image of the Mth space.

[0383] It can be understood that the point cloud data is actually a series of three-dimensional coordinate points, and any three-dimensional coordinate point can be represented by a three-dimensional Cartesian coordinate (x, y, z), where x, y, and z are the coordinate values ​​of the x-axis, y-axis, and z-axis, which have a common zero point and are orthogonal to each other.

[0384] In an optional embodiment, the target point cloud data of the Mth space is mapped to a two-dimensional plane to obtain an initial two-dimensional point cloud image of the Mth space, including: converting the three-dimensional coordinate points (x, y, z) corresponding to the target point cloud data of the Mth space into two-dimensional coordinate points (x, y), for example: setting the z-axis coordinate value of the three-dimensional coordinate points to 0, and then obtaining the initial two-dimensional point cloud image of the Mth space based on the converted two-dimensional coordinate points. Alternatively, a three-dimensional spatial structure diagram of the Mth space is first generated based on the three-dimensional coordinate points (x, y, z) corresponding to the target point cloud data of the Mth space, and then obtaining a top view of the three-dimensional spatial structure diagram as the initial two-dimensional point cloud image of the Mth space.

[0385] The two-dimensional point cloud image of the Mth space can also be used to obtain the Mth space contour of the Mth space, for example, by performing edge detection on the two-dimensional point cloud image of the Mth space to obtain the Mth space contour.

[0386] For the contour lines in the Mth space contour that do not match the actual wall position, the shape and / or position of the contour lines in the Mth space contour can be adjusted based on the preset contour line editing options, or the contour lines for which there are no corresponding wall lines can be deleted, or the contour lines corresponding to a certain wall line can be added.

[0387] For example, Figure 17 is a schematic diagram of the Mth spatial outline provided by the fourth embodiment of the present invention. As shown in the left figure of Figure 17, before the contour lines of the Mth spatial outline are edited, contour line l in the Mth spatial outline does not coincide with wall line L in the 2D point cloud image of the Mth spatial outline. However, contour line l and wall line L actually correspond to the same wall in the Mth spatial outline. Based on this, the length and position of contour line l can be adjusted using the preset contour line editing options to make contour line l coincide with wall line L in the 2D point cloud image, resulting in the edited Mth spatial outline, as shown in the right figure of Figure 17.

[0388] Furthermore, as shown in the left image of FIG17 , before the contour lines of the M-th spatial contour are edited, there is no contour line in the M-th spatial contour corresponding to the wall line H in the 2D point cloud image. Based on this, the preset contour line editing option is used to add contour line h, and contour line h coincides with the wall line H in the 2D point cloud image, as shown in the right image of FIG17 . In this embodiment, the contour lines of the M-th spatial contour are finally connected.

[0389] The above is the process of determining the Mth spatial profile. The following describes the process of determining the mapping medium in the Mth spatial profile.

[0390] For the Mth space, the corresponding mapping medium in the floor plan can be obtained in the following way: first, the corresponding target medium is identified from the panoramic image through methods such as image recognition, that is, the image of the physical medium (door and window) in the Mth space in the panoramic image; then, the corresponding mapping medium is determined based on the target medium.

[0391] In practical applications, the target medium may be included in the panoramic images corresponding to more than one shooting point in the Mth space. To improve the control terminal's efficiency in identifying the target medium, the control terminal may optionally determine a target panoramic image from the panoramic images of at least one shooting point before identifying the target medium in the panoramic image. The target panoramic image is a panoramic image that meets preset recognition requirements, such as a panoramic image with the widest field of view, the best lighting, or a panoramic image that includes user-marked information (e.g., the best panoramic image).

[0392] For the Mth space, the shooting points corresponding to the target panoramic image can be the same as or different from the shooting points corresponding to the point cloud data used to generate the spatial outline. Assume that the Mth space contains two shooting points, shooting point A and shooting point B. Panoramic image A1 and point cloud data A2 are acquired at shooting point A, and panoramic image B1 and point cloud data B2 are acquired at shooting point B. If the Mth spatial outline is generated based on point cloud data A2, then panoramic image A1 can be determined to be the target panoramic image, or panoramic image B1 can be determined to be the target panoramic image. Similarly, if the Mth spatial outline is generated based on point cloud data B2, then panoramic image A1 can be determined to be the target panoramic image, or panoramic image B1 can be determined to be the target panoramic image.

[0393] Because the relative pose between the laser sensor and camera is pre-calibrated before point cloud data and panoramic image acquisition, the coordinate mapping between the 3D point cloud coordinates corresponding to the acquired point cloud data and the panoramic pixel coordinates of the panoramic image can be determined for the Mth space based on the pre-calibrated relative pose and the relative positional relationship between the actual captured points in the Mth space.

[0394] Furthermore, based on the coordinate mapping between the point cloud data of the Mth space and the target panoramic image, a mapping between the target panoramic image and the Mth space outline can be established, that is, the mapping relationship between the target panoramic image of the Mth space and the Mth space outline is predetermined.

[0395] Thus, the mapping medium of the target medium in the Mth spatial contour is obtained according to the target medium, including: according to the mapping relationship between the target panoramic image of the Mth space and the Mth spatial contour, the panoramic pixel coordinates corresponding to the target medium in the target panoramic image and the mapped spatial contour coordinates are obtained to determine the mapping medium corresponding to the target medium in the Mth spatial contour to obtain the floor plan of the Mth space.

[0396] The mapping medium is adapted to the target identification and target display size of the target medium, and the target identification is used to distinguish target media of different types (doors or windows).

[0397] In the embodiments of the present invention, the specific method of mapping the coordinates of the panoramic image and the point cloud data is not limited. Optionally, the panoramic pixel coordinates can be directly mapped to the three-dimensional point cloud coordinates, and the three-dimensional point cloud coordinates can be mapped to the panoramic pixel coordinates based on the relative posture relationship between the devices that obtain the panoramic image and the point cloud data. Alternatively, the panoramic pixel coordinates can be first mapped to the intermediate coordinates, and then mapped to the three-dimensional point cloud coordinates, with the help of the relative posture relationship and the intermediate coordinate system. Alternatively, the three-dimensional point cloud coordinates can be first mapped to the intermediate coordinates, and then mapped to the intermediate coordinates. Here, the specific type of the intermediate coordinate system is not limited, nor is the specific method used in the coordinate mapping process. The mapping method used will also be different depending on the different intermediate coordinate systems and the different relative posture relationships.

[0398] After determining the floor plan for the Mth space, determine whether the Mth space is the last space among the N spaces for which a floor plan was generated. If M is less than N, indicating that there are still spaces for which floor plans have not been obtained, M is assigned to M+1, and the floor plan for the M+1th space is obtained. If M is equal to N, indicating that there are no spaces for which floor plans have not been obtained, the first floor plan, the second floor plan, ..., and the Nth floor plan are concatenated in the same spatial coordinate system to form the floor plan for the target physical space.

[0399] In this embodiment, the connection relationship between the N subspaces in the target physical space can be determined based on the point cloud data of each space; or based on the panoramic image of each space. Then, based on this connection relationship, the N floor plan layouts are converted to the same spatial coordinate system and spliced ​​in this spatial coordinate system.

[0400] It is understandable that the N spaces in the target physical space are connected to each other through doors or windows. For example, the Eth space and the Fth space are connected through the same target door. Since the target door is open when collecting point cloud data and panoramic images, the point cloud data 1 of the Eth space and the point cloud data 2 of the Fth space may contain feature points m corresponding to the same object, and the panoramic image 3 of the Eth space and the panoramic image 4 of the Fth space may contain images n corresponding to the same object. Therefore, the connection relationship between the Eth space and the Fth space can be determined based on the point cloud data 1 of the Eth space and the point cloud data 2 of the Fth space, and / or the panoramic image 3 of the Eth space and the panoramic image 4 of the Fth space, for example, through feature matching. In this case, the same object is within the collection range of the camera or laser sensor, whether it is collecting data in the Eth space or collecting information in the Fth space.

[0401] To facilitate understanding of the floor plan generation method of this embodiment, an example is provided with reference to FIG18 . FIG18 is a schematic diagram of a floor plan generation process provided by the fourth embodiment of the present invention, wherein a in FIG18 is the actual spatial structure of the target physical space, b in FIG18 is a floor plan structure diagram of three spaces in the target physical space, and c in FIG18 is a floor plan diagram of the target physical space.

[0402] As shown in Figure 18(a), assume that the target physical space contains three spaces: bedroom 1, bedroom 2, and living room 3. Bedroom 1 and living room 3 are connected by door 1, bedroom 2 and living room 3 are connected by door 2, and bedroom 1 and bedroom 2 are connected by window 3. Assume that based on the corresponding point cloud data collection time, bedroom 1 is determined to be the first space, bedroom 2 is the second space, and living room 3 is the third space.

[0403] Based on the method provided by the embodiment of the present invention, the floor plan diagram for the first space, the floor plan diagram for the second space, and the floor plan diagram for the third space are generated in order from 1 to 3, as shown in FIG18(b). Each floor plan diagram includes a space outline, which is marked with mapping media representing doors and windows. Since there are no spaces for which floor plan diagrams have not been obtained after the floor plan diagram for the third space is determined, the floor plan diagrams for the first space, the second space, and the third space are spliced ​​together in the same spatial coordinate system based on the connection relationship between bedroom 1, bedroom 2, and living room 3 to obtain the floor plan diagram of the target physical space, as shown in FIG18(c).

[0404] The following describes in detail the floor plan generation device according to one or more embodiments of the present invention. Those skilled in the art will appreciate that these devices can be constructed using commercially available hardware components and configured according to the steps taught in this solution.

[0405] Figure 19 is a structural schematic diagram of a floor plan generation device provided in the fourth embodiment of the present invention. The device is used to generate a floor plan of a target physical space, where the target physical space includes at least N spaces and is applied to a control terminal. As shown in Figure 19, the device includes: an acquisition module 41, a first processing module 42 and a second processing module 43.

[0406] The acquisition module 41 is used to acquire point cloud data and panoramic images collected by the information collection terminal in each of the N spaces, wherein the point cloud data and panoramic images are collected from at least one shooting point in each of the spaces.

[0407] The first processing module 42 is used to obtain the Mth space outline of the Mth space in the N spaces for display for editing, and the Mth space outline is obtained based on point cloud data and / or panoramic images collected from at least one shooting point of the Mth space; obtain the target medium identified in the target panoramic image of the Mth space, so as to obtain the mapping medium of the target medium in the Mth space outline based on the target medium, so as to edit the Mth space outline based on the mapping medium to obtain the floor plan of the Mth space; the target panoramic image is a panoramic image for identifying the target medium in the panoramic image collected from at least one shooting point of the Mth space, and the target medium is an image of the physical medium in the Mth space in the target panoramic image.

[0408] The second processing module 43 is used to determine whether the Mth space is the last space among the N spaces for which a floor plan diagram is generated; if not, M is assigned a value of M+1 and the process returns to execute the first processing module; if so, a floor plan diagram of the target physical space composed of the N floor plan diagrams is obtained for display, and the process ends; wherein M and N are natural numbers, and 1≤M≤N.

[0409] Optionally, the first processing module 42 is specifically used to display the Mth space contour of the Mth space in the two-dimensional point cloud image of the Mth space in the N spaces; wherein, the two-dimensional point cloud image is obtained by plane mapping the point cloud data of at least one shooting point in the Mth space; in response to an editing operation on the Mth space contour, adjust the contour line of the Mth space contour so that the contour line coincides with the wall line in the two-dimensional point cloud image.

[0410] Optionally, the first processing module 42 is specifically used to fuse the point cloud data of at least one shooting point of the Mth space among the N spaces to determine the target point cloud data of the Mth space; map the target point cloud data to a two-dimensional plane to obtain an initial two-dimensional point cloud image of the Mth space; and determine the two-dimensional point cloud image of the Mth space based on the user's correction operation on the initial two-dimensional point cloud image.

[0411] Optionally, the first processing module 42 is specifically used to obtain the panoramic pixel coordinates corresponding to the target medium in the target panoramic image and the mapped spatial contour coordinates based on the mapping relationship between the target panoramic image of the Mth space and the Mth spatial contour, so as to determine the mapping medium corresponding to the target medium in the Mth spatial contour; wherein the mapping medium is adapted to the target identifier and target display size of the target medium, the target identifier is used to distinguish target media belonging to different types, and the mapping relationship is a mapping between the target panoramic image and the Mth spatial contour established based on the coordinate mapping between the point cloud data of the Mth space and the target panoramic image.

[0412] Optionally, the first processing module 42 is specifically configured to sort the N spaces according to acquisition time points of the point cloud data and / or panoramic images respectively corresponding to the N spaces.

[0413] The device shown in FIG19 can execute the steps in the aforementioned embodiments. For detailed execution process and technical effects, please refer to the description in the aforementioned embodiments and will not be repeated here.

[0414] Fifth embodiment

[0415] Referring to the floor plan generation system shown in FIG. 7 of the second embodiment, the floor plan generation system includes an information collection terminal and a control terminal. The information collection terminal can be directly integrated with the control terminal, functioning as a single entity. Alternatively, the information collection terminal can be decoupled from the control terminal and separately provided, communicating with the control terminal via, for example, Bluetooth or a Wireless Fidelity (WiFi) hotspot.

[0416] The information acquisition terminal includes: a laser sensor, a camera, a motor and a processor (such as a CPU). Among them, the laser sensor and the camera serve as sensing devices, and are used to collect point cloud data and panoramic images corresponding to multiple subspaces in the target physical space, that is, scene information of multiple subspaces. In order to obtain complete spatial information in each subspace, more than one shooting point may be set in any subspace, for example: subspace X corresponds to shooting point 1, shooting point 2 and shooting point 3. Therefore, in this embodiment, the point cloud data and panoramic image of any subspace refer to the point cloud data and panoramic image collected at at least one shooting point in any subspace.

[0417] In actual applications, the setting of shooting points can be customized by the user based on the current collection situation when collecting scene information of each subspace through the information collection terminal; or the information collection terminal or control terminal can automatically generate reference shooting points for the subspace based on the description information of the subspace input by the user (such as the size of the space, etc.).

[0418] The process of collecting point cloud data and panoramic images at any shooting point in any subspace by the information collection terminal is actually the same. For ease of understanding, we will use a shooting point Y as an example to illustrate.

[0419] For shooting point Y, the information collection terminal responds to the information collection instruction by driving the motor to rotate the laser sensor 360 degrees to collect point cloud data corresponding to the target shooting point Y; the driving motor also drives the camera 360 degrees to collect a panoramic image corresponding to the target shooting point Y. The information collection instruction is sent by the control terminal or automatically triggered in response to a user operation on the information collection terminal.

[0420] At the shooting point Y, the motor can drive the laser sensor and the camera to rotate at the same time to collect point cloud data and panoramic images at the same time, or it can drive the laser sensor and the camera to rotate in sequence, for example: first drive the laser sensor to rotate and then drive the camera to rotate, or first drive the camera to rotate and then drive the laser to rotate, so as to collect point cloud data and panoramic images respectively. This embodiment does not impose any restrictions on this.

[0421] Optionally, if point cloud data is collected first and then the panoramic image is captured, to improve the quality of the panoramic image, the camera can be turned on simultaneously during the point cloud data acquisition process to collect scene lighting information at the current shooting point, perform light measurement, and determine the corresponding exposure parameters. The camera then captures the panoramic image based on the determined exposure parameters.

[0422] Optionally, the camera in the information collection terminal is a panoramic camera or a non-panoramic camera. If the camera in the information collection terminal is a non-panoramic camera, then during the above-mentioned 360-degree rotation process, the camera is controlled to capture images corresponding to the target shooting point Y at multiple preset angles, and the above-mentioned processor can stitch the images captured at multiple preset angles into a panoramic image through a panoramic image stitching algorithm such as a feature matching algorithm. Among them, multiple preset angles can be customized by the user according to the viewing angle of the camera. For example, if the viewing angle of the camera is 180 degrees, a certain reference direction can be set as 0 degrees, and a degree and (a+180) degrees based on the reference direction are determined as preset angles. In this embodiment, the images captured based on multiple preset angles contain scene information within a 360-degree range of the current shooting point,

[0423] Optionally, when generating a panoramic image, High Dynamic Range Imaging (HDRI) may be combined to generate a high-quality panoramic image.

[0424] Optionally, the information acquisition terminal also includes an inertial measurement unit (IMU). The IMU is used to correct the pose information corresponding to the collected point cloud data and image data to reduce errors caused by environmental or human factors (such as: the information acquisition terminal is not placed horizontally, etc.).

[0425] The control terminal is used to generate a floor plan diagram for each subspace in turn based on the point cloud data and panoramic images corresponding to multiple subspaces in the target physical space sent by the information collection terminal, and when generating the floor plan diagram, it is spliced ​​with the previously generated floor plan diagram to finally generate a floor plan diagram for the target physical space.

[0426] The control terminal can be a terminal device with data processing capabilities, such as a smart phone, tablet computer, or laptop computer.

[0427] Optionally, the floor plan generation system shown in Figure 7 may also include a cloud server, which may be a physical server or a virtual server in the cloud. The control terminal communicates with the cloud server by accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks.

[0428] In an optional embodiment, the cloud server can receive point cloud data and panoramic images corresponding to multiple subspaces forwarded by the control terminal to generate a floor plan of the target physical space and feed the floor plan back to the control terminal for display. In practice, the cloud server generates the floor plan of the target physical space using the same process as the control terminal generates the spatial structure diagram. However, due to the cloud server's greater computing power, it generates the floor plan more efficiently, further enhancing the user experience.

[0429] In another optional embodiment, the cloud server may also be directly connected to the information collection terminal to directly obtain the point cloud data and panoramic images corresponding to the multiple subspaces collected by the information collection terminal to generate a floor plan of the target physical space.

[0430] The following describes the process of generating a floor plan of a target physical space based on a control terminal. The following describes the process of generating a floor plan of a target physical space based on a control terminal.

[0431] FIG20 is a flowchart of a method for generating a floor plan according to a fifth embodiment of the present invention, which is applied to a control terminal. As shown in FIG20 , the method for generating a floor plan includes the following steps:

[0432] 2001. Obtain point cloud data and panoramic images corresponding to multiple subspaces in a target physical space obtained by an information collection terminal, wherein the point cloud data and panoramic image of any subspace are collected at at least one shooting point in any subspace.

[0433] 2002. During the process of sequentially stitching together the floor plan diagrams of a plurality of subspaces, for a first subspace to be stitched together, a target spatial outline of the first subspace is obtained based on point cloud data and / or panoramic images collected at at least one shooting point of the first subspace.

[0434] 2003. Obtain a target medium identified in a target panoramic image, where the target medium is an image of a physical medium in a first subspace in the target panoramic image. The target panoramic image is a panoramic image captured at at least one shooting point in the first subspace and is used to identify the target medium.

[0435] 2004. Determine a mapping medium for representing a target medium on the target space contour of the first subspace to obtain a floor plan diagram of the first subspace.

[0436] 2005. Splice the apartment structure diagram of the first subspace with the apartment structure diagram of the second subspace, where the second subspace is the subspace to which the apartment structure diagram of the first subspace has been spliced.

[0437] 2006. If there is no subspace in the multiple subspaces that has not been subjected to the splicing process of the floor plan structure diagram, the splicing process result is determined as the floor plan diagram of the target physical space.

[0438] In step 2001, the specific process of the information collection terminal collecting point cloud data and panoramic images of multiple subspaces in the target physical space can be referred to the above embodiment and will not be repeated in this embodiment.

[0439] Among them, if the information acquisition terminal is integrated into the target control terminal, the control terminal can directly and synchronously obtain the point cloud data and panoramic images of multiple subspaces obtained by the information acquisition terminal; if the information acquisition terminal is connected to the control terminal through a communication link, the control terminal receives the point cloud data and panoramic images of multiple subspaces sent by the information acquisition terminal based on the pre-established communication link.

[0440] In this embodiment, when generating a floor plan for the target physical space, floor plan diagrams for multiple subspaces are generated one by one. Each time a floor plan diagram for a subspace is generated, it is spliced ​​with the previously generated floor plan diagram until the floor plan diagram for the last subspace is generated and the splicing is complete. Finally, the final splicing result is determined to be the floor plan for the target physical space. Generating a floor plan diagram for a single subspace requires fewer computing resources and can accommodate the computing processing capabilities of most control devices. Furthermore, the splicing is performed while the floor plan diagram is being generated, facilitating immediate user confirmation of the splicing results and ensuring the accuracy of the generated floor plan for the target physical space.

[0441] In practice, the process of generating and splicing the floor plan diagrams for each subspace is similar. In this embodiment, the subspace currently to be spliced ​​is referred to as the first subspace, and the subspace to which the floor plan diagrams have been spliced ​​before the first subspace is referred to as the second subspace. It will be understood that the subspaces corresponding to the first and second subspaces are constantly updated.

[0442] Here, the generation process and the splicing process of the apartment structure diagram of the first subspace are first described, and the update of the first subspace and the second subspace are described in subsequent embodiments.

[0443] The floor plan of the target physical space is composed of floor plan diagrams for multiple subspaces. Both the floor plan of the target physical space and the floor plan diagrams for each subspace can be understood as two-dimensional floor plans of that space. The difference between the two is that the floor plan of the target physical space corresponds to more subspaces, making the two-dimensional floor plan "larger," while the floor plan diagram for each subspace is only a two-dimensional floor plan of the current subspace, making it "smaller." In scenarios such as AR house viewing, a two-dimensional floor plan is more commonly understood as a bird's-eye view of the physical space.

[0444] The floor plan diagram for a subspace includes the spatial outlines representing the walls in the subspace and the mapping medium representing the doors and windows in the subspace. Therefore, to determine the floor plan diagram for a subspace, the spatial outline and mapping medium for the subspace must first be obtained.

[0445] In an optional embodiment, the target spatial contour of the first subspace can be obtained based on the point cloud data and / or panoramic image collected at at least one shooting point in the first subspace. Specifically, the first spatial contour of the first subspace can be determined based on the point cloud data collected at at least one shooting point in the first subspace; and the second spatial contour of the first subspace can be determined based on the panoramic image collected at at least one shooting point in the first subspace. Thereafter, the target spatial contour of the first subspace is determined based on the first spatial contour and / or the second spatial contour. For example: the first spatial contour is determined to be the target spatial contour; or the second spatial contour is determined to be the target spatial contour; or a spatial contour with better contour line quality is selected from the above-mentioned first spatial contour and the above-mentioned second spatial contour as the target spatial contour of the target subspace; or the contour lines of the above-mentioned first spatial contour and the above-mentioned second spatial contour are fused to obtain a spatial contour with better contour line quality, and the fused spatial contour is determined to be the target spatial contour.

[0446] In practical applications, the target space outline contains multiple contour lines. Some of these contour lines may not match the actual positions of walls in the first subspace. To ensure that the target space outline accurately reflects the first subspace, the contour lines in the target space outline can be edited. Therefore, after determining the target space outline for the first subspace based on the first and / or second space outlines, manual or automated editing can be performed on the target space outline.

[0447] To facilitate editing of the target space outline, the target space outline of the first subspace can optionally be displayed in the 2D point cloud image of the first subspace. Subsequently, in response to a user's editing operation on the target space outline, the contour lines of the target space outline are adjusted so that the contour lines of the target space outline coincide with the wall lines in the 2D point cloud image. The wall lines in the 2D point cloud image correspond to the walls in the first subspace.

[0448] The two-dimensional point cloud image of the first subspace is obtained by plane mapping the point cloud data of at least one shooting point in the first subspace.

[0449] It can be understood that point cloud data is actually a series of three-dimensional coordinate points. Any three-dimensional coordinate point can be represented by three-dimensional Cartesian coordinates (x, y, z), where x, y, and z are the coordinate values ​​of the x-axis, y-axis, and z-axis, which have a common zero point and are orthogonal to each other.

[0450] In an optional embodiment, mapping the point cloud data collected at at least one shooting point in the first subspace to a two-dimensional plane to obtain a two-dimensional point cloud image of the first subspace includes: converting the three-dimensional coordinate points (x, y, z) corresponding to the point cloud data collected at at least one shooting point in the first subspace into two-dimensional coordinate points (x, y), for example, setting the z-axis coordinate value of the three-dimensional coordinate points to 0, and then obtaining the two-dimensional point cloud image of the first subspace based on the converted two-dimensional coordinate points. Alternatively, a three-dimensional spatial structure diagram of the first subspace is first generated based on the three-dimensional coordinate points (x, y, z) corresponding to the point cloud data collected at at least one shooting point in the first subspace, and then obtaining a top view of the three-dimensional spatial structure diagram as the two-dimensional point cloud image of the first subspace.

[0451] Since the relative positional relationship of at least one shooting point in the first subspace is known, the point cloud data collected at the at least one shooting point can be fused based on this relative positional relationship to obtain dense point cloud data, and then mapped to obtain a two-dimensional point cloud image. Specifically, the point cloud data of at least one shooting point in the first subspace are first fused to determine the target point cloud data of the first subspace; then, the target point cloud data is mapped to a two-dimensional plane to obtain an initial two-dimensional point cloud image of the first subspace; then, based on the user's correction operation on the initial two-dimensional point cloud image (for example, cropping out areas with unclear boundaries in the two-dimensional point cloud image, highlighting unclear wall lines in the two-dimensional point cloud image, etc.), the target two-dimensional point cloud image obtained after the correction operation is determined to be the two-dimensional point cloud image of the first subspace.

[0452] The two-dimensional point cloud image of the first subspace may also be used to obtain a first spatial contour of the first subspace, for example, by performing edge detection on the two-dimensional point cloud image of the first subspace to obtain the first spatial contour.

[0453] For contour lines in the target space outline that do not match the actual wall positions, you can adjust the shape and / or position of the contour lines in the target space outline based on the preset contour line editing options, or delete the contour lines that do not have corresponding wall lines, or add contour lines corresponding to a certain wall line.

[0454] FIG21 is a schematic diagram of the target space contour provided by the fifth embodiment of the present invention. For example, as shown in the left figure of FIG21 , before the contour line of the target space contour is edited, the contour line l in the target space contour does not coincide with the wall line L in the two-dimensional point cloud image of the first subspace, but in fact the contour line l and the wall line L correspond to the same wall in the first subspace. For another example, before the contour line of the target space contour is edited, there is no contour line in the target space contour corresponding to the wall line H in the two-dimensional point cloud image. Based on this, the size and position of the contour line l can be adjusted through the preset contour line editing option so that the contour line l coincides with the wall line L in the two-dimensional point cloud image; the contour line h can be added through the preset contour line editing option, and the contour line h coincides with the wall line H in the two-dimensional point cloud image. The edited target space contour is shown in the right figure of FIG21 . In this embodiment, the contour lines of the target space contour finally determined are connected to each other.

[0455] The process of determining the mapping medium in the floor plan corresponding to the first subspace is as follows: first, identify the corresponding target medium from the panoramic image through methods such as image recognition, that is, the image of the physical medium (door and window) in the first subspace in the panoramic image; then, determine the mapping medium corresponding to the target medium, that is, the corresponding identification of the door and window in the space outline.

[0456] In actual applications, the target medium may be identified in the panoramic images corresponding to more than one shooting point in the first subspace. For example, the three panoramic images corresponding to shooting points 1, 2 and 3 in the first subspace all contain images corresponding to the door and window in the first subspace.

[0457] It's understandable that capturing panoramic images at at least one capture point within the same subspace is intended to ensure the integrity of the scene information within each subspace. Typically, the captured panoramic images are sufficient for determining the target medium. Therefore, when determining the target medium within the first subspace, it's not necessary to identify the target medium in all panoramic images within the first subspace.

[0458] To improve the control terminal's efficiency in identifying the target medium, the control terminal can optionally determine a target panoramic image from the panoramic images of at least one capture point before identifying the target medium in the panoramic image. The target panoramic image is a panoramic image that meets preset recognition requirements, such as a panoramic image with the widest field of view, the best lighting, or a panoramic image that includes user-marked information (e.g., the best panoramic image).

[0459] For the first subspace, the shooting points corresponding to the target panoramic image can be the same as or different from the shooting points corresponding to the point cloud data used to generate the target space contour of the first subspace. Assume that the first subspace contains two shooting points, shooting point A and shooting point B. Panoramic image A1 and point cloud data A2 were acquired at shooting point A, and panoramic image B1 and point cloud data B2 were acquired at shooting point B. If the target space contour is generated based on point cloud data A2, then panoramic image A1 can be determined to be the target panoramic image, or panoramic image B1 can be determined to be the target panoramic image. Similarly, if the target space contour is generated based on point cloud data B2, then panoramic image A1 can be determined to be the target panoramic image, or panoramic image B1 can be determined to be the target panoramic image.

[0460] Because the relative pose between the laser sensor and camera is pre-calibrated before point cloud data and panoramic image acquisition, the coordinate mapping between the 3D point cloud coordinates corresponding to the acquired point cloud data and the panoramic pixel coordinates of the panoramic image can be determined for the first subspace based on the pre-calibrated relative pose and the relative positional relationship between the actual captured points in the first subspace.

[0461] Furthermore, based on the coordinate mapping between the point cloud data of the first subspace and the target panoramic image, a mapping between the target panoramic image and the target space contour of the first subspace can be established, that is, the mapping relationship between the target panoramic image of the first subspace and the target space contour can be predetermined.

[0462] Thus, a mapping medium for representing the target medium is determined on the target space contour of the first subspace, including: according to the mapping relationship between the target panoramic image and the target space contour of the first subspace, obtaining the panoramic pixel coordinates corresponding to the target medium in the target panoramic image and the mapped space contour coordinates, so as to determine the mapping medium corresponding to the target medium in the target space contour of the first subspace.

[0463] The mapping medium is adapted to the target identifier and target display size of the target medium, and the target identifier is used to distinguish target media of different types (doors or windows).

[0464] In the embodiments of the present invention, the specific method of mapping the coordinates of the panoramic image and the point cloud data is not limited. Optionally, the panoramic pixel coordinates can be directly mapped to the three-dimensional point cloud coordinates, and the three-dimensional point cloud coordinates can be mapped to the panoramic pixel coordinates based on the relative posture relationship between the devices that obtain the panoramic image and the point cloud data. Alternatively, the panoramic pixel coordinates can be first mapped to the intermediate coordinates, and then mapped to the three-dimensional point cloud coordinates, with the help of the relative posture relationship and the intermediate coordinate system. Alternatively, the three-dimensional point cloud coordinates can be first mapped to the intermediate coordinates, and then mapped to the intermediate coordinates. Here, the specific type of the intermediate coordinate system is not limited, nor is the specific method used in the coordinate mapping process. The mapping method used will also be different depending on the different intermediate coordinate systems and the different relative posture relationships.

[0465] After the floor plan diagram of the first subspace is generated, the floor plan diagram of the first subspace is spliced ​​with the floor plan diagram of the second subspace, wherein the second subspace is the subspace to which the floor plan diagram has been spliced ​​before the first subspace.

[0466] Because multiple spaces within the target physical space are connected to each other through doors or windows, for example, subspace E and subspace F are connected through the same target door. When collecting point cloud data and panoramic images, the target door is open. Therefore, the panoramic image of subspace E contains an object within subspace E. Therefore, in practical applications, the adjacency relationship between the multiple subspaces can be determined based on the panoramic image of at least one shooting point in the multiple subspaces, for example, through feature matching. Subsequently, based on the adjacency relationship between the multiple subspaces, the floor plan diagram of the first subspace is spliced ​​with the floor plan diagram of the second subspace.

[0467] The above is the process of generating and splicing the floor plan diagram of the first subspace to be spliced.

[0468] The following describes the updating of the first subspace and the second subspace, and the process of obtaining the floor plan of the target physical space with reference to FIG. 22 to FIG. 24 .

[0469] Figure 22 is a schematic diagram of the actual spatial structure of a target physical space provided by the fifth embodiment of the present invention. As shown in Figure 22, the target physical space contains three subspaces, namely bedroom 1, bedroom 2 and living room 3, among which bedroom 1 and living room 3 are connected by door body 1, bedroom 2 and living room 3 are connected by door body 2, and bedroom 1 and bedroom 2 are connected by window body 3.

[0470] When starting to generate the floor plan of the target physical space, the subspace corresponding to the point cloud data or panoramic image with the earliest acquisition time point can be determined as the first subspace for generating the floor plan structure diagram based on the acquisition time point of the point cloud data or panoramic image of each subspace; or, based on the number of adjacent subspaces corresponding to each subspace, the subspace with the largest number of adjacent subspaces can be determined as the first subspace for generating the floor plan structure diagram; or, a subspace can be randomly selected from multiple subspaces as the first subspace for generating the floor plan structure diagram.

[0471] In this embodiment, assuming that the first subspace for generating a floor plan diagram is bedroom 1, bedroom 1 is used as the first subspace to generate floor plan diagram 1 of bedroom 1.

[0472] Figure 23 is a schematic diagram of a floor plan provided in the fifth embodiment of the present invention. As shown in the left figure in Figure 23, the floor plan 1 includes a target space outline of the bedroom 1, and the target space outline is marked with mapping media representing the door 1 and the window 3 in the bedroom 1.

[0473] Since there is no second subspace with the floor plan spliced ​​before bedroom 1, floor plan 1 is the splicing result 1 of the floor plan of the first subspace and the floor plan of the second subspace.

[0474] Afterwards, it is determined whether there is a subspace in the three subspaces that has not been spliced ​​with the floor plan diagram.

[0475] Based on the assumptions in this embodiment, there are two subspaces among the three subspaces that have not been spliced ​​with the apartment structure diagram, namely bedroom 2 and living room 3.

[0476] If there are subspaces among the multiple subspaces that have not undergone floor plan splicing, first, the floor plan of the second subspace is updated to the splicing result, that is, the floor plan of the second subspace is determined to be splicing result 1. Then, a new first subspace to be spliced ​​is determined from the subspaces that have not undergone floor plan splicing. Specifically, a subspace is newly determined from bedroom 2 and living room 3 as the first subspace, and the floor plan of the newly determined first subspace is spliced ​​with the updated floor plan of the second subspace.

[0477] During the specific implementation process, optionally, the adjacent relationship between the multiple subspaces can be determined based on the panoramic image of at least one shooting point corresponding to each of the multiple subspaces; based on the adjacent relationship, a first subspace to be spliced ​​is re-determined from the subspace that has not undergone the household structure diagram splicing process, wherein the re-determined first subspace is adjacent to the second subspace. Alternatively, based on the acquisition time point of the point cloud data and / or panoramic image corresponding to each subspace in the subspace that has not undergone the household structure diagram splicing process, a first subspace to be spliced ​​is re-determined from the subspace that has not undergone the household structure diagram splicing process, wherein the difference between the acquisition time point corresponding to the re-determined first subspace and the current moment is greater than the difference between the acquisition time point corresponding to other subspaces in the subspace that has not undergone the household structure diagram splicing process and the current moment. Alternatively, a subspace is randomly selected from the subspace that has not undergone the household structure diagram splicing process as the new first subspace to be spliced.

[0478] In this embodiment, assuming that bedroom 2 is determined to be the new first subspace from the neighboring relationship between bedroom 2 and living room 3, a floor plan diagram 2 for bedroom 2 is generated and then spliced ​​with the floor plan diagram for the second subspace (i.e., splicing result 1). Splicing result 2 is shown in the right figure of FIG5 . Bedroom 2 is the first subspace, and bedroom 1 is the second subspace.

[0479] Afterwards, it is determined whether there is a subspace in the three subspaces that has not been spliced ​​with the floor plan diagram.

[0480] Based on the assumptions in this embodiment, among the three subspaces, there is still one subspace that has not been spliced ​​with the apartment structure diagram, namely, living room 3.

[0481] Thus, the floor plan of the second subspace is updated to the splicing result 2. Since there is only one subspace left that has not undergone the floor plan splicing process, the living room 3 is directly determined as the new first subspace, and the floor plan 3 of the living room 3 is generated. The floor plan 3 of the living room 3 is then spliced ​​with the updated floor plan of the second subspace (i.e., the splicing result 2). The splicing result 3 is shown in FIG24 , which is a schematic diagram of another floor plan provided by the fifth embodiment of the present invention, wherein the living room 3 is the first subspace, and the second subspace includes the bedroom 1 and the bedroom 2.

[0482] Afterwards, it is determined whether there is a subspace in the three subspaces that has not been spliced ​​with the floor plan diagram.

[0483] Based on the assumptions in this embodiment, there are no subspaces in the multiple subspaces that have not undergone the floor plan splicing process. Therefore, splicing result 3 is determined as the floor plan of the target physical space, that is, Figure 24 is determined as the floor plan of the target physical space. Splicing result 3 in Figure 24 includes the floor plans of the three subspaces in the target physical space.

[0484] In this embodiment, when generating a floor plan for the target physical space, floor plan diagrams for multiple subspaces are generated one by one. Each time a floor plan diagram for a subspace is generated, it is spliced ​​with the previously generated floor plan diagram until the floor plan diagram for the last subspace is generated and the splicing is complete. Finally, the final splicing result is determined to be the floor plan for the target physical space. Generating a floor plan diagram for a single subspace requires fewer computing resources and can accommodate the computing processing capabilities of most control devices. Furthermore, the splicing is performed while the floor plan diagram is being generated, facilitating immediate user confirmation of the splicing results and ensuring the accuracy of the generated floor plan for the target physical space. In addition, in this embodiment, the mapping medium on the spatial outline in the floor plan of each subspace is determined based on the panoramic image. Compared with point cloud data, the panoramic image can better reflect the actual position of doors and windows (i.e., target media) in the actual space. Therefore, based on the assistance of the panoramic image, the floor plan of each subspace is marked with more accurate door and window information, which can better reflect the scene information in the actual space. Therefore, the floor plan determined based on the floor plan of multiple subspaces can also accurately reflect the actual spatial structure of the target physical space.

[0485] The following describes in detail the floor plan generation device according to one or more embodiments of the present invention. Those skilled in the art will appreciate that these devices can be constructed using commercially available hardware components and configured according to the steps taught in this solution.

[0486] Figure 25 is a structural schematic diagram of a floor plan generation device provided in the fifth embodiment of the present invention, which is used to generate a floor plan of a target physical space, wherein the target physical space includes multiple subspaces and is applied to a control terminal. As shown in Figure 25, the device includes: an acquisition module 51, a splicing module 52 and a processing module 53.

[0487] The acquisition module 51 is used to acquire the point cloud data and panoramic images corresponding to the multiple subspaces obtained by the information acquisition terminal, wherein the point cloud data and panoramic image of any subspace are collected from at least one shooting point in any subspace.

[0488] The stitching module 52 is used to, in the process of stitching the floor plan diagrams of the multiple subspaces in sequence, obtain, for the first subspace to be stitched, the target space outline of the first subspace based on the point cloud data and / or panoramic image collected at at least one shooting point of the first subspace; obtain the target medium identified in the target panoramic image, the target medium being the image of the physical medium in the first subspace in the target panoramic image, the target panoramic image being a panoramic image for identifying the target medium in the panoramic image collected at at least one shooting point of the first subspace; determine the mapping medium for representing the target medium on the target space outline of the first subspace to obtain the floor plan diagram of the first subspace; stitch the floor plan diagram of the first subspace with the floor plan diagram of the second subspace, the second subspace being the subspace to which the floor plan diagram has been stitched before.

[0489] The processing module 53 is configured to determine the splicing result as the floor plan of the target physical space if there is no subspace in the multiple subspaces that has not been spliced ​​with the floor plan structure diagram.

[0490] Optionally, the processing module 53 is further used to update the floor plan structure diagram of the second subspace to the splicing processing result if there is a subspace among the multiple subspaces that has not undergone floor plan structure splicing processing; and to re-determine a first subspace to be spliced ​​from the subspace that has not undergone floor plan structure splicing processing, so as to splice the floor plan structure diagram of the re-determined first subspace with the updated floor plan structure diagram of the second subspace.

[0491] Optionally, the processing module 53 is specifically used to determine the adjacent relationship between multiple subspaces based on the panoramic image of at least one shooting point corresponding to each of the multiple subspaces; based on the adjacent relationship, re-determine a first subspace to be spliced ​​from the subspaces that have not undergone the household structure diagram splicing processing, wherein the re-determined first subspace is adjacent to the second subspace.

[0492] Optionally, the processing module 53 is further specifically configured to re-determine a first subspace to be spliced ​​from the subspace that has not undergone the household structure diagram splicing processing, based on the acquisition time point of the point cloud data and / or panoramic image corresponding to each subspace in the subspace that has not undergone the household structure diagram splicing processing, wherein the difference between the acquisition time point corresponding to the re-determined first subspace and the current moment is greater than the difference between the acquisition time point corresponding to other subspaces in the subspace that has not undergone the household structure diagram splicing processing and the current moment.

[0493] Optionally, the stitching module 52 is specifically used to determine the first spatial contour of the first subspace based on the point cloud data collected at at least one shooting point in the first subspace; determine the second spatial contour of the first subspace based on the panoramic image collected at at least one shooting point in the first subspace; and determine the target spatial contour of the first subspace based on the first spatial contour and / or the second spatial contour.

[0494] Optionally, the stitching module 52 is further specifically used to determine the two-dimensional point cloud image of the first subspace based on the point cloud data collected at at least one shooting point in the first subspace; and determine the first spatial contour of the first subspace based on the two-dimensional point cloud image.

[0495] Optionally, the stitching module 52 is further specifically used to obtain the panoramic pixel coordinates corresponding to the target medium in the target panoramic image and the mapped spatial contour coordinates based on the mapping relationship between the target panoramic image and the target spatial contour of the first subspace, so as to determine the mapping medium corresponding to the target medium in the target spatial contour of the first subspace; wherein the mapping medium is adapted to the target identifier and target display size of the target medium, the target identifier is used to distinguish target media belonging to different types, and the mapping relationship is a mapping between the target panoramic image and the spatial contour established based on the coordinate mapping between the point cloud data of the first subspace and the target panoramic image.

[0496] The device shown in Figure 25 can execute the steps in the aforementioned embodiments. The detailed execution process and technical effects can be found in the description of the aforementioned embodiments and will not be repeated here.

[0497] Sixth embodiment

[0498] FIG26 is a flowchart of a floor plan generation method provided by a sixth embodiment of the present invention, which is used to generate a floor plan of a target physical space, where the target physical space includes at least N spaces and is applied to a control terminal. As shown in FIG26 , the floor plan generation method includes the following steps:

[0499] Step S261: Acquire point cloud data and panoramic images collected by the information collection terminal in each of N spaces in the target physical space, wherein the point cloud data and panoramic image are collected at at least one shooting point in each space.

[0500] Step S262: Obtain the Mth space outline for display for editing; wherein the Mth space outline is the space outline of the Mth space among the N spaces; the Mth space outline is obtained based on point cloud data and / or panoramic images collected from at least one shooting point of the Mth space.

[0501] Step S263: Acquire the first target medium identified in the first target panoramic image, so as to obtain the first mapping medium of the first target medium in the Mth space outline according to the first target medium, so as to edit the Mth space outline according to the first mapping medium to obtain the floor plan of the Mth space.

[0502] Step S264: Obtain the M+1th spatial outline for display for editing; wherein the M+1th spatial outline is the spatial outline of the M+1th space in the N spaces, the M+1th space is an adjacent space to the Mth space, and the M+1th spatial outline is obtained based on point cloud data and / or panoramic images collected from at least one shooting point in the M+1th space.

[0503] Step S265: Acquire the second target medium identified in the second target panoramic image, so as to obtain a second mapping medium of the second target medium in the M+1th space outline according to the second target medium, so as to edit the M+1th space outline according to the second mapping medium to obtain the floor plan of the M+1th space.

[0504] Step S266: Splice the floor plan diagram of the M+1th space with the floor plan diagram of the Mth space, and determine whether the M+1th space is the last space among the N spaces to generate a floor plan diagram; if not, execute step S267; if so, execute step S268.

[0505] Step S267: Merge the Mth space and the M+1th space as the Mth space, and return to execute step S264.

[0506] Step S268: Use the splicing result as the floor plan of the target physical space for display, and the process ends.

[0507] The first target panoramic image in step S263 is a panoramic image captured at at least one shooting point in the Mth space, and is used to identify the first target medium; the first target medium is the image of the physical medium in the Mth space in the first target panoramic image. The second target panoramic image in step S265 is a panoramic image captured at at least one shooting point in the M+1th space, and is used to identify the second target medium; the second target medium is the image of the physical medium in the M+1th space in the second target panoramic image.

[0508] The control terminal can be a terminal device with data processing capabilities, such as a smart phone, a tablet computer, a laptop computer, etc. The control terminal can be connected to the information collection terminal through communication methods such as Bluetooth or Wireless Fidelity (WiFi) hotspots.

[0509] In this embodiment, when generating a floor plan for a target physical space, floor plan diagrams for N spaces are generated one by one. Each time a floor plan diagram is generated, it is spliced ​​with the previously generated floor plan diagram until the floor plan diagram for the last space is generated and the splicing is complete. Finally, the final spliced ​​result is determined to be the floor plan for the target physical space. Generating a floor plan diagram for a single space requires fewer computing resources and can accommodate the computing processing capabilities of most control devices. Furthermore, the splicing is performed while the floor plan diagram is generated, facilitating immediate user confirmation of the splicing results and ensuring the accuracy of the generated floor plan for the target physical space. Furthermore, in this embodiment, the mapping medium on the spatial outline of each space floor plan diagram is determined based on a panoramic image. Since panoramic images better reflect the actual location of doors and windows (i.e., the target medium) in the actual space compared to point cloud data, the floor plan diagram for each space, aided by the panoramic image, includes more accurate door and window information, better reflecting the scene information in the actual space.

[0510] Before specifically introducing the floor plan generation method provided by the embodiment of the present invention, a brief description of the Mth space and the M+1th space, etc. in the embodiment of the present invention is first given.

[0511] In this embodiment, N represents the number of spaces included in the target physical space, and the value of N is an integer greater than or equal to 1. For example, assuming the target physical space is a house, if the house contains only one space (e.g., a living room), the value of N is 1; if the house contains three spaces (e.g., a living room, a bedroom, and a bathroom), the value of N is 3. It will be understood that when the value of N is greater than or equal to 2, any space in the target physical space must have an adjacent space.

[0512] When N is 1, the Mth space is actually the only space contained in the target physical space. In this case, the M+1th space does not exist. Therefore, when generating the floor plan of the target physical space, the floor plan of the Mth space is the floor plan of the target physical space.

[0513] When N is 2, assuming the target physical space includes space a and space b, if the Mth space is space a, then the M+1th space is space b; conversely, if the Mth space is space b, then the M+1th space is space a. Therefore, when generating the floor plan of the target physical space, the floor plan of the Mth space and the floor plan of the M+1th space are concatenated to form the floor plan of the target physical space.

[0514] When the value of N is greater than or equal to 3, the space corresponding to the Mth space and the M+1th space is actually continuously updated during the splicing process and does not specifically refer to a certain space among the N spaces.

[0515] Assuming that the value of N is 3, for example: assuming that the target physical space includes space a, space b and space c, where space a is adjacent to space b, and space b is adjacent to space c.

[0516] When generating the floor plan for the target physical space, assume that space a is the first space for which floor plan diagrams are generated. This means that space a is first determined to be the Mth space. After generating floor plan diagram A for space a, the M+1th space is determined. Since space b is adjacent to space a, it is determined to be the M+1th space, and floor plan diagram B for space b is generated. Next, floor plan diagrams A and B are concatenated to produce concatenated result AB.

[0517] Since the floor plan for space c has not yet been generated, space b (also known as space M+1) is not the last of the three spaces to generate a floor plan. Therefore, space M and space M+1 are merged to form space M. This means that the merged space M includes space a and space b. The floor plan for this merged space M is the above-mentioned concatenation result AB.

[0518] When re-determining the M+1th space based on the merged Mth space, since space c is adjacent to space b, space c is also adjacent to the merged Mth space. Therefore, space c is determined to be the M+1th space, and a floor plan C for space c is generated. Next, floor plan C is spliced ​​with the floor plan of the Mth space (i.e., splicing result AB) to obtain splicing result ABC. Since there are no spaces for which a floor plan has not been generated, splicing result ABC is the floor plan of the target physical space.

[0519] In the above process, the space corresponding to the Mth space is updated from space a to space a and space b, and the M+1th space is updated from space b to space c.

[0520] The following describes the method for generating the floor plan shown in FIG26 in conjunction with the schematic diagram of the scene for generating the floor plan shown in FIG27 .

[0521] FIG27 is a schematic diagram of a scenario for generating a floor plan provided by the sixth embodiment of the present invention. As shown in FIG27 , the information collection terminal and the control terminal are decoupled from each other. The information collection terminal collects point cloud data and panoramic images in each of N spaces and sends them to the control terminal. This allows the control terminal to generate floor plan diagrams for N spaces one by one, for example, sequentially generating floor plan diagram a for space 1, floor plan diagram b for space 2, and so on; and while generating the floor plan diagrams, the generated floor plan diagrams are spliced ​​together to ultimately obtain the floor plan of the target physical space. The information collection terminal can be connected to the control terminal in communication with the control terminal via, for example, Bluetooth, Wireless Fidelity (WiFi) hotspot, or the like.

[0522] Alternatively, the information collection terminal can be directly integrated with the control terminal. The control terminal, acting as a single entity, can directly and synchronously acquire the point cloud data and panoramic images collected by the information collection terminal in each of the N spaces, eliminating the need to transmit the point cloud data and panoramic images over an established communication connection. This can improve the efficiency of the control terminal in generating floor plans for the target physical space.

[0523] Optionally, a floor plan of the target physical space can be generated through a cloud server. The cloud server can be a physical server or a virtual server in the cloud, and the control terminal communicates with the cloud server by accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G, or other mobile communication network.

[0524] The cloud server can receive the point cloud data and panoramic images corresponding to the N spaces forwarded by the control terminal to generate a floor plan of the target physical space, and feed the floor plan back to the control terminal for display. Alternatively, the cloud server can also be directly connected to the information collection terminal to directly obtain the point cloud data and panoramic images corresponding to the multiple subspaces collected by the information collection terminal to generate a floor plan of the target physical space. In fact, the process of the cloud server generating the floor plan of the target physical space is the same as the process of the control terminal generating the spatial structure diagram, but because the cloud server has stronger computing power, it is more efficient in generating the floor plan of the target physical space, which can further enhance the user experience. In this embodiment, the control terminal generates the floor plan of the target physical space as an example for explanation, but it is not limited to this.

[0525] In practical applications, to obtain complete spatial information for each space, more than one shooting point may be set in any space. For example, space X may have shooting points 1, 2, and 3. Therefore, in this implementation, the point cloud data and panoramic image of any space refer to the point cloud data and panoramic image collected at at least one shooting point in any space.

[0526] In actual applications, the setting of shooting points can be customized by the user based on the current collection situation when collecting scene information of each space through the information collection terminal; or the information collection terminal or control terminal can automatically generate reference shooting points for the space based on the space description information (such as space size, etc.) input by the user.

[0527] When the information collection terminal collects point cloud data and panoramic images at each shooting point, the corresponding information collection process is the same. In this embodiment, a certain shooting point Y is taken as an example for explanation.

[0528] First, a brief description of the basic structure of the information acquisition terminal is given. The information acquisition terminal includes: a laser sensor, a camera, a motor, and a processor (such as a CPU). Among them, the laser sensor and the camera are used as sensing devices to collect scene information of each space at each shooting point, that is, point cloud data and panoramic images; the motor is used to drive the laser sensor and the camera to rotate so as to collect point cloud data and panoramic images from various angles. Optionally, the information acquisition terminal also includes an inertial measurement unit (IMU). The IMU is used to correct the posture information corresponding to the collected point cloud data and image data to reduce errors caused by environmental or human factors (such as: the information acquisition terminal is not placed horizontally, etc.).

[0529] Next, the data collection process of the information collection terminal at the shooting point Y is described.

[0530] For shooting point Y, the information collection terminal responds to the information collection instruction by driving the motor to rotate the laser sensor 360 degrees to collect point cloud data corresponding to the target shooting point Y; the driving motor also drives the camera 360 degrees to collect a panoramic image corresponding to the target shooting point Y. The information collection instruction is sent by the user through the control terminal, or is triggered in response to a trigger operation by the user on the information collection terminal.

[0531] At the shooting point Y, the motor can drive the laser sensor and the camera to rotate at the same time to collect point cloud data and panoramic images at the same time, or it can drive the laser sensor and the camera to rotate in sequence, for example: first drive the laser sensor to rotate and then drive the camera to rotate, or first drive the camera to rotate and then drive the laser to rotate, so as to collect point cloud data and panoramic images respectively. This embodiment does not impose any restrictions on this.

[0532] In an optional embodiment, if point cloud data is collected first and then the panoramic image is captured, to improve the quality of the captured panoramic image, the camera can be activated simultaneously with the point cloud data acquisition process to collect scene lighting information at the current capture point, perform light measurement, and determine corresponding exposure parameters. The camera then captures the panoramic image based on the determined exposure parameters.

[0533] Optionally, the camera in the information collection terminal is a panoramic camera or a non-panoramic camera. If the camera in the information collection terminal is a non-panoramic camera, then during the above-mentioned 360-degree rotation process, the camera is controlled to capture images corresponding to the target shooting point Y at multiple preset angles, and the above-mentioned processor can stitch the images captured at multiple preset angles into a panoramic image through a panoramic image stitching algorithm such as a feature matching algorithm. For example: if the camera's viewing angle is 120 degrees, a certain reference direction can be set as 0 degrees, and a degree, (a+120) degrees, and (a+240) degrees based on the reference direction are determined as preset angles, and the camera is controlled to capture images 1, 2, and 3 at these three preset angles; then, images 1 to 3 are stitched together to obtain a panoramic image.

[0534] It is understood that the smaller the intervals between the preset angles, the more images are captured, and the more accurate the resulting panoramic image after stitching. In specific implementations, the number of preset angles can be customized by the user based on the camera's viewing angle. Images captured at multiple preset angles contain scene information within a 360-degree range at the current point.

[0535] Optionally, when generating a panoramic image, High Dynamic Range Imaging (HDRI) may be combined to generate a high-quality panoramic image.

[0536] After the information collection terminal has collected point cloud data and panoramic images at a shooting point, it can directly send the point cloud data and panoramic images of the shooting point to the control terminal; or, it can first store them, and then after completing the collection of point cloud data and panoramic images at all shooting points in the current space, send the point cloud data and panoramic images of all shooting points in the space to the control terminal. This embodiment does not impose any restrictions on this.

[0537] In practical applications, the control terminal generates a floor plan diagram for any of N spaces based on point cloud data and panoramic images. In this embodiment, the floor plan diagram generation process for space Z is described using an example. Space Z can be either the Mth space or the M+1th space.

[0538] To represent the physical space corresponding to space Z, the floor plan diagram for space Z includes the spatial outline of space Z and a mapping medium. The spatial outline represents the walls in the physical space, while the mapping medium represents the windows and doors in the physical space. Therefore, the process of obtaining the floor plan diagram for space Z can be further divided into the process of determining the spatial outline of space Z and the process of determining the mapping medium of space Z, each of which is explained below.

[0539] First, the process of obtaining the spatial contour of space Z is introduced.

[0540] In an optional implementation, the spatial contour of space Z can be obtained based on the point cloud data and / or panoramic image collected at at least one shooting point in space Z. Specifically, the first spatial contour of space Z can be determined based on the point cloud data collected at at least one shooting point in space Z; and the second spatial contour of space Z can be determined based on the panoramic image collected at at least one shooting point in space Z. Afterwards, the spatial contour of space Z is determined based on the first spatial contour and / or the second spatial contour. For example: the first spatial contour is determined to be the spatial contour of space Z; or the second spatial contour is determined to be the spatial contour of space Z; or a spatial contour with better contour line quality is selected from the above-mentioned first spatial contour and the above-mentioned second spatial contour as the spatial contour of space Z; or the contour lines of the above-mentioned first spatial contour and the above-mentioned second spatial contour are fused to obtain a spatial contour with better contour line quality, and the spatial contour after the fusion processing is determined to be the spatial contour of space Z.

[0541] In practical applications, the spatial outline of space Z contains multiple contour lines. Some of these contour lines may not match the actual positions of the walls of space Z. To ensure that the spatial outline accurately reflects space Z, the contour lines in the spatial outline of space Z can be edited. Therefore, after determining the spatial outline of space Z based on the first spatial outline and / or the second spatial outline, manual or automated editing can be performed on the spatial outline of space Z.

[0542] To facilitate editing of the spatial outline of space Z, the spatial outline of space Z may optionally be displayed in the two-dimensional point cloud image of space Z. Subsequently, in response to a user's editing operation on the spatial outline of space Z, the contour lines of the spatial outline of space Z are adjusted so that the contour lines of the spatial outline of space Z coincide with the wall lines in the two-dimensional point cloud image. The wall lines in the two-dimensional point cloud image correspond to the walls in space Z.

[0543] The two-dimensional point cloud image of space Z is obtained by plane mapping the point cloud data of at least one shooting point in space Z.

[0544] Since the relative position relationship of at least one shooting point in space Z is known, the point cloud data collected at at least one shooting point can be fused based on the relative position relationship to obtain dense point cloud data, and then mapped to obtain a two-dimensional point cloud image.

[0545] Specifically, the point cloud data of at least one shooting point in space Z are first fused to determine the target point cloud data of space Z; then, the target point cloud data is mapped to a two-dimensional plane to obtain an initial two-dimensional point cloud image of space Z; thereafter, based on the user's correction operation on the initial two-dimensional point cloud image (for example: cropping out areas with unclear boundaries in the two-dimensional point cloud image, highlighting unclear wall lines in the two-dimensional point cloud image, etc.), the target two-dimensional point cloud image obtained after the correction operation is determined to be the two-dimensional point cloud image of space Z.

[0546] It can be understood that the point cloud data is actually a series of three-dimensional coordinate points, and any three-dimensional coordinate point can be represented by a three-dimensional Cartesian coordinate (x, y, z), where x, y, and z are the coordinate values ​​of the x-axis, y-axis, and z-axis, which have a common zero point and are orthogonal to each other.

[0547] In an optional embodiment, the target point cloud data of space Z is mapped to a two-dimensional plane to obtain an initial two-dimensional point cloud image of space Z, including: converting the three-dimensional coordinate points (x, y, z) corresponding to the target point cloud data of space Z into two-dimensional coordinate points (x, y), for example: setting the z-axis coordinate value of the three-dimensional coordinate points to 0, and then obtaining the initial two-dimensional point cloud image of space Z based on the converted two-dimensional coordinate points. Alternatively, a three-dimensional spatial structure diagram of space Z is first generated based on the three-dimensional coordinate points (x, y, z) corresponding to the target point cloud data of space Z, and then obtaining a top view of the three-dimensional spatial structure diagram as the initial two-dimensional point cloud image of space Z.

[0548] The two-dimensional point cloud image of space Z may also be used to obtain a first spatial contour of space Z. For example, the first spatial contour of space Z may be obtained by performing edge detection on the two-dimensional point cloud image of space Z.

[0549] For the contour lines in the spatial outline of space Z that do not match the actual wall positions, you can adjust the shape and / or position of the contour lines in the spatial outline of space Z based on the preset contour line editing options, or delete the contour lines for which there are no corresponding wall lines, or add contour lines corresponding to a certain wall line.

[0550] For ease of understanding, for example, FIG28 is a schematic diagram of the spatial outline of space Z provided in the sixth embodiment of the present invention. As shown in the left figure of FIG28 , before the contour lines of the spatial outline of space Z are edited, contour line l in the spatial outline of space Z does not coincide with wall line L in the two-dimensional point cloud image of space Z. However, contour line l and wall line L actually correspond to the same wall in space Z. For another example, before the contour lines of the spatial outline of space Z are edited, there is no contour line in the spatial outline of space Z corresponding to wall line H in the two-dimensional point cloud image. Based on this, the size and position of contour line l can be adjusted through the preset contour line editing options so that contour line l coincides with wall line L in the two-dimensional point cloud image; contour line h can be added through the preset contour line editing options, and contour line h coincides with wall line H in the two-dimensional point cloud image. The edited spatial outline of space Z is shown in the right figure of FIG28 . In this embodiment, the contour lines of the spatial outline of space Z are finally connected to each other.

[0551] Afterwards, the process of obtaining the mapping medium of space Z is introduced.

[0552] In summary, to obtain the mapping medium of space Z, we must first identify the corresponding target medium from the panoramic image through methods such as image recognition, that is, the image of the physical medium (door and window) in space Z in the panoramic image; then, determine the mapping medium corresponding to the target medium.

[0553] In actual applications, the target medium may be included in the panoramic images corresponding to more than one shooting point in space Z. For example, the three panoramic images corresponding to shooting points 1, 2 and 3 in space Z all contain images corresponding to the door and window in space Z.

[0554] It's understandable that capturing panoramic images at at least one point in the same subspace is intended to ensure the integrity of the scene information in each subspace. Typically, the captured panoramic images are sufficient for determining the target medium. Therefore, when determining the target medium in space Z, it's not necessary to identify the target medium in all panoramic images of space Z.

[0555] To improve the efficiency of the control terminal in identifying the target medium, optionally, before the control terminal identifies the target medium in the panoramic image, a target panoramic image can be determined from the panoramic images of at least one shooting point in space Z for use in identifying the target medium. The target panoramic image is a panoramic image that meets preset recognition requirements, such as a panoramic image with the widest field of view, the best lighting, or a panoramic image that includes user-marked information (e.g., the best panoramic image).

[0556] For space Z, the shooting points corresponding to the target panoramic image can be the same as or different from the shooting points corresponding to the point cloud data used to generate the spatial outline. Assume that space Z contains two shooting points, namely shooting point A and shooting point B. Panoramic image A1 and point cloud data A2 are acquired at shooting point A, and panoramic image B1 and point cloud data B2 are acquired at shooting point B. If the spatial outline of space Z is generated based on point cloud data A2, then panoramic image A1 can be determined to be the target panoramic image, or panoramic image B1 can be determined to be the target panoramic image. Similarly, if the spatial outline of space Z is generated based on point cloud data B2, then panoramic image A1 can be determined to be the target panoramic image, or panoramic image B1 can be determined to be the target panoramic image.

[0557] Because the relative pose between the laser sensor and camera is pre-calibrated before point cloud data and panoramic image acquisition, the coordinate mapping between the 3D point cloud coordinates corresponding to the acquired point cloud data in Space Z and the panoramic pixel coordinates of the panoramic image can be determined based on the pre-calibrated relative pose and the relative positional relationship between the actual captured points in Space Z.

[0558] Furthermore, based on the coordinate mapping between the point cloud data of space Z and the target panoramic image, a mapping between the target panoramic image of space Z and the spatial outline of space Z can be established, that is, the mapping relationship between the target panoramic image of space Z and the spatial outline can be predetermined.

[0559] Therefore, based on the mapping relationship between the target panoramic image and the spatial contour of space Z, the panoramic pixel coordinates corresponding to the target medium in the target panoramic image and the mapped spatial contour coordinates can be obtained to determine the mapping medium corresponding to the target medium in the spatial contour of space Z, so as to obtain the floor plan of space Z.

[0560] The mapping medium is adapted to the target identification and target display size of the target medium, and the target identification is used to distinguish target media of different types (doors or windows).

[0561] In the embodiments of the present invention, the specific method of mapping the coordinates of the panoramic image and the point cloud data is not limited. Optionally, the panoramic pixel coordinates can be directly mapped to the three-dimensional point cloud coordinates, and the three-dimensional point cloud coordinates can be mapped to the panoramic pixel coordinates based on the relative posture relationship between the devices that obtain the panoramic image and the point cloud data. Alternatively, the panoramic pixel coordinates can be first mapped to the intermediate coordinates, and then mapped to the three-dimensional point cloud coordinates, with the help of the relative posture relationship and the intermediate coordinate system. Alternatively, the three-dimensional point cloud coordinates can be first mapped to the intermediate coordinates, and then mapped to the intermediate coordinates. Here, the specific type of the intermediate coordinate system is not limited, nor is the specific method used in the coordinate mapping process. The mapping method used will also be different depending on the different intermediate coordinate systems and the different relative posture relationships.

[0562] The above is the process of obtaining the floor plan of any space Z in the target physical space.

[0563] The following describes a process of obtaining a floor plan of a target physical space containing at least N spaces with reference to FIG. 22 and FIG. 29 to FIG. 31 in the fifth embodiment.

[0564] As shown in Figure 22, the target physical space contains three spaces, namely bedroom 1, bedroom 2 and living room 3, among which bedroom 1 and living room 3 are connected through door 1, bedroom 2 and living room 3 are connected through door 2, and bedroom 1 and bedroom 2 are connected through window 3.

[0565] When starting to generate the floor plan of the target physical space, the space corresponding to the point cloud data or panoramic image with the earliest collection time point can be determined as the first space to generate the floor plan structure diagram based on the collection time point of the point cloud data or panoramic image of each space; or, based on the number of adjacent spaces corresponding to each space, the space with the largest number of adjacent spaces can be determined as the first space to generate the floor plan structure diagram; or, a space can be randomly selected from multiple spaces as the first space to generate the floor plan structure diagram.

[0566] In this embodiment, assuming that the first space for which a floor plan diagram is generated is bedroom 1, bedroom 1 is taken as the Mth space, and a floor plan diagram R for the Mth space (i.e., bedroom 1) is generated, as shown in the left figure of Figure 29. Figure 29 is a schematic diagram of a floor plan diagram provided in accordance with the sixth embodiment of the present invention.

[0567] Afterwards, the space adjacent to the Mth space is determined as the M+1th space, and a floor plan diagram of the M+1th space is generated.

[0568] It is understood that multiple spaces within the target physical space are connected to each other through doors or windows. For example, space E and space F are connected through the same target door. When collecting point cloud data and panoramic images, the target door is open, so the panoramic image of space E includes an object in space E. Therefore, in practical applications, the adjacency relationship between multiple spaces can be determined based on the panoramic image of at least one capture point in the multiple spaces, for example, through feature matching.

[0569] Therefore, when determining the M+1th space adjacent to the Mth space, as an optional implementation method, the M+1th space can be determined from the spaces for which floor plan diagrams have not been generated based on a panoramic image of at least one shooting point in the Mth space and a panoramic image of at least one shooting point in each of the remaining N spaces for which floor plan diagrams have not been generated. Specifically, based on a panoramic image of at least one shooting point in the Mth space and a panoramic image of at least one shooting point in each of the remaining N spaces for which floor plan diagrams have not been generated, the total number of adjacent spaces corresponding to each space in the spaces for which floor plan diagrams have not been generated is determined; and the space for which the total number of adjacent spaces is greater than or equal to a set threshold is determined to be the M+1th space. During specific implementation, if the total number of adjacent spaces corresponding to multiple spaces is equal, a space can be randomly selected from them as the M+1th space.

[0570] Based on the assumption in this embodiment, the remaining spaces for which the floor plan diagram has not been generated are bedroom 2 and living room 3. Therefore, the M+1th space outline needs to be determined from bedroom 2 and living room 3.

[0571] Since bedroom 2 is adjacent to bedroom 1 and living room 3, and living room 3 is adjacent to bedroom 1 and bedroom 2, the number of adjacent spaces corresponding to bedroom 2 and living room 3 is 2. Assuming that bedroom 2 is randomly selected as the M+1th space, the floor plan S of the M+1th space (i.e., bedroom 2) is generated, as shown in the right figure in Figure 29.

[0572] Afterwards, the floor plan diagram R is spliced ​​with the floor plan diagram S to obtain a splicing result RS, as shown in Figure 30. A determination is then made as to whether the M+1th space is the last of the three spaces for which a floor plan diagram is generated. Figure 30 is a schematic diagram of a spliced ​​floor plan diagram provided in accordance with the sixth embodiment of the present invention.

[0573] Optionally, based on the adjacent relationship between the M+1th space and the Mth space, the floor plan structure diagram of the M+1th space and the floor plan structure diagram of the Mth space may be spliced ​​in the same spatial coordinate system.

[0574] Based on the assumption in this embodiment, the corresponding floor plan diagram has not yet been generated for living room 3 among the three spaces. Therefore, the M+1th space (i.e., bedroom 2) is not the last space among the three spaces to generate a floor plan diagram.

[0575] Therefore, we merge the Mth and M+1th spaces into the Mth space. That is, we consider Bedroom 1 and Bedroom 2 as a whole, and this whole is referred to as the Mth space. At this point, the Mth space includes Bedroom 1 and Bedroom 2. The floor plan diagram of the Mth space is the above-mentioned splicing result RS, as shown in Figure 30.

[0576] Next, the M+1th space adjacent to the Mth space (including Bedroom 1 and Bedroom 2) is re-identified. Since Living Room 3 is adjacent to both Bedroom 1 and Bedroom 2, Living Room 3 is also adjacent to the Mth space (including Bedroom 1 and Bedroom 2). Furthermore, Living Room 3 is determined to be the M+1th space, and a floor plan diagram T for the M+1th space (Living Room 3) is generated, as shown in the left figure of Figure 31. Figure 31 is a schematic diagram of another spliced ​​floor plan diagram provided by the sixth embodiment of the present invention.

[0577] Afterwards, the floor plan structure diagram T of the M+1th space is spliced ​​with the floor plan structure diagram of the Mth space (i.e., the splicing result RS) to obtain the splicing result RST, as shown in the right figure in Figure 31; and it is determined whether the M+1th space is the last space among the three spaces to generate a floor plan structure diagram.

[0578] Based on the assumptions in this embodiment, after the floor plan diagram of the living room 3 is generated, there is no space in the target physical space for which the floor plan diagram has not been generated. Therefore, the splicing result RST is the floor plan diagram of the target physical space.

[0579] The process of obtaining the floor plan diagrams of bedroom 1, bedroom 2 and living room 3 can refer to the aforementioned embodiment and will not be repeated here.

[0580] In this embodiment, when generating a floor plan for a target physical space, floor plan diagrams for N spaces are generated one by one. Each time a floor plan diagram is generated, it is spliced ​​with the previously generated floor plan diagram until the floor plan diagram for the last space is generated and the splicing is complete. Finally, the final spliced ​​result is determined to be the floor plan for the target physical space. Generating a floor plan diagram for a single space requires fewer computing resources and can accommodate the computing processing capabilities of most control devices. Furthermore, the splicing is performed while the floor plan diagram is generated, facilitating immediate user confirmation of the splicing results and ensuring the accuracy of the generated floor plan for the target physical space. Furthermore, in this embodiment, the mapping medium on the spatial outline of each space floor plan diagram is determined based on a panoramic image. Since panoramic images better reflect the actual location of doors and windows (i.e., the target medium) in the actual space compared to point cloud data, the floor plan diagram for each space, aided by the panoramic image, includes more accurate door and window information, better reflecting the scene information in the actual space.

[0581] The following describes in detail the floor plan generation device according to one or more embodiments of the present invention. Those skilled in the art will appreciate that these devices can be constructed using commercially available hardware components and configured according to the steps taught in this solution.

[0582] Figure 32 is a structural schematic diagram of a floor plan generation device provided in the sixth embodiment of the present invention. The device is used to generate a floor plan of a target physical space, where the target physical space includes at least N spaces and is applied to a control terminal. As shown in Figure 32, the device includes: a first acquisition module 61, a second acquisition module 62 and a processing module 63.

[0583] The first acquisition module 61 is used to acquire the point cloud data and panoramic images collected by the information collection terminal in each of the N spaces, wherein the point cloud data and panoramic images are collected at at least one shooting point in each of the spaces; acquire the Mth space outline for display for editing; wherein the Mth space outline is the space outline of the Mth space in the N spaces, and the Mth space outline is acquired based on the point cloud data and / or panoramic image collected by at least one shooting point in the Mth space; acquire the first target medium identified in the first target panoramic image, so as to acquire the first mapping medium of the first target medium in the Mth space outline based on the first target medium, so as to edit the Mth space outline based on the first mapping medium to obtain the floor plan of the Mth space; the first target panoramic image is a panoramic image for identifying the first target medium in the panoramic image collected by at least one shooting point in the Mth space, and the first target medium is an image of the physical medium in the Mth space in the first target panoramic image.

[0584] The second acquisition module 62 is used to obtain the M+1th space outline for display for editing; wherein, the M+1th space outline is the space outline of the M+1th space in the N spaces, the M+1th space is an adjacent space of the Mth space, and the M+1th space outline is obtained based on point cloud data and / or panoramic images collected from at least one shooting point of the M+1th space; the second target medium identified in the second target panoramic image is obtained, so that the second mapping medium of the second target medium in the M+1th space outline is obtained based on the second target medium, so as to edit the M+1th space outline according to the second mapping medium to obtain the floor plan of the M+1th space; the second target panoramic image is a panoramic image for identifying the second target medium in the panoramic image collected from at least one shooting point of the M+1th space, and the second target medium is an image of the physical medium in the M+1th space in the first target panoramic image.

[0585] The processing module 63 is used to splice the floor plan diagram of the M+1th space with the floor plan diagram of the Mth space, and determine whether the M+1th space is the last space among the N spaces to generate a floor plan diagram; if not, the Mth space and the M+1th space are merged as the Mth space, and the process returns to execute the second acquisition module 62; if so, the splicing result is used as the floor plan diagram of the target physical space for display, and the process ends.

[0586] Optionally, the second acquisition module 62 is also used to determine the M+1th space from the spaces for which the floor plan diagram has not been generated based on a panoramic image of at least one shooting point of the Mth space and a panoramic image of at least one shooting point of each of the remaining spaces in the N spaces for which the floor plan diagram has not been generated.

[0587] Optionally, the second acquisition module 62 is specifically used to determine the total number of adjacent spaces corresponding to each space in the space where the floor plan diagram has not been generated based on a panoramic image of at least one shooting point in the Mth space and a panoramic image of at least one shooting point in each of the remaining spaces in the N spaces where the floor plan diagram has not been generated; and determine that the space where the total number of adjacent spaces is greater than or equal to a set threshold is the M+1th space.

[0588] Optionally, the first acquisition module 61 is specifically used to display the outline of the Mth space in the two-dimensional point cloud image of the Mth space, and in response to the user's editing operation on the outline of the Mth space, adjust the outline line of the Mth space outline so that the outline line coincides with the wall line in the two-dimensional point cloud image; wherein, the two-dimensional point cloud image of the Mth space is obtained after planar mapping the point cloud data of at least one shooting point in the Mth space.

[0589] The second acquisition module 62 is specifically used to display the M+1th space outline in the two-dimensional point cloud image of the M+1th space, and in response to the user's editing operation on the M+1th space outline, adjust the contour line of the M+1th space outline so that the contour line coincides with the wall line in the two-dimensional point cloud image of the M+1th space; wherein, the two-dimensional point cloud image of the M+1th space is obtained after planar mapping the point cloud data of at least one shooting point in the M+1th space.

[0590] Optionally, the first acquisition module 61 is specifically used to obtain the panoramic pixel coordinates corresponding to the first target medium in the first target panoramic image and the mapped spatial contour coordinates based on the mapping relationship between the first target panoramic image and the Mth spatial contour, so as to determine the first mapping medium corresponding to the first target medium in the Mth spatial contour, so as to obtain the floor plan of the Mth space; wherein, the first mapping medium is adapted to the target identification and target display size of the first target medium, the target identification is used to distinguish target media belonging to different types, and the mapping relationship is a mapping between the first target panoramic image and the Mth spatial contour established based on the coordinate mapping between the point cloud data of the Mth space and the first target panoramic image.

[0591] Optionally, the second acquisition module 62 is specifically used to obtain the panoramic pixel coordinates corresponding to the second target medium in the second target panoramic image and the mapped spatial contour coordinates based on the mapping relationship between the second target panoramic image and the M+1th spatial contour, so as to determine the second mapping medium corresponding to the second target medium in the M+1th spatial contour, so as to obtain the floor plan of the M+1th space; wherein, the second mapping medium is adapted to the target identification and target display size of the second target medium, the target identification is used to distinguish target media belonging to different types, and the mapping relationship is a mapping between the second target panoramic image and the M+1th spatial contour established based on the coordinate mapping between the point cloud data of the M+1th space and the second target panoramic image.

[0592] Optionally, the processing module 63 is specifically configured to splice the floor plan diagram of the M+1th space with the floor plan diagram of the Mth space in the same spatial coordinate system according to the adjacent relationship between the M+1th space and the Mth space.

[0593] The device shown in Figure 32 can execute the steps in the aforementioned embodiments. For detailed execution process and technical effects, please refer to the description in the aforementioned embodiments and will not be repeated here.

[0594] In one possible design, the spatial structure diagram generation device shown in FIG6 and the floor plan diagram generation device shown in FIG11, FIG14, FIG19, FIG25, and / or FIG32 can each be implemented as an electronic device. As shown in FIG33, the electronic device may include: a memory 71, a processor 72, and a communication interface 73. The memory 71 stores executable code. When the executable code is executed by the processor 72, the processor 72 can at least implement the spatial structure diagram generation method and / or floor plan diagram generation method provided in the aforementioned embodiments.

[0595] In addition, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the spatial structure diagram generation method provided in the aforementioned embodiment.

[0596] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Those skilled in the art can understand and implement the present invention without inventive effort.

[0597] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by a combination of hardware and software. Based on this understanding, the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0598] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a spatial structure diagram, characterized in that: Applied to a control terminal, the method includes: Acquire point cloud data and panoramic images obtained by the information acquisition terminal at least one shooting point in the target space; Acquiring a spatial contour of the target space according to the point cloud data of the at least one shooting point; Acquire a target medium identified in a target panoramic image, where the target medium is an image of a physical medium in the target space in the target panoramic image, and the target panoramic image is a panoramic image for identifying the target medium in the panoramic image of the at least one shooting point; A mapping medium corresponding to the target medium is determined in the spatial outline to obtain a spatial structure diagram of the target space.

2. The method according to claim 1, characterized in that The acquiring the spatial contour of the target space according to the point cloud data of the at least one shooting point includes: fusing the point cloud data of the at least one shooting point to determine target point cloud data of the target space; Mapping the target point cloud data to a two-dimensional plane to obtain a point cloud image of the target space; A spatial contour of the target space is determined according to the point cloud image.

3. The method according to claim 2, characterized in that Determining a spatial contour of the target space according to the point cloud image includes: receiving a user's correction operation on the point cloud image; The spatial contour of the target space is determined based on the target point cloud image obtained after the correction operation.

4. The method according to claim 2, characterized in that The spatial contour is composed of a plurality of contour lines, the spatial contour includes a target contour line that does not correspond to a wall line in the point cloud image, and the method further includes: In response to an editing operation on the target contour line on the spatial contour of the target space, the shape and / or position of the target contour line is adjusted so that the target contour line coincides with a wall line in the point cloud image.

5. The method according to claim 1, wherein The determining, in the spatial profile, a mapping medium corresponding to the target medium includes: According to the mapping relationship between the target panoramic image and the spatial contour, the panoramic pixel coordinates corresponding to the target medium in the target panoramic image and the mapped spatial contour coordinates are obtained to determine the mapping medium corresponding to the target medium in the spatial contour; wherein the mapping medium is adapted to the target identification and target display size of the target medium, the target identification is used to distinguish target media belonging to different types, and the mapping relationship is a mapping between the target panoramic image and the spatial contour established based on the coordinate mapping between the point cloud data and the target panoramic image.

6. A spatial structure diagram generating device, applied to a control terminal, characterized in that: The device comprises: An acquisition module is used to acquire point cloud data and a panoramic image obtained by the information acquisition terminal at least one shooting point in the target space; A processing module is used to obtain the spatial outline of the target space based on the point cloud data of the at least one shooting point; obtain the target medium identified in the target panoramic image, wherein the target medium is the image of the physical medium in the target space in the target panoramic image, and the target panoramic image is a panoramic image for identifying the target medium in the panoramic image of the at least one shooting point; determine the mapping medium corresponding to the target medium in the spatial outline to obtain a spatial structure diagram of the target space.

7. An electronic device, characterized in that: include: A memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the spatial structure diagram generating method according to any one of claims 1 to 5.

8. A non-transitory machine-readable storage medium, characterized in that The non-transitory machine-readable storage medium stores executable code. When the executable code is executed by a processor of an electronic device, the processor executes the spatial structure diagram generating method according to any one of claims 1 to 5.

9. A method for generating a floor plan, characterized in that: Applied to a target control terminal, the method includes: Acquiring point cloud data and panoramic images corresponding to a plurality of subspaces in the target physical space obtained by the information collection terminal, respectively, to determine a plurality of spatial contours corresponding to the plurality of subspaces; wherein the plurality of subspaces correspond one-to-one to the plurality of spatial contours, and the point cloud data and panoramic image of any subspace are collected from at least one shooting point in the any subspace; displaying a plurality of space contours corresponding to the plurality of subspaces for editing; For a target subspace among the multiple subspaces, obtaining a target medium identified in a target panoramic image, where the target medium is an image of a physical medium in the target subspace in the target panoramic image, and the target panoramic image is a panoramic image captured by at least one shooting point in the target subspace and used to identify the target medium; Determining a mapping medium corresponding to the target medium in a spatial contour of the target subspace to generate a spatial structure diagram of the target subspace; In response to the completion of the operation of acquiring the multiple space structure diagrams corresponding to the multiple subspaces, a floor plan of the target physical space obtained by splicing the multiple space structure diagrams is acquired.

10. The method according to claim 9, characterized in that The method further comprises: sending the plurality of space contours corresponding to the plurality of subspaces to other control terminals, so as to display the plurality of space contours corresponding to the plurality of subspaces on the other control terminals for synchronous editing with the target control terminal; Acquire the spatial structure diagram of the target subspace acquired by the other control terminal.

11. The method according to claim 10, characterized in that The method further comprises: The terminal device identifiers corresponding to the multiple space outlines are displayed, where the terminal device identifiers are used to indicate the terminal device currently editing each space outline, and the terminal device includes the target control terminal and the other control terminals.

12. The method according to claim 9, characterized in that Determining a spatial contour of a target subspace among the plurality of subspaces includes: Acquire a first space contour according to point cloud data of at least one shooting point in the target subspace; Acquiring a second space contour according to a panoramic image of at least one shooting point of the target subspace; The spatial contour of the target subspace is determined according to the first spatial contour and the second spatial contour.

13. The method according to any one of claims 9 to 12, characterized in that Editing the spatial contour of the target subspace includes: In response to an editing operation on a target contour line on the spatial contour of the target subspace, the shape and / or position of the target contour line is adjusted so that the target contour line coincides with a wall line in a point cloud image, wherein the point cloud image is determined based on point cloud data of at least one shooting point of the target subspace, and the spatial contour of the target subspace is composed of multiple contour lines.

14. The method according to claim 9, characterized in that The acquiring of the floor plan of the target physical space obtained by splicing the multiple space structure diagrams includes: determining a spatial connection relationship between the plurality of subspaces according to the point cloud data and / or panoramic images respectively corresponding to the plurality of subspaces; According to the spatial connection relationship, the multiple spatial structure diagrams are spliced ​​to obtain a floor plan of the target physical space.

15. The method according to claim 9, characterized in that The determining the mapping medium corresponding to the target medium in the spatial contour of the target subspace includes: According to the mapping relationship between the target panoramic image and the spatial contour of the target subspace, the panoramic pixel coordinates corresponding to the target medium in the target panoramic image and the mapped spatial contour coordinates are obtained to determine the mapping medium corresponding to the target medium in the spatial contour of the target subspace; wherein the mapping medium is adapted to the target identification and target display size of the target medium, the target identification is used to distinguish target media belonging to different types, and the mapping relationship is a mapping between the target panoramic image and the spatial contour established based on the coordinate mapping between the point cloud data of the target subspace and the target panoramic image.

16. A floor plan generating device, characterized in that: Applied to a target control terminal, the device comprises: an acquisition module, configured to acquire point cloud data and panoramic images corresponding to a plurality of subspaces in the target physical space obtained by the information acquisition terminal, so as to determine a plurality of spatial contours corresponding to the plurality of subspaces; wherein the plurality of subspaces correspond one-to-one to the plurality of spatial contours, and the point cloud data and panoramic image of any subspace are acquired from at least one shooting point in the any subspace; A display module, configured to display a plurality of space contours corresponding to the plurality of subspaces for editing; A processing module is used to obtain, for a target subspace among multiple subspaces, a target medium identified in a target panoramic image, where the target medium is an image of a physical medium in the target subspace in the target panoramic image, and the target panoramic image is a panoramic image captured by at least one shooting point of the target subspace, for identifying the target medium; determine a mapping medium corresponding to the target medium in the spatial contour of the target subspace to generate a spatial structure diagram of the target subspace; and in response to completion of an operation to obtain multiple spatial structure diagrams corresponding to the multiple subspaces, obtain a floor plan of the target physical space obtained by splicing the multiple spatial structure diagrams.

17. A method for generating a floor plan, characterized in that: Applied to a control terminal, the method includes: Acquiring point cloud data and panoramic images corresponding to a plurality of subspaces in the target physical space obtained by the information collection terminal, wherein the point cloud data and panoramic image of any subspace are collected from at least one shooting point in the subspace; In the process of sequentially performing spatial structure graph acquisition processing on the plurality of subspaces, for a target subspace currently to be edited, acquiring a target space contour of the target subspace based on point cloud data and / or panoramic images collected at at least one shooting point in the target subspace; Acquire a target medium identified in a target panoramic image, where the target medium is an image of a physical medium in the target subspace in the target panoramic image, and the target panoramic image is a panoramic image used to identify the target medium in a panoramic image captured at at least one shooting point in the target subspace; determining a mapping medium for representing the target medium on a target space contour of the target subspace to obtain a spatial structure diagram of the target subspace; In response to the completion of the acquisition operation of the spatial structure diagram of the target subspace, if there is no subspace among the multiple subspaces for which the spatial structure diagram has not been acquired, a floor plan of the target physical space obtained by splicing the spatial structure diagrams of the multiple spaces is acquired.

18. The method according to claim 17, characterized in that The method further comprises: In response to the completion of the acquisition operation of the spatial structure diagram of the target subspace, if there is at least one subspace among the multiple subspaces whose spatial structure diagram has not been acquired, a subspace to be edited is determined from the at least one subspace to acquire the spatial structure diagram of the subspace to be edited.

19. The method according to claim 18, characterized in that The determining a subspace to be edited from the at least one subspace includes: A subspace to be edited is determined from the at least one subspace based on the acquisition time points of the point cloud data and the panoramic image corresponding to the at least one subspace, wherein a difference between the acquisition time point corresponding to the subspace to be edited and the current moment is greater than a difference between the acquisition time points corresponding to other subspaces in the at least one subspace and the current moment.

20. The method according to claim 18, wherein The determining a subspace to be edited from the at least one subspace includes: determining a connection relationship between the plurality of subspaces according to the point cloud data respectively corresponding to the plurality of subspaces; According to the connection relationship, a subspace to be edited is determined from the at least one subspace, wherein the subspace to be edited is connected to the target subspace.

21. The method according to claim 17, wherein The step of obtaining a target space contour of the target subspace based on point cloud data collected at at least one shooting point in the target subspace includes: Mapping point cloud data collected at at least one shooting point in the target subspace to a two-dimensional plane to determine a two-dimensional point cloud image of the target subspace; Displaying a spatial outline recognized based on the two-dimensional point cloud image, wherein the spatial outline is composed of a plurality of contour lines; In response to a user's correction operation on the space outline, adjusting the shape and / or position of a target contour line in the space outline so that the target contour line coincides with a wall line in the two-dimensional point cloud image; A spatial contour formed by contour lines coinciding with the wall line is determined as a target spatial contour of the target subspace.

22. The method according to claim 17, wherein The determining of a mapping medium for representing the target medium on a target space contour of the target subspace includes: According to the mapping relationship between the target panoramic image and the spatial contour of the target subspace, the panoramic pixel coordinates corresponding to the target medium in the target panoramic image and the mapped spatial contour coordinates are obtained to determine the mapping medium corresponding to the target medium in the spatial contour of the target subspace; wherein the mapping medium is adapted to the target identification and target display size of the target medium, the target identification is used to distinguish target media belonging to different types, and the mapping relationship is a mapping between the target panoramic image and the spatial contour established based on the coordinate mapping between the point cloud data of the target subspace and the target panoramic image.

23. A floor plan generating device, characterized in that: Applied to a control terminal, the device comprises: an acquisition module, configured to acquire point cloud data and panoramic images corresponding to a plurality of subspaces in the target physical space obtained by the information acquisition terminal, wherein the point cloud data and panoramic image of any subspace are acquired from at least one shooting point in the subspace; A processing module is used to obtain a target space outline of the target subspace according to the point cloud data and / or panoramic image collected at at least one shooting point of the target subspace in the process of sequentially performing spatial structure diagram acquisition processing on the multiple subspaces; obtain a target medium identified in the target panoramic image, wherein the target medium is an image of the physical medium in the target subspace in the target panoramic image, and the target panoramic image is a panoramic image used to identify the target medium in the panoramic image collected at at least one shooting point of the target subspace; determine a mapping medium for representing the target medium on the target space outline of the target subspace to obtain a spatial structure diagram of the target subspace; in response to the completion operation of the acquisition of the spatial structure diagram of the target subspace, if there is no subspace among the multiple subspaces for which the spatial structure diagram has not been obtained, obtain a floor plan of the target physical space obtained by splicing the spatial structure diagrams of the multiple spaces.

24. A method for generating a floor plan, the method being used to generate a floor plan of a target physical space, the target physical space comprising at least N spaces, and being applied to a control terminal, characterized in that: The method comprises: Step 1: Acquire point cloud data and panoramic images collected by an information collection terminal in each of the N spaces, wherein the point cloud data and panoramic images are collected from at least one shooting point in each of the spaces; Step 2: Obtaining an M-th space outline of an M-th space among the N spaces and displaying it for editing, wherein the M-th space outline is obtained based on point cloud data and / or panoramic images collected from at least one shooting point of the M-th space; Step 3: Acquire a target medium identified in the target panoramic image of the Mth space, so as to obtain a mapping medium of the target medium in the outline of the Mth space based on the target medium, so as to edit the outline of the Mth space based on the mapping medium to obtain a floor plan of the Mth space; the target panoramic image is a panoramic image for identifying the target medium in a panoramic image captured by at least one shooting point in the Mth space, and the target medium is an image of a physical medium in the Mth space in the target panoramic image; Step 4: Determine whether the Mth space is the last space among the N spaces for which a floor plan diagram is generated; If not, execute step 5, assign M to M+1 and return to step 2; If so, execute step 6 to obtain a floor plan of the target physical space consisting of N floor plan structure diagrams for display, and the process ends; wherein M and N are natural numbers, and 1≤M≤N.

25. The method according to claim 24, characterized in that The obtaining of the Mth space outline of the Mth space in the N spaces for displaying for editing includes: Displaying an Mth space outline of the Mth space in a two-dimensional point cloud image of the Mth space among the N spaces; wherein the two-dimensional point cloud image is obtained by plane mapping point cloud data of at least one shooting point in the Mth space; In response to the editing operation on the Mth space contour, a contour line of the Mth space contour is adjusted so that the contour line coincides with a wall line in the two-dimensional point cloud image.

26. The method according to claim 25, characterized in that The two-dimensional point cloud image is obtained by the following method: Fusing point cloud data of at least one shooting point in an M-th space among the N spaces to determine target point cloud data in the M-th space; Mapping the target point cloud data to a two-dimensional plane to obtain an initial two-dimensional point cloud image of the M-th space; The two-dimensional point cloud image of the M-th space is determined according to the user's correction operation on the initial two-dimensional point cloud image.

27. The method according to claim 24, characterized in that The acquiring, according to the target medium, a mapping medium of the target medium in the Mth spatial profile includes: According to the mapping relationship between the target panoramic image of the Mth space and the Mth space contour, the panoramic pixel coordinates corresponding to the target medium in the target panoramic image and the mapped space contour coordinates are obtained to determine the mapping medium corresponding to the target medium in the Mth space contour; wherein, the mapping medium is adapted to the target identification and target display size of the target medium, the target identification is used to distinguish target media belonging to different types, and the mapping relationship is a mapping between the target panoramic image and the Mth space contour established based on the coordinate mapping between the point cloud data of the Mth space and the target panoramic image.

28. The method according to claim 24, characterized in that The method comprises: The N spaces are sorted according to the acquisition time points of the point cloud data and / or panoramic images respectively corresponding to the N spaces.

29. A floor plan generating device, the device being used to generate a floor plan of a target physical space, the target physical space comprising at least N spaces, applied to a control terminal, characterized in that: The device comprises: an acquisition module, configured to acquire point cloud data and panoramic images collected by the information collection terminal in each of the N spaces, wherein the point cloud data and panoramic images are collected from at least one shooting point in each of the spaces; A first processing module is configured to obtain an M-th spatial outline of an M-th space among the N spaces for displaying for editing, wherein the M-th spatial outline is obtained based on point cloud data and / or a panoramic image acquired from at least one shooting point of the M-th space; obtain a target medium identified in a target panoramic image of the M-th space, so as to obtain a mapping medium of the target medium in the M-th spatial outline based on the target medium, so as to edit the M-th spatial outline based on the mapping medium, so as to obtain a floor plan diagram of the M-th space; the target panoramic image is a panoramic image for identifying the target medium in a panoramic image acquired from at least one shooting point of the M-th space, wherein the target medium is an image of a physical medium in the M-th space in the target panoramic image; The second processing module is used to determine whether the Mth space is the last space among the N spaces for generating a floor plan diagram; if not, M is assigned a value of M+1 and the process returns to execute the first processing module; if so, the floor plan of the target physical space composed of the N floor plan diagrams is obtained for display, and the process ends; wherein M and N are natural numbers, and 1≤M≤N.

30. A method for generating a floor plan, the method being used to generate a floor plan of a target physical space, wherein the target physical space includes a plurality of subspaces, and being applied to a control terminal, characterized in that: The method comprises: Acquiring point cloud data and panoramic images corresponding to the multiple subspaces respectively obtained by the information collection terminal, wherein the point cloud data and panoramic image of any subspace are collected from at least one shooting point in the any subspace; During the process of sequentially stitching the floor plan diagrams of the plurality of subspaces, for a first subspace to be stitched, obtaining a target spatial outline of the first subspace based on point cloud data and / or panoramic images collected at at least one shooting point in the first subspace; Acquire a target medium identified in a target panoramic image, where the target medium is an image of a physical medium in the first subspace in the target panoramic image, and the target panoramic image is a panoramic image captured at at least one shooting point in the first subspace and used to identify the target medium; Determining a mapping medium for representing the target medium on a target space outline of the first subspace to obtain a floor plan of the first subspace; splicing the floor plan diagram of the first subspace with the floor plan diagram of the second subspace, where the second subspace is the subspace to which the floor plan diagram of the first subspace has been spliced; If there is no subspace among the multiple subspaces that has not been subjected to the floor plan structure splicing process, the splicing process result is determined as the floor plan of the target physical space.

31. The method according to claim 30, wherein The method further comprises: If there is a subspace among the multiple subspaces that has not undergone the splicing process for the floor plan structure diagram, updating the floor plan structure diagram of the second subspace to the splicing process result; A first subspace to be spliced ​​is re-determined from the subspaces that have not been spliced ​​with the floor plan structure diagram, so as to splice the floor plan structure diagram of the re-determined first subspace with the updated floor plan structure diagram of the second subspace.

32. The method according to claim 31, characterized in that The step of re-determining a first subspace to be spliced ​​from the subspaces that have not been spliced ​​with the floor plan diagram includes: determining an adjacent relationship between the plurality of subspaces according to the panoramic image of at least one shooting point corresponding to each of the plurality of subspaces; According to the adjacent relationship, a first subspace to be spliced ​​is re-determined from the subspaces that have not been spliced ​​with the floor plan diagram, wherein the re-determined first subspace is adjacent to the second subspace.

33. The method according to claim 31, characterized in that The step of re-determining a first subspace to be spliced ​​from the subspaces that have not been spliced ​​with the floor plan diagram includes: Based on the acquisition time point of the point cloud data and / or panoramic image corresponding to each subspace in the subspace that has not undergone the apartment structure diagram stitching processing, a first subspace to be stitched is re-determined from the subspace that has not undergone the apartment structure diagram stitching processing, wherein the difference between the acquisition time point corresponding to the re-determined first subspace and the current moment is greater than the difference between the acquisition time point corresponding to other subspaces in the subspace that has not undergone the apartment structure diagram stitching processing and the current moment.

34. The method according to claim 30, wherein The acquiring of the target space contour of the first subspace according to the point cloud data and / or the panoramic image collected at at least one shooting point in the first subspace includes: determining a first spatial contour of the first subspace based on point cloud data collected at at least one shooting point in the first subspace; determining a second spatial contour of the first subspace based on a panoramic image captured at at least one shooting point in the first subspace; A target space contour of the first subspace is determined according to the first space contour and / or the second space contour.

35. The method according to claim 34, wherein Determining a first spatial contour of the first subspace based on point cloud data collected at at least one shooting point in the first subspace includes: determining a two-dimensional point cloud image of the first subspace based on point cloud data collected at at least one shooting point in the first subspace; A first spatial contour of the first subspace is determined according to the two-dimensional point cloud image.

36. The method according to claim 30, wherein The determining of a mapping medium for representing the target medium on the target space contour of the first subspace includes: According to the mapping relationship between the target panoramic image and the target spatial contour of the first subspace, the panoramic pixel coordinates corresponding to the target medium in the target panoramic image and the mapped spatial contour coordinates are obtained to determine the mapping medium corresponding to the target medium in the target spatial contour of the first subspace; wherein the mapping medium is adapted to the target identification and target display size of the target medium, the target identification is used to distinguish target media belonging to different types, and the mapping relationship is a mapping between the target panoramic image and the spatial contour established based on the coordinate mapping between the point cloud data of the first subspace and the target panoramic image.

37. A floor plan generating device, the device being used to generate a floor plan of a target physical space, wherein the target physical space includes a plurality of subspaces, and being applied to a control terminal, characterized in that: The device comprises: an acquisition module, configured to acquire point cloud data and panoramic images corresponding to the plurality of subspaces respectively obtained by the information acquisition terminal, wherein the point cloud data and panoramic image of any subspace are acquired from at least one shooting point in the any subspace; A stitching module is used to, in the process of stitching the floor plan diagrams of the multiple subspaces in sequence, obtain, for a first subspace to be stitched, a target space outline of the first subspace based on point cloud data and / or panoramic images collected at at least one shooting point of the first subspace; obtain a target medium identified in the target panoramic image, wherein the target medium is an image of a physical medium in the first subspace in the target panoramic image, and the target panoramic image is a panoramic image for identifying the target medium in the panoramic image collected at at least one shooting point of the first subspace; determine a mapping medium for representing the target medium on the target space outline of the first subspace to obtain a floor plan diagram of the first subspace; stitch the floor plan diagram of the first subspace with that of the second subspace, wherein the second subspace is a subspace to which the floor plan diagram has been stitched before by the first subspace; The processing module is configured to determine the splicing result as the floor plan of the target physical space if there is no subspace in the plurality of subspaces that has not been spliced ​​with the floor plan structure diagram.

38. A method for generating a floor plan, the method being used to generate a floor plan of a target physical space, wherein the target physical space includes at least N spaces, and being applied to a control terminal, characterized in that: The method comprises: Step 1: Acquire point cloud data and panoramic images collected by an information collection terminal in each of the N spaces, wherein the point cloud data and panoramic images are collected from at least one shooting point in each of the spaces; Step 2: Obtain an Mth spatial outline for display for editing; wherein the Mth spatial outline is a spatial outline of the Mth space among the N spaces, and the Mth spatial outline is obtained based on point cloud data and / or panoramic images collected from at least one shooting point of the Mth space; Step 3: Acquire a first target medium identified in the first target panoramic image, so as to obtain a first mapping medium of the first target medium in the outline of the Mth space based on the first target medium, so as to edit the outline of the Mth space based on the first mapping medium to obtain a floor plan of the Mth space; the first target panoramic image is a panoramic image for identifying the first target medium in the panoramic image captured by the at least one shooting point of the Mth space, and the first target medium is an image of a physical medium in the Mth space in the first target panoramic image; Step 4: Obtain an M+1th spatial outline for display for editing; wherein the M+1th spatial outline is a spatial outline of the M+1th space among the N spaces, the M+1th space is an adjacent space to the Mth space, and the M+1th spatial outline is obtained based on point cloud data and / or panoramic images collected from at least one shooting point of the M+1th space; Step 5: Acquire a second target medium identified in the second target panoramic image, so as to obtain a second mapping medium of the second target medium in the outline of the M+1th space based on the second target medium, so as to edit the outline of the M+1th space based on the second mapping medium to obtain a floor plan of the M+1th space; the second target panoramic image is a panoramic image for identifying the second target medium in the panoramic image captured by the at least one shooting point of the M+1th space, and the second target medium is an image of a physical medium in the M+1th space in the second target panoramic image; Step 6: splice the floor plan diagram of the M+1th space with the floor plan diagram of the Mth space, and determine whether the M+1th space is the last space among the N spaces for which a floor plan diagram is generated; If not, execute step 7, merge the Mth space and the M+1th space as the Mth space, and return to step 4; If so, execute step 8, use the splicing result as the floor plan of the target physical space for display, and the process ends.

39. The method according to claim 38, characterized in that The method further comprises: The M+1th space is determined from the spaces for which floor plan diagrams have not been generated based on a panoramic image of at least one shooting point of the Mth space and a panoramic image of at least one shooting point of each of the remaining N spaces for which floor plan diagrams have not been generated.

40. The method according to claim 39, wherein The determining the M+1th space from the spaces for which floor plan diagrams are not generated based on the panoramic image of at least one shooting point of the Mth space and the panoramic image of at least one shooting point of each of the remaining spaces in the N spaces for which floor plan diagrams are not generated, includes: Determining the total number of adjacent spaces corresponding to each space in the space for which a floor plan diagram has not been generated, based on the panoramic image of at least one shooting point in the Mth space and the panoramic image of at least one shooting point in each of the remaining N spaces for which floor plan diagrams have not been generated; A space whose total number of adjacent spaces is greater than or equal to a set threshold is determined as the M+1th space.

41. The method according to claim 38, wherein The obtaining of the Mth spatial contour for displaying for editing includes: displaying an outline of the Mth space in the two-dimensional point cloud image of the Mth space, and adjusting a contour line of the Mth space contour in response to a user editing operation on the Mth space contour so that the contour line coincides with a wall line in the two-dimensional point cloud image; wherein the two-dimensional point cloud image of the Mth space is obtained by plane mapping point cloud data of at least one shooting point in the Mth space; The obtaining of the M+1th spatial contour for displaying for editing includes: The outline of the M+1th space is displayed in the two-dimensional point cloud image of the M+1th space, and in response to a user's editing operation on the outline of the M+1th space, a contour line of the outline of the M+1th space is adjusted so that the contour line coincides with a wall line in the two-dimensional point cloud image of the M+1th space; wherein the two-dimensional point cloud image of the M+1th space is obtained by plane mapping the point cloud data of at least one shooting point in the M+1th space.

42. The method according to claim 38, wherein The acquiring, according to the first target medium, a first mapping medium of the first target medium in the M-th space outline, so as to edit the M-th space outline according to the first mapping medium to obtain the floor plan of the M-th space, includes: Obtaining, based on a mapping relationship between the first target panoramic image and the Mth spatial outline, panoramic pixel coordinates corresponding to the first target medium in the first target panoramic image and the mapped spatial outline coordinates, so as to determine, in the Mth spatial outline, a first mapping medium corresponding to the first target medium, thereby obtaining a floor plan diagram of the Mth space; In which, the first mapping medium is adapted to the target identification and target display size of the first target medium, the target identification is used to distinguish target media belonging to different types, and the mapping relationship is a mapping between the first target panoramic image and the Mth space outline established based on the coordinate mapping between the point cloud data of the Mth space and the first target panoramic image.

43. The method according to claim 38, wherein The acquiring, according to the second target medium, a second mapping medium of the second target medium in the M+1th space outline, so as to edit the M+1th space outline according to the second mapping medium to obtain the floor plan of the M+1th space, includes: Obtaining, based on a mapping relationship between the second target panoramic image and the M+1th spatial outline, panoramic pixel coordinates corresponding to the second target medium in the second target panoramic image and the mapped spatial outline coordinates, so as to determine a second mapping medium corresponding to the second target medium in the M+1th spatial outline, thereby obtaining a floor plan of the M+1th space; In which, the second mapping medium is adapted to the target identification and target display size of the second target medium, the target identification is used to distinguish target media belonging to different types, and the mapping relationship is a mapping between the second target panoramic image and the M+1 space outline established based on the coordinate mapping between the point cloud data of the M+1 space and the second target panoramic image.

44. The method according to claim 38, wherein The step of splicing the floor plan diagram of the M+1th space with the floor plan diagram of the Mth space includes: According to the adjacent relationship between the M+1th space and the Mth space, the floor plan structure diagram of the M+1th space is spliced ​​with the floor plan structure diagram of the Mth space in the same spatial coordinate system.

45. A floor plan generating device, the device being used to generate a floor plan of a target physical space, wherein the target physical space includes at least N spaces, and being applied to a control terminal, characterized in that: The device comprises: A first acquisition module is configured to acquire point cloud data and panoramic images collected by the information collection terminal in each of the N spaces, wherein the point cloud data and panoramic images are collected from at least one shooting point in each of the spaces; acquire an Mth space outline for display for editing; wherein the Mth space outline is the space outline of the Mth space in the N spaces, and the Mth space outline is acquired based on the point cloud data and / or panoramic image collected from at least one shooting point in the Mth space; acquire a first target medium identified in a first target panoramic image, so as to acquire a first mapping medium of the first target medium in the Mth space outline based on the first target medium, so as to edit the Mth space outline based on the first mapping medium to obtain a floor plan of the Mth space; the first target panoramic image is a panoramic image for identifying the first target medium in the panoramic image collected from at least one shooting point in the Mth space, and the first target medium is an image of a physical medium in the Mth space in the first target panoramic image; A second acquisition module is configured to acquire an M+1th spatial outline for display for editing; wherein the M+1th spatial outline is the spatial outline of the M+1th space among the N spaces, the M+1th space is an adjacent space to the Mth space, and the M+1th spatial outline is acquired based on point cloud data and / or panoramic images collected from at least one shooting point of the M+1th space; a second target medium identified in a second target panoramic image is acquired, so that a second mapping medium of the second target medium in the M+1th spatial outline is acquired based on the second target medium, so as to edit the M+1th spatial outline based on the second mapping medium to acquire a floor plan of the M+1th space; the second target panoramic image is a panoramic image for identifying the second target medium in the panoramic image collected from at least one shooting point of the M+1th space, and the second target medium is an image of a physical medium in the M+1th space in the second target panoramic image; A processing module is configured to splice the floor plan diagram of the M+1th space with the floor plan diagram of the Mth space, and determine whether the M+1th space is the last space among the N spaces for which a floor plan diagram is generated; if not, merge the Mth space and the M+1th space as the Mth space, and return to execute the second acquisition module; if so, use the splicing result as the floor plan diagram of the target physical space for display, and the process ends.

46. ​​An electronic device, characterized in that include: A memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the floor plan generation method as described in any one of claims 9 to 15, 17 to 22, 24 to 28, 30 to 36 and / or 38 to 44.

47. A non-transitory machine-readable storage medium, characterized in that The non-transitory machine-readable storage medium stores executable code. When the executable code is executed by a processor of an electronic device, the processor executes the floor plan generation method as described in any one of claims 9 to 15, 17 to 22, 24 to 28, 30 to 36 and / or 38 to 44.

Images