An indoor mapping method and apparatus

Abstract

This application discloses an indoor mapping method and apparatus. The method includes: generating vectorized floor plan data for a target house using a deep neural network based on a standard floor plan of the target house; obtaining a first radar map by scanning a first area of ​​traversable space using radar, the first area containing at least one room of the target house; and performing image matching and fusion processing based on the first radar map and the floor plan data to obtain a floor plan display image of the target house. Using this application can improve the accuracy and completeness of the generated floor plan display image.

Description

TECHNICAL FIELD

[0001] The present application relates to computer application technology, in particular to an indoor mapping method and device. BACKGROUND

[0002] With the increasing application of sweeping robots in the family, the real-time radar map of the house structure can be obtained by using the radar carried by the sweeping robot itself. At present, the mapping methods of sweeping robots on the market mainly include visual mapping and radar mapping and other methods, among which, radar mapping is the main method.

[0003] The inventor found that the house type display map obtained by using the existing radar mapping method has the problems of low accuracy and incompleteness, and the reasons are as follows through research and analysis:

[0004] Due to the complexity of the placement of various furniture and household appliances in the actual home environment, the moving range of the sweeping robot is limited by the blockage of various objects on the ground, and it cannot traverse all areas in the room, so that the radar map established has many noises and is incomplete. SUMMARY

[0005] Therefore, the main purpose of the present application is to provide an indoor mapping method and device, which can improve the accuracy and completeness of the generation of the house type display map.

[0006] In order to achieve the above purpose, the technical scheme provided by the embodiments of the present application is:

[0007] An indoor mapping method comprises:

[0008] Based on the standard house type map of the target house, a vectorized house structure data of the target house is generated by using a deep neural network;

[0009] A first radar map is obtained by scanning the first area of the available space by using the radar, and the first area contains at least one room of the target house;

[0010] Based on the first radar map and the house structure data, image matching and fusion processing is performed to obtain a house type display map of the target house.

[0011] The embodiments of the present application also provide an indoor mapping device, which comprises:

[0012] A standard house type data generation unit is configured to generate a vectorized house structure data of the target house by using a deep neural network based on the standard house type map of the target house;

[0013] a radar map data generation unit, configured to acquire a first radar map by scanning a first region containing at least one room of the target house using a radar;

[0014] a matching fusion unit, configured to perform image matching fusion processing based on the first radar map and the vectorized house structure data to obtain a house type display map of the target house.

[0015] The embodiment of the present application further provides an indoor mapping device, comprising a processor and a memory.

[0016] The memory stores an application executable by the processor, so that the processor executes the indoor mapping method.

[0017] The embodiment of the present application further provides a computer readable storage medium, which stores computer readable instructions for executing the indoor mapping method.

[0018] In summary, the indoor mapping scheme provided by the embodiment of the present application introduces a standard house type map of a target house, matches and fuses the vectorized house structure data corresponding to the standard house type map with radar map data to obtain a house type display map of the target house. In this way, the radar map data can be corrected by using the complete data of the standard house type map, the influence of a complex indoor environment on the radar map is effectively overcome, and thus a house type display map that matches and is complete with an actual indoor environment is obtained. Therefore, the technical scheme of the present application can effectively improve the accuracy and completeness of the house type display map generation. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The figure is a method flowchart of the embodiment of the present application.

[0020] Figure 2 The figure is a schematic diagram of generating vectorized house structure data using a deep neural network in the embodiment of the present application.

[0021] Figure 3 The figure is a schematic diagram of constructing all standard house type sub-maps of a target house based on the house structure data output by the deep neural network in the embodiment of the present application.

[0022] Figures 4 to 7 The figure is an implementation schematic diagram in scenario one in the embodiment of the present application.

[0023] Figure 8 The figure is a device structure schematic diagram of the embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the objects, technical solutions and advantages of the present application clearer, the following will further describe the present application in detail with reference to the accompanying drawings and specific embodiments.

[0025] Figure 1 The method flowchart of the embodiment of the present application is shown in Figure 1 The indoor mapping method realized by the embodiment mainly includes the following steps:

[0026] Step 101, based on the standard house plan of the target house, a vectorized house plan structure data is generated for the target house by using a deep neural network.

[0027] In this step, the vectorized house plan structure data corresponding to the standard house plan of the target house is generated, so that in the subsequent steps, the radar map data is matched, fused and corrected based on the vectorized house plan structure data of the standard house plan.

[0028] Figure 2 The generation schematic diagram of the vectorized house plan structure data in this step is shown in Figure 2 The standard house plan of the target house is input into the pre-trained deep neural network to generate the house plan structure, the positions of the walls, doors and windows are identified, and the vectorized structure information diagram is constructed, including the room category diagram, the wall, door and window position diagram of the target house. In this way, by using the standard house plan as the input and combining the deep learning algorithm, the vectorized structure data of each room (i.e. the wall structure information including the positions of the doors and windows) can be obtained.

[0029] In actual application, the deep neural network can be obtained by using existing methods, which will not be described here.

[0030] Step 102, by scanning the first area of the travelable space by using the radar, a first radar map is obtained, and the first area contains at least one room of the target house.

[0031] In this step, the radar map of part or all of the rooms of the target house is obtained. In specific implementation, the laser radar map of the target house can be generated by the sweeping robot, and the laser radar map can be realized by using existing methods such as simultaneous localization and mapping (SLAM) algorithm, but is not limited thereto.

[0032] Considering that in actual indoor environment, noise data such as corner burrs may occur due to the shielding of various furniture or the laser radar colliding with glass transparent materials or high-reflective objects, thereby affecting the accuracy of the house plan, in order to improve the accuracy of the radar map, the original radar map can be denoised. Accordingly, in one embodiment, the first radar map can be obtained by scanning the first area of the travelable space by using the radar:

[0033] The first region of the drivable space is scanned by using a radar to obtain an original radar map; and a noise region in the original radar map is filtered to obtain the first radar map.

[0034] In an implementation, the noise region in the original radar map can be filtered by using a region growing method, but is not limited thereto, and other noise region filtering methods can also be used.

[0035] In actual application, there is no execution sequence requirement between step 101 and step 102, that is, step 101 and step 102 are not limited to the above sequence.

[0036] Step 103: performing image matching fusion processing based on the first radar map and the house type structure data to obtain a house type display map of the target house.

[0037] In this step, the radar map obtained in step 102 is subjected to image matching fusion processing by using the vectorized house type structure data of a standard house type, so as to obtain an accurate and complete house type display map.

[0038] In an implementation, the image matching fusion processing can be performed by using the following method:

[0039] Step 1031: constructing all standard house type subgraphs of the target house based on the house type structure data; the standard house type subgraph contains at least one room of the target house.

[0040] In this step, all standard house type subgraphs of the target house are constructed based on the generated vectorized house type structure data, so as to provide the radar map data with standard house type data of a matching region in a subsequent step, and to fuse and correct the radar map data.

[0041] Figure 3 For the house type structure data output based on the deep neural network in step 101, all standard house type subgraph diagrams of the target house are constructed. As shown in FIG. 6, for constructing all standard house type subgraphs of the target house, specifically, rooms of the target house are combined in an arbitrary manner to obtain a room subset in a standard house type diagram, each room subset is composed of at least one room, and a corresponding standard house type subgraph is obtained based on the vectorized structure data of each room subset. Figure 3

[0042] ​It should be noted that, in the prior art, the indoor map is obtained by traversing all areas in the indoor space, and then the radar data is used to generate the indoor map. Thus, the mapping efficiency of the sweeping robot is low, especially when the target area is large. Therefore, in order to improve the mapping efficiency, only part of the room can be scanned by the radar, that is, the radar map can only include part of the room, for example, only the living room and the kitchen area can be scanned by the sweeping robot. Accordingly, in step 1031, all standard house type sub-maps that can be constructed are constructed to obtain the layout of any subset of rooms of the standard house type, so as to provide the standard house type data of the matching area for the radar map data in the subsequent step, and to fuse and correct the radar map data.

[0043] In an embodiment, considering that the scanned area by the radar is usually connected, in order to improve the processing speed and save the computational cost, only the standard house type sub-map with connectivity can be constructed, that is, the rooms in the standard house type sub-map need to have connectivity.

[0044] In step 1032, the first standard house type sub-map that is most matched with the first radar map is selected from the standard house type sub-maps, and the corresponding matching adjustment angle of the first radar map is determined. The matching adjustment angle is the angle by which the first radar map needs to be rotated and / or flipped to be consistent with the direction of the first standard house type sub-map.

[0045] In this step, the angle by which the first radar map needs to be rotated and / or flipped to be consistent with the direction of the first standard house type sub-map, that is, the matching adjustment angle, is determined.

[0046] In an embodiment, step 1032 can be implemented by steps 10321-10324 as follows:

[0047] In step 10321, the first radar map is segmented by room, and the largest area room obtained by the segmentation is taken as the main area.

[0048] In step 10322, the first similarity between the main area and each standard house type sub-map is determined by taking the matching adjustment angle as the similarity parameter, and the first similarity greater than the preset coarse similarity threshold is selected. The matching adjustment angle corresponding to the selected first similarity is taken as the candidate matching adjustment angle corresponding to the corresponding standard house type sub-map.

[0049] The step is used for coarse matching based on the main area to obtain the first several standard house type subgraphs with high similarity, and the matching adjustment angle corresponding to the standard house type subgraphs is taken as a candidate matching adjustment angle, so that in the subsequent step 10323, fine matching is performed based on the candidate matching adjustment angle to obtain the most matched standard house type subgraph.

[0050] In actual application, the coarse similarity threshold value can be set to a proper value by a person skilled in the art according to the matching needs in actual application, which will not be described here.

[0051] In step 10323, the first radar map is adjusted based on each candidate matching adjustment angle respectively, and a second similarity between the adjusted radar map and the corresponding standard house type subgraph is calculated, the second similarity being obtained by weighted calculation based on preset fine similarity parameters, the fine similarity parameters including intersection over union, coverage rate and / or aspect ratio.

[0052] In this step, for each candidate matching adjustment angle, the first radar map is adjusted by using the candidate matching adjustment angle, and then the similarity between the adjusted radar map and the standard house type subgraph corresponding to the candidate matching adjustment angle is calculated according to each fine similarity parameter respectively, and finally the comprehensive similarity (i.e. the second similarity) between the adjusted radar map and the corresponding standard house type subgraph is obtained by weighted calculation based on the similarities corresponding to all fine similarity parameters, so that in the subsequent step 10324, the standard house type subgraph most matched with the first radar map can be screened out based on the comprehensive similarity.

[0053] In step 10324, the standard house type subgraph corresponding to the maximum value of the second similarity is taken as the standard house type subgraph matched with the first radar map, and the corresponding candidate matching adjustment angle is taken as the corresponding matching adjustment angle of the first radar map.

[0054] In step 1033, for each first room in the first radar map, a corresponding room in the first standard house type subgraph is determined, and the center position and the length and width of the first room are iteratively fitted to obtain the size matching parameter of the first room, the size matching parameter including the center position, the length, the width and the room identification information of the room.

[0055] In this step, for each room in the first radar map, the size matching parameter of the room is generated based on the corresponding room in the matched first standard house type subgraph, so that the size of the room in the radar map is optimized and improved by using the corresponding room in the first standard house type subgraph.

[0056] In an implementation, the following method can be used to determine the corresponding room (i.e., the matched room) of the first room in the first standard house type subgraph for each first room in the first radar map:

[0057] The similarity between the first room and each room in the first standard house type subgraph is calculated using the intersection-over-union ratio as the similarity parameter, and the room corresponding to the maximum similarity is taken as the corresponding room of the first room in the first standard house type subgraph.

[0058] Step 1034, based on the matching adjustment angle and the size matching parameter of the first room, the first radar map is adjusted accordingly to obtain the house type display graph of the target house.

[0059] In this step, the radar map is adjusted in terms of direction and size based on the matching adjustment angle and the size matching parameter of each room obtained in steps 1032 and 1033, and finally the house type display graph with accurate room semantic information is obtained.

[0060] Since the size matching parameter includes not only the position and size data of the room, but also the semantic information (i.e., room identification information) of the room, the house type display graph obtained in this step has the semantic information of each room, thereby facilitating the use of various services based on spatial structure understanding.

[0061] In actual application scenarios, the radar map may only contain map data of part of the rooms. In this case, for other rooms in the target house that are not in the radar map, the size of the standard house type of the other rooms in the target house that are not in the radar map can be adjusted using the ratio between the matching size of each room in the radar map and the corresponding standard house type size, to obtain the corresponding size matching parameter in the house type display graph. Accordingly, the above image matching fusion processing method can further include the following steps:

[0062] Step x1, for each first room, determine the size ratio between the standard house type size of the first room and the size matching parameter; the second room is a room in the target house that is not included in the first radar map; the standard house type size is the size data of the corresponding room of the first room in the first standard house type subgraph.

[0063] Step x2, adjust the corresponding vectorized house type structure data of each second room according to the average value of the size ratio to obtain the size matching parameter of the second room.

[0064] Step x3, based on the size matching parameter of the second room, the corresponding room is added to the house type display graph.

[0065] Based on the above scheme, the method of the embodiment of the application can introduce the standard house type drawing of the target house, match and fuse the vectorized house type structure data corresponding to the standard house type drawing with the radar map data, and obtain the house type display drawing of the target house. In this way, the influence of the complex indoor environment on the accuracy and integrity of the radar map can be effectively solved, so that the house type display drawing that matches and is complete with the actual indoor environment is obtained. Moreover, the size matching parameters of each room in the house type display drawing have room identification information, so that the house type display drawing has room semantic information, that is, the house type display drawing is a semantic map. In this way, the house type display drawing obtained by the embodiment of the application can create an indoor 3D map, and is also convenient for various services based on understanding of the spatial structure of the target house, such as intelligent control of home Internet of Things (IoT) devices, indoor intelligent interaction of robots, etc. Therefore, the technical scheme of the application can not only obtain a house type display drawing with semantic information, which is conducive to understanding the spatial structure of the target house, but also effectively improve the accuracy and integrity of the house type display drawing generation. The specific application of the embodiment of the application will be described in detail in combination with two specific application scenarios.

[0066] Scenario one: precisely control the scanning area of the intelligent sweeper through the service application program (app) of the mobile phone end, for example, the following steps a1-a4 can be used to achieve.

[0067] Step a1, the sweeper scans the target room to build an indoor map by using a laser radar, and the mobile phone end service app obtains the corresponding radar map, as shown in Figure 4 .

[0068] Step a2, the mobile phone end service app inputs the standard house type drawing of the target room, as shown in Figure 5 .

[0069] Step a3, the mobile phone end fuses and matches the generated result drawing, as shown in Figure 6 .

[0070] Step a4, the user can specify the room area to be scanned by the sweeper, as shown in Figure 7 .

[0071] Scenario two: automatically control the IoT device through the service app of the mobile phone end, for example, the following steps b1-b3 are used to control the intelligent television to turn on.

[0072] Step b1, based on the method embodiment of the application, the accurate house type display drawing of the target house is obtained, and based on the 3D map corresponding to the house type display drawing, the IoT device (such as an intelligent television, a refrigerator, a curtain, a lamp, etc.) is added to the corresponding room.

[0073] Step b2, the intelligent television is taken as a central device, and the 3D map is deployed in the central device.

[0074] Step b3: The user inputs a voice command through the smart TV to instruct the TV in the living room to be turned on. Based on the semantic information of the 3D map, the IoT service locates the smart device indicated by the voice command, determines that the target device to be turned on is the TV in the living room, and controls the TV to be turned on.

[0075] Based on the above embodiments of the indoor mapping method, the present invention correspondingly proposes an indoor mapping device, such as... Figure 8 As shown, the device includes:

[0076] The standard house type data generation unit 801 is used to generate vectorized house type structure data for the target house based on the standard house type diagram and using a deep neural network.

[0077] The radar map data generation unit 802 is used to obtain a first radar map by scanning the drivable space of a first area using radar, the first area containing at least one room of the target house.

[0078] The matching and fusion unit 803 is used to perform image matching and fusion processing based on the first radar map and the house structure data to obtain a house layout display image of the target house.

[0079] It should be noted that the above methods and apparatus are based on the same inventive concept. Since the methods and apparatus solve problems in similar ways, the implementation of the apparatus and methods can refer to each other, and the repeated parts will not be described again.

[0080] Based on the above method embodiments, this invention also proposes an indoor mapping device, including a processor and a memory; the memory stores an application program executable by the processor, which causes the processor to execute the indoor mapping method as described above. Specifically, a system or device equipped with a storage medium can be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium. Furthermore, the operating system or other system operating on the computer can perform some or all of the actual operations through instructions based on the program code. The program code read from the storage medium can also be written to a memory located in an expansion board inserted into the computer or to a memory located in an expansion unit connected to the computer. Subsequently, the CPU or other system installed on the expansion board or expansion unit executes some or all of the actual operations based on the instructions of the program code, thereby realizing the functions of any of the above embodiments of the indoor mapping method.

[0081] The memory can be embodied as an electrically erasable programmable read-only memory (EEPROM), a flash memory, a programmable read-only memory (PROM), or the like. The processor can be embodied as one or more central processing units or one or more field programmable gate arrays (FPGAs) that integrate one or more central processing unit cores. Specifically, the central processing unit or the central processing unit core can be embodied as a CPU or an MCU.

[0082] The embodiments of the present application also implement a computer program product including computer programs / instructions that, when executed by a processor, implement the steps of the indoor mapping method as described above.

[0083] It should be noted that not all steps and modules in the above processes and structural diagrams are necessary, and some steps or modules can be omitted according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The division of each module is only for the convenience of description of the adopted functional division, and in actual implementation, one module can be implemented by multiple modules, and the functions of multiple modules can be implemented by the same module. These modules can be located in the same device or in different devices.

[0084] The hardware modules in each embodiment can be implemented mechanically or electronically. For example, a hardware module can include a specially designed permanent circuit or logic device (such as a dedicated processor, e.g., an FPGA or an ASIC) for completing a specific operation. A hardware module can also include a programmable logic device or circuit temporarily configured by software (such as a general-purpose processor or other programmable processor) for performing a specific operation. As for the specific implementation of mechanical means, or the use of dedicated permanent circuits, or the use of temporarily configured circuits (such as configured by software), the implementation of hardware modules can be determined according to cost and time considerations.

[0085] In the present document, "schematically" means "to serve as an example, instance or illustration", and any diagram, embodiment described as "schematically" in the present document should not be interpreted as a more preferred or more advantageous technical solution. In order to make the drawings simple, only the parts related to the present application are schematically shown in each drawing, and do not represent the actual structure of the product. In addition, in order to make the drawings simple and easy to understand, in some drawings, only one of the parts having the same structure or function is schematically shown, or only one of them is marked. In the present document, "one" does not mean to limit the number of the parts related to the present application to "only one", and "one" does not mean to exclude the case where the number of the parts related to the present application is "more than one". In the present document, "upper", "lower", "front", "rear", "left", "right", "inner", "outer" and the like are used only to indicate the relative positional relationship between the parts, and not to limit the absolute position of the parts.

[0086] In the present specification and embodiments, if the scheme described involves processing of personal information, the processing will be performed on the premise of having a legal basis (for example, obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and will be performed only within the prescribed or agreed scope. Refusal of the user to process personal information other than the necessary information required for the basic function will not affect the user's use of the basic function.

[0087] In conclusion, the above is only a preferred embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method of indoor mapping, characterized by, The method comprises: based on the standard house plan of the target house, a deep neural network is used to generate vectorized house structure data for the target house; by scanning the first area of the available space with a radar, a first radar map is obtained, the first area containing at least one room of the target house; based on the first radar map and the house structure data, image matching fusion processing is performed to obtain a house layout display of the target house; wherein the image matching fusion processing comprises: based on the house structure data, all standard house subgraphs of the target house are constructed; the standard house subgraph contains at least one room of the target house; from the standard house subgraph, the first standard house subgraph that best matches the first radar map is selected, and the corresponding matching adjustment angle of the first radar map is determined, the matching adjustment angle being the angle by which the first radar map needs to be rotated and / or flipped to be consistent with the direction of the first standard house subgraph; for each first room in the first radar map, the corresponding room in the first standard house subgraph is determined, and the center position, length and width of the first room are iteratively fitted to obtain the size matching parameters of the first room; the size matching parameters include the center position, length, width and room identification information of the room; based on the matching adjustment angle and the size matching parameters of the first room, the first radar map is adjusted accordingly to obtain the house layout display of the target house.

2. The method of claim 1, wherein, The first radar map is obtained by scanning the first area of the available space with a radar, which comprises: scanning the first area of the available space with a radar to obtain an original radar map; filtering the noise area in the original radar map to obtain the first radar map.

3. The method of claim 1, wherein, The first radar map that best matches the first radar map is selected from the standard house subgraph, and the corresponding matching adjustment angle of the first radar map is determined, which comprises: performing room segmentation on the first radar map, and taking the largest area room as the main area; taking the matching adjustment angle as the similarity parameter to determine the first similarity between the main area and each standard house subgraph, and selecting the first similarity greater than the preset coarse similarity threshold value; the matching adjustment angle corresponding to the selected first similarity is taken as the candidate matching adjustment angle corresponding to the standard house subgraph; adjusting the first radar map based on each candidate matching adjustment angle, and calculating the second similarity between the adjusted radar map and the corresponding standard house subgraph, the second similarity being obtained by weighted calculation based on the preset fine similarity parameter, the fine similarity parameter including intersection over union, coverage and / or aspect ratio; the standard house subgraph corresponding to the maximum value of the second similarity is taken as the standard house subgraph that best matches the first radar map, and the corresponding candidate matching adjustment angle is taken as the corresponding matching adjustment angle of the first radar map.

4. The method of claim 1, wherein, For each first room in the first radar map, determining a corresponding room of the first room in the first standard house type subgraph includes: Taking the intersection-over-union ratio as the similarity parameter, calculating the similarity between the first room and each room in the first standard house type subgraph, and taking the room corresponding to the maximum similarity as the corresponding room of the first room in the first standard house type subgraph.

5. The method of claim 1, wherein, The image matching fusion processing further includes: When there are second rooms, for each first room, determining the size ratio between the standard house type size of the first room and the size matching parameter; the second room is a room in the target house that is not included in the first radar map; the standard house type size is the size data of the corresponding room of the first room in the first standard house type subgraph; According to the average value of the size ratio, adjusting the corresponding vectorized house type structure data of each second room to obtain the size matching parameter of the second room, Based on the size matching parameter of the second room, the corresponding room is added to the house type display graph.

6. The method of claim 1, wherein, The rooms in the standard house type graph have connectivity.

7. An indoor mapping device, characterized by It includes: A standard house type data generation unit for generating vectorized house type structure data for the target house based on the standard house type graph of the target house using a deep neural network; A radar map data generation unit for obtaining a first radar map by scanning the first area of the available space using a radar; the first area contains at least one room of the target house; A matching fusion unit for performing image matching fusion processing based on the first radar map and the house type structure data to obtain a house type display graph of the target house; The matching fusion unit is specifically configured to, when performing image matching fusion processing, construct all standard house type subgraphs of the target house based on the house type structure data; the standard house type subgraph contains at least one room of the target house; from the standard house type subgraph, the first standard house type subgraph that best matches the first radar map is selected, and the corresponding matching adjustment angle of the first radar map is determined, which is the angle that the first radar map needs to be rotated and / or flipped to be consistent with the direction of the first standard house type subgraph; for each first room in the first radar map, determine the corresponding room of the first room in the first standard house type subgraph, and take the size of the corresponding room as the target to optimize and iteratively fit the center position, length and width of the first room to obtain the size matching parameter of the first room; the size matching parameter includes the center position, length, width and room identification information of the room; based on the matching adjustment angle and the size matching parameter of the first room, the first radar map is adjusted accordingly to obtain the house type display graph of the target house.

8. An indoor mapping device, comprising: It includes a processor and a memory; The memory stores an application program executable by the processor, which is used to make the processor execute the indoor mapping method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable instructions stored therein are used to perform the indoor mapping method as claimed in any one of claims 1 to 6.

Images