A planar graph generating method, device and computer readable storage medium

Abstract

The present application discloses a plan view generation method, device, and computer-readable storage medium. The plan view generation method includes: acquiring multiple frames of images from different perspectives; generating a dense semantic point cloud model based on the multiple frames of images from different perspectives, wherein the dense semantic point cloud model includes: a first point cloud whose semantic label represents the ground, and a second point cloud; generating several candidate planes as floor planes based on the first point cloud in the dense semantic point cloud model; dividing the data points of the second point cloud in the dense semantic point cloud model into floors corresponding to the floor planes according to their distance relationship with the several candidate planes; and generating a plan projection map based on the floor planes and the data points on the corresponding floors. The present application uses this plan view generation method to automatically detect multi-story structures in a scene and automatically generate a plan projection map for each floor, greatly improving mapping efficiency.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for generating a plane image, and a computer-readable storage medium. Background Art

[0002] Floor plans or CAD (Computer Aided Design) drawings are essential elements in AR (Augmented Reality) navigation, planning, and design. For example, AR navigation requires CAD drawings for path planning, while planning and design require a holistic understanding of the upgrade and renovation scenario. All of these require operations or presentation on floor plans or CAD drawings.

[0003] In recent years, some research has begun to project human trajectories onto floor plans to analyze human behavior and achieve intelligent monitoring and management of urban crowds. However, currently, generating floor plans or CAD drawings often requires a lot of manpower, is costly, has a long production cycle, and cannot be dynamically updated, which limits mapping efficiency. Summary of the Invention

[0004] The present application provides a plan view generation method, device, and computer-readable storage medium.

[0005] To solve the above technical problems, the first technical solution provided by this application is to provide a plan view generation method, which includes:

[0006] Acquire multiple frames of images from different perspectives;

[0007] Generating a dense semantic point cloud model based on the multiple frames of images with different perspectives, the dense semantic point cloud model comprising: a first point cloud whose semantic label is represented as ground, and a second point cloud;

[0008] generating a plurality of candidate planes as floor planes based on the first point cloud in the dense semantic point cloud model;

[0009] Dividing the data points of the second point cloud in the dense semantic point cloud model into floors corresponding to the respective floor planes according to their distance relationships with the candidate planes;

[0010] A plan projection image is generated based on the floor plan and data points on the corresponding floor.

[0011] Through this method, the floor plan generation device automatically captures images from different perspectives, builds dense semantic point cloud models based on these images, and then uses this dense semantic point cloud model to segment the data points of different floors to generate floor plan projections for each floor. This entire process requires no human intervention and can automatically detect multi-story structures in the scene and automatically generate floor plan projections for each floor, greatly improving mapping efficiency.

[0012] The step of generating a dense semantic point cloud model based on the multiple frames of images from different perspectives includes:

[0013] A dense semantic point cloud model is generated based on the multiple frames of images with different perspectives and the semantic labels of the images.

[0014] In the above manner, the semantic labels of the image are used to generate the semantics of the data points in the point cloud model, which can avoid semantic segmentation of the point cloud model and improve the efficiency of the planar map generation method.

[0015] The generating of a dense semantic point cloud model based on the multiple frames of images at different perspectives and the semantic labels of the images includes:

[0016] generating a dense point cloud model based on the multiple frames of images at different perspectives, wherein each pixel of the multiple frames of images at different perspectives has a semantic label;

[0017] Obtaining a semantic label for a data point in the dense point cloud model according to a semantic label of a pixel in the multiple frames of images at different perspectives;

[0018] The dense semantic point cloud model is generated based on the dense point cloud model and the semantic labels of all data points therein.

[0019] In this way, the semantic labels of data points in the dense semantic point cloud model are comprehensively evaluated according to the semantic labels of pixel points in images with different perspectives, which can improve the accuracy of the semantic labels of data points and improve the mapping accuracy.

[0020] The step of obtaining the semantic label of the data point according to the semantic label of the pixel of the data point in the multiple frames of images with different perspectives in the dense point cloud model includes:

[0021] Selecting visibility images of vertices in the dense point cloud model from the multiple frames of images at different viewing angles;

[0022] Projecting the vertices in the dense point cloud model onto the visibility image, and obtaining semantic labels of all pixel points corresponding to the vertices projected onto the visibility image;

[0023] According to the semantic category, the semantic label that appears most frequently among the semantic labels of all the pixels is determined to correspond to the semantic label in the dense point cloud model.

[0024] In this way, the vertices of the dense point cloud model are used to determine the visibility image as a benchmark, thereby ensuring that the projection of images from most perspectives on the visibility image is as clear as possible, reducing occlusions and improving the accuracy of semantic labels.

[0025] The step of generating a plurality of candidate planes based on the first point cloud in the dense semantic point cloud model includes:

[0026] Connecting data points of the first point cloud in the dense semantic point cloud model into a plurality of facets;

[0027] Select one of the facets as the seed facet;

[0028] Determining whether there is a bin that meets a first preset condition among other bins in the neighborhood of the seed bin;

[0029] If so, taking the bin that meets the first preset condition as the first bin associated with the seed bin;

[0030] The candidate plane is fitted using the seed bin and a first bin associated with the seed bin.

[0031] In the above manner, the region growing method is adopted through the seed surface element to traverse all the associated surface elements in the neighborhood of the seed surface element, so as to fit them into the candidate plane, ensuring that as many similar surface elements as possible are fitted to the same candidate plane, thereby improving the generation efficiency of the candidate plane.

[0032] The step of fitting the candidate plane using the seed bin and the first bin associated with the seed bin includes:

[0033] Determining whether any other bins in the neighborhood of the first bin meet the first preset condition;

[0034] If so, taking the bins satisfying the first preset condition among other bins in the neighborhood of the first bin as second bins associated with the seed bin;

[0035] The candidate plane is fitted using the seed bin, and a first bin and a second bin associated with the seed bin.

[0036] In the above manner, the first surfel that is already associated with the seed surfel is used to further search for other second surfels that can be associated with the seed surfel, thereby broadening the surfels for fitting the candidate plane.

[0037] Among them, the first preset condition is that for two surface elements that are neighbors of each other, the angle between the normal vectors of the plane where one surface element is located and the plane where the other surface element is located is less than or equal to the first preset angle threshold, and the planar distance between the planes where the two surface elements are located is less than the first preset distance threshold.

[0038] The step of fitting the candidate plane using the seed bin, and the first bin and the second bin associated with the seed bin includes:

[0039] Setting the plane where the seed bin is located as the initial plane;

[0040] updating the initial plane based on a first bin associated with the seed bin;

[0041] The candidate plane is fitted based on a second bin associated with the seed bin and the updated initial plane.

[0042] Through the above method, a specific method for screening neighborhood-associated surface elements is provided, which improves the screening accuracy of neighborhood-associated surface elements by simultaneously executing two judgment conditions: normal vector angle difference and plane distance.

[0043] The step of dividing the data points of the second point cloud in the dense semantic point cloud model into floors corresponding to the respective floor planes according to their distance relationships with the candidate planes includes:

[0044] Calculating the absolute distance between the data points of the second point cloud and each floor plane;

[0045] The data points of the second point cloud are divided into floors corresponding to the floor planes having the smallest absolute distance.

[0046] In the above method, all data points whose semantic labels are not ground are divided into the nearest floors through absolute value distance judgment, thereby improving the richness of the plane projection image.

[0047] After generating a plurality of candidate planes based on the first point cloud in the dense semantic point cloud model, the planar graph generation method further includes:

[0048] Determine whether any two candidate planes satisfy a second preset condition;

[0049] If yes, the two candidate planes are fused;

[0050] The second preset condition is that the angle difference between the normal vectors of the two candidate planes is less than a second preset angle threshold, and the distance between the two candidate planes is less than a second preset distance threshold.

[0051] In the above method, by fusing the candidate planes, the misjudgment of floor classification caused by the existence of point cloud position error and semantic error is reduced.

[0052] After generating a plurality of candidate planes based on the first point cloud in the dense semantic point cloud model, the planar graph generation method further includes:

[0053] Get the number of data points in each candidate plane, and get the maximum number of data points in all candidate planes;

[0054] Calculating the ratio of the number of data points in each candidate plane to the maximum number;

[0055] Eliminate candidate planes whose ratio is smaller than a preset ratio.

[0056] In the above manner, candidate planes with too few data points are eliminated, thereby excluding planes that are identified as potentially erroneous based on experience, thereby improving the accuracy of the floor plan.

[0057] The generating of a plane projection image based on the floor plan and the data points on the corresponding floor includes:

[0058] Taking the normal vector of the floor plane as the first coordinate axis;

[0059] selecting a vector in the floor plane as a second coordinate axis;

[0060] Determine a third coordinate axis based on the first coordinate axis and the second coordinate axis, and construct a floor plane coordinate system using the first coordinate axis, the second coordinate axis, and the third coordinate axis;

[0061] The data points on the floor corresponding to the floor plan are projected onto the plane where the second coordinate axis and the third coordinate axis in the floor plan coordinate system are located to obtain a plane projection diagram of the floor.

[0062] Through the above method, the projections of the data points divided into relevant floors are added to the plane projection map to enrich the content of the plane projection map.

[0063] In order to solve the above technical problems, the second technical solution provided by this application is: to provide a plan view generation device, which includes an image acquisition module, a model generation module, a floor generation module and a plan view generation module; wherein,

[0064] The image acquisition module is used to acquire multiple frames of images from different perspectives;

[0065] The model generation module is configured to generate a dense semantic point cloud model based on the multiple frames of images with different perspectives, wherein the dense semantic point cloud model includes: a first point cloud whose semantic label is represented as ground, and a second point cloud;

[0066] The floor generation module is configured to generate a plurality of candidate planes as floor planes based on the first point cloud in the dense semantic point cloud model;

[0067] The floor generation module is further configured to divide the data points of the second point cloud in the dense semantic point cloud model into floors corresponding to the respective floor planes according to their distance relationships with the plurality of candidate planes;

[0068] The plan view generating module is used to generate a plan projection view based on the floor plan and the data points on the corresponding floor.

[0069] To solve the above technical problems, the third technical solution provided in this application is: to provide a plan view generation device, which includes a processor and a memory connected to the processor, wherein the memory stores program instructions; the processor is used to execute the program instructions stored in the memory to implement the above plan view generation method.

[0070] In order to solve the above technical problems, the fourth technical solution provided in this application is: providing a computer-readable storage medium, wherein the storage medium stores program instructions, and the program instructions implement the above-mentioned plan view generation method when executed.

[0071] In the floor plan generation method provided by the present application, a floor plan generation device obtains multiple frames of images from different perspectives; generates a dense semantic point cloud model based on the multiple frames of images from different perspectives, the dense semantic point cloud model including: a first point cloud whose semantic label represents the ground, and a second point cloud; generates several candidate planes as floor planes based on the first point cloud in the dense semantic point cloud model; divides the data points of the second point cloud in the dense semantic point cloud model into floors corresponding to each floor plane according to the distance relationship with the several candidate planes; and generates a plane projection map based on the floor planes and the data points on the corresponding floors. The present application automatically detects the multi-story structure in the scene through this floor plan generation method and automatically generates a plane projection map for each floor, greatly improving mapping efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. Among them:

[0073] Figure 1 This is a flow chart of an embodiment of a plan view generation method provided by the present application;

[0074] Figure 2 This is a flow chart of another embodiment of the plan view generation method provided by the present application;

[0075] Figure 3 This is the plane projection provided by this application Figure 1 Schematic diagram of an embodiment;

[0076] Figure 4 It is a structural diagram of an embodiment of a plan view generating device provided by the present application;

[0077] Figure 5 is a structural schematic diagram of another embodiment of the plan view generating device provided by the present application;

[0078] Figure 6 It is a structural diagram of the computer-readable storage medium provided by this application. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0080] The present application is described in detail below with reference to the accompanying drawings and embodiments.

[0081] See Figure 1 , Figure 1 The figure is a flow chart of an embodiment of a plan view generation method provided by the present application. The plan view generation method described in the embodiment of the present application is applied to a plan view generation device. The plan view generation device of the present application can be a server or a system comprising a server and a terminal device. Accordingly, the various components of the plan view generation device, such as the various units, subunits, modules, and submodules, can be entirely located in the server or separately located in the server and the terminal device.

[0082] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here. In some possible implementations, the floor plan generation method of the embodiment of the present application can be implemented by a processor calling computer-readable instructions stored in a memory.

[0083] like Figure 1 As shown, the specific steps of the plan view generation method in the embodiment of the present application are as follows:

[0084] Step S11: Acquire multiple frames of images from different perspectives.

[0085] In an embodiment of the present application, the plan view generating device collects images of the target building from different directions and perspectives through an image acquisition device, such as a camera, a video monitor, or the like.

[0086] Step S12: Generate a dense semantic point cloud model based on multiple frames of images from different perspectives. The dense semantic point cloud model includes: a first point cloud whose semantic label is represented as the ground, and a second point cloud.

[0087] In this embodiment of the present application, the planar map generation device can generate a dense point cloud model using the MVS (Multi View System) algorithm. The semantic label of each data point in the dense point cloud model is then comprehensively evaluated based on the semantic information of images from different viewpoints. The overall process of the MVS algorithm is as follows: data acquisition - sparse reconstruction - depth map estimation - dense reconstruction.

[0088] Specifically, the specific process of generating a dense semantic point cloud model based on multiple frames of images with different perspectives can be found in Figure 2 , Figure 2 It is a flow chart of another embodiment of the plan view generation method provided in this application.

[0089] like Figure 2 As shown, the specific steps of the plan view generation method in the embodiment of the present application are as follows:

[0090] Step S21: Generate a dense point cloud model based on multiple frames of images with different perspectives, where each pixel of the multiple frames of images with different perspectives has a semantic label.

[0091] In an embodiment of the present application, the plan view generating device obtains the pixel semantics of each frame of the image, wherein the pixel semantics of the image can be identified and generated by any semantic deep learning model, which will not be repeated here.

[0092] Step S22: Obtain a semantic label for the data point based on the semantic labels of the pixel points of the data point in multiple frames of images with different perspectives in the dense point cloud model.

[0093] In an embodiment of the present application, the planar map generation device can use a voting method to obtain a semantic label for each data point in the dense point cloud model. Each data point in the dense point cloud model can be searched for a corresponding pixel in images from different perspectives. Therefore, the planar map generation device can generate semantic labels for data points in the dense point cloud model using the semantic information of the pixels in the images from different perspectives.

[0094] For example, the plan view generating device may count the semantic types of the data points corresponding to the images of different viewing angles, and then use the semantic type that appears most frequently in the images of different viewing angles as the semantic label of the data point.

[0095] In order to further improve the accuracy of semantic labels, the plan view generating device may also project images from different perspectives using the visibility image as a projection reference, thereby performing a voting method based on the projection results.

[0096] Specifically, the planar map generation device selects visibility images of the vertices of the dense point cloud model from multiple frames of images at different viewpoints. The visibility of the visibility images can be obtained through a multi-view stereo geometry process. Specifically, during the generation of the dense point cloud model, images with appropriate viewpoints are selected as visibility images, and the visibility images are projected onto the vertices of the dense point cloud model without occlusion.

[0097] The planar graph generation device projects any data point X in the dense point cloud model onto the visibility image according to projection parameters, such as the projection matrix, and projects images from other perspectives onto the visibility image according to the same projection parameters. The planar graph generation device counts the pixel semantics of the projection point of the data point on the visibility image and obtains the pixel semantics with the most votes as the semantic label of the data point. The voting method in this process is expressed as follows:

[0098]

[0099] Where n is the number of images; v i is the visibility of the image, which is 1 if the image is visible and 0 if the image is invisible; P i represents the projection matrix of the i-th frame image; l(x) represents the label value operation; γ represents the label consistency threshold, and the specific value in the embodiment of the present application can be set to 0.2, where the larger γ is, the more accurate the semantic label of the data point is. However, if the vote ratio of the highest semantic label cannot meet the label consistency threshold, the semantic labels of some data points will be lost.

[0100] Step S23: Generate a dense semantic point cloud model based on the dense point cloud model and the semantic labels of all data points therein.

[0101] In an embodiment of the present application, after the plan view generating device obtains the semantic labels of all data points in the dense point cloud model, it can generate a dense semantic point cloud model by combining the dense point cloud model and the semantic labels.

[0102] Step S13: Generate several candidate planes as floor planes based on the first point cloud in the dense semantic point cloud model.

[0103] In an embodiment of the present application, the plan view generating device obtains all data points with semantic labels as ground from the dense semantic point cloud model, that is, the first point cloud, performs plane fitting, and obtains the final candidate plane.

[0104] Specifically, the plane map generating device takes out all ground points, constructs face elements based on adjacent ground points, and then uses the region growing method to detect and fit multiple candidate planes. The plane map generating device randomly selects a seed face element, creates a queue containing the seed face element, and initializes the plane equation according to the centroid and normal vector of the seed face element. Then, the plane map generating device takes out the first element in the queue, such as the seed face element, and traverses its one-ring neighborhood, that is, searches for other directly adjacent face elements to see whether there are face elements that meet the first preset condition. Among them, the first preset condition can be reflected in the following formula:

[0105]

[0106] in, is the normal vector of the seed surface element, C f is the centroid coordinate of the seed surface element; is the normal vector of the neighboring surface element, C t is the centroid coordinate of the neighborhood face element; θ1 is the deviation angle threshold, which is set to 10° in the embodiment of the present application; δ1 is the distance threshold.

[0107] The plane graph generating device places the neighboring surface elements that satisfy the above formula (2) at the end of the queue and updates the plane equation at the same time. The plane equation is updated in the following formula:

[0108]

[0109] in, is the normal vector of the plane where the associated face element is located before updating, is the normal vector of the plane where the associated face element is located after the update, C t-1 is the centroid coordinate of all associated facets before updating, C t The coordinates of the center of gravity of all associated surface elements after updating.

[0110] Then, the plane map generating device extracts a neighboring surface element from the queue as a new seed surface element and searches for other related surface elements according to the above process until all the surface elements in the queue have no neighboring surface elements that satisfy the above formula (2). At this point, the plane map generating device can fit a candidate plane using all the surface elements in the queue and the latest plane equation.

[0111] The plane map generating device traverses all data points in the dense semantic point cloud model, and repeatedly performs the above process of searching for associated neighborhood facets until the entire dense semantic point cloud model is traversed to obtain several candidate planes.

[0112] Due to positional and semantic errors in the point cloud, the multiple candidate planes obtained in the above process inevitably contain errors or redundancies. Therefore, the floor plan generation device can also merge multiple parallel and spatially close candidate planes into a single candidate plane, and eliminate planes with too few points within the candidate planes, thereby generating the final multiple floor plans.

[0113] The criterion for determining whether two candidate planes need to be merged is expressed in the following formula:

[0114]

[0115] in, and are the plane parameters of the candidate planes to be merged, θ2 is the deviation angle threshold, which is set to 2° in the embodiment of the present application, and δ is a very small distance error threshold.

[0116] Furthermore, after all candidate planes are merged, some candidate planes in the candidate plane set may still have too few data points. These candidate planes are empirically identified as potentially erroneous planes. Therefore, the planar graph generation device needs to remove these candidate planes from the candidate plane set.

[0117] The criteria for determining whether candidate planes need to be eliminated are expressed in the following formula:

[0118]

[0119] Where K is the number of planes in the candidate plane set, C inlier (L j ) represents the number of data points in the j-th candidate plane, and α is the elimination ratio threshold.

[0120] Step S14: Divide the data points of the second point cloud in the dense semantic point cloud model into floors corresponding to the respective floor planes according to their distance relationships with the candidate planes.

[0121] In this embodiment of the present application, after the floor plan generation device detects all candidate planes, it needs to classify all non-ground points (i.e., data points whose semantic label is not "ground") to distinguish between floors. Specifically, if two data points located in a floor plane (i.e., the space above and below a candidate plane) have opposite distance signs to the candidate plane, they can be used to classify the point cloud into floors.

[0122] Specifically, the floor plan generation device calculates the absolute distances from the non-ground point to all floor planes. The plane with the smallest absolute distance is the floor plane corresponding to the non-ground point. Finally, the floor plan generation device re-estimates the ground parameters based on the selected floor ground points and non-ground points, ensuring more accurate ground parameters.

[0123] Step S15: Generate a plan projection image based on the floor plan and the data points on the corresponding floor.

[0124] In the embodiment of the present application, the plan view generating device needs to convert all data points into the corresponding floor plane coordinate system based on the floor plane, and then project from far to near according to the orthogonal projection method to generate the final plane projection map, such as Figure 3 shown.

[0125] The establishment of the floor plane coordinate system and the projection of the data points can be achieved by the following steps:

[0126] (1) The normal vector of the floor plane is used as the first coordinate axis.

[0127] (2) Select a vector in the floor plane as the second coordinate axis.

[0128] (3) Determine the third coordinate axis based on the first coordinate axis and the second coordinate axis, and construct a floor plane coordinate system using the first coordinate axis, the second coordinate axis, and the third coordinate axis.

[0129] (4) Project the data points on the corresponding floor of the floor plane onto the plane where the second coordinate axis and the third coordinate axis are located in the floor plane coordinate system to obtain the plane projection diagram of the floor.

[0130] In an embodiment of the present application, a plan view generating device acquires multiple frames of images from different perspectives; generates a dense semantic point cloud model based on the multiple frames of images from different perspectives; generates several candidate planes as floor planes based on the data points with the semantic label of ground in the dense semantic point cloud model; divides the data points in the dense semantic point cloud model except for the data points with the semantic label of ground into floors corresponding to each floor plane; generates a plane projection map based on the floor plane and the data points in the corresponding floor. The present application automatically detects the multi-story structure in the scene through the plan view generating method, and automatically generates a plane projection map for each floor, which greatly improves the mapping efficiency; the plan view generating method can provide an accurate scene plane sketch without the need for tedious on-site measurements, providing a basis for accurate CAD production, and in some cases can even completely replace traditional CAD drawings.

[0131] The above embodiment is only one common case of the present application and does not limit the technical scope of the present application. Therefore, any minor modifications, equivalent changes or modifications made to the above content based on the essence of the present application solution are still within the scope of the technical solution of the present application.

[0132] Please continue to see Figure 4 , Figure 4 This is a schematic diagram of the structure of an embodiment of the planar graph generating device provided by this application. Figure 4 As shown, the plan view generating device 40 includes an image acquisition module 41 , a model generating module 42 , a floor generating module 43 and a plan view generating module 44 .

[0133] The image acquisition module 41 is used to acquire multiple frames of images from different perspectives.

[0134] The model generation module 42 is configured to generate a dense semantic point cloud model based on the multiple frames of images from different perspectives, wherein the dense semantic point cloud model includes a first point cloud whose semantic label represents the ground, and a second point cloud.

[0135] The floor generation module 43 is configured to generate a plurality of candidate planes as floor planes based on the first point cloud in the dense semantic point cloud model.

[0136] The floor generation module 44 is further configured to divide the data points of the second point cloud in the dense semantic point cloud model into floors corresponding to the respective floor planes according to the distance relationship between the data points and the plurality of candidate planes.

[0137] The plan view generating module 45 is configured to generate a plan projection view based on the floor plan and the data points on the corresponding floor.

[0138] The model generation module 42 is configured to generate a dense semantic point cloud model based on the multiple frames of images from different perspectives and the semantic labels of the images.

[0139] The model generation module 42 is used to generate a dense point cloud model based on the multiple frames of images with different perspectives, wherein each pixel point of the multiple frames of images with different perspectives has a semantic label; obtain the semantic label of the data point in the dense point cloud model according to the semantic label of the pixel point in the multiple frames of images with different perspectives; and generate the dense semantic point cloud model based on the dense point cloud model and the semantic labels of all data points therein.

[0140] The model generation module 42 is used to select the visibility image of the vertices in the dense point cloud model from the multiple frames of images with different perspectives; project the vertices in the dense point cloud model onto the visibility image, and obtain the semantic labels of all pixel points corresponding to the vertices projected onto the visibility image; and confirm that the vertex corresponds to the semantic label in the dense point cloud model according to the semantic label that appears the most times in the semantic labels of all pixel points according to the semantic category.

[0141] The floor generation module 43 is used to connect the data points of the first point cloud in the dense semantic point cloud model into a plurality of facets; select one of the facets as a seed facet; determine whether there is a facet that meets a first preset condition among other facets in the neighborhood of the seed facet; if so, use the facet that meets the first preset condition as the first facet associated with the seed facet; and fit the candidate plane using the seed facet and the first facet associated with the seed facet.

[0142] The floor generation module 43 is used to determine whether there are other face elements in the neighborhood of the first face element that meet the first preset condition; if so, use the face element in the neighborhood of the first face element that meets the first preset condition as the second face element associated with the seed face element; and use the seed face element, and the first face element and the second face element associated with the seed face element to fit the candidate plane.

[0143] The first preset condition is that for two surface elements that are neighbors of each other, the angle between the normal vectors of the plane where one surface element is located and the plane where the other surface element is located is less than or equal to the first preset angle threshold, and the plane distance between the planes where the two surface elements are located is less than the first preset distance threshold.

[0144] The floor generation module 43 is used to set the plane where the seed surface element is located as the initial plane; update the initial plane based on the first surface element associated with the seed surface element; and fit the candidate plane based on the second surface element associated with the seed surface element and the updated initial plane.

[0145] The floor generation module 43 is configured to calculate the absolute distance between the data points of the second point cloud and each floor plane; and to divide the data points of the second point cloud into floors corresponding to the floor plane with the smallest absolute distance.

[0146] The floor generation module 43 is used to determine whether a second preset condition is satisfied between any two candidate planes; if so, the two candidate planes are merged; wherein the second preset condition is that the angle difference between the normal vectors of the two candidate planes is less than a second preset angle threshold, and the distance between the two candidate planes is less than a second preset distance threshold.

[0147] The floor generation module 43 is used to obtain the number of data points in each candidate plane and the maximum number of data points in all candidate planes; calculate the ratio of the number of data points in each candidate plane to the maximum number; and eliminate the candidate planes whose ratio is less than a preset ratio.

[0148] The floor plan generation module 45 is used to use the normal vector of the floor plane as the first coordinate axis; select a vector in the floor plane as the second coordinate axis; determine the third coordinate axis based on the first coordinate axis and the second coordinate axis, and construct a floor plan coordinate system with the first coordinate axis, the second coordinate axis and the third coordinate axis; project the data points in the corresponding floor of the floor plan to the plane where the second coordinate axis and the third coordinate axis are located in the floor plan coordinate system to obtain a plan projection of the floor.

[0149] See Figure 5 , Figure 5 FIG2 is a schematic diagram of another embodiment of a planar graph generating device provided by the present application. The planar graph generating device includes a memory 52 and a processor 51 connected to each other.

[0150] The memory 52 is used to store program instructions for implementing any one of the above-mentioned plan view generation methods.

[0151] The processor 51 is configured to execute program instructions stored in the memory 52 .

[0152] The processor 51 may also be referred to as a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip with signaling processing capabilities. The processor 51 may also be a general-purpose processor, a digital signaling processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor or any conventional processor.

[0153] The memory 52 can be a memory stick, a TF card, etc., which can store all the information in the string matching prediction device, including the input raw data, computer programs, intermediate operation results and final operation results. It is stored in the memory. It stores and retrieves information according to the location specified by the controller. With the memory, the string matching prediction device has a memory function and can ensure normal operation. The memory of the string matching prediction device can be divided into main memory (internal memory) and auxiliary memory (external memory) according to its purpose. There is also a classification method of dividing it into external memory and internal memory. External memory is usually a magnetic medium or an optical disk, etc., which can store information for a long time. Memory refers to the storage component on the motherboard, which is used to store the data and programs currently being executed, but is only used to temporarily store programs and data. If the power is turned off or the power is cut off, the data will be lost.

[0154] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0155] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0156] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, system server, or network device, etc.) or a processor to execute all or part of the steps of the various embodiments of the present application.

[0158] See also Figure 6 , which is a structural diagram of the computer-readable storage medium of the present application. The storage medium of the present application stores a program file 61 that can implement all the above-mentioned plan view generation methods, wherein the program file 61 can be stored in the above-mentioned storage medium in the form of a software product, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to execute all or part of the steps of the various implementation methods of the present application. The aforementioned storage device includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or a plan view generation device such as a computer, a server, a mobile phone, or a tablet.

[0159] The above is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A planar graph generation method, characterized in that: The plan view generating method comprises: Acquire multiple frames of images from different perspectives; Generating a dense semantic point cloud model based on the multiple frames of images with different perspectives, the dense semantic point cloud model comprising: a first point cloud whose semantic label is represented as ground, and a second point cloud; generating a plurality of candidate planes as floor planes based on the first point cloud in the dense semantic point cloud model; Dividing the data points of the second point cloud in the dense semantic point cloud model into floors corresponding to the respective floor planes according to their distance relationships with the candidate planes; generating a planar projection image based on the floor plan and data points on the corresponding floor; Generating a dense semantic point cloud model based on the multiple frames of images from different perspectives includes: A dense semantic point cloud model is generated based on the multiple frames of images with different perspectives and the semantic labels of the images.

2. The planar graph generation method according to claim 1, wherein: Generating a dense semantic point cloud model based on the multiple frames of images at different perspectives and the semantic labels of the images includes: generating a dense point cloud model based on the multiple frames of images at different perspectives, wherein each pixel of the multiple frames of images at different perspectives has a semantic label; Obtaining a semantic label for a data point in the dense point cloud model according to a semantic label of a pixel in the multiple frames of images at different perspectives; The dense semantic point cloud model is generated based on the dense point cloud model and the semantic labels of all data points therein.

3. The planar graph generation method according to claim 2, characterized in that: Obtaining the semantic label of the data point according to the semantic label of the pixel of the data point in the multiple frames of images with different perspectives in the dense point cloud model includes: Selecting visibility images of vertices in the dense point cloud model from the multiple frames of images at different viewing angles; Projecting the vertices in the dense point cloud model onto the visibility image, and obtaining semantic labels of all pixel points corresponding to the vertices projected onto the visibility image; According to the semantic category, the semantic label that appears most frequently among the semantic labels of all the pixels is determined to correspond to the semantic label in the dense point cloud model.

4. The planar graph generation method according to claim 1, wherein: The generating of a plurality of candidate planes based on the first point cloud in the dense semantic point cloud model includes: Connecting data points of the first point cloud in the dense semantic point cloud model into a plurality of facets; Select one of the facets as the seed facet; Determining whether there is a bin that meets a first preset condition among other bins in the neighborhood of the seed bin; If so, taking the bin that meets the first preset condition as the first bin associated with the seed bin; The candidate plane is fitted using the seed bin and a first bin associated with the seed bin.

5. The planar graph generation method according to claim 4, characterized in that: The step of fitting the candidate plane using the seed bin and a first bin associated with the seed bin comprises: Determining whether any other bins in the neighborhood of the first bin meet the first preset condition; If so, taking the bins satisfying the first preset condition among other bins in the neighborhood of the first bin as second bins associated with the seed bin; The candidate plane is fitted using the seed bin, and a first bin and a second bin associated with the seed bin.

6. The planar graph generating method according to claim 4 or 5, characterized in that: The first preset condition is that for two surface elements that are neighbors of each other, the angle between the normal vectors of the plane where one surface element is located and the plane where the other surface element is located is less than or equal to the first preset angle threshold, and the plane distance between the planes where the two surface elements are located is less than the first preset distance threshold.

7. The planar graph generating method according to claim 5, wherein: The step of fitting the candidate plane using the seed bin, and the first bin and the second bin associated with the seed bin comprises: Setting the plane where the seed bin is located as the initial plane; updating the initial plane based on a first bin associated with the seed bin; The candidate plane is fitted based on a second bin associated with the seed bin and the updated initial plane.

8. The planar graph generating method according to claim 1, wherein: The step of dividing the data points of the second point cloud in the dense semantic point cloud model into floors corresponding to the respective floor planes according to the distance relationship with the plurality of candidate planes comprises: Calculating the absolute distance between the data points of the second point cloud and each floor plane; The data points of the second point cloud are divided into floors corresponding to the floor planes having the smallest absolute distance.

9. The planar graph generating method according to claim 1, wherein: After generating a plurality of candidate planes based on the first point cloud in the dense semantic point cloud model, the planar graph generation method further includes: Determine whether any two candidate planes satisfy a second preset condition; If yes, the two candidate planes are fused; The second preset condition is that the angle difference between the normal vectors of the two candidate planes is less than a second preset angle threshold, and the distance between the two candidate planes is less than a second preset distance threshold.

10. The planar graph generating method according to claim 9, wherein: After generating a plurality of candidate planes based on the first point cloud in the dense semantic point cloud model, the planar graph generation method further includes: Get the number of data points in each candidate plane, and get the maximum number of data points in all candidate planes; Calculating the ratio of the number of data points in each candidate plane to the maximum number; Eliminate candidate planes whose ratio is smaller than a preset ratio.

11. The planar graph generating method according to claim 1, wherein: Generating a planar projection image based on the floor plan and data points on the corresponding floor includes: Taking the normal vector of the floor plane as the first coordinate axis; selecting a vector in the floor plane as a second coordinate axis; Determine a third coordinate axis based on the first coordinate axis and the second coordinate axis, and construct a floor plane coordinate system using the first coordinate axis, the second coordinate axis, and the third coordinate axis; The data points on the floor corresponding to the floor plan are projected onto the plane where the second coordinate axis and the third coordinate axis in the floor plan coordinate system are located to obtain a plane projection diagram of the floor.

12. A plan view generating device, characterized in that: The plan view generating device includes an image acquisition module, a model generating module, a floor generating module and a plan view generating module; wherein, The image acquisition module is used to acquire multiple frames of images from different perspectives; The model generation module is configured to generate a dense semantic point cloud model based on the multiple frames of images with different perspectives, wherein the dense semantic point cloud model includes: a first point cloud whose semantic label is represented as ground, and a second point cloud; The floor generation module is configured to generate a plurality of candidate planes as floor planes based on the first point cloud in the dense semantic point cloud model; The floor generation module is further configured to divide the data points of the second point cloud in the dense semantic point cloud model into floors corresponding to the respective floor planes according to their distance relationships with the plurality of candidate planes; The plan view generating module is configured to generate a plan projection map based on the floor plan and data points on the corresponding floor; The model generation module is further used to generate a dense semantic point cloud model based on the multiple frames of images with different perspectives and the semantic labels of the images.

13. A plan view generating device, characterized in that: The plan view generating device includes a processor and a memory connected to the processor, wherein: The memory stores program instructions; The processor is used to execute the program instructions stored in the memory to implement the plan view generation method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that The storage medium stores program instructions, and when the program instructions are executed, the plan view generation method according to any one of claims 1 to 11 is implemented.

Images