A method for generating site plan based on depth camera
The field three-dimensional point cloud is acquired and segmented by depth cameras, combined with JSIS3D network and ORB-SLAM2, and the rapid and accurate generation of field floor plans is achieved, solving the inaccuracy and time-consuming problems of traditional manual measurements, improving the efficiency of on-site surveys and providing Intelligent criminal investigation application.
Patent Information
- Application Number
- CN202211739212.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-30
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-12-30
AI Technical Summary
Traditional manual measurement and drawing on site floor plans have problems such as inaccurate, irregular and time-consuming results, and it has failed to actively obtain information.
The depth camera is used to obtain the field three-dimensional point cloud, and the target segmentation of the field three-dimensional point cloud is performed through the point cloud segmentation model, and a field floor plan is generated. The point cloud segmentation network training is used using the JSIS3D network structure and the S3DIS dataset, combining ORB-SLAM2 and PCL The library performs point cloud projection and dimensionality reduction processing, uses Sobel operator for contour extraction, and finally draws the field plan through OpenCV.
It realizes rapid and accurate generation of on-site floor plans without manual measurement, improves on-site investigation efficiency, and provides the value of intelligent criminal investigation application.
Smart Images

Figure CN115953292B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image generation, and in particular to a method for generating a site plan view based on a depth camera. Background Art
[0002] Crime scene investigation is the starting point of case investigation, and the scene plan is an important part of the scene inspection record. According to the forensic science scene mapping specification GA / T1509-2018, the scene plan includes a plan diagram, a plan scale diagram, and a plan expansion diagram. The scene investigation rules clearly stipulate that the on-site investigators should prepare a scene plan diagram and choose to prepare a scene plan scale diagram according to the situation. Therefore, the scene plan in the case Figure 1 Generally refers to a plan view, that is, it does not pursue accurate drawing proportions, but records the case scene concisely and completely. Therefore, the research object site plan view refers to a plan view. The traditional method of manually measuring and drawing site plans has the problems of inaccurate results, non-standardization and long time consumption. The research on site plans also stays at the stage of using software to standardize the drawing, without realizing the active acquisition of information.
[0003] Therefore, it is necessary to provide a new method for generating site plan views based on a depth camera to solve the above technical problems. Summary of the invention
[0004] The technical problem solved by the present invention is to provide a method for generating a site plan map without manual measurement. The device is light and easy to carry, and the generation task can be completed quickly and effectively without contact. It provides a certain reference for intelligent criminal investigation and has a high application value for rapid on-site investigation. In order to solve the above technical problems, the method for generating a site plan map based on a depth camera provided by the present invention includes the following steps:
[0005] S1: Use depth camera to obtain 3D point cloud on site;
[0006] S2: Train the point cloud segmentation model and use the model to segment the target of the 3D point cloud at the scene;
[0007] S3: Generate site plan from 3D point cloud.
[0008] Preferably, in S1, a depth camera is used to acquire a three-dimensional point cloud on site, which includes the following steps: S11: filling in calibration parameters of the depth camera; S12: acquiring a point cloud on site.
[0009] Preferably, in S12, the acquisition of the on-site point cloud includes the following steps: S121: install the camera and the micro-tripod, enter the Ubuntu system, and open the terminal in the ORB-SLAM2 folder; S122: enter the command in the terminal, run the launch file, and start the D435i node; S123: open a new terminal, enter the command in the new terminal, and start ORB-SLAM2 in the depth camera mode; S124: the program has no error, keep the camera micro-tripod perpendicular to the ground, the camera plane is parallel to the ground, hold the micro-tripod, start moving from a position in the room, and rotate slowly and at a constant speed at the turning point of the perspective to ensure that continuous feature points always appear in the picture and the whole room is completely scanned; S125: when the camera lens is about to pass through the wall, step back to ensure that other objects that can be extracted as feature points are included in the field of view; S126: circle the room, when returning to the starting position and the path shows a loop, end the program and save the point cloud.
[0010] Preferably, in said S2, JSIS3D is used to perform on-site point cloud segmentation, including the following steps: S21: Using a multi-layer neural network algorithm to learn data; S22: Using a multi-task point-by-point network and a multi-value conditional random field to solve the segmentation problem. In terms of the multi-task point-by-point network, the network performs two tasks simultaneously: predicting the semantic categories of three-dimensional points in the point cloud, and embedding these points into high-dimensional vectors so that points of the same object instance are represented by similar embeddings; based on the multi-task point-by-point network, JSIS3D proposes a multi-value conditional random field model that combines semantic and instance labels, and expresses the semantic and instance segmentation problems as joint optimization labels in the field model. S23: The original data is parsed into a semantically meaningful space, and the space is aligned to a standardized reference coordinate system.
[0011] Preferably, in said S3, a site plan is generated using a three-dimensional point cloud, comprising the following steps: S31: performing point cloud projection; S32: generating a site plan; S33: extracting contours using a Sobel operator; S35: drawing an image using a rectangular bounding box.
[0012] Compared with the related art, the method for generating a scene plan view based on a depth camera provided by the present invention has the following beneficial effects:
[0013] The present invention provides a method for generating a site plan map based on a depth camera, using a point cloud segmentation network JSIS3D network structure and an open source data set S3DIS, setting parameters based on the data set and completing the training of the point cloud segmentation network. On this basis, ORB-SLAM2 segments the acquired room point cloud structure, obtains relatively accurate on-site object information, and lays the foundation for obtaining the site plan map. Subsequently, a method for obtaining a plane structure based on projection and interception is introduced in detail. After obtaining the room point cloud segmentation result, the PCL point cloud library is called to remove the background and projection dimension reduction of the point cloud perpendicular to the plane, and the room plane structure is obtained. Subsequently, the edge detection algorithm of the Sobel operator is introduced, and the contour of the acquired room bird's-eye view is extracted, and then the shape coordinate width and height function and the contour drawing function are obtained in OpenCV, and finally the automatic generation of the site plan map is realized. By comparing the generated point cloud scale with the actual measurement result, the generated site map with the actual measurement result, the possible sources of error in the whole process are analyzed in detail, proving that the method has a certain accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 A schematic diagram of camera movement of a method for generating a scene plan view based on a depth camera provided by the present invention;
[0015] Figure 2 A flowchart of obtaining point clouds at a simulated site for the method for generating a site plan based on a depth camera provided by the present invention;
[0016] Figure 3 A scene 1 acquisition structure diagram of the method for generating a site plan view based on a depth camera provided by the present invention;
[0017] Figure 4 A structural diagram of scene 2 acquisition of the method for generating a site plan view based on a depth camera provided by the present invention;
[0018] Figure 5 A scene point cloud segmentation result diagram of the scene plane map generation method based on the depth camera provided by the present invention;
[0019] Figure 6 A scene two-point cloud segmentation result diagram of the scene plan view generation method based on the depth camera provided by the present invention;
[0020] Figure 7 A bird's-eye view diagram of the method for generating a site plan view based on a depth camera provided by the present invention;
[0021] Figure 8 The method for generating a scene plan view based on a depth camera provided by the present invention is based on a capture method to obtain a planar structure;
[0022] Fig. 9The method for generating a scene plan view based on a depth camera provided by the present invention is based on a projection method to obtain a planar structure;
[0023] Fig.10 A flow chart of background removal and dimensionality reduction of room point cloud in the method for generating a scene plan based on a depth camera provided by the present invention;
[0024] Fig.11 The scene-a room point cloud segmentation result before projection of the method for generating a site plan view based on a depth camera provided by the present invention;
[0025] Fig.12 A scene point cloud bird's-eye view and a point cloud segmentation and background-removed projection map of the scene plan view generation method based on the depth camera provided by the present invention;
[0026] Fig.13 The room point cloud segmentation result before projection of scene 2 of the method for generating a site plan based on a depth camera provided by the present invention;
[0027] Fig.14 A scene two point cloud bird's-eye view and point cloud segmentation and background removal projection of the scene plan view generation method based on the depth camera provided by the present invention;
[0028] Fig.15 A schematic diagram of contour extraction of a method for generating a site plan view based on a depth camera provided by the present invention;
[0029] Fig.16 A scene contour extraction result of the scene plane map generation method based on the depth camera provided by the present invention;
[0030] Fig.17 The second contour extraction result of the scene of the scene plan view generation method based on the depth camera provided by the present invention;
[0031] Fig.18 A schematic diagram of generating a circumscribed rectangle of the method for generating a scene plan view based on a depth camera provided by the present invention;
[0032] Fig.19 A scene plan view generation result of a scene plan view generation method based on a depth camera provided by the present invention;
[0033] Fig. 20 The scene plan map generation result of scene 2 of the scene plan map generation method based on the depth camera provided by the present invention;
[0034] Fig.21 This is a principle block diagram of the method for generating a site plan view based on a depth camera provided by the present invention. DETAILED DESCRIPTION
[0035] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.
[0036] Please refer to Figure 1-21 . Method for generating site plan based on depth camera:
[0037] Fill in the camera calibration parameters:
[0038] Input the calibration parameters and alignment results into SLAM. In the ORB-SLAM2 workspace, corresponding to the camera parameter file given by the initial ORB-SLAM2, create a new configuration file named realsense.yaml. In the configuration file, fill in the camera internal parameter data after D435i calibration according to the parameter format.
[0039] Without calibration and alignment, the point cloud obtained by the depth camera is scattered and cannot reflect the normal open room structure and object color information. After filling the D435i camera calibration parameters into the yaml file, the effect of calibrating and aligning the depth head and color head can more realistically reflect the room structure and color information.
[0040] Steps for obtaining on-site point cloud:
[0041] To maintain the stability of the camera during use, the D435i needs to be used with a micro tripod. The tripod is configured at the factory. The bolts need to be tightened before use, and the camera should be kept parallel to the bottom of the tripod as much as possible. The computer camera is turned off during use. From the introduction of SLAM related theories, it can be seen that the depth camera mode needs to be initialized, and the first frame is the key frame. In order to ensure that one of the xyz planes where the point cloud center is located is parallel to the ground and the other two are perpendicular to the ground, the camera needs to be placed on a horizontal plane, such as a table, in the simulation embodiment.
[0042] In order to ensure the integrity of the final point cloud structure, it is necessary to ensure the consistency between the starting and ending parts of the room scan. During the tracking process, the depth camera moving too fast or changing the angle too much may cause tracking failure. Therefore, try to keep the moving speed within a constant range during the scanning process of the room, and do not make the deflection angle too large when scanning up and down.
[0043] like Figure 1As shown in the figure, the black cube is used as a simulated room, the blue ellipse is the ideal movement route of the camera after scanning the room, the arrow direction indicates the movement direction, and the arrow position represents the initial position of the camera (the initial key frame in the depth camera mode). In the wrong movement diagram, the red line represents the camera movement route. The camera has a large angle and moves violently, which is easy to lose tracking. In the correct movement diagram, the green line represents the camera movement route. The angle is relatively gentle, which is close to the ideal movement route and is not prone to tracking loss. Since the thread needs to select feature points as the basis, for the range of walls with a large proportion and few feature points, try to place other objects with bright colors or shape outlines within the depth camera's field of view during scanning to assist in the effective extraction of feature points and avoid tracking loss due to too few feature points.
[0044] According to the above principles and schematics, the experimental steps for obtaining on-site point cloud are as follows, and the flow chart is as follows: Figure 2 As shown:
[0045] 1. Install the camera and micro-tripod, enter the Ubuntu system, and open the terminal in the ORB-SLAM2 folder.
[0046] 2. Enter the command in the terminal, run the launch file, and start the D435i node.
[0047] 3. Open a new terminal and enter the command in the new terminal to start ORB-SLAM2 in depth camera mode.
[0048] 4. The program reports no errors, keep the camera micro-tripod perpendicular to the ground, and the camera plane parallel to the ground. Hold the micro-tripod and start moving from one position in the room. Rotate slowly and at a constant speed at the turning point of the view to ensure that there are always continuous feature points in the picture, and try to ensure that the whole room is completely scanned.
[0049] 5. When the camera lens is about to pass through the wall, you can step back slightly to ensure that other objects that can be extracted as feature points are included in the field of view.
[0050] 6. Go around the room once. When you return to the starting position and the path shows a loop, end the program and save the point cloud.
[0051] On-site point cloud structure acquisition results
[0052] In order to be as close to the crime scene environment as possible, in this embodiment, the study and living room in an ordinary residence are selected as the simulation experiment scene. The study and living room are places that often appear in real crime scenes, and both rooms are common square structures and are universal. Figure 3 , Figure 4 shown.
[0053] Figure 3The experimental scene 1 is shown. Scene 1 is a study room. Perspective 1 reflects the bookcase on the wall, the white desk next to the bookcase, the stool in front of the desk, and the blue room door next to the desk. Perspective 2 reflects the white sofa and the black table in front of the sofa. It can be seen that the reconstructed 3D point cloud basically reflects the bookcase, desk, door and other items and colors in the study room, with a high degree of restoration. Ideally, the wall should be uniformly white, but in reality, the wall material and surface are uneven, so the color will be uneven after ranging imaging. In perspective 2, since the D435i uses infrared structured light imaging, there will be unstable ranging for glass objects, so the messy transparent glass part of the bookcase in the lower right corner is reflected in Figure (b). The missing part on the door side is because the lower half of the desk is blocked by the stool, so the front view of the desk can obtain the complete structure, and there is a missing part on the back.
[0054] Figure 4 The experimental scene 2 is shown. The experimental scene 2 is a living room, which reflects the sofa, the cabinet close to one side, the white table and the gray room door on one side. It can be seen that the reconstructed 3D point cloud basically reflects the cabinets, sofas, doors and other items and colors in the living room. Ideally, the wall part should be uniformly white, but in reality, the wall material and surface are uneven, so there will be uneven color after ranging imaging. In perspective 1, because the sofa is placed against the wall in the scene, and the sofa is an opaque object higher than the ground, there is a missing in the lower left corner of (a); similarly, the missing in the lower left corner of (b) in perspective 2 is also caused by the placement of the sofa in the room, and because the D435i uses infrared structured light imaging, there will be unstable ranging for glass objects, so in (b) it is reflected as the messy transparent glass part of the bookcase in the lower right corner.
[0055] On-site point cloud segmentation based on JSIS3D:
[0056] Deep learning describes an algorithm that uses multi-layer neural networks to learn data. Deep learning is an algorithm developed on the basis of neural networks. Generally speaking, neural networks are divided into single-layer networks and deep networks. A single-layer neural network is a simple structure neural network that only contains input and output. The initial node directly outputs to the neuron as output. The deep network contains multiple hidden layers. As a type of deep neural network, it breaks through the limitations of traditional algorithms and continues the practice of sharing weights to reduce training parameters. It demonstrates a powerful ability to obtain good training results in a small sample size. The current application objects include text, voice, image, video, etc. Convolutional neural networks have local coupling, and the corresponding sparse response characteristics greatly reduce the dependence on the amount of training data in parameter learning.
[0057] JSIS3D Network Structure
[0058] The application of deep learning technology in two-dimensional vision has been relatively mature. In the field of three-dimensional scene understanding, although there have been attempts at PointNet-related networks, previous point cloud segmentation networks have performed poorly when faced with large-scale, noisy and multi-scene tasks. Among the related problems of three-dimensional scene understanding, semantic segmentation is described as identifying the category label or object category of each 3D point in the scene, while instance segmentation clusters the scene into object instances. Usually, these two problems are solved separately and instance segmentation is a post-processing task of semantic segmentation. However, in practice, the determination of the shape and appearance of object instances helps to determine the object category, so semantics and instances can be combined for application. JSIS3D (Joint Semantic-Instance Segmentation of 3D Point Clouds)
[74] is a joint semantic and instance 3D point cloud segmentation solution that uses a multi-task point-by-point network (MTPNet) and a multi-valued conditional random field (MV-CRF) to solve the segmentation problem. In terms of the multi-task point-by-point network, the network performs two tasks simultaneously: predicting the semantic category of 3D points in the point cloud and embedding these points into high-dimensional vectors so that points of the same object instance are represented by similar embeddings. Based on the multi-task point-by-point network, JSIS3D proposes a multi-value conditional random field model that combines semantic and instance labels.
[0059] The segmentation problem is formulated as a joint optimization label in the field model. Experimental results show that JSIS3D has good segmentation effect and good robustness for different datasets.
[0060] In the input stage, for a given point cloud, we first scan the entire point cloud through overlapping 3D windows, obtain each window and its associated 3D vertices and pass them into the multi-task point-by-point network for the next stage.
[0061] The multi-task point-by-point network is proposed based on the point network feed-forward architecture proposed by Qi et al. Specifically, for an input point cloud of size N, a feature map of size N×D is first calculated, where D is the feature dimension of each point. The network is divided into two different branches, performing two tasks: predicting the semantic labels of 3D points and creating their point-by-point instance embeddings. Therefore, the multi-task point-by-point network loss is expressed as the sum of the losses of the two network branches.
[0062] In terms of embedding functions, JSIS3D uses a joint discriminant function to define the embedding function, hoping to make the three-dimensional points belonging to the same object instance pull each other and maximize the difference between different object instances.
[0063] The multi-valued conditional random field part associates points in the point cloud with semantic and instance labels.
[0064] The energy function involves many potential functions, and contains physical constraints (surface smoothness, geometric proximity) and semantic constraints (shape consistency between object classes and instances, etc.) in semantic and instance-specific operations. Minimizing the energy function is equivalent to maximizing the posterior condition. JSIS3D uses the mean field variation method to solve the optimization problem.
[0065] S3DIS dataset:
[0066] S3DIS is the product of a joint project at Stanford University. The project uses a hierarchical approach to perform semantic analysis on 3D point clouds of large-scale indoor buildings. The data set collects more than 6,000 square meters of indoor buildings, including more than 200 rooms in 6 large scenes, with a total of 215 million point clouds, each of which specifies semantic segmentation categories and instance categories: First, the original data is parsed into semantically meaningful spaces (such as rooms, etc.), which are aligned to a standardized reference coordinate system. Secondly, the space is parsed into its structural and architectural elements, so that when the appearance of the building changes dramatically, application migration can still be achieved through common elements shared by the room. In the process of establishing the specific data set, the researchers used Matterport cameras and combined them with 3 structured light sensors with different spacings to perform a full scan at each scanning position, and each point position can capture 18 color and depth images. Among them, each full scan angle is superimposed one by one, and each position provides 6 triplet color and depth data. Based on the scanning results, data such as 3D texture meshes, original color images and depth images are generated, and room point clouds are made through grid sampling, and semantic labels are added to each point in the point cloud. S3DIS has promoted the development of indoor point cloud segmentation with its huge data volume and precise annotation.
[0067] JSIS3D training and on-site point cloud segmentation results:
[0068] Network parameter settings:
[0069] The parameters of the network need to be set based on experience, and the quality of parameter selection has a direct impact on the training performance and efficiency of the network. The learning rate refers to the speed at which the weights are updated after back propagation. It determines whether the objective function can converge to the local minimum and the convergence time. When the learning rate is too large, the gradient may exceed the optimal value. If it is too small, the convergence speed will be reduced and the training time will be extended. Referring to the original setting of JSIS3D, the initial value of the learning rate in this embodiment is 0.01, and the weight decay coefficient is set to 0.0005. The setting of this parameter is to avoid overfitting of the model during the weight update process. In addition, if the number of training iterations is too small, the model training will be insufficient, and if the number of iterations is too large, it will lead to overfitting. When epoc is 60, the model loss value is reduced to about 0.15 and remains relatively stable.
[0070] On-site point cloud segmentation experiment
[0071] In the network segmentation result display part, Open3D is introduced to determine the color of the objects represented by the semantic categories in the segmentation point cloud, and the format is RGB. The predicted label category is determined in sequence according to the semantic categories in S3DIS, and the color list data is specified to correspond to the semantic categories in sequence. The label color can be changed as needed. According to the semantic label color, the corresponding object category of the area can be determined according to the segmentation result color display.
[0072] Figure 5 This is a comparison of scene 1 before and after segmentation. Experimental scene 1 is a study room. The upper picture shows the bookcase on the wall, the desk next to the bookcase, the stool in front of the desk, and the room door next to the desk; the lower picture shows the sofa and the table in front of the sofa. In the segmentation results, the bookcases, doors, walls, and paintings on the walls with fixed structures and relatively complete structures are relatively complete. (a) The sundries and curtains on the desk on the door side of the picture and (b) the computer and curtains on the table on the window side of the picture cannot be segmented due to the lack of corresponding data set training. (b) The missing part on the door side of the picture is because the lower half of the desk is blocked by the stool. Therefore, the front view of the desk can obtain the complete structure, but the back view is missing. In the segmentation, due to the existence of scattered points and partial missing when obtaining the point cloud, it appears as black particles in the segmentation result.
[0073] Figure 6 This is a comparison before and after the segmentation of scene 2. Experimental scene 2 is a living room, which reflects the sofa, the cabinet next to one side, the table and the room door on one side. In the segmentation results, the common bookcases, doors, walls, wall paintings, sofas, etc. with fixed structures and relatively complete structures are relatively complete, but the sundries on the desk next to the door and the sundries on the cabinet next to the wall cannot be identified, and due to the lack of corresponding data set categories, the curtains are identified as walls. Since the sofa is placed against the wall in the scene and the sofa is an opaque object above the ground, there is a missing in the lower left corner of (a). Similarly, the missing in the lower left corner of (b) is also determined by the placement of the sofa in the room, and since the D435i uses infrared structured light imaging, there will be unstable ranging for glass objects, so it is reflected in (b) as the messy situation of the glass part of the bookcase in the lower right corner.
[0074] Method for generating site plan based on 3D point cloud
[0075] Point cloud projection
[0076] Point cloud is a group of points in three-dimensional space that describes spatial color, position and other information. The ultimate goal of this paper is to achieve the generation of site plan. In field survey work, site plan is mostly expressed in the form of bird's-eye view, that is, the ground plane is the plane of the site space, and the ground projection is formed by the light perpendicular to the ground falling on the ground. Figure 7 shown.
[0077] The blue plane represents the plane where the ground is located in the space, the pink cube represents the object in the space, and the yellow line represents the light perpendicular to the ground plane. The projection of the object in the space on the ground plane is based on the yellow light. As shown in the figure, the bird's-eye view can fully reflect the plane structure of the space.
[0078] Projection is to project the point cloud onto the plane with the ground as the plane to obtain the structure. In the interception method, the point cloud is intercepted at a specified height with a plane parallel to the ground as the reference. In an ideal room structure, the upper and lower structures are consistent and the height of the objects are uniform. Projection or direct interception can obtain the complete room structure. However, when there are objects with inconsistent upper and lower structures such as chairs and tables in the room, interception cannot obtain the complete structure. Figure 8 As shown:
[0079] The blue plane represents the plane where the ground is located in the space, and the pink cube represents the objects in the space. The object on the left can represent the common objects with inconsistent upper and lower structures in the room, such as chairs, and the cube on the right can represent the objects with consistent upper and lower structures in the room, such as bookcases. The gray plane represents the interception plane parallel to the ground. In common cases, the heights of objects in the room are different. The direct interception method will result in the loss of some objects and it is impossible to obtain a true and complete room structure diagram.
[0080] Based on the projection method, the plane structure is obtained. Fig. 9 As shown:
[0081] The blue plane represents the plane where the ground is located in the space, and the pink cube represents the objects in the space. The object on the left can represent the common objects with inconsistent upper and lower structures in the room, such as chairs, and the cube on the right can represent the objects with consistent upper and lower structures in the room, such as bookcases. The yellow line indicates the projection direction. In the projection-based method, it is not affected by the shape, height, and consistency of the upper and lower structures of the objects in the space, and can reflect the room structure more completely and comprehensively. Therefore, this experiment adopts the projection-based method.
[0082] PCL (Point Cloud Library) is an independent, large-scale processing project based on point cloud generation. It is mainly used for the development and utilization of two-dimensional and three-dimensional images including point clouds. The content covers point cloud filtering, point cloud features, key points, point cloud marking, segmentation, foreground detection, surface texture, point cloud visualization, etc. Using PCL related functions, it is relatively easy to realize the point cloud plane projection task.
[0083] In the first frame, the xz plane of the point cloud center is set to be parallel to the ground, and the y axis is perpendicular to the ground. Since the final generated plan needs to remove the large area of floor background interference, the point cloud filtering function pass.setFilterLimits is used to set the filter range to (0.7, 1.0), remove the ground point cloud, and compress the y axis coordinate to 0. At this time, the point cloud only contains xz axis information. The screenshot can be saved to obtain the plane structure with background removal and dimensionality reduction. The experimental flow chart is as follows: Fig.10 shown.
[0084] The results of background removal and dimensionality reduction of the room structure point cloud are as follows Fig.11 , Fig.13 shown.
[0085] The plane structure diagram obtained from the experimental scene is as follows: Fig.12 , Fig.14 As shown in the figure, after background removal and dimensionality reduction operations, the large area of floor interference in the room is removed, and a relatively complete bird's-eye view of the room is obtained. The JSIS3D network presets the corresponding color value of the object to determine the location of the object in the room. Ideally, the thickness of the yellow wall should be 0, but in reality, the uneven reflection of light caused by the uneven wall leads to the thickness of the wall in the reconstruction result, so the wall part still appears in the segmentation and projection results.
[0086] Site plan generation
[0087] The site plan is surrounded by simple lines such as rectangular frames, reflecting information such as room structure and item location. After obtaining the background-removed bird's-eye view of the room in 4.3.1, it is necessary to use the relevant functions in OpenCV to extract the room contour based on the segmented color blocks. Based on the obtained contour, a rectangular bounding box is drawn to generate the site plan.
[0088] Contour Extraction Based on Sobel Operator
[0089] The image contour can be extracted by edge detection algorithm, such as Fig.15 As shown, the irregular figure E is an arbitrary figure on the pixel plane, Figure (a) represents the unprocessed original image, and Figure (b) represents the image contour extracted by the edge detection algorithm;
[0090] Rectangular bounding box drawing
[0091] After obtaining the outline of each part of the bird's-eye view, you need to draw the corresponding circumscribed rectangular bounding box. The operation is as follows: Fig.18 As shown. The irregular figure E is an arbitrary figure on the pixel plane. Figure (a) shows the image after the contour extraction operation, and Figure (b) shows the circumscribed rectangular bounding box ABCD of the generated contour, which can be completed by calling the OpenCV functions cv.boundingRect and cv.drawContours.
[0092] The specific process is as follows:
[0093] 1. Input image contour E.
[0094] 2. Call the cv.boundingRect function. Detect the minimum coordinate points in the x-axis and y-axis directions and output the coordinates; that is, obtain the coordinates (x, y) of point B in the upper left corner of the figure. Search for the maximum coordinate point along the x-axis direction. If the corresponding y-axis coordinate value of this point is also the largest in the figure, then point D is the lower right corner of figure A. If it is not the largest, search for the maximum coordinate point along the y-axis and output it as point D. Then you can get the width w and height h of the figure.
[0095] 3. Call the cv2.drawContours function. In the circumscribed rectangle ABCD, the coordinates of point B are (x+w, y), the coordinates of point C are (x+w, y+h), and the coordinates of point D are (x, y+h). After obtaining the coordinates of points ABCD, the corresponding rectangle can be quickly drawn based on the cv2.drawContours function.
[0096] 4. Output the image.
[0097] Compared with the related art, the method for generating a scene plan view based on a depth camera provided by the present invention has the following beneficial effects:
[0098] The present invention provides a method for generating a site plan based on a depth camera, using a point cloud segmentation network JSIS3D network structure and an open source data set S3DIS, setting parameters based on the data set and completing the training of the point cloud segmentation network. On this basis, the room point cloud structure obtained by ORB-SLAM2 is segmented, and relatively accurate on-site object information is obtained, laying the foundation for obtaining the site plan. Subsequently, a method for obtaining a plane structure based on projection and interception is introduced in detail. After obtaining the room point cloud segmentation result, the PCL point cloud library is called to remove the background and projection dimension reduction of the point cloud perpendicular to the plane, and the room plane structure is obtained. Subsequently, the edge detection algorithm of the Sobel operator is introduced, and the contour of the obtained room bird's-eye view is extracted, and then the shape coordinate width and height function and the contour drawing function are obtained in OpenCV, and finally the automatic generation of the site plan is realized. By comparing the generated point cloud scale with the actual measurement result, the generated site map with the actual measurement result, the possible sources of error in the whole process are analyzed in detail, proving that the method has a certain accuracy.
[0099] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A method for generating a site plan view based on a depth camera, characterized in that: The following steps are involved: S1: Use depth camera to obtain 3D point cloud on site; S2: Train the point cloud segmentation model and use the model to segment the target of the 3D point cloud at the scene; Using JSIS3D to perform on-site point cloud segmentation includes the following steps: S21: Use multi-layer neural network algorithm to learn data; S22: Solve the segmentation problem using multi-task point-by-point networks and multi-valued conditional random fields. In terms of multi-task point-by-point networks, the network performs two tasks simultaneously: predicting the semantic categories of 3D points in the point cloud and embedding these points into high-dimensional vectors so that points of the same object instance are represented by similar embeddings; Based on the multi-task point-by-point network, JSIS3D proposes a multi-valued conditional random field model that combines semantic and instance labels together, and formulates the semantic and instance segmentation problem as a joint optimization label in the field model; S23: Parse the raw data into a semantically meaningful space and align the space into a canonical reference coordinate system; S3: Generate site plan from 3D point cloud; In S1, a depth camera is used to acquire a three-dimensional point cloud on site, including the following steps: S11: Fill in the depth camera calibration parameters; Input the calibration parameters and alignment results into SLAM; in the ORB-SLAM2 workspace, create a new configuration file named realsense.yaml corresponding to the camera parameter file given by the initial ORB-SLAM2; fill in the camera internal parameter data after D435i calibration in the configuration file according to the parameter format; S12: Acquisition of on-site point cloud; In S3, the three-dimensional point cloud is used to generate a site plan, including the following steps: S31: perform point cloud projection; S32: Generate a site plan; S33: Use Sobel operator to extract contour; S35: Draw the image using a rectangular bounding box.
2. The method for generating a site plan view based on a depth camera according to claim 1, characterized in that: In S12, the acquisition of the on-site point cloud includes the following steps: S121: Install the camera and micro-trip, enter the Ubuntu system, and open the terminal in the ORB-SLAM2 folder; S122: Enter the command in the terminal, run the launch file, and start the D435i node; S123: Open a new terminal, enter commands in the new terminal, and start ORB-SLAM2 in depth camera mode; S124: The program reports no error. Keep the camera micro-tripod perpendicular to the ground, the camera plane parallel to the ground, hold the micro-tripod, start moving from one position in the room, and rotate slowly and at a constant speed at the turning point of the view to ensure that continuous feature points always appear in the picture and the whole room is completely scanned; S125: When the camera lens is about to pass through the wall, it moves back to ensure that other objects that can be extracted as feature points are included in the field of view; S126: Go around the room once, and when it returns to the starting position and the path shows a loop, end the program and save the point cloud.
