Three-dimensional plane reconstruction method, three-dimensional plane reconstruction system and computing device

Abstract

A three-dimensional plane reconstruction method, a three-dimensional plane reconstruction system and a computing device. The three-dimensional plane reconstruction method comprises the following steps: a sensing device is used to continuously obtain color information, depth information and orientation information of a dynamic scene. A plurality of feature points are calculated based on the color information and depth information, and non-planar objects are marked. Non-planar objects include dynamic objects and fragmentary objects. A point cloud is calculated based on the feature points and orientation information of non-planar objects that are not marked as non-planar objects, and the point cloud is converted into a three-dimensional grid in real time. Based on the three-dimensional grid information surrounding or adjacent to the non-planar object, the three-dimensional grid is expanded to fill the vacant part corresponding to the non-planar object.

Description

Technical Field

[0001] The invention relates to a three-dimensional plane reconstruction method, a three-dimensional plane reconstruction system and a computing device. Background Art

[0002] Generally, a mobile camera module is used to scan a scene to establish a 3D scene model. However, when scanning a scene, people often move around, and thus a correct 3D scene model cannot be reconstructed.

[0003] In many applications, only planar structures in a 3D scene need to be reconstructed without including cluttered small objects or people.

[0004] The amount of point cloud data generated by scanning a scene using a general method is very large, and all point clouds must be obtained before they can be converted into a mesh, which will take up a lot of memory and processor resources and cannot be effectively applied to head-mounted devices.

[0005] Therefore, researchers are committed to developing a new technology for scanning and reconstructing three-dimensional planar structures in dynamic scenes to reduce the memory and processor resources required for calculations and improve processing efficiency and quality. Summary of the invention

[0006] The present invention relates to a 3D plane reconstruction method, a 3D plane reconstruction system and a computing device, which combine color information, depth information and feature point analysis to remove dynamic objects and fragmentary objects to accurately obtain a 3D plane structure. After obtaining a local point cloud, it can be converted into a 3D grid in real time to reduce the memory required to store the point cloud. At the same time, the 3D grid is continuously updated according to the newly generated point cloud to improve processing efficiency.

[0007] According to one embodiment of the present invention, a three-dimensional plane reconstruction method is proposed, comprising the following steps: continuously obtaining color information, depth information and orientation information of a dynamic scene by a sensing device. Based on the color information and the depth information, a plurality of feature points are calculated, and non-planar objects are marked. Non-planar objects include dynamic objects and fragmentary objects. Point cloud information is calculated based on feature points that are not marked as non-planar objects and the orientation information, and the point cloud information is converted into a three-dimensional grid in real time. Based on the three-dimensional grid information surrounding or adjacent to the non-planar object, the three-dimensional grid is expanded to fill the vacant part corresponding to the non-planar object.

[0008] According to one embodiment of the present invention, a three-dimensional plane reconstruction system is proposed, including: a sensing device and a computing device. The sensing device is used to continuously obtain color information, depth information and orientation information of a dynamic scene. The computing device includes a feature point calculation unit, a non-planar object marking unit, a grid calculation unit and a grid filling unit. The feature point calculation unit is used to calculate a plurality of feature points based on color information and depth information. The non-planar object marking unit is used to mark non-planar objects, including dynamic objects and fragmentary objects. The grid calculation unit is used to calculate point cloud information based on feature points and orientation information that are not marked as non-planar objects, and to convert the point cloud information into a three-dimensional grid in real time. The grid filling unit expands the three-dimensional grid based on the three-dimensional grid information surrounding or adjacent to the non-planar object to fill the missing part corresponding to the non-planar object.

[0009] According to one embodiment of the present invention, a computing device is provided. The computing device includes a feature point computing unit, a non-planar object marking unit, a grid computing unit, and a grid filling unit. The feature point computing unit is used to calculate a plurality of feature points based on color information and depth information. The non-planar object marking unit is used to mark non-planar objects, including dynamic objects and fragmentary objects. The grid computing unit is used to calculate point cloud information based on feature points and orientation information that are not marked as non-planar objects, and to convert the point cloud information into a three-dimensional grid in real time. The grid filling unit is used to expand the three-dimensional grid based on the three-dimensional grid information surrounding or adjacent to the non-planar object to fill the missing part corresponding to the non-planar object.

[0010] In order to better understand the above and other aspects of the present invention, embodiments are listed below and described in detail with reference to the accompanying drawings as follows: BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A schematic diagram of a system for reconstructing a three-dimensional plane of a dynamic scene according to an embodiment is shown;

[0012] Figure 2 A flowchart of a method for 3D plane reconstruction of a dynamic scene according to an embodiment is shown;

[0013] Figure 3 A partial flow chart of a dynamic object marking procedure according to an embodiment is shown;

[0014] Figures 4-5 A schematic diagram of an object according to an embodiment is shown;

[0015] Figure 6 A partial flow chart of a marking procedure for fragmentary objects according to an embodiment is shown;

[0016] Figure 7 Draw Figure 2 A partial flow chart of step S103;

[0017] Figures 8 to 11 Draw Figure 7 Example diagrams of each step;

[0018] Figures 12-13 Draw Figure 2 Schematic diagram of step S104.

[0019] [Description of Reference Numerals]

[0020] 1000: 3D Plane Reconstruction System

[0021] 100: Sensing device

[0022] 110: Color Camera

[0023] 120: Depth Camera

[0024] 130: Position sensor

[0025] 200: Computing Device

[0026] 210: Feature point calculation unit

[0027] 220: Non-planar object marking unit

[0028] 221: Repetition Level Recorder

[0029] 222: Dynamic Object Marker

[0030] 223: Size Calculator

[0031] 224: Fragmentary Object Marker

[0032] 230: Grid computing unit

[0033] 240: Grid filling unit

[0034] CP1, CP2: point cloud information

[0035] F1: Feature points

[0036] F N-2 、F N-1 、F N :image

[0037] MS1, MS2, MS3: 3D mesh

[0038] O1: Feature points of non-dynamic objects

[0039] O2: Non-dynamic and non-fragmentary object feature points

[0040] OB1: Moving Objects

[0041] S1: Color information

[0042] S2: Depth Information

[0043] S3: Position information

[0044] S4: Trustworthiness Information

[0045] S5: Three-dimensional dimensions

[0046] S6: 3D Grid

[0047] S101, S102, S1021, S1022, S1023, S1024, S103, S1031, S1032, S1033, S104: Steps DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0049] Please refer to Figure 1 , which depicts a schematic diagram of a three-dimensional plane reconstruction system 1000 according to an embodiment. The three-dimensional plane reconstruction system 1000 includes a sensing device 100 and a computing device 200. The sensing device 100 includes a color camera 110, a depth camera 120, and an orientation sensor 130. The color camera 110 is, for example, a video camera or a camera with a continuous shooting function. The depth camera 120 is, for example, a dual camera, a structured light depth camera, or a time of flight (ToF) depth sensor. The orientation sensor 130 can sense at least three-axis rotation direction and three-axis movement position, for example, it is composed of one or more wide-angle or fisheye cameras and an inertial measurement unit (IMU). The IMU is, for example, a multi-degree of freedom (DOF) micro-electromechanical device (MEMS), which is composed of a combination of gyroscopes, accelerometers, magnetometers, etc.

[0050] like Figure 1As shown, the computing device 200 includes a feature point calculation unit 210, a non-planar object marking unit 220, a grid calculation unit 230 and a grid filling unit 240. The computing device 200 is, for example, a computer, a server, a cloud computing center, a processor, a chip, a circuit and the like in a head-mounted device. The feature point calculation unit 210 calculates a plurality of feature points based on color information and depth information. The non-planar object marking unit 220 includes a repetition degree recorder 221, a dynamic object marker 222, a size calculator 223 and a fragmentary object marker 224. The repetition degree recorder 221 compares a plurality of images obtained continuously, calculates the reliability of each feature point, and then uses the dynamic object marker 222 to exclude dynamic objects and retain the non-dynamic object feature point O1. The size calculator 223 calculates the three-dimensional size of the object based on the color information and depth information (for example, depth maps), and then uses the fragmentary object marker 224 to exclude fragmentary objects and retain the non-fragmentary object feature points. The grid calculation unit 230 then calculates point clouds based on the feature points O2 and orientation information (e.g., yaw / roll / pitch rotation amount and direction, XYZ movement position, etc.) of these non-dynamic and non-fragmentary objects, and converts them into corresponding three-dimensional grids in real time. For the missing parts of dynamic objects and fragmentary objects, the grid filling unit 240 expands the three-dimensional grid to fill them based on the three-dimensional grid information surrounding or adjacent to the vacant area to establish a complete three-dimensional plane structure. In this embodiment, the feature point calculation unit 210, the non-planar object marking unit 220, the grid calculation unit 230 and the grid filling unit 240 are processed using continuous update technology to reduce the memory and processor resources required for calculations. The following is also accompanied by a flow chart to explain in detail the operation of the above-mentioned components.

[0051] Figure 2 A flow chart of a three-dimensional plane reconstruction method according to an embodiment is depicted. In step S101, the color information S1, depth information S2 and orientation information S3 of a dynamic scene are continuously obtained by using the sensing device 100. In this step, the color information S1, depth information S2 and orientation information S3 are continuously obtained from multiple images. For example, the user wears a head-mounted device equipped with the sensing device 100 and extracts images while walking to continuously obtain multiple images. The so-called dynamic scene in this embodiment refers to a scene with moving objects, such as walking people. The color information S1, depth information S2 and orientation information S3 of each image correspond one to one, and the changes of dynamic objects in the scene can be observed through the relationship between these information.

[0052] Next, in step S102, the feature point calculation unit 210 calculates a plurality of feature points F1 based on the color information and the depth information, and the non-planar object marking unit 220 marks the non-planar objects (including dynamic objects and fragmentary objects). In this step, the non-planar object marking unit 220 compares a plurality of consecutive color information, depth information and corresponding feature points for a period of time, marks the feature points in the picture that are determined to be dynamic objects or fragmentary objects and deletes them. Dynamic objects are, for example, people and vehicles at work; fragmentary objects are, for example, messy stationery. The following also illustrates how to mark dynamic objects and fragmentary objects through a partial flowchart.

[0053] Please refer to Figure 3 , which illustrates a partial flow chart of a dynamic object marking procedure according to one embodiment. Figure 2 The step S102 includes steps S1021 and S1022. In step S1021, the repetition degree recorder 221 compares the positions and the repetition times of the feature points F1 to obtain a plurality of reliability information S4 of the feature points F1.

[0054] For example, see Figures 4-5 , which is a schematic diagram of determining the reliability of feature points according to an embodiment. Figure 4 It shows that in the absence of interference from the moving object OB1, the feature points of the static scene can be obtained from multiple consecutive images F N-2 、F N-1 、F N Obtained, its reliability is high (image F N-2 、F N-1 、F N The feature points in will not be blocked by the moving object OB1). Figure 5 It shows that if there is a moving object OB1, its corresponding feature point will not stably appear in multiple consecutive images F N-2 、F N-1 、F N and will interfere with the feature points of static scenes, so its reliability is low (for example, image F N-2 The feature points in are blocked by the moving object OB1).

[0055] In step S1022 , the dynamic object marker 222 marks and deletes the feature points F1 with a value less than the reliability threshold as dynamic objects based on the reliability information S4 , and retains the non-dynamic object feature points O1 to mark and delete the fragmentary objects.

[0056] Please refer to Figure 6 , which illustrates a partial flow chart of a fragmentary object labeling procedure according to one embodiment. Figure 2Step S102 also includes steps S1023 and S1024. In step S1023, the size calculator 223 cuts the object according to the color information S1 and the depth information S2 corresponding to the feature point O1 of the non-dynamic object, and calculates its three-dimensional size S5. For example, the color or texture obvious change can be found in the color information S1 and temporarily determined as the edge of the object, and then the corresponding discontinuous boundary of the depth information S2 can be found to segment the object and estimate its three-dimensional size S5.

[0057] In step S1024, the fragmentary object marker 224 marks the feature point F1 whose three-dimensional size S5 is smaller than the size threshold as a fragmentary object and deletes it. After steps S1022 and S1024, the non-dynamic and non-fragmentary object feature point O2 is retained, and the point cloud and mesh are calculated.

[0058] Feature points F1 marked as dynamic objects or fragmentary objects will be excluded and will not be used to create / modify the 3D mesh. Figure 2 In step S103, the mesh calculation unit 230 uses the non-dynamic and non-fragmentary object feature points O2 and their orientation information S3 to calculate point cloud information, and instantly converts the point cloud information into a three-dimensional mesh S6.

[0059] Please refer to Figures 7 to 11 , Figure 7 Draw Figure 2 A local flow chart of converting point cloud information into a three-dimensional mesh in step S103, Figures 8 to 11 Draw Figure 7 In step S1031, as shown in FIG. Figures 8-9 As shown, the grid computing unit 230 deletes the point cloud information CP1 located in the established three-dimensional grid MS1, which can reduce the computing resources for processing these point clouds without affecting the structure of the three-dimensional grid.

[0060] In step S1032, if Figures 9-10 As shown, the grid calculation unit 230 adds a three-dimensional grid MS2 according to the point cloud information CP2 outside the three-dimensional grid MS1.

[0061] Next, in step S1033, Figures 10-11 As shown, the grid calculation unit 230 compares the direction vector of the newly added three-dimensional grid MS2 with the direction vector of the adjacent established three-dimensional grid MS1. If the difference in the direction vectors is less than the difference threshold, they are merged into the three-dimensional grid MS3 to reduce the number of grids.

[0062] Next, explain Figure 2 Step S104, please refer to Figures 12-13In step S104, the mesh filling unit 240 expands the three-dimensional mesh according to the three-dimensional mesh S6 surrounding or adjacent to the non-planar object (e.g., a dynamic object) to fill the vacant portion corresponding to the non-planar object (e.g., a dynamic object). In this step, the difference in direction vector between the filled three-dimensional mesh and the three-dimensional mesh adjacent to or surrounding the non-planar object (e.g., a dynamic object) is less than the difference threshold.

[0063] In this way, there is no need to store a large amount of point cloud information during the calculation process, and only a small amount of memory and processor resources are needed to establish a three-dimensional plane structure. In summary, the three-dimensional plane reconstruction method, three-dimensional plane reconstruction system and computing device of the present invention combine the analysis of color information, depth information and feature points to remove dynamic objects and fragmentary objects to accurately obtain a three-dimensional plane structure. And after obtaining a local point cloud, it can be converted into a three-dimensional grid in real time to reduce the memory required to store the point cloud. At the same time, the three-dimensional grid is continuously updated according to the newly generated point cloud to improve processing efficiency.

[0064] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. A person skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the claims.

Claims

1. A three-dimensional plane reconstruction method, comprising: Continuously obtain color information, depth information and orientation information of dynamic scenes using a sensing device; Calculating a plurality of feature points according to the color information and the depth information, and marking non-planar objects, wherein the non-planar objects include dynamic objects and fragmentary objects; Calculating point cloud information from which the non-planar object has been removed based on the feature points not marked as the non-planar object and the orientation information, and instantly converting the point cloud information from which the non-planar object has been removed into a three-dimensional grid; as well as According to the three-dimensional mesh information surrounding or adjacent to the non-planar object, the three-dimensional mesh is enlarged to fill the vacant part corresponding to the non-planar object; The steps of calculating the point cloud information from which the non-planar object has been removed based on the feature points not marked as the non-planar object and the orientation information, and instantly converting the point cloud information from which the non-planar object has been removed into a three-dimensional grid include: Deleting the point cloud information located in the established three-dimensional grid; Adding a three-dimensional grid based on the deleted point cloud information; and The direction vectors of the newly added three-dimensional grids are compared with the direction vectors of the established three-dimensional grids, and the three-dimensional grids whose direction vector differences are less than a difference threshold are merged.

2. The three-dimensional plane reconstruction method as claimed in claim 1, wherein the sensing device comprises a color camera, a depth camera and an orientation sensor.

3. The three-dimensional plane reconstruction method as claimed in claim 1, wherein the step of calculating the feature points based on the color information and the depth information and marking the non-planar object comprises: Compare the positions and the number of repetitions of the feature points to obtain multiple reliability information of the feature points; as well as According to the reliability information of the feature points, part of the feature points are marked as the dynamic object, and the reliability information of the feature points corresponding to the dynamic object is less than the reliability threshold value.

4. The three-dimensional plane reconstruction method as claimed in claim 1, wherein the step of calculating the feature points based on the color information and the depth information and marking the non-planar object comprises: Calculating the three-dimensional size of the object according to the color information and the depth information; as well as According to the three-dimensional size, some of the feature points are marked as the fragmentary objects, and the three-dimensional size of the fragmentary objects is smaller than the size threshold.

5. The three-dimensional plane reconstruction method as described in claim 1, wherein in the step of augmenting the three-dimensional grid with the three-dimensional grid information surrounding or adjacent to the non-planar object to fill the missing part corresponding to the non-planar object, the difference in direction vector between the filled three-dimensional grid and the three-dimensional grid adjacent to or surrounding the non-planar object marked as the non-planar object is less than a difference threshold value.

6. A three-dimensional plane reconstruction system, comprising: A sensing device for continuously acquiring color information, depth information and orientation information of a dynamic scene; as well as A computing device comprising: A feature point calculation unit, used for calculating a plurality of feature points according to the color information and the depth information; A non-planar object marking unit, used to mark non-planar objects, including dynamic objects and fragmentary objects; A mesh calculation unit, used for calculating point cloud information from which the non-planar object has been removed based on the feature points not marked as the non-planar object and the orientation information, and instantly converting the point cloud information from which the non-planar object has been removed into a three-dimensional mesh; and A mesh filling unit, used for expanding the three-dimensional mesh according to the three-dimensional mesh information surrounding or adjacent to the non-planar object, so as to fill the vacant part corresponding to the non-planar object; The grid calculation unit deletes the point cloud information within the established three-dimensional grid; adds a three-dimensional grid based on the deleted point cloud information; compares the direction vector of the newly added three-dimensional grid with the direction vector of the established three-dimensional grid, and merges the three-dimensional grids whose direction vector differences are less than a difference threshold value. 7 . The three-dimensional plane reconstruction system as claimed in claim 6 , wherein the sensing device comprises a depth color camera, a depth camera and an orientation sensor.

8. The three-dimensional plane reconstruction system as claimed in claim 6, wherein the non-planar object marking unit comprises: A repetition degree recorder is used to compare the positions and repetition times of the feature points to obtain multiple reliability information of the feature points; as well as The dynamic object marker is used to mark part of the feature points as the dynamic object according to the reliability information of the feature points, and the reliability information of the feature points corresponding to the dynamic object is less than the reliability threshold value.

9. The three-dimensional plane reconstruction system as claimed in claim 6, wherein the non-planar object marking unit comprises: A size calculator, used for calculating the three-dimensional size of the object according to the color information and the depth information; as well as The fragmentary object marker is used to mark some of the feature points as the fragmentary objects according to the three-dimensional size, and the three-dimensional size of the fragmentary object is smaller than the size threshold value. 10 . The three-dimensional plane reconstruction system as claimed in claim 6 , wherein a difference in direction vector between the three-dimensional mesh filled by the mesh filling unit and the three-dimensional mesh adjacent to or surrounding the non-planar object is less than a difference threshold.

11. A computing device comprising: A feature point calculation unit, used for calculating a plurality of feature points according to the color information and the depth information; A non-planar object marking unit, used to mark non-planar objects, including dynamic objects and fragmentary objects; A mesh calculation unit, used for calculating point cloud information from which the non-planar object has been removed based on the feature points and the orientation information that are not marked as the non-planar object, and instantly converting the point cloud information from which the non-planar object has been removed into a three-dimensional mesh; as well as A mesh filling unit, used for expanding the three-dimensional mesh according to the three-dimensional mesh information surrounding or adjacent to the non-planar object, so as to fill the vacant part corresponding to the non-planar object; The grid calculation unit deletes the point cloud information within the established three-dimensional grid; adds a three-dimensional grid based on the deleted point cloud information; and compares the direction vector of the newly added three-dimensional grid with the direction vector of the established three-dimensional grid, and merges the three-dimensional grids whose direction vector differences are less than a difference threshold value.

12. The computing device of claim 11, wherein the non-planar object marking unit comprises: A repetition degree recorder is used to compare the positions and repetition times of the feature points to obtain multiple reliability information of the feature points; as well as The dynamic object marker is used to mark part of the feature points as the dynamic object according to the reliability information of the feature points, and the reliability information of the feature points corresponding to the dynamic object is less than the reliability threshold value.

13. The computing device of claim 11, wherein the non-planar object marking unit comprises: A size calculator, used for calculating the three-dimensional size of the object according to the color information and the depth information; as well as The fragmentary object marker is used to mark some of the feature points as the fragmentary objects according to the three-dimensional size, and the three-dimensional size of the fragmentary object is smaller than the size threshold value. 14 . The computing device of claim 11 , wherein a difference in direction vector between the three-dimensional mesh filled by the mesh filling unit and a three-dimensional mesh adjacent to or surrounding the non-planar object is less than a difference threshold.

Images