Method for creating three-dimensional model data

Abstract

To provide a method for creating three-dimensional model data by synthesizing point group data and photographic data.SOLUTION: A method includes: preparing an optical device 10 which projects a laser beam on a measurement object in an optical path passing a mechanical origin 18, and guides reflected laser along the optical path; acquiring point group data at a laser irradiation position by using reflected laser; acquiring photographic data on a measurement object from light that is incident via an origin; and creating model data by using the point group data and the photographic data. Further, the method includes: synthesizing point group data acquired at a plurality of measurement positions; synthesizing photographic data acquired at the plurality of measurement positions; dividing the synthesized photographic data on the basis of the synthesized photographic data and instrument information on the optical device when acquiring the photographic data at each of the plurality of measurement positions; associating the synthesized point group data and the divided image data, according to position information on each of the plurality of divided image data pieces; generating a mesh model from the synthesized point group data; and generating model data by pasting the divided image data on the mesh model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a method for creating high-quality three-dimensional model data by combining point cloud data and photographic data. [Background technology]

[0002] In point cloud data acquired by a three-dimensional laser scanner (3DLS), each point that makes up the point cloud has three-dimensional coordinates. Therefore, by synthesizing point cloud data stereoscopically (three-dimensionally), it is possible to reproduce a subject from any viewpoint (position and angle). However, point clouds do not have as high a quality as photographic images taken with a high-quality camera. For this reason, a method has been proposed to create high-quality model data by synthesizing point cloud data with photographic data.

[0003] To create high-quality model data, point cloud data obtained by measuring the subject using a 3D LLS and high-quality photographic data (RGB data) obtained by photographing the subject from various positions and angles using a digital camera are used.

[0004] Specifically, a conventional procedure for creating high-quality model data will be described with reference to FIGS.

[0005] Step 1 (3DLS measurement) A 3DLS is used to measure a subject and obtain point cloud data of the subject (S1). During measurement, the 3DLS is moved to and set up at multiple measurement positions, and the subject is photographed from each measurement position (angle). For example, when measuring the exterior structure of a building, the 3DLS is moved to and set up at multiple measurement positions around the building, and the exterior of the building is measured from each angle. Similarly, when measuring the internal structure of a building, the 3DLS is moved to and set up at multiple measurement positions inside the building, and the interior (walls, ceiling, floor, interior, etc.) is measured from each angle. In this way, the three-dimensional coordinates of each part of the building are obtained.

[0006] Step 2 (Processing point cloud data) Using point cloud data processing software (for example, the point cloud data processing software "Trimble RealWorks" provided by Nikon-Trimble Corporation), the point cloud data photographed at each measurement position is stitched together to create composite point cloud data (S2). The composite point cloud data obtained by measuring the subject contains various noises. Therefore, the noise is removed from the composite point cloud data to create noise-removed composite point cloud data (S3).

[0007] Step 3 (taking a photo) Before or after, or simultaneously with, the subject is photographed from various positions and angles using a high-resolution digital camera (S4) to obtain multiple pieces of photographic data. For example, if the subject is a building, the subject is photographed from all angles while moving around the building.

[0008] Step 4 (Processing photo data) Using photo data processing software (for example, "Adobe Photoshop" image editing software provided by Adobe Inc.), unusable photo data (for example, broken images and image data with unwanted reflections or glare) is removed from the multiple photo data, and only usable photo data is selected (S5). Next, the selected photo data is adjusted (S7). For example, if the reproduced image is insufficiently exposed, its brightness is adjusted (S6).

[0009] Step 5 (Creating 3D model data) Using point cloud editing software (for example, the 3D image editing software "RealityCapture" provided by Epic Games, Inc.), the noise-removed composite point cloud data obtained in step 2 is synthesized with the adjusted photo data obtained in step 4 to create three-dimensional model data.

[0010] In step 5, first, feature points are extracted from the photographic images reproduced from the multiple adjusted photographic data, and the multiple photographic images are aligned so that the feature points match, to create composite photographic data (S7).

[0011] Next, the composite photograph data and the composite point cloud data obtained in step 2 are subjected to an alignment process (S8). In this alignment process, a plurality of adjustment points (control points) that appear in common in both the point cloud image and the photographic image are set, and the composite photographic data is adjusted so that the adjustment points appearing in the point cloud image and the adjustment points appearing in the photographic image match.

[0012] Next, the composite point cloud data is converted into mesh data to create a mesh model (S9). A mesh is a 3D object that is composed of vertices, edges, and faces.

[0013] Next, an image (texture) of a composite photograph image (composite photograph data) corresponding to each mesh is attached to the corresponding mesh to generate three-dimensional model data (S10).

[0014] The 3D model data created in this way contains 3D coordinates and high-definition color information. Therefore, images reproduced using the 3D model data are more realistic than point cloud images, which are simply displayed as a collection of points.

[0015] However, as described above, in the conventional method of creating model data, step 1 of acquiring point cloud data of the subject using 3D LLS and step 2 of photographing the subject from various positions and angles are performed at different times and in different locations. For this reason, it takes a lot of time and effort to acquire the original data (point cloud data and photographic data).

[0016] In addition, because the measurement position (machine origin) where the point cloud data is acquired and the shooting position (photography origin) where the photo data is acquired are different, when aligning the point cloud and photos, it is difficult to determine which photo to paste for which point cloud position, which makes the alignment process take a lot of time and effort. Summary of the Invention [Problem to be solved by the invention]

[0017] An object of the present invention is to provide a new method for creating model data that solves the above problems. [Means for solving the problem]

[0018] Therefore, an embodiment of the present invention is a method of preparing an optical device (10) having an optical system (22) that projects a laser onto a measurement target along an optical path that passes through an origin (18) and guides the laser reflected by the measurement target along the optical path, acquiring point cloud data of a laser irradiation position using the reflected laser, and acquiring photographic data of the measurement target by receiving light incident via the origin (18), the method comprising the steps of: The method comprises: a point cloud synthesis step (S103) of synthesizing the point cloud data acquired at a plurality of measurement positions to generate synthesized point cloud data; a photo synthesis step (S107) of synthesizing the photo data acquired at the plurality of measurement positions to generate synthetic photo data; a division step (S110) of dividing the composite photographic data to obtain a plurality of divided image data based on the composite photographic data and instrument information of the optical device (10) when the photographic data was obtained at each of the plurality of measurement positions; An alignment step (S111) of associating the composite point cloud data with the divided image data based on position information of each of the plurality of divided image data; A mesh model generating step (S112) of generating a mesh model from the composite point cloud data; and a model data generating step of generating model data by attaching the divided image data to the mesh model. Effect of the Invention

[0019] According to this method, since the optical device has both the functions of point cloud acquisition and photography, it is possible to acquire the point cloud and take the photograph in a short time. In addition, since the coordinate reference when acquiring the point cloud data is the same as the coordinate reference when acquiring the photographic data, it becomes extremely easy to align the divided image data with the composite point cloud data. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a block diagram showing the configuration of a 3DLS for implementing a method according to an embodiment of the present invention. [Diagram 2] 5A to 5C are diagrams showing the steps of a method for creating three-dimensional model data. [Diagram 3] FIG. 3 is a diagram showing, together with FIG. 2, the steps of a method for creating three-dimensional model data. [Figure 4] FIG. 2-3 shows the steps of a method for creating three-dimensional model data. [Diagram 5] FIG. 5 is a diagram showing, together with FIGS. 2-4, the steps of a method for creating three-dimensional model data. [Figure 6] A diagram showing the steps of a method for creating three-dimensional model data together with Figures 2-5. [Figure 7] FIG. 1 is a diagram showing the steps of a conventional method for creating three-dimensional model data. [Figure 8] FIG. 8 is a diagram showing, together with FIG. 7, the steps of a conventional method for creating three-dimensional model data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, a method and system for generating high-quality three-dimensional model data according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0022] The three-dimensional model data is created by combining point cloud data acquired by a three-dimensional laser scanner and photographic data acquired by a digital camera. In the conventional three-dimensional model data creation method described above, the point cloud data and photographic data are acquired using separate devices, but in this embodiment, the point cloud data and photographic data are acquired by an optical device that has both the functions of a three-dimensional laser scanner and a digital camera, for example, a three-dimensional laser scanner with a built-in camera. In the following description, the three-dimensional laser scanner with a built-in camera is referred to as a "3DLS."

[0023] [A.3DLS Configuration and Operation] As the 3DLS, for example, 3DLS "Focus Premium" (product name) commercially available from FARO Technologies Inc. of Lake Mary, Florida, USA, can be suitably used.

[0024] 1, the 3DLS 10 has a lower fixed part 12 that is detachably fixed to a support means 11 such as a tripod, and an upper rotating part 13 that is supported by the lower fixed part 12. The upper rotating part 13 is connected to the lower fixed part 12 so as to be rotatable relative to the lower fixed part 12 around a vertical axis 14, which is a first axis.

[0025] The lower fixed part 12 and the upper rotating part 13 are provided with a first rotation mechanism required to rotate the upper rotating part 13 relative to the lower fixed part 12. In the embodiment, the first rotation mechanism includes a motor 15 and a rotation transmission mechanism (not shown) that converts the rotation of the motor 15 into the rotation of the upper rotating part 13 relative to the lower fixed part 12. In the embodiment, the motor 15 is housed in, for example, the upper rotating part 13, and the rotation transmission mechanism is interposed between the lower fixed part 12 and the upper rotating part 13.

[0026] The upper rotating part 13 has a cylindrical mirror 17 supported for rotation around a horizontal axis 16, which is a second axis. The horizontal axis 16 is perpendicular to the vertical axis 14, and the intersection of the vertical axis 14 and the horizontal axis 16 forms the mechanical origin 18 of the 3DLS 10. The coordinates of the mechanical origin 18 are the "machine coordinates."

[0027] The cylindrical mirror 17 has a reflecting surface 19 that passes through a mechanical origin 18 and intersects the horizontal axis 16 and the vertical axis 14 at an angle of 45 degrees.

[0028] The cylindrical mirror 17 is coupled to a second rotation mechanism that rotates the cylindrical mirror 17 around the horizontal axis 16. In the embodiment, the second rotation mechanism includes a motor 20 and a rotation transmission mechanism (not shown) that transmits the rotation of the motor 20 to the cylindrical mirror 17.

[0029] The upper rotating unit 13 also has a laser oscillator 21 that oscillates a laser at regular time intervals, and an optical system 22 that guides the oscillated laser. The optical system 22 is configured to guide the laser oscillated by the laser oscillator 21 along the optical path shown by the dotted line so that the laser is incident on the mechanical origin 18 of the reflecting surface 19 along the horizontal axis 16.

[0030] With the above configuration, the laser incident on the mechanical origin 18 along the horizontal axis 16 is projected while rotating around the mechanical origin 18 along a vertical plane that includes the mechanical origin 18 and is perpendicular to the horizontal axis 16, based on the drive of the motor 20. At the same time, the projected laser rotates around the vertical axis 14 that passes through the mechanical origin 18, based on the drive of the motor 15. By combining these two rotational movements, the laser is projected into the entire sky space centered on the mechanical origin 18. In addition, the laser projected from the 3DLS 10 is reflected by a measurement target in the surrounding environment, and a portion of the reflected laser returning to the mechanical origin 18 is guided in the opposite direction along the optical path of the optical system 22.

[0031] The upper rotating unit 13 also has a light receiving unit 23 that detects reflected laser returning from the surrounding environment, a communication unit 25 that receives signals from a remote controller 24 held by an operator, a processor 26 that drives the above-mentioned multiple devices in accordance with a predetermined procedure and stores and processes the acquired point cloud data as described later, and a power supply unit 27 that supplies power to those devices.

[0032] The 3DLS 10 also includes an image sensor (digital camera) 28 having a high-resolution CCD sensor or CMOS sensor. The image sensor 28 is positioned to capture an image incident along the optical path of the reflected laser from the reflective surface 19 or along an optical path branching therefrom.

[0033] In addition, the 3DLS 10 is equipped with an inclinometer 29 that automatically detects the amount of tilt of the upper rotating part 13 relative to the horizontal plane (the amount of rotation relative to three orthogonal axes, i.e., the vertical axis Z in the direction of gravity, a first horizontal axis X that is perpendicular to the vertical axis Z, and a second horizontal axis Y that is perpendicular to the vertical axis Z and the first horizontal axis X) and a compass 30 that automatically detects in which direction (azimuth) on the earth the upper rotating part 13 is facing.

[0034] According to the 3DLS 10 having such a configuration, when acquiring point cloud data, the communication unit 25 receives an activation signal transmitted from the remote controller 24, and when the information is transmitted to the processor 26, the processor 26 drives the motors 15 and 20. As a result, the upper rotating unit 13 rotates around the vertical axis 14, and the cylindrical mirror 17 rotates around the horizontal axis 16. In this state, the processor 26 activates the laser oscillator 21 to oscillate the laser. The optical system 22 guides the laser oscillated by the laser oscillator 21 along the horizontal axis 16 to the reflecting surface 19, and projects it in all sky directions via the reflecting surface 19. The laser reflected in the surrounding environment is detected by the light receiving unit 23 from the reflecting surface 19 via the optical system 22. For example, in a time-of-flight type scanner, the processor 26 calculates the distance from the machine origin 18 to each laser irradiation position based on the time until the oscillated laser is detected by the light receiving unit 23. In addition, based on the amount of rotation of motors 15 and 20, the processor 26 calculates the movement angle (the amount of rotation about vertical axis 14 and the amount of rotation about horizontal axis 16) from the laser projection direction when the 3DLS 10 is started. The processor 26 acquires and stores these distances and movement angles as three-dimensional position information (point cloud data) of each measurement position.

[0035] Furthermore, after or before acquiring the point cloud data, the 3DLS 10 has the processor 26 drive the motors 15, 20 at the measurement position where the point cloud data is acquired, and while the upper rotating part 13 rotates about the vertical axis 14 and the cylindrical mirror 17 rotates about the horizontal axis 16, the image sensor 28 acquires and stores two-dimensional images (photographic data) of the entire sky direction centered on the 3DLS 10. The shooting conditions (shooting interval, rotation speed) are determined so as to capture the surrounding environment without any gaps.

[0036] The processor 26 creates three-dimensional model data from the point cloud data and photographic data thus acquired.

[0037] For example, when creating three-dimensional model data of a building, first, the 3DLS 10 is installed at a first measurement position. The machine coordinates of the 3DLS 10 installed at the first measurement position are (X1, Y1, Z1). In this state, the processor 26 starts the 3DLS 10 based on an instruction from the user, and drives the motors 15 and 20 while oscillating the laser at a predetermined time interval to obtain point cloud data PC1 in all sky directions. The point cloud data PC1 includes, for each of a plurality of positions (point clouds) irradiated with the laser, the distance (slant distance) from the machine coordinates (X1, Y1, Z1) to the laser irradiation position, and the laser movement angle (corresponding to the azimuth angle and elevation angle when the laser irradiation position is viewed from the machine coordinates).

[0038] After acquiring the point cloud data PC1, next, while driving the motors 15 and 20 without oscillating the laser, omnidirectional photo data PI1 is acquired at a predetermined time interval. The photo data PI1 includes a plurality of photographic images captured and the photographing angles of each photographic image (the azimuth angle and elevation angle of the photograph relative to the machine coordinate system).

[0039] Next, the 3DLS10 is installed at the second measurement position. The machine coordinates of the 3DLS10 installed at the second measurement position are (X2, Y2, Z2). In this state, the 3DLS10 is started based on an instruction from the user, and the motors 15 and 20 are driven while oscillating the laser to obtain point cloud data PC2 of the entire sky direction, and then the motors 15 and 20 are driven without oscillating the laser to obtain photo data PI2 of the entire sky direction. The point cloud data PC2 includes the distance from the machine coordinates (X2, Y2, Z2) to each laser irradiation point and the moving angle of the laser (the azimuth angle when the laser irradiation position point is viewed from the machine coordinates is the elevation angle). The photo data PI2 includes the multiple photographic images taken and the shooting angle of each photographic image (the azimuth angle and elevation angle of the photo shoot relative to the machine coordinates).

[0040] By repeating these steps, point cloud data and photographic data are obtained at multiple measurement positions for the entire building, and the obtained point cloud data and photographic data are then combined according to the procedure described below to create a single 3D model data.

[0041] [B. Creating 3D model data]

[0042] 2, the processor 26 executes the above-mentioned measurement (S101) to acquire point cloud data and photographic data of the subject (S102). The acquired measurement data including the point cloud data (D101) and photographic data (D102) are processed according to the following procedure.

[0043] Step 101 Referring to FIG. 3, in step 101, point cloud data acquired at each measurement position (3DLS installation point) is synthesized using point cloud data processing computer software (e.g., "Trimble RealWorks" provided by Nikon-Trimble Corporation, "FARO SCENE" provided by FARO Technologies Inc., and "Infipoints" provided by Elysium Corporation) to generate synthetic point cloud data (S103). The machine coordinates [(X1, Y1, Z1), (X2, Y2, Z2)...] of each measurement position are used to generate the synthetic point cloud data. Preferably, tilt data measured by the inclinometer 29 at each measurement position and orientation data measured by the compass 30 at each measurement position are used to correct data variations caused by installation errors of the 3DLS 10.

[0044] Next, noise is removed from the composite point cloud data (S104), and the composite point cloud data after noise removal is output (S105). In the following description, "composite point cloud data" refers to the composite point cloud data from which noise has been removed. In addition, the machine coordinates of each measurement position used when synthesizing the point cloud data (hereinafter, this information will be referred to as "synthesis information") are output (S106).

[0045] Step 102 Referring to FIG. 4, step 102 is performed using panoramic image creation computer software (for example, "FARO SCENE" available from FARO Technologies Inc., or "Trimble RealWorks" available from Nikon-Trimble Corporation).

[0046] Specifically, in step 102, the multiple photos taken at each measurement position are stitched together to create a 360-degree panoramic image (composite photo data) (S107). In creating the 360-degree panoramic image, multiple photos are combined by extracting feature points commonly contained in the multiple images and matching feature points with high similarity. Preferably, the 360-degree panoramic image is created using the tilt angle and azimuth angle measured by the inclinometer 29 and compass 30 at each measurement position (hereinafter, these tilt angle and azimuth angle are collectively referred to as "instrument information"), and a 360-degree panoramic image with the tilt and orientation appropriately adjusted using this instrument information is synthesized.

[0047] Step 103 Referring to FIG. 5, step 103 is a process for programming a general-purpose programming language (e.g., Python, C ++ , C#). First, in step 103, a 360-degree panoramic image, instrument information (tilt of the 3DLS) at the start of measurement at each measurement position, and synthesis information (machine coordinates) at each measurement position are read (S109). Next, the 360-degree panoramic image is converted into a celestial sphere image (S110). Next, the origin direction of the celestial sphere image is specified using the instrument information and synthesis information (S111). Next, the celestial sphere image is divided based on the specified origin direction to obtain multiple divided images (S112), and position information is created for each divided image (S113).

[0048] Step 104 6, step 104 is performed using point cloud editing software (for example, "MetaShape" provided by Agisoft LLC, "RealityCapture" provided by Epic games, Inc., and "Recap Photo" provided by Autodesk). First, in step 104, point cloud composite data, divided images, and position information of the divided images are acquired (S114), and based on the position information of the divided images and the coordinate information of the point cloud data, each divided image data is positionally associated with the corresponding composite point cloud data (alignment process) (S115).

[0049] Next, the composite point cloud data is converted into mesh data to create a mesh model (S116).

[0050] Next, the texture images associated by the alignment process are attached to each mesh to generate three-dimensional model data (S117).

[0051] According to the three-dimensional model data creation method of the above-mentioned embodiment, the coordinate reference when acquiring the point cloud data is the same as the coordinate reference when acquiring the photographic data. Therefore, the work of aligning the divided image data with the composite point cloud data becomes extremely easy. In addition, since all processing can be performed using a single reference coordinate, there is no need to convert the coordinate system during processing, and the time and processing required to create the three-dimensional model data are significantly reduced. [Explanation of symbols]

[0052] 10:3DLS (optical device) 11: Support means 12; Lower fixing part 13: Upper rotating part 14: Vertical axis (first axis) 15:Motor 16: Horizontal axis (second axis) 17: Cylindrical mirror 18: Machine origin 19: Reflective surface 20:Motor 21: Laser oscillator 22:Optical system 23: Light receiving part 24: Controller 25: Communications Department 26: Processor 27: Power supply section 28: Image sensor 29: Inclinometer 30: Compass

Claims

1. A method for creating model data using the point cloud data and the photographic data acquired by the optical device, comprising: preparing an optical device (10) having an optical system (22) that projects a laser onto a measurement object along an optical path that passes through a mechanical origin (18) and guides the laser reflected by the measurement object along the optical path; acquiring point cloud data of a laser irradiation position using the reflected laser; and acquiring photographic data of the measurement object by receiving light incident through the mechanical origin (18); The method comprises: a point cloud synthesis step (S103) of synthesizing the point cloud data acquired at a plurality of measurement positions to generate synthesized point cloud data; a photo synthesis step (S107) of synthesizing the photo data acquired at the plurality of measurement positions to generate synthetic photo data; A division step (S110) of dividing the composite photographic data to obtain a plurality of divided image data based on the composite photographic data and instrument information of the optical device (10) when the photographic data was acquired at each of the plurality of measurement positions; An alignment step (S115) of associating the composite point cloud data with the divided image data based on position information of each of the plurality of divided image data; A mesh model generating step (S112) of generating a mesh model from the composite point cloud data; and a model data generating step of generating model data by attaching the divided image data to the mesh model.

2. The method of claim 1 , wherein the synthetic photograph data is data for a 360 degree panoramic image.

3. A conversion step of converting the 360-degree panoramic image into a spherical image, The method according to claim 2 , wherein the dividing step includes a step of dividing the spherical image to obtain the plurality of divided image data.

4. In the dividing step, the plurality of divided image data are obtained by utilizing combination information used in creating the combined point cloud data; The method of claim 1 , wherein the composite information includes machine coordinates of the measurement locations.

Images